Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning

Authors: jiawei  yao
qi  qian
juhua  hu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:43:14

Abstract: Multiple clustering aims to discover various latent structures of data from different aspects. Deep multiple clustering methods have achieved remarkable performance by exploiting complex patterns and relationships in data. However, existing works struggle to flexibly adapt to diverse user-specific needs in data grouping, which may require manual understanding of each clustering. To address these limitations, we introduce Multi-Sub, a novel end-to-end multiple clustering approach that incorporates a multi-modal subspace proxy learning framework in this work. Utilizing the synergistic capabilities of CLIP and GPT-4, Multi-Sub aligns textual prompts expressing user preferences with their corresponding visual representations. This is achieved by automatically generating proxy words from large language models that act as subspace bases, thus allowing for the customized representation of data in terms specific to the user’s interests. Our method consistently outperforms existing baselines across a broad set of datasets in visual multiple clustering tasks. Our code is available at https://github.com/Alexander-Yao/Multi-Sub.

Main paper: 
 multiple clustering: 
 Multiple clustering, a methodology capable of unveiling alternative data perspectives, has garnered significant interest. Traditional approaches for multiple clustering [Hu and Pei, 2018] employ shallow models to identify diverse data groupings. Some methods, such as COALA [Bae and Bailey, 2006] and [Qi and Davidson, 2009] , utilize constraints to generate alternative clusterings. Other techniques leverage distinct feature subspaces to produce multiple clusterings, as exemplified by [Hu et al., 2017] and MNMF [Yang and Zhang, 2017] . Information theory has also been applied to generate multiple clusterings, as demonstrated by [Gondek and Hofmann, 2003] and [Dang and Bailey, 2010] .Recent advancements have seen the application of deep learning to discover multiple clusterings, yielding improved clustering performance. For instance, [Wei et al., 2020] proposed a deep matrix factorization method that utilizes multi-view data to identify multiple clusterings. ENRC [Miklautz et al., 2020] employs an auto-encoder to learn object features and optimizes a clustering objective function to find multiple clusterings. iMClusts [Ren et al., 2022] leverages auto-encoders and multi-head attention to learn features from various perspectives and discover multiple clusterings.AugDMC [Yao et al., 2023] uses data augmentation to generate diverse image aspects and learns representations to uncover multiple clusterings. DDMC [Yao and Hu, 2024 ] employs a variational Expectation-Maximization framework with disentangled representations to achieve superior clustering outcomes. However, almost all of these methods necessitate substantial user efforts to understand and select the appropriate clustering for different application purposes. Recently, Multi-MaP [Yao et al., 2024] leverages CLIP encoders to align a user's interest with visual data by learning representations close to the interested concept but far away from a contrastive concept, significantly improving the efficiency of capturing user-desired clusterings. However, Multi-MaP requires the user to input a contrastive concept for better performance, which is often not applicable. More importantly, it separates the representation learning and clustering as two distinct stages, which may result in sub-optimal performance. These issues will be mitigated in this work.Multi-modal learning involves acquiring representations from various input modalities like image, text, or speech. Here, we focus on how vision models benefit from natural language supervision. A key model in this area is CLIP [Radford et al., 2021] , which aligns images with their corresponding text using contrastive learning on a dataset of 400 million text-image pairs.Fine-tuning adapts vision-language models, such as CLIP, for specific image recognition tasks. This is seen in CoOp [Zhou et al., 2022] and CLIP-Adapter [Gao et al., 2023] , the latter using residual style feature blending to enhance performance. TeS [Wang et al., 2023] highlights the efficacy of fine-tuning in improving visual comprehension through natural language supervision. With limited labeled data, zero-shot learning has gained attention. Some approaches surpass CLIP by integrating other large pre-trained models. For example, VisDesc [Menon and Vondrick, 2022] uses GPT-3 to generate contextual descriptions for class names, outperforming basic CLIP prompts. UPL [Huang et al., 2022] and TPT [Shu et al., 2022] utilize unlabeled data to optimize text prompts. InMaP [Qian et al., 2024] and the online variant [Qian and Hu, 2024] aid class proxies in vision space with text proxies. Recent advancements have significantly improved vision-language pre-training using largescale noisy datasets. ALIGN [Jia et al., 2021] employs over one billion image alt-text pairs without expensive filtering, showing that corpus scale can offset noise. Similarly, BLIP-2 [Li et al., 2023] uses a novel framework to bootstrap captions from noisy web data, enhancing both vision-language understanding and generation tasks. While these methods strive to enhance the performance of vision classification tasks, clustering presents a distinct scenario where class names are not available to extract useful information from multi-modal information as in this work.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper discusses the limitation in Section 5, addressing potential biases from large language models. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the proposed method: 
 Given a dataset of images {x i } n i=1 and user-defined preferences for data grouping (such as color and species), our goal is to generate clustering results that are specifically tailored to each preference. Thereafter, end users can directly use them for different application purposes without additional manual selection efforts. This process poses significant challenges, as it requires accurately aligning the complex, multi-dimensional data of images with the subjective and varied textual preferences of users. Traditional clustering methods often fail to capture these nuances, leading to a generic and less informative categorization for specific user applications.Recently, the CLIP model [Radford et al., 2021] facilitated a more natural alignment between textual interests and visual representations. Our method, Multi-Sub, extends this alignment through a novel multi-modal subspace proxy learning approach. Fig. 2 outlines the overall framework of Multi-Sub, which is tailored to capture and respond to the diverse interests of users in clustering tasks. Multi-Sub employs a two-phase iterative approach to align and cluster images based on user-defined preferences such as color and species as described below. processes each image x i with user-defined textual prompts through a partially learnable image encoder (with a learnable projection layer) and a frozen text encoder. The latent factor p i calculates weights {a i,k } K k=1 based on the similarity to reference word embeddings {z i } K k=1 , which are then aggregated to form the proxy word embedding w i . This proxy word embedding, combined with the image representation x i , establishes the Aligned Feature Subspace for better alignment between the text and image under the user's interest. In Phase II (Clustering), given the learned proxy word embeddings {w i } from Phase I to form pseudo-labels, the projection layer of the image encoder is further refined using the clustering loss. In Phase I, both the latent factor p and the projection layer learn 100 epochs, after which the projection layer further learns 10 epochs using the clustering loss in Phase II. This alternative process repeats until convergence.: Multi-Modal Pre-Training in CLIP Let {x i , t i } ni=1 be a set of image-text pairs, where x i denotes an image and t i denotes its corresponding text description. We can obtain the vision and text representations of each pair by applying two encoders, f (•) and h(•), as x i = f (x i ) and t i = h(t i ). Both f (•) and h(•) are encoders that optimize the vision and text representations, respectively, such that x i and t i are unit vectors. The primary goal during this pre-training phase is to minimize the contrastive loss, formulated asmin f,h i -log exp(x ⊤ i t i /τ ) j exp(x ⊤ i t j /τ ) -log exp(t ⊤ i x i /τ ) j exp(t ⊤ i x j /τ ) (1)where τ is a temperature parameter. The contrastive loss encourages the alignment of the image and its description while penalizing the similarity of the image with irrelevant texts [Qian et al., 2019] . The efficacy of this contrastive approach is vital for the subsequent phases of proxy word learning and fine-grained clustering, as it ensures that the foundational embeddings accurately reflect the inherent content and context of each modality.We build upon the pre-trained image and text encoders from CLIP and investigate whether we can leverage the image-text alignment to extract user-specific information. Specifically, given a fruit image [Hu et al., 2017] as illustrated in Fig. 2 , different users may have different interests of its attributes, such as color, species, etc. However, the pre-trained image encoder in CLIP can only produce a single image embedding, which may not capture a user's interest exactly, not mentioning capturing different aspects. Furthermore, unlike classification tasks, clustering tasks do not come with concrete cluster names or numbers. Therefore, we cannot directly use the pre-trained text encoder of CLIP to generate the corresponding text embedding.To address these challenges, we propose a subspace proxy word learning method to learn new embedding under the preferred aspect provided by the user. Thereafter, the main challenge is, given only a high-level concept like 'color' as in Fig. 2 , how to effectively represent its subspace. Since the high-level concept itself cannot reflect different details under this concept in different images, it is difficult to do effective alignment between the high-level concept and images to figure out the corresponding vision subspace. Therefore, we propose to figure out the text subspace at first. Concretely, given pre-trained large language models like GPT-4 as low-cost experts, we can quickly gather common categories under a high-level concept using only one query like 'what are the common fruit colors' in Fig. 2 . However, we cannot directly use the returned categories to do grouping, since they may not cover all existing categories in the data. Instead, we consider that most categories in the data under this concept are residing in the same subspace as the returned ones. Therefore, we can apply suggested categories as basis or reference words in the subspace. Then, each image's category under the desired concept can be represented by a linear combination of these reference words.Assuming GPT-4 provides K reference words as {z k } K k=1 , the proxy word of image x i can be calculated asEQUATIONwhere ϕ(z k ) is the token embedding of reference word z k and {a i,k } K k=1 are weights corresponding to each reference word as a basis. A higher weight a i,k indicates that the image x i 's category is closer to the reference word z k . Here, we introduce trainable latent factor p i to learn the weight a i,k , and it can be calculated asEQUATIONwhere z k = ϕ(z k ). Thereafter, w i is representing the token embedding of image x i 's proxy word under the preferred user concept. Once p i is well obtained, the image's proxy word representation under the preferred user concept is also obtained. Next, we discuss how to learn p i using CLIP.As mentioned above, CLIP's text and image encoders were learned by aligning the text prompt with its corresponding image. The standard text prompt of CLIP is designed as "a photo of a fruit" for an image containing "fruit". Now, given a user's preference (e.g., color), we can rewrite the prompt as "a fruit with the color of *" denoted by t * i for image x i , where "*" is the placeholder for the unknown proxy word of image x i under concept 'color' and its token embedding w i can be formulated as the linear superposition of reference words' token embeddings as discussed above.Thereafter, the prompt text embedding after the text encoder can be formulated asEQUATIONTo effectively learn p i , the trainable latent factors, we utilize the alignment capabilities of CLIP by adjusting these factors so that the weighted sum of reference word embeddings closely aligns with the visual representation of the image. This process involves iteratively adjusting p i to maximize the cosine similarity between the image's representation x i and its corresponding proxy word embedding w i . The optimization is conducted with the following loss function:EQUATIONIt should be noted that this optimization procedure can be conducted with both the text encoder and image encoder frozen, which is very efficient. However, the image embedding extracted directly from the pre-trained image encoder may not reflect its representation under the desired user interest. Therefore, during the optimization procedure, we do freeze the text encoder but open the image encoder. Nevertheless, to preserve the strong capacity of the pre-trained image encoder in CLIP, we open only the projection layer of the image encoder, while its remaining parameters are frozen as shown in the 'Phase I' of Fig. 2 .To enhance the clustering performance of Multi-Sub, in 'Phase II', we leverage pseudo-labels assigned using the currently learned proxy word embeddings {w i } and image embeddings {x i } from 'Phase I'. Concretely, each image x i can be represented by the concatenation of its currently learned proxy word embedding w i and image embedding x i , denoted asv i = [w i , x i ].The pseudo-labels can be obtained by an offline k-means on {v i }, which is however not efficient. Considering that proxy words for data points within the same cluster should show similar relationships to reference words, we obtain the pseudo-labels using the highest cosine similarity between the currently learned proxy word embeddings {w i } and the reference word embeddings {z k }.Given the pseudo-labels, the image embeddings can be further optimized by opening only the projection layer of the image encoder for improved compactness and separability in clusters. This loss consists of two primary components: intra-cluster loss and inter-cluster loss, aimed at refining cluster cohesion and separation, respectively. It should be noted that to better represent each image under the desired user concept, we define the clustering loss over v i containing both textual and visual information.Intra-cluster Loss: The intra-cluster loss is designed to minimize the distances between embeddings within the same cluster, encouraging cluster compactness. It is calculated using the following formula:EQUATIONHere, ∥v i -v j ∥ 2 is the squared Euclidean distance between embeddings x i and x j of data points i and j within the same cluster, and N intra denotes the number of intra-cluster pairs.Inter-cluster Loss: This component aims to maximize the distances between embeddings from different clusters, thus enhancing separability. The inter-cluster loss is defined by a margin-based hinge loss as follows:EQUATIONwhere max(0, m -∥v i -v j ∥) computes the hinge loss for each pair of embeddings from different clusters, ensuring a minimum margin m between them. N inter is the count of inter-cluster pairs.Total Loss: The overall clustering loss combines the intra-and inter-cluster losses, moderated by a balancing factor λ:L total = λ • L intra + (1 -λ) • L inter (8)Optimizing this loss function in 'Phase II' helps regularize the embedding space where clusters are both internally dense and well-separated from each other. It should be noted that in this phase we aim to learn a better projection layer only for the image encoder, while all others are fixed as shown in 'Phase II' of Fig. 2 .Previous methods often use a two-stage strategy that separates representation learning and clustering to simplify the optimization process. This separation, however, can lead to sub-optimal clustering results, since the learned representations may not be fully aligned with the clustering objective without refinement. In this work, we obtain both the proxy word and the clustering alternatively and simultaneously. Concretely, we first learn the proxy word in a user-preferred subspace. Then, we fix the proxy word and refine the image encoder further to obtain better image representations using the clustering objective. These two phases are repeated alternatively until convergence, where 'Phase I' learns 100 epochs and 'Phase II' learns 10 epochs in each alternating according to the empirical experience as summarized in Fig. 2 . Hu et al. [2017] and Fruit360 Yao et al. [2023] . Stanford Cars contains two different clustering types, one for car color (e.g., red, blue, black) and one for car type (e.g., sedan, SUV, convertible), comprising 1,200 annotated car images. Card includes 8,029 images of playing cards, with two clustering types: one based on rank (e.g., Ace, King, Queen) and another on suit (e.g., clubs, diamonds, hearts, spades). CMUface provides 640 facial images with clustering options for pose (e.g., front-facing, side-facing), identity, glasses (with/without), and emotion (e.g., happy, neutral, sad). Flowers comprises 1,600 flower images with two clustering types: one for color (e.g., red, blue, yellow) and another for species (e.g., iris, aster). Fruit includes 105 images of fruits with two clustering criteria: species (e.g., apples, bananas, grapes) and color (e.g., green, red, yellow). Fruit360, similar to the Fruit dataset, contains 4,856 images annotated for species (e.g., apple, banana, cherry) and color.Additionally, we created a multiple clustering dataset from CIFAR-10 Krizhevsky et al. [2009] by organizing the images into clusters based on type and environment. For type, the clusters are transportation and animals. For environment, the clusters are land, air, and water. The dataset characteristics about data size, handcrafted features, and cluster information are also summarized in Table 1 .It should be noted that some data may face challenges in extraction of meaningful candidate categories from GPT-4, or their labels lack semantic features. Taking the identity clustering on the CMUface dataset Günnemann et al. [2014] as an example, different identities correspond to different individuals, and the names' semantic meanings should not affect clustering outcomes. In such cases, following the Multi-Map setting Yao et al. [2024] , we randomly select 10 words from WordNet Fellbaum [2010] as reference categories.Baselines We compare our Multi-Sub with seven state-of-the-art multiple clustering methods. These methods are: MSC Hu et al. [2017] is a traditional multiple clustering method that uses handcrafted features to automatically find different feature subspace for different clusterings; MCV Guérin and Boots [2018] leverages multiple pre-trained feature extractors as different views of the same data; ENRC Miklautz et al. [2020] integrates auto-encoder and clustering objective to generate different clusterings; iMClusts Ren et al. [2022] is a deep multiple clustering method that leverages the expressive representational power of deep autoencoders and multi-head attention to generate multiple salient embedding matrices and multiple clusterings therein; AugDMC Yao et al. [2023] leverages data augmentations to automatically extract features related to different aspects of the data using a self-supervised prototype-based representation learning method; DDMC Yao and Hu [2024] combines disentangled representation learning with a variational Expectation-Maximization (EM) framework; Multi-MaP Yao et al. [2024] relies on a contrastive user-defined concept to learn a proxy better tailored to a user's interest. It is worth noting that, in our experiments, we apply both traditional and deep learning baselines. Traditional methods rely on hand-crafted features, while deep learning methods directly utilize the original images as input.Hyperparameter For each user's preference, we train the model for 1000 epochs using Adam optimizer with a momentum of 0.9. We tune all the hyper-parameters based on the loss score of Multi-Sub, where the learning rate is selected from {1e-1,5e-2,1e-2,5e-3,1e-3,5e-4}, weight decay is chosen from {5e-4,1e-4,5e-5,1e-5, 0} for all the experiments. Most methods obtain each clustering by applying k-means Lloyd [1982] to the newly learned representations, while ours is end-to-end. The experiments are performed on four NVIDIA GeForce RTX 2080 Ti GPUs.Evaluation metrics Considering the randomness of k-means for those applicable baselines, we run k-means 10 times and report the average clustering performance using two metrics, namely, Normalized Mutual Information (NMI) White et al. [2004] and Rand index (RI) Rand [1971] . These metrics range from 0 to 1 with higher value indicating better performance compared to the groundtruth.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
performance comparison: 
 Table 2 reports the clustering results. During the clustering stage, after we obtain the proxy word embedding of each image for a desired concept, we can concatenate the image embedding and the token embedding of proxy word. The results show that Multi-Sub consistently outperforms the baselines, demonstrating the superiority of the proposed method. This also indicates a strong generalization ability of the pre-trained model by CLIP, which can capture the features of data from different perspectives.Our methodology uses the CLIP encoder and GPT-4 to derive clustering results, prompting an evaluation of their performance in a zero-shot manner. We introduce two zero-shot variants of CLIP: CLIP GPT and CLIP label . CLIP GPT uses GPT-4 to generate candidate labels and performs zero-shot classification, while CLIP label uses ground truth labels directly, providing an optimal setting. As shown in Table 3 , CLIP label generally outperforms CLIP GPT due to its use of accurate labels, while CLIP GPT introduces noise. Both variants perform equally on the Card dataset as GPT-4's labels match the groundtruth. Multi-Sub surpasses CLIP GPT and even outperforms CLIP label in all cases, demonstrating its ability to capture user-interest-based data aspects and confirming its efficacy. This superiority can be attributed to Multi-Sub's proxy word learning mechanism, which automatically adjusts textual embeddings based on user-defined interests, creating more accurate proxy word embeddings. This approach reduces noise compared to CLIP GPT , which suffers from label mismatches. Additionally, Multi-Sub's iterative learning process refines these embeddings, optimizing alignment between text and image representations.Different ways of constructing subspace The subspace of the proposed method can be expanded by different embeddings, i.e., the token embedding of the proxy word ϕ(w i ), the text embedding of the proxy word h(ϕ(w i )), and the text embedding of the prompt t * i = h(ϕ(t * i )∥ϕ(w i )). These three kinds of embeddings can also be used to evaluate the clustering results in each case. In addition, we can use different combinations of learned embeddings (e.g., different concatenations of text and image embedding) as the final embedding for clustering. The results are shown in Table 5 . It can be seen that using word token embedding usually achieves better results. This is expected since the word proxy directly reflects the image's category under the desired concept. The token word embedding subspace is also aligning well with CLIP's training method. In contrast, prompt embedding performs the worst as it introduces noise from user interest, dataset, and reference words, which are unnecessary for clustering. Additionally, most methods perform better when the same approach is used for constructing subspace and evaluating clustering results. Combining text and image embeddings generally enhances performance, capturing user interests from both aspects effectively.Effect of text encoder Table 4 compares the performance of three text encoders-CLIP, ALIGN, and BLIP-across various datasets. The results indicate that ALIGN generally outperforms CLIP and BLIP in most tasks. This suggests that ALIGN's text encoder effectively captures and aligns textual and visual representations, enhancing clustering performance. ALIGN tends to excel in tasks that require distinguishing subtle visual differences influenced by textual descriptions, such as emotions and accessories in the CMUface dataset, and colors in the Fruit360 dataset. CLIP shows a strong tendency in identity-related tasks and complex object categorization, as evidenced by its performance in the CMUface identity task and Standford Cars type clustering. BLIP, while competitive, seems to perform better in categorical distinctions rather than abstract attributes, performing relatively well in species-related tasks across various datasets. These findings underscore the importance of effective text embeddings in multi-modal clustering frameworks.We conducted an additional analysis using the Maximum Mean Discrepancy (MMD) metric to quantify the differences in the feature spaces generated by different text encoders (i.e., CLIP, ALIGN, and BLIP) in Table 6 . The MMD results indicate that although our text prompts are simple, the feature spaces generated by different text encoders exhibit significant distributional differences. The effectiveness of a text encoder can vary depending on the specific clustering task. For example, ALIGN tends to excel in tasks with more abstract attributes, such as colors and emotions, while CLIP shows strong performance in identity-related tasks. This variability underscores the importance of selecting an appropriate text encoder based on the specific application requirements. The difference between text encoders may come from the different corresponding pre-training tasks and this will be an interesting future direction.  
conclusion and limitations: 
 In conclusion, our study mitigates an important challenge in multiple clustering: effectively identifying desired clustering results based on user interests or application purposes. We introduce Multi-Sub, a novel approach that integrates user-defined preferences into a customized multi-modal subspace proxy learning framework. By leveraging the synergy between CLIP and GPT-4, Multi-Sub automatically aligns textual prompts expressing user interests with corresponding visual representations. First, we observe reference words for user's interests from large language models. Given the absence of concrete class names in clustering tasks, our method uses these reference words to learn both text and vision embeddings tailored to user preferences. Extensive experiments across various visual multiple clustering tasks demonstrate that Multi-Sub consistently outperforms state-of-the-art techniques.However, our approach has certain limitations. The reliance on large language models like GPT-4 can introduce biases inherent in these models, potentially affecting the clustering outcomes. Additionally, the field of multiple clustering lacks large, diverse datasets, which limits comprehensive evaluation.Although we have annotated CIFAR-10, more extensive datasets are needed. 
acknowledgment: 
 Yao's research was funded in part by J.P. Morgan Chase & Co and Advata Gift funding. Any views or opinions expressed herein are solely those of the authors listed, and may differ from the views and opinions expressed by J.P. Morgan Chase & Co. or its affiliates. This material is not a product of the Research Department of J.P. Morgan Securities LLC. This material should not be construed as an individual recommendation for any particular client and is not intended as a recommendation of particular securities, financial instruments or strategies for a particular client. This material does not constitute a solicitation or offer in any jurisdiction. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Since some baselines involve randomness using k-means, the paper did 95% significance test using 10 repeated results. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper discusses the potential positive impacts of improving clustering techniques for various applications and mentions possible negative impacts such as biases introduced by large language models in Section 5. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper does not involve the release of data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The paper properly credits the creators of the datasets and models used, and mentions the licenses and terms of use, as detailed in Section 4.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The new datasets and code introduced in the paper are well documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
multi-sub: 
 Which aspect would you like to group? User I would like to group by color.Common colors of fruit include red, yellow, green, orange, purple, and blue.Multi-Sub I have learned the embedding of each image in the subspace spanned by the reference words. Clustering is a fundamental technique for analyzing data based on certain similarities, attracting extensive attention due to the abundance of unlabeled data. Traditional clustering methods [MacQueen et al., 1967 , Ng et al., 2001 , Bishop and Nasrabadi, 2006] rely on general-purpose handcrafted features that may not suit specific tasks well. Deep clustering algorithms have improved clustering performance by employing Deep Neural Networks (DNNs) [Xie et al., 2016 , Guérin and Boots, 2018 , Qian et al., 2022 , Qian, 2023] to learn task-specific features. However, most of these algorithms assume a single partition of the data, while real data can be clustered differently according to different aspects, e.g., fruits in Fig. 1 can be grouped differently by color or by species.Multiple clustering algorithms [Bae and Bailey, 2006, Hu et al., 2017] address this challenge by producing multiple partitions of the data for various applications,showing the capability of discovering multiple clusterings from a single dataset. For instance, in e-commerce, products can be clustered by category for inventory management or by customer preferences for personalized recommendations. Recently, there has been a growing interest in incorporating deep learning techniques into multiple clustering. These techniques mainly use autoencoders and data augmentation methods to extract a wide range of feature representations, which enhance the quality of multiple clustering [Miklautz et al., 2020 , Ren et al., 2022 , Yao et al., 2023] .For real-world applications, a key challenge for end users is efficiently identifying the desired clustering from multiple results based on their interests or application purposes. We observe that users are willing to indicate their interest using succinct keywords (e.g., color or species for fruits in Fig. 1 ). However, it is difficult to use only a concise keyword to directly extract the corresponding image representations. Fortunately, the recent development of multi-modal models like CLIP [Radford et al., 2021] that align images with their text descriptions can help bridge this gap. Nevertheless, unlike methods that can use labeled data to fine-tune pre-trained models [Gao et al., 2023 , Wang et al., 2023] to learn new task-specific representations, multiple clustering often faces scenarios with ambiguous or unspecified label categories and quantities. Therefore, given only a high-level concept from the user, it is intractable to fine-tune pre-trained models to capture a particular aspect of the data in an unsupervised manner. Very recently, Multi-MaP [Yao et al., 2024] leverages CLIP to learn textual and image embeddings simultaneously that follow the user's high-level textual concept. However, to achieve better performance, they require the user to provide a contrastive concept that is different from the desired concept, which may not be feasible in many real-world applications. Moreover, they obtain the new representations at first and then apply the traditional clustering method like k-means in a separate stage. This insufficient optimization lacking refinement between stages makes the clustering performance sub-optimal.To mitigate these challenges, in this work, we first assume that the desired image and textual representations are residing in the same subspace according to the user's specific concept. Thereafter, to capture the desired subspace better, we can ask low-cost experts like Google or large language models (LLMs) (e.g., GPT-4) for common categories under the desired concept, as illustrated in Fig. 1 . Although those returned common categories may not directly capture the clustering targets, they can be applied as the subspace basis to help search the appropriate representations inside. More importantly, during the learning under the desired subspace, we also incorporate the clustering loss to learn the representations and obtain the clustering simultaneously, which significantly enhances the model's clustering performance and efficiency. The main contributions of this work can be summarized as follows.• We present a novel multiple clustering method, Multi-Sub, that can explicitly capture a user's clustering interest by aligning the textual interest with the visual features of images. Concretely, we propose to learn the desired clustering proxy in the subspace spanned by the common categories under a user's interest.• Unlike most existing multiple clustering methods that require distinct stages for representation learning and clustering, Multi-Sub can obtain both the desired representations and clustering simultaneously, which can significantly improve the clustering performance and efficiency.• Extensive experiments on all publicly available multiple clustering tasks empirically demonstrate the superiority of the proposed Multi-Sub, with a precise capturing of a user's interest.2 Related Work 
 
 Appendix: A.1 Further Analysis Parameters Analysis. To show the sensitivity of the balancing factor λ that is the only hyperparameter in our proposed method, the experiments were conducted on CIFAR-10. We varied the value of λ from 0.1 to 1.0 in increments of 0.1 to observe its effect on the model's performance. As shown in Fig. 4 , we can observe that different values of λ can work with our method and the optimal performance for "Type" & "Environment" clustering is achieved when λ is set to 0.5. When λ is too low, the model focuses too much on maximizing the distances between different clusters, which can lead to less cohesive clusters. Conversely, when λ is too high, the model emphasizes compactness within clusters at the expense of inter-cluster separation, leading to less distinct clusters. Therefore, we set λ to be 0.5 for all datasets, which confirms the robustness of our method.Balancing Factor λ Model Adaptability. To evaluate how Multi-Sub adapts to new user demands not originally provided in the ground-truth of the dataset, we conducted an additional experiment using the Fruit dataset. Specifically, we introduced a new demand based on the shape of the fruits, with the prompt set as "fruit with the shape of *". We categorized the fruits into two shapes, that is, round and elongated. Although this specific demand may not be common in practical applications, it serves as an exploratory experiment to test the adaptability of our method.The results in Table 7 demonstrate that Multi-Sub successfully adapted to the new user demand of shape. The model learned to align the textual descriptions of shapes with the visual features, resulting in a clustering under the new subspace of shape. Justification: The main claims in the abstract and introduction are supported by detailed descriptions, empirical evaluations, and theoretical analysis provided in the body of the paper (Sections 1, 3, and 4). Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper received 4 reviews from experts in the field. The paper received the following reviews:  1 Accept, 1 Weak Accept and 2 Borderline Accepts. The paper has clearly positive reviews and the decision is to accept. Please incorporate reviewer comments in your final camera ready submission.

Reviews: review: paper_summary: paper_summary: This paper introduces an end-to-end multi-clustering method that integrates a multimodal subspace proxy learning framework. It combines text prompts expressing user preferences with corresponding visual representations to achieve clustering based on user interests.
strength_weakness: strength_weakness: 1.The contributions of the paper are not very clear. At first glance, it appears to merely combine CLIP and GPT, lacking innovative architecture.
2.The baseline methods chosen for comparison are neither cited nor introduced.
3.Section 5 discusses only limitations, lacking a discussion on broader impacts.
questions: questions: The evaluation metrics mentioned in the paper require comparing results with ground truth values. How were the multiple clustering ground truth values in the dataset obtained? How is the accuracy of these ground truth values ensured?
limitations: limitations: The authors only address the limitations of their work and do not discuss broader impacts.
paper_summary: paper_summary: The paper is about Multiple Clustering, which is an interesting topic. The authors propose a novel end-to-end multiple clustering approach that incorporates a multi-modal subspace proxy learning framework. The paper is well written and well organized. However, there are several concerns in the current version of the paper that addressing them will increase the quality of this paper.
strength_weakness: strength_weakness: 1 The authors point out that different clustering results can be given for different customization needs of users. Then it will bring several associations (not necessarily accurate): a. What should be done if the user's demand is exactly opposite to the potential clustering distribution? b. The experiments do give different clustering results for different demand types, if the user proposes a new type of demand, can the model also adaptively adjust?
2 Figure 2 is well drawn but could be further improved, some icons and fonts need to be adjusted.
3 The authors point out that their model is capable of outputting clustering results directly, and then there should be a corresponding formula to represent this. In addition, it is hoped that the authors will discuss further why, if it is not a difficult task to output clustering results directly, few previous methods have done so.
4 Authors should add details about the dataset, such as data size, feature types, etc.
questions: questions: Considering that the authors did not add an appendix, are there any other discussions or experiments?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents an innovative approach for addressing the limitations of existing multiple clustering methods. By leveraging the synergistic capabilities of CLIP and GPT-4, Multi-Sub aligns textual prompts with visual representations to cater to diverse user-specific clustering needs. This method introduces a novel multi-modal subspace proxy learning framework, which automatically generates proxy words from large language models to represent data in terms specific to user interests. The experimental results demonstrate that Multi-Sub consistently outperforms existing baselines across various datasets. Overall, I believe this paper makes a substantial contribution to the field of deep clustering and holds significant practical application value.
strength_weakness: strength_weakness: Despite its strengths, there are some areas where the paper could be improved to enhance its clarity and applicability:
1. Although the paper mentions the hyperparameters used, a more detailed analysis and discussion on the sensitivity of the method to these parameters would be beneficial.
2. Given the method's iterative nature and the use of large models, there is a risk of overfitting, especially on smaller datasets. I am curious whether regularization techniques were used to address this issue?
3. Table 3 compares the impact of different text encoders on performance. Clearly, there are significant performance differences when using different encoders, and the authors have indeed analyzed this issue. However, I believe the reasons behind this phenomenon could be explored in depth. Intuitively, given that the input text is quite simple, the overall performance should not be particularly sensitive to the choice of text encoder.
questions: questions: 1. Although the paper mentions the hyperparameters used, a more detailed analysis and discussion on the sensitivity of the method to these parameters would be beneficial.
2. Given the method's iterative nature and the use of large models, there is a risk of overfitting, especially on smaller datasets. I am curious whether regularization techniques were used to address this issue?
3. Table 3 compares the impact of different text encoders on performance. Clearly, there are significant performance differences when using different encoders, and the authors have indeed analyzed this issue. However, I believe the reasons behind this phenomenon could be explored in depth. Intuitively, given that the input text is quite simple, the overall performance should not be particularly sensitive to the choice of text encoder.
limitations: limitations: The limitations are discussed in the paper by the authors. There is no potential negative societal impact
paper_summary: paper_summary: This paper incorporates a multi-model subspace proxy learning (Multi-Sub) to design a novel end-to-end multiple clustering approach and utilizes the synergistic capabilities of CLIP and GPT-4 to align textual prompts expressing user preferences with corresponding visual representations. The main contributions of Multi-Sub can be summarized as follows:
1. Capturing user’s clustering interest: Existing works struggle to adapt to diverse user-specific needs in data grouping. To overcome this limitation, Multi-Sub explicitly captures a user’s clustering interest by learning the desired clustering proxy under a user’s interest and aligning textual interest with corresponding visual features.
2. Simultaneous optimized framework: The previous methods separated the representation learning and clustering stages. Different from them, Multi-Sub obtains both the desired representations and clustering simultaneously, which significantly improve the clustering performance and efficiency.
3. Extensive experimental validation: Extensive experiments on all public multiple clustering tasks demonstrate that Multi-Sub outperform other methods. Moreover, a series of ablation studies further verify the effectiveness of Multi-Sub.
strength_weakness: strength_weakness: 1. To change the two-stage learning approach of previous works, Multi-Sub aims to learn representation and clustering simultaneously. However, Multi-Sub employs a two-phase iterative approach to align and cluster images in training process, including (1) Phase I: Learning and Alignment; (2) Phase II: Clustering and Optimization. I wonder if this is another form of two-stage task.
2. The description of Clustering Loss is not very clear in Section 3.4, how to determine that samples belong to the same class? By pseudo-label? Where did the pseudo-label come from?
3. In this paper, the authors introduced large language models (LLMs) to learn representations and bridge the gap of textual and image features. But does the direct use of a pre-trained large language model introduce a priori information about the category, which can lead to unsupervised scenarios being corrupted?
questions: questions: Please refer to the Weaknesses.
limitations: limitations: The authors have discussed social impacts and limitations.

Comments: comment: We highly appreciate your acknowledgment and will carefully take your insightful feedback to improve our revision.
Thank you for he response. All my concerns have been addressed, and I recognize the contributions of this paper. Therefore, I will raise my score.
Many thanks again for your insightful comments for improving our work. If applicable, your further suggestion on any remaining issues would be greatly appreciated.
Thanks for the response. Some of my concerns have been addressed. After reading the responses and the comments from other reviewers, I would maintain the original score.
We are very grateful for your appreciation and endorsement. Your feedback holds significant value in helping us enhance our work. We will carefully incorporate them in our revision.
I appreciate the authors' detailed responses to my feedback. After carefully reviewing the rebuttal, I am satisfied that they have fully addressed all of my concerns. The datasets used in this work is quite comprehensive, and the detailed description provided so far has greatly helped my understanding. Overall, the Multi-Sub shows strong potential for user-friendly clustering tasks, which I find highly valuable. The method's contribution is clear: it effectively aligns user needs with image-level features by using textual information as a bridge, all within an end-to-end training process that integrates both textual representations and clustering-oriented fusion.

In light of the novelty of the proposed method and the improvements made, I'm raising my rating to 7.