Agent-Centric Personalized Multiple Clustering with Multi-Modal LLMs

## Abstract

Abstract Personalized multiple clustering aims to generate diverse partitions of a dataset based on different user-specified aspects, rather than a single clustering. It has recently drawn research interest for accommodating varying user clustering preferences.
Recent approaches primarily use CLIP embeddings with proxy learning to extract representations biased toward user interests. However, CLIP primarily focuses on coarse image-text alignment, lacking a deep contextual understanding of user interests.
To overcome these limitations, we propose an agent-centric personalized clustering framework that leverages multi-modal large language models (MLLMs) as agents to comprehensively traverse a relational graph to search for clusters based on user interests. Due to the advanced reasoning mechanism of MLLMs, the obtained clusters align more closely with user-defined criteria than those obtained from CLIP-based representations. To reduce computational overhead, we shorten the agents’ traversal path by constructing a relational graph using user-interest-biased embeddings extracted by MLLMs.
A large number of weak edges can be filtered out based on embedding similarity, facilitating an efficient traversal search for agents. Experimental results show that the proposed method achieves NMI scores of 0.9667 0.9667 0.9667 0.9667 and 0.9481 0.9481 0.9481 0.9481 on the Card Order and Card Suits benchmarks, respectively, largely improving the SOTA model by over 140 % percent 140 140\% 140 % .

## 1 Introduction

Clustering is a fundamental technique that partitions data into meaningful groups, uncovering underlying structures within a dataset. Traditional clustering methods  [23, 3, 4, 5] rely on handcrafted features or fixed representations, which may fail to capture the inherent complex data relationships. Recent advances [20, 6, 25, 24, 26, 18, 8] leverage learning-based techniques to obtain more expressive representations, leading to significant performance improvements. Most of these approaches produce only one data partition from a single perspective.

Figure: Figure 1: The workflow of the proposed agent-centric multiple clustering framework, which obtains a personalized clustering by using MLLMs as agents to traverse a relational graph based on user preferences. The relational graph is constructed from MLLM embeddings biased toward user interests.
Refer to caption: x1.png

However, real-world data exhibit inherent complexity, making it impossible for any single clustering method to capture all relevant structures. Consequently, identifying multiple valid clusterings within a dataset is essential to address various analytical needs. For example, the fruits in Fig. [1](https://arxiv.org/html/2503.22241v3#S1.F1) can be grouped by color or species, depending on the perspective. This need has driven the development of personalized multiple clustering, which aims to generate diverse partitions of a dataset based on different user clustering preferences, rather than a single fixed solution.

Typical multiple clustering methods [22, 29, 35] primarily leverage autoencoders and data augmentation to extract diverse data representations, enabling clustering from different perspectives. However, identifying which clustering outcome aligns with user interests remains challenging, as these interests are often expressed through abstract keywords—such as color or species in the case of fruits, as illustrated in Fig. [1](https://arxiv.org/html/2503.22241v3#S1.F1). Recent approaches [37, 36] integrate CLIP [27, 16, 15, 17] embeddings with proxy learning to obtain data representations tailored to user clustering preferences. While effective, CLIP is trained on coarse-grained image-text pairs and is designed for overall cross-modal alignment, limiting its ability to focus solely on fine-grained, user-specified aspects.

To address these limitations, we propose an agent-centric personalized multiple clustering framework that employs shared multi-modal large language models (MLLMs) as agents to assess data relationships based on user interests and search for clusters accordingly. As shown in Fig. [1](https://arxiv.org/html/2503.22241v3#S1.F1), we first construct a relational graph where nodes represent input images and edges encode pairwise similarities. Each agent is assigned to a connected component of the graph and initializes a cluster from the highest-degree node within that component. It then expands the cluster by traversing neighboring nodes, evaluating their membership based on user interests, and updating the cluster boundaries accordingly. Once a cluster is completed, the agent selects the next highest-degree unassigned node in the component to start a new cluster, repeating the traversal process until all nodes are assigned. After all agents complete, another agent performs a global review to merge semantically redundant clusters. Errors in graph connectivity are corrected through agent-based graph traversal and cluster merging. Leveraging the reasoning capabilities of MLLMs, our method yields clusters more faithfully aligned with user-defined criteria than those from CLIP-based approaches.

A naive approach to constructing the relational graph is to fully connect all nodes, resulting in an overly dense graph with many spurious edges. This significantly increases the traversal steps needed for agents to discard falsely connected neighbors, thereby compromising clustering efficiency. To address this, we build a weighted graph where edge weights reflect similarities between data embeddings extracted by MLLMs conditioned on user interests. By pruning weak edges based on similarity scores, we retain a compact set of high-quality neighbors, substantially reducing the traversal steps required for agents to evaluate them and improving clustering efficiency. Specifically, for each input image, the MLLM generates a textual description containing an embedding token based on user interests, where the hidden state of this token is projected as the data embedding. Unlike CLIP-based embeddings, which are limited by coarse-grained modality alignment, MLLM-based embeddings adaptively capture fine-grained, user-specified semantics through contextual reasoning, resulting in a sparser relational graph with primarily meaningful edges.

We evaluate our method on five publicly available visual datasets commonly used for multiple clustering tasks, and the results demonstrate its state-of-the-art performance. For instance, our method achieves NMI scores of $0.9667$ and $0.9481$ on the Card Order and Card Suits benchmarks, respectively, surpassing the current state-of-the-art model by over $140\%$. To the best of our knowledge, this is the first multiple clustering method built on MLLMs, providing a promising approach to unify personalized clustering and agent-based searching.

## 2 Related Work

#### Multiple Clustering

Multiple clustering explores diverse data partitions from different perspectives, gaining increasing attention. Early methods rely on hand-crafted rules and representations. For example, COALA [2] generates new clusters using existing ones as a constraint, Hu et al. [13] maximized eigengap across subspaces, and Dang et al. [7] utilizes an expectation-maximization framework to optimize mutual information. Recent approaches leverage learning-based techniques for better representations. For instance, ENRC [22] optimizes clustering objectives within a latent space learned by an auto-encoder, iMClusts [29] leverages auto-encoders and multi-head attention to learn diverse feature representations, and AugDMC [35] applies data augmentation to generate diverse image perspectives. However, it remains challenging to identify the clustering most relevant to user interests. Recently, Multi-MaP [37] and Multi-Sub [36] integrate CLIP embeddings with proxy learning to generate data representations aligned with user interests. While effective, CLIP lacks the deep contextual understanding necessary to capture abstract and nuanced user interests.

#### LLM-Driven Agent Search

LLM-driven agent search integrates Large Language Models (LLMs) into search processes, enabling agents to reason, plan, and interact with tools for efficient information retrieval. For example, PaSa [11] employs LLM-driven agents to search, read papers, and select references, ensuring comprehensive and accurate results for scholarly queries. Toolformer [30] demonstrates how LLMs can interact with external APIs (e.g., search engines, calculators) to enhance factual accuracy. ReAct [38] combines reasoning and acting, allowing agents to iteratively retrieve and process information for more effective decision-making. Recently, this field has shifted toward multi-agent search, emphasizing cooperative problem-solving, knowledge sharing, and decentralized decision-making. For example, CAMEL [14] introduces LLM agents with specialized roles (e.g., teacher and student) to refine search strategies collaboratively. Voyager [31] applies LLMs in an open-world exploration setting (Minecraft), where agents autonomously collect, analyze, and apply new knowledge. In this paper, we apply MLLM-driven agents to search clusters based on user clustering preferences.

## 3 Method

In this section, we introduce our agent-centric personalized multiple clustering framework. We begin with an overview of the framework, followed by a detailed description of each component: agent-centric graph traversal, MLLM-based graph construction, and agent-based membership assessment.

Figure: Figure 2: Overview of the Agent-Centric Personalized Multiple Clustering Framework. (a) MLLM-based graph construction, where image embeddings are extracted using MLLM based on user interests, from which a relational graph is constructed. (b) Agent-centric graph traversal, where agents search for clusters by traversing the graph. (c) Rollout of the graph traversal process, where agents expand cluster iteratively by assessing neighboring nodes based on user-defined criteria.
Refer to caption: x2.png

### 3.1 Overall Framework

The framework, illustrated in Fig. [2](https://arxiv.org/html/2503.22241v3#S3.F2), begins by extracting image embeddings using an MLLM-based embedding extractor, tailoring representations to user interests. A relational graph is then constructed from these embeddings, where a large number of weak edges are filtered out based on embedding similarities. Next, multiple MLLM-based agents traverse this graph to search for clusters aligned with user preferences, with each agent specializing in a different connected component of the graph. After all agents complete, another agent performs a global review to merge semantically redundant clusters. This agent-centric approach produces clusters that better adhere to user-defined criteria than those obtained from CLIP-based representations.

### 3.2 Agent-Centric Graph Traversal

Here, we describe our agent-centric graph traversal approach for cluster discovery. We begin with a relational graph $\mathcal{G}=\{\mathcal{V},\mathcal{E},\mathcal{W}\}$ constructed as detailed in Sec [3.3](https://arxiv.org/html/2503.22241v3#S3.SS3), where each node $u\in\mathcal{V}$ represents an input image, and each edge $\{u,v\}\in\mathcal{E}$ carries a weight $w(u,v)\in\mathcal{W}$ indicating a precomputed, embedding-based similarity score between $u$ and $v$. Leveraging shared multi-modal large language models (MLLMs) as agents, we traverse $\mathcal{G}$ to search for clusters $\mathcal{S}$ aligned with user interests $T$.

As outlined in Algorithm [1](https://arxiv.org/html/2503.22241v3#algorithm1), we first identify the connected components of the relational graph $\mathcal{G}$, resulting in a total of $M$ components:

$$ $\{\mathcal{C}_{i}\}_{i=1}^{M}=\text{ConnectedComponents}(\mathcal{G}).$ (1) $$

For each connected component $\mathcal{C}_{i}$, we select the highest-degree node $v_{i}^{*}$ and designate it as the seed for an initial cluster $\mathcal{S}_{i}$ as follows:

$$ $v_{i}^{*}=\arg\max_{v\in\mathcal{C}_{i}}\sum_{u\in\mathcal{N}(v)}w(v,u),\quad \mathcal{S}_{i}=\big{\{}v_{i}^{*}\big{\}},$ (2) $$

where $\mathcal{N}(v)$ is the neighborhood of node $v$. We then assign an MLLM-based agent $\mathcal{A}_{i}$ to each initial cluster $\mathcal{S}_{i}$, resulting in a total of $M$ agents, denoted as $\{\mathcal{A}_{i}\}_{i=1}^{M}$. Each agent $\mathcal{A}_{i}$ iteratively expands its corresponding cluster $\mathcal{S}_{i}$ by evaluating the membership of its neighboring nodes in $\mathcal{S}_{i}$ based on user interests $T$. As shown in Fig. [2](https://arxiv.org/html/2503.22241v3#S3.F2) (c) and Algorithm 1, at each step, the agent $\mathcal{A}_{i}$ first determines the neighborhood of the current cluster $\mathcal{S}_{i}$, defined as:

$$ $\mathcal{N}(\mathcal{S}_{i})=\bigcup_{v\in\mathcal{S}_{i}}\mathcal{N}(v) \setminus\mathcal{S}_{i}.$ (3) $$

Next, the agent selects a candidate neighboring node $v_{j}^{*}$ from $\mathcal{N}(\mathcal{S}_{i})$ that exhibits the highest weighted connectivity to $\mathcal{S}_{i}$, given by:

$$ $v_{j}^{*}=\arg\max_{v\in\mathcal{N}(\mathcal{S}_{i})}\sum_{u\in\mathcal{S}_{i} }w(v,u).$ (4) $$

The agent then assesses whether $v_{j}^{*}$ should be included in $\mathcal{S}_{i}$ based on user-defined criteria $T$, as detailed in Sec. [3.4](https://arxiv.org/html/2503.22241v3#S3.SS4). If the candidate node $v_{j}^{*}$ is deemed a valid member, it is merged into the current cluster $\mathcal{S}_{i}$, updating $\mathcal{S}_{i}$ as follows:

$$ $\mathcal{S}_{i}\leftarrow\mathcal{S}_{i}\cup\big{\{}v_{j}^{*}\big{\}}.$ (5) $$

Otherwise, the edges between the candidate node $v_{j}^{*}$ and the current cluster $\mathcal{S}_{i}$ are removed from the edge set $\mathcal{E}$, yielding an updated edge set:

$$ $\mathcal{E}\leftarrow\mathcal{E}\setminus\{(u,v_{j}^{*})\in\mathcal{E}\mid u \in\mathcal{S}_{i}\}.$ (6) $$

The process from Eq. ([3](https://arxiv.org/html/2503.22241v3#S3.E3)) to Eq. ([6](https://arxiv.org/html/2503.22241v3#S3.E6)) is repeated until the neighborhood $\mathcal{N}(\mathcal{S}_{i})$ is empty, determining the cluster $\mathcal{S}_{i}$. The agent then removes the nodes in $\mathcal{S}_{i}$ from the component $\mathcal{C}_{i}$ as follows:

$$ $\mathcal{C}_{i}\leftarrow\mathcal{C}_{i}\setminus\mathcal{S}_{i}.$ (7) $$

Next, the agent selects the highest-degree unassigned node in $\mathcal{C}_{i}$ to initiate a new cluster, repeating the traversal process from Eq. ([2](https://arxiv.org/html/2503.22241v3#S3.E2)) to Eq. ([7](https://arxiv.org/html/2503.22241v3#S3.E7)) until all nodes in $\mathcal{C}_{i}$ are assigned. The $M$ agents, $\{\mathcal{A}_{i}\}_{i=1}^{M}$, traverse the graph $\mathcal{G}$ in parallel, with the searched clusters collected into the set $\mathcal{S}$. Finally, another agent performs a global review to merge semantically redundant clusters in $\mathcal{S}$ based on user interests $T$. Incorrect or missing edges in the graph $\mathcal{G}$ are corrected through agent-based graph traversal and cluster merging. Leveraging the contextual reasoning capabilities of MLLMs, our method yields clusters more faithfully aligned with user interests compared to CLIP-based approaches.

Figure: Algorithm 1 Agent-Centric Graph Traversal for Personalized Multiple Clustering

### 3.3 MLLM-Based Graph Construction

Here, we detail the construction of the relational graph $\mathcal{G}=\{\mathcal{V},\mathcal{E},\mathcal{W}\}$ used for agent traversal. Each node $u\in\mathcal{V}$ corresponds to an input image. Each edge $(u,v)\in\mathcal{E}$ is assigned a weight $w(u,v)\in W$, determined by the similarity between the embeddings $\mathcal{H}(u)$ and $\mathcal{H}(v)$. These embeddings are extracted by the MLLM for nodes $u$ and $v$ based on user interests $T$.

As shown in Fig. [3](https://arxiv.org/html/2503.22241v3#S3.F3) (a), we first design an embedding generation instruction based on user interests $T$. For example, if the user intends to cluster fruit images by “color”, the instruction is formulated as: “Describe the color of the fruit in the provided image in detail and generate <embedding> based on the description.” We then provide the image $u$ and the instruction as inputs to the MLLM, which responds with a reasoning statement followed by an <embedding> token. The hidden state of the <embedding> token is then projected to obtain the user-interest-biased embedding $\mathcal{H}(u)$. After obtaining the embeddings $\mathcal{H}(u)$ and $\mathcal{H}(v)$, the edge weight $w(u,v)$ is computed as:

$$ $w(u,v)=\sigma(\beta\cdot\mathcal{H}(u)^{\top}\mathcal{H}(v)),$ (8) $$

where $\beta$ is a learnable logit scale, and $\sigma$ denotes the sigmoid function. An edge $(u,v)$ is retained in the edge set $\mathcal{E}$ only if its weight $w(u,v)$ exceeds a predefined threshold $\tau$.

As shown in Fig. [3](https://arxiv.org/html/2503.22241v3#S3.F3) (b), for training data generation, we perform hard negative mining to select the top $1,024$ image pairs with the highest similarity entropy in each epoch. GPT-4 [1] evaluates the similarity of these pairs based on user interests $T$, from which binary pseudo labels are inferred. It also generates detailed descriptions for each image, with an appended <embedding> token. During training, the MLLM produces descriptions and embeddings for each image pair conditioned on $T$. We compute embedding similarities following Eq. ([8](https://arxiv.org/html/2503.22241v3#S3.E8)) and supervise them using GPT-4’s pseudo labels via binary cross-entropy loss. The generated descriptions are aligned with GPT-4’s outputs using cross-entropy loss. The MLLM is finetuned with LoRA [12].

Figure: Figure 3: Illustration of image embedding extraction using MLLM based on user interests.
Refer to caption: x3.png

Figure: Figure 4: Illustration of Agent-based assessment of candidate membership according to user interests.
Refer to caption: x4.png

### 3.4 Agent-Based Membership Assessment

Here, we describe how the agent $\mathcal{A}_{i}$ assesses whether a candidate node $v_{j}^{*}$ should be included in the current cluster $\mathcal{S}_{i}$ based on user interests $T$. As illustrated in Fig. [4](https://arxiv.org/html/2503.22241v3#S3.F4) (a), we first select the top-$K$ nodes with the highest degrees within $\mathcal{S}_{i}$ to form a set of representative nodes:

$$ $\mathcal{R}_{i}=\arg\max_{\begin{subarray}{c}R\subseteq\mathcal{S}_{i}\\ |\mathcal{R}|=min(K,|\mathcal{S}|)\end{subarray}}\sum_{v\in R}\sum_{u\in \mathcal{N}(v)\cap\mathcal{S}_{i}}w(v,u),$ (9) $$

where $\mathcal{R}_{i}$ serves as a compact representation of the current cluster $\mathcal{S}_{i}$. We then design a membership assessment instruction tailored to user interests $T$. For instance, if the user aims to cluster fruit images by “color”, the instruction is: “Determine whether the candidate fruit should be included in the existing fruit cluster based on color similarity. Respond with ‘yes’ or ‘no’.” The candidate node $v_{j}^{*}$ and the representative node set $\mathcal{R}_{i}$ are provided to the agent $\mathcal{A}_{i}$ along with this instruction. The agent responds with a reasoning statement followed by a binary decision. Based on the agent’s assessment, we update the current cluster $\mathcal{S}_{i}$ as described in Eqs. ([5](https://arxiv.org/html/2503.22241v3#S3.E5)) and ([6](https://arxiv.org/html/2503.22241v3#S3.E6)).

As shown in Fig. [4](https://arxiv.org/html/2503.22241v3#S3.F4) (b), for training data generation, we begin by constructing a relational graph using embeddings extracted by the MLLM. We then uniformly sample a subgraph containing $1,024$ nodes from the relational graph. GPT-4 traverses this subgraph according to Eqs. ([1](https://arxiv.org/html/2503.22241v3#S3.E1))–([7](https://arxiv.org/html/2503.22241v3#S3.E7)), assessing the membership of each neighboring node with respect to the current cluster based on user interests $T$, as defined in Eq. [9](https://arxiv.org/html/2503.22241v3#S3.E9). We collect these assessments at each traversal step as training samples. During training, the MLLM predicts membership for each sampled neighboring node and its corresponding cluster based on $T$. These predictions are supervised using GPT-4’s assessments via cross-entropy loss. The MLLM is finetuned using LoRA [12].

## 4 Experiment

**Table 1: Comparison with state-of-the-art methods across multiple clustering benchmarks.**
|  |  | Fruit | Fruit360 | Card | CMUface | CIFAR10-MC |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Color | Species | Color | Species | Order | Suits | Emotion | Sunglass | Identity | Pose | Type | Environment |
| MSC[13] | NMI | 0.6886 | 0.1627 | 0.2544 | 0.2184 | 0.0807 | 0.0497 | 0.1284 | 0.1420 | 0.3892 | 0.3687 | 0.1547 | 0.1136 |
| RI | 0.8051 | 0.6045 | 0.6054 | 0.5805 | 0.7805 | 0.3587 | 0.6736 | 0.5745 | 0.7326 | 0.6322 | 0.3296 | 0.3082 |  |
| MCV[9] | NMI | 0.6266 | 0.2733 | 0.3776 | 0.2985 | 0.0792 | 0.0430 | 0.1433 | 0.1201 | 0.4637 | 0.3254 | 0.1618 | 0.1379 |
| RI | 0.7685 | 0.6597 | 0.6791 | 0.6176 | 0.7128 | 0.3638 | 0.5268 | 0.4905 | 0.6247 | 0.6028 | 0.3305 | 0.3344 |  |
| ENRC[22] | NMI | 0.7103 | 0.3187 | 0.4264 | 0.4142 | 0.1225 | 0.0676 | 0.1592 | 0.1493 | 0.5607 | 0.2290 | 0.1826 | 0.1892 |
| RI | 0.8511 | 0.6536 | 0.6868 | 0.6984 | 0.7313 | 0.3801 | 0.6630 | 0.6209 | 0.7635 | 0.5029 | 0.3469 | 0.3599 |  |
| iMClusts[29] | NMI | 0.7351 | 0.3029 | 0.4097 | 0.3861 | 0.1144 | 0.0716 | 0.0422 | 0.1929 | 0.5109 | 0.4437 | 0.2040 | 0.1920 |
| RI | 0.8632 | 0.6743 | 0.6841 | 0.6732 | 0.7658 | 0.3715 | 0.5932 | 0.5627 | 0.8260 | 0.6114 | 0.3695 | 0.3664 |  |
| AugDMC[35] | NMI | 0.8517 | 0.3546 | 0.4594 | 0.5139 | 0.1440 | 0.0873 | 0.0161 | 0.1039 | 0.5875 | 0.1320 | 0.2855 | 0.2927 |
| RI | 0.9108 | 0.7399 | 0.7392 | 0.7430 | 0.8267 | 0.4228 | 0.5367 | 0.5361 | 0.8334 | 0.5517 | 0.4516 | 0.4689 |  |
| DDMC[34] | NMI | 0.8973 | 0.3764 | 0.4981 | 0.5292 | 0.1563 | 0.0933 | 0.1726 | 0.2261 | 0.6360 | 0.4526 | 0.3991 | 0.3782 |
| RI | 0.9383 | 0.7621 | 0.7472 | 0.7703 | 0.8326 | 0.6469 | 0.7593 | 0.7663 | 0.8907 | 0.7904 | 0.5827 | 0.5547 |  |
| Multi-Map[37] | NMI | 0.8619 | 1.0000 | 0.6239 | 0.5284 | 0.3653 | 0.2734 | 0.1786 | 0.3402 | 0.6625 | 0.4693 | 0.4969 | 0.4598 |
| RI | 0.9526 | 1.0000 | 0.8243 | 0.7582 | 0.8587 | 0.7039 | 0.7105 | 0.7068 | 0.9496 | 0.6624 | 0.7104 | 0.6737 |  |
| Multi-Sub[36] | NMI | 0.9693 | 1.0000 | 0.6654 | 0.6123 | 0.3921 | 0.3104 | 0.2053 | 0.4870 | 0.7441 | 0.5923 | 0.5271 | 0.4828 |
| RI | 0.9964 | 1.0000 | 0.8821 | 0.8504 | 0.8842 | 0.7941 | 0.8527 | 0.8324 | 0.9834 | 0.8736 | 0.7394 | 0.7096 |  |
| Ours | NMI | 1.0000 | 1.0000 | 0.7214 | 0.6532 | 0.9667 | 0.9481 | 0.2196 | 0.9720 | 0.7631 | 0.7789 | 0.6385 | 0.5931 |
| RI | 1.0000 | 1.0000 | 0.8715 | 0.8436 | 0.9952 | 0.9882 | 0.8415 | 0.9936 | 0.9763 | 0.9002 | 0.7812 | 0.7439 |  |

### 4.1 Datasets

We evaluate our method on all publicly available multiple-clustering benchmarks: Card [35], CMUface [10], Fruit [13], Fruit360 [35], and CIFAR10-MC [36]. The Card dataset contains 8,029 playing card images with two clustering criteria: order (Ace–King) and suits (clubs, diamonds, hearts, spades). CMUface includes 640 facial images annotated for pose (left, right, straight, up), identity (20 individuals), sunglasses (with/without), and emotion (angry, sad, happy, neutral). Fruit consists of 105 images clusterable by species (apple, banana, grape) or color (green, red, yellow). Fruit360 extends this with 4,856 images labeled by species (apple, banana, grape, cherry) and color (green, red, yellow, burgundy). CIFAR10-MC includes 60,000 images grouped by type (transportation, animals) and environment (land, air, water).

### 4.2 Implementation Details

We adopt LLaVA [19] as our MLLM model, using Qwen2-7B [33] as the language model. We fine-tune only the language model with LoRA [12]. Both the embedding extractor and the agent are trained for $50$ epochs on $4$ A100 GPUs over $3$ days, with a total batch size of $32$. Optimization is performed using AdamW [21] with a learning rate of $1\mathrm{e}{-5}$ and a cosine decay schedule. We set the embedding similarity threshold $\tau$ to $0.6$ to filter weak edges. Clustering performance is evaluated using Normalized Mutual Information (NMI) [32] and Rand Index (RI) [28].

### 4.3 Results

As shown in Table [1](https://arxiv.org/html/2503.22241v3#S4.T1), our method consistently outperforms state-of-the-art models across all multiple clustering benchmarks. On the Fruit dataset, it achieves perfect NMI and RI scores ($1.0000$) for both color- and species-based clustering. On Fruit360, it attains an NMI of $0.7214$ for color-based clustering, exceeding Multi-Map and Multi-Sub by $15.6\%$ and $8.4\%$, respectively, and an NMI of $0.6532$ for species-based clustering, outperforming them by $24.3\%$ and $6.7\%$. On CIFAR10-MC, our method achieves an NMI of $0.6385$ and RI of $0.7812$ for type-based clustering, surpassing Multi-Sub by $21.1\%$ and $5.7\%$, and an NMI of $0.5931$ and RI of $0.7439$ for environment-based clustering, exceeding Multi-Sub by $22.8\%$ and $4.8\%$. On Card, it reaches $0.9667$ NMI and $0.9952$ RI for order-based clustering, and $0.9481$ NMI and $0.9882$ RI for suits-based clustering, outperforming Multi-Sub by up to $205.4\%$ in NMI and $24.4\%$ in RI. On CMUface, it achieves $0.9720$ NMI and $0.9936$ RI for sunglass-based clustering, and $0.7789$ NMI and $0.9002$ RI for pose-based clustering, showing up to $99.6\%$ and $31.5\%$ improvements over Multi-Sub.

These significant improvements highlight the effectiveness of our agent-centric clustering framework in capturing user interests (e.g., order, suits, sunglass, pose) with advanced contextual understanding capabilities from MLLMs, while minimizing interference from irrelevant information.

Figure: Figure 5: Graph density vs. clustering metrics and traversal steps.
Refer to caption: x5.png

Figure: Figure 6: T-SNE visualization of MLLM-based and CLIP-based embeddings on the Fruit dataset.
Refer to caption: x6.png

### 4.4 Ablation Studies

#### Density of Relational Graph.

We investigate how graph density affects agent traversal by varying the embedding similarity threshold $\tau$ to construct relational graphs of different densities. Experiments are conducted on Fruit360. As shown in Fig. [5](https://arxiv.org/html/2503.22241v3#S4.F5), increasing graph density (i.e., relaxing $\tau$) improves NMI and RI for both color- and species-based clustering until performance saturates. For instance, Color RI and Species RI quickly rise and plateau around $0.87$ and $0.84$, while Color NMI and Species NMI converge near $0.72$ and $0.65$. This suggests that once sufficient edges form connected components of semantically similar nodes, further densification yields diminishing returns.

However, denser graphs incur higher traversal costs. As graph density increases from $0.001$ to $0.015$, the number of agent traversal steps (dark blue curve) rises from $3,000$ to nearly $10,000$ due to a significant increase in spurious edges, indicating substantial computational overhead. A moderate $\tau$ strikes a balance: clustering accuracy stabilizes while reducing unnecessary traversal. These findings underscore the benefits of using MLLM-based embeddings to construct relational graphs. By pruning weak edges with low similarity scores, agents avoid exhaustive searches in irrelevant regions. At the same time, the context-aware nature of MLLM embeddings retains high-quality edges, aligning clusters with user interests while keeping traversal efficient.

**Table 2: Ablation study on the Fruit360, Card, and CMUface datasets.**
| Embedding | Dataset | Aspect | K-Means | HDBSCAN | Agent-Centric Graph Traversal |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NMI | RI | NMI | RI | NMI | RI | Traversal Steps |  |  |  |
| CLIP Emb | Fruit360 | Color | 0.6239 | 0.8243 | 0.6541 | 0.8408 | 0.6922 | 0.8573 | 14821 |
| Species | 0.5284 | 0.7582 | 0.5702 | 0.7851 | 0.6158 | 0.8180 | 14712 |  |  |
| Card | Order | 0.3653 | 0.8587 | 0.4163 | 0.8708 | 0.8464 | 0.9679 | 29456 |  |
| Suits | 0.2734 | 0.7039 | 0.3368 | 0.7275 | 0.8132 | 0.9313 | 14969 |  |  |
| CMUface | Sunglass | 0.3402 | 0.7068 | 0.3952 | 0.7301 | 0.8456 | 0.9362 | 1281 |  |
| Pose | 0.4693 | 0.6624 | 0.4935 | 0.6804 | 0.7168 | 0.8526 | 4015 |  |  |
| MLLM Emb | Fruit360 | Color | 0.6629 | 0.8432 | 0.6912 | 0.8578 | 0.7214 | 0.8715 | 6404 |
| Species | 0.5783 | 0.7924 | 0.6138 | 0.8179 | 0.6532 | 0.8436 | 6964 |  |  |
| Card | Order | 0.7262 | 0.9406 | 0.7743 | 0.9515 | 0.9667 | 0.9952 | 12837 |  |
| Suits | 0.6782 | 0.8745 | 0.7322 | 0.8972 | 0.9481 | 0.9882 | 9970 |  |  |
| CMUface | Sunglass | 0.7193 | 0.9005 | 0.7698 | 0.9191 | 0.9720 | 0.9936 | 798 |  |
| Pose | 0.6551 | 0.8051 | 0.6799 | 0.8241 | 0.7789 | 0.9002 | 1452 |  |  |

**Table 3: Ablation study on number of representative nodes $K$.**
| Num of Representatives | Fruit360-Color | Fruit360-Species | Card-Order | Card-Suits |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NMI | RI | NMI | RI | NMI | RI | NMI | RI |  |
| $K=1$ | 0.7131 | 0.8643 | 0.6424 | 0.8361 | 0.9625 | 0.9930 | 0.9421 | 0.9860 |
| $K=2$ | 0.7185 | 0.8691 | 0.6490 | 0.8402 | 0.9652 | 0.9943 | 0.9461 | 0.9873 |
| $K=3$ | 0.7214 | 0.8715 | 0.6532 | 0.8436 | 0.9667 | 0.9952 | 0.9481 | 0.9882 |
| $K=4$ | 0.7240 | 0.8730 | 0.6559 | 0.8457 | 0.9673 | 0.9957 | 0.9492 | 0.9888 |

#### MLLM-based vs. CLIP-based Embeddings.

We compare our MLLM-based embeddings with CLIP-based ones across various clustering methods. We use Multi-Map [37] as the CLIP baseline, as both approaches decouple embedding extraction from clustering for fair ablation. As shown in Table [2](https://arxiv.org/html/2503.22241v3#S4.T2), MLLM-based embeddings consistently outperform CLIP-based ones. For example, on CMUface (“Sunglass” aspect), MLLM-based embeddings improve NMI from $0.3402$, $0.3952$, and $0.8456$ to $0.7193$, $0.7698$, and $0.9720$ for K-means, HDBSCAN, and agent-centric clustering, respectively—demonstrating superior alignment with user-defined criteria. Fig. [6](https://arxiv.org/html/2503.22241v3#S4.F6) further illustrates this: MLLM-based embeddings yield compact, well-separated clusters in t-SNE visualization on the Fruit dataset, while CLIP-based embeddings show higher intra-cluster variance, particularly in color-based clustering. These results confirm that MLLM embeddings more effectively capture user interests (e.g., color, species), leading to improved clustering accuracy.

Moreover, MLLM-based embeddings significantly reduce agent traversal steps in agent-centric clustering. On Fruit360 (“Color” aspect), agent traversal completes in $6,404$ steps with MLLM embeddings, compared to $14,821$ with CLIP-based ones. This efficiency arises from the ability of MLLM embeddings to better prune noisy edges, retaining a compact set of high-quality neighbors and reducing the number of membership assessments needed during graph traversal.

#### Agent-Centric Clustering vs. K-Means & HDBSCAN.

We compare our agent-centric graph traversal clustering with classical K-Means and HDBSCAN. As shown in Table [2](https://arxiv.org/html/2503.22241v3#S4.T2), the agent-centric approach consistently outperforms both baselines in NMI and RI across all datasets and embedding types. On the Card dataset (“Order” aspect) with MLLM-based embeddings, it achieves an NMI of $0.9667$, outperforming K-Means ($0.7262$) and HDBSCAN ($0.7743$). The improvement is even larger with CLIP-based embeddings, reaching an NMI of $0.8464$ vs. $0.3653$ (K-Means) and $0.4163$ (HDBSCAN), demonstrating the robustness of the agent-centric approach. Furthermore, it requires fewer traversal steps with MLLM-based embeddings than with CLIP-based ones, improving efficiency.

#### Number of Representative Nodes.

Table [3](https://arxiv.org/html/2503.22241v3#S4.T3) shows the effect of varying $K$ on clustering performance (NMI and RI). Using a single representative node may fail to capture the cluster’s diversity, leading to lower performance. Increasing $K$ improves both metrics by better reflecting the cluster’s structure, although gains beyond three or four representatives are typically modest. Thus, a moderate $K$ strikes the best balance between capturing diversity and avoiding redundant information.

## 5 Impacts and Limitations

Our main limitation is relying on GPT-4 for generating pseudo labels for MLLM’s embedding and agent training, as current open-source MLLMs like LLaVA lack the ability to produce accurate or semantically rich descriptions. Future work will explore stronger open-source MLLMs capable of generating their own supervision signals, enabling self-supervised embedding training. This paradigm offers a flexible and scalable solution for clustering. Our approach supports customizable clustering criteria and is well-suited for industrial applications such as search and graph mining.

## 6 Conclusion

This paper presents an agent-centric clustering framework that leverages MLLMs to traverse relational graphs and search for clusters based on user interests. By leveraging MLLMs’ reasoning abilities, our method better captures user clustering preferences than CLIP-based approaches. To improve efficiency, we construct relational graphs from user-interest-biased embeddings derived from MLLMs, and reduce agents’ traversal steps by filtering weak connections. Experiments on multiple benchmarks demonstrate the state-of-the-art performance of our method, validating its effectiveness.

## References

- [1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya,
Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman,
Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv preprint arXiv:2303.08774, 2023.
- [2]
Eric Bae and James Bailey.
Coala: A novel approach for the extraction of an alternate clustering
of high quality and high dissimilarity.
In Sixth International Conference on Data Mining (ICDM’06),
pages 53–62. IEEE, 2006.
- [3]
Christopher M Bishop and Nasser M Nasrabadi.
Pattern recognition and machine learning, volume 4.
Springer, 2006.
- [4]
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.
Deep clustering for unsupervised learning of visual features.
In Proceedings of the European conference on computer vision
(ECCV), pages 132–149, 2018.
- [5]
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and
Armand Joulin.
Unsupervised learning of visual features by contrasting cluster
assignments.
Advances in neural information processing systems,
33:9912–9924, 2020.
- [6]
Tianzhe Chu Chu, Shengbang Tong, Tianjiao Ding, Xili Dai, Benjamin Haeffele,
Rene Vidal, and Yi Ma.
Image clustering via the principle of rate reduction in the age of
pretrained models.
International Conference on Learning Representations (ICLR), 2024.
- [7]
Xuan Hong Dang and James Bailey.
Generation of alternative clusterings using the cami approach.
In Proceedings of the 2010 SIAM International Conference on Data
Mining, pages 118–129. SIAM, 2010.
- [8]
Yiqun Duan, Sameera Ramasinghe, Stephen Gould, and Ajanthan Thalaiyasingam.
Scaling prompt instructed zero shot composed image retrieval with
image-only data.
arXiv preprint arXiv:2504.00812, 2025.
- [9]
Joris Guérin and Byron Boots.
Improving image clustering with multiple pretrained cnn feature
extractors.
arXiv preprint arXiv:1807.07760, 2018.
- [10]
Stephan Günnemann, Ines Färber, Matthias Rüdiger, and Thomas Seidl.
Smvc: semi-supervised multi-view clustering in subspace projections.
In Proceedings of the 20th ACM SIGKDD international conference
on Knowledge discovery and data mining, pages 253–262, 2014.
- [11]
Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, et al.
Pasa: An llm agent for comprehensive academic paper search.
arXiv preprint arXiv:2501.10120, 2025.
- [12]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
ICLR, 1(2):3, 2022.
- [13]
Juhua Hu, Qi Qian, Jian Pei, Rong Jin, and Shenghuo Zhu.
Finding multiple stable clusterings.
Knowledge and Information Systems, 51:991–1021, 2017.
- [14]
G Li, HAAK Hammoud, H Itani, D Khizbullin, and B Ghanem.
Camel: communicative agents for" mind" exploration of large scale
language model society.(2023).
arXiv preprint arXiv:2303.17760.
- [15]
Yi Li, Hualiang Wang, Yiqun Duan, and Xiaomeng Li.
Clip surgery for better explainability with enhancement in
open-vocabulary tasks.
arXiv e-prints, pages arXiv–2304, 2023.
- [16]
Yi Li, Hualiang Wang, Yiqun Duan, Hang Xu, and Xiaomeng Li.
Exploring visual interpretability for contrastive language-image
pre-training.
arXiv preprint arXiv:2209.07046, 2022.
- [17]
Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li.
A closer look at the explainability of contrastive language-image
pre-training.
Pattern Recognition, page 111409, 2025.
- [18]
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng.
Contrastive clustering.
In Proceedings of the AAAI conference on artificial
intelligence, volume 35, pages 8547–8555, 2021.
- [19]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
Advances in neural information processing systems, 36, 2024.
- [20]
Yaroslava Lochman, Carl Olsson, and Christopher Zach.
Learned trajectory embedding for subspace clustering.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 19092–19102, 2024.
- [21]
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
arXiv preprint arXiv:1711.05101, 2017.
- [22]
Lukas Miklautz, Dominik Mautz, Muzaffer Can Altinigneli, Christian Böhm,
and Claudia Plant.
Deep embedded non-redundant clustering.
In Proceedings of the AAAI conference on artificial
intelligence, volume 34, pages 5174–5181, 2020.
- [23]
Andrew Ng, Michael Jordan, and Yair Weiss.
On spectral clustering: Analysis and an algorithm.
Advances in neural information processing systems, 14, 2001.
- [24]
Rabah Ouldnoughi, Chia-Wen Kuo, and Zsolt Kira.
Clip-gcd: Simple language guided generalized category discovery.
arXiv preprint arXiv:2305.10420, 2023.
- [25]
Qi Qian.
Stable cluster discrimination for deep clustering.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pages 16645–16654, 2023.
- [26]
Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin.
Unsupervised visual representation learning by online constrained
k-means.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 16640–16649, 2022.
- [27]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
supervision.
In International conference on machine learning, pages
8748–8763. PMLR, 2021.
- [28]
William M Rand.
Objective criteria for the evaluation of clustering methods.
Journal of the American Statistical association,
66(336):846–850, 1971.
- [29]
Liangrui Ren, Guoxian Yu, Jun Wang, Lei Liu, Carlotta Domeniconi, and
Xiangliang Zhang.
A diversified attention model for interpretable multiple clusterings.
IEEE Transactions on Knowledge and Data Engineering,
35(9):8852–8864, 2022.
- [30]
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria
Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools.
Advances in Neural Information Processing Systems, 36, 2024.
- [31]
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu,
Linxi Fan, and Anima Anandkumar.
Voyager: An open-ended embodied agent with large language models.
arXiv preprint arXiv:2305.16291, 2023.
- [32]
JV White, Sam Steingold, and CG Fournelle.
Performance metrics for group-detection algorithms.
Proceedings of Interface, 2004:5, 2004.
- [33]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng
Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan
Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang,
Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He,
Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng
Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin,
Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu,
Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin
Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan,
Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan.
Qwen2 technical report, 2024.
- [34]
Jiawei Yao and Juhua Hu.
Dual-disentangled deep multiple clustering.
In Proceedings of the 2024 SIAM International Conference on Data
Mining (SDM), pages 679–687. SIAM, 2024.
- [35]
Jiawei Yao, Enbei Liu, Maham Rashid, and Juhua Hu.
Augdmc: Data augmentation guided deep multiple clustering.
Procedia Computer Science, 222:571–580, 2023.
- [36]
Jiawei Yao, Qi Qian, and Juhua Hu.
Customized multiple clustering via multi-modal subspace proxy
learning.
arXiv preprint arXiv:2411.03978, 2024.
- [37]
Jiawei Yao, Qi Qian, and Juhua Hu.
Multi-modal proxy learning towards personalized visual multiple
clustering.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 14066–14075, 2024.
- [38]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan,
and Yuan Cao.
React: Synergizing reasoning and acting in language models.
arXiv preprint arXiv:2210.03629, 2022.

## Appendix

The Appendix is organized as follows:

- •
Appendix [A](https://arxiv.org/html/2503.22241v3#A1): Additional implementation details of our proposed framework, including key code snippets and explanations for each core component:
–
Embedding & Agent Training Data Generation ([A.1](https://arxiv.org/html/2503.22241v3#A1.SS1)): Describes how to generate labeled data for embedding and agent training with GPT-4.
–
Multi-Modal Large Language Model ([A.2](https://arxiv.org/html/2503.22241v3#A1.SS2)): Describes the initialization and LoRA-based fine-tuning of the MLLM used for both embedding extraction and agent-based decision making.
–
MLLM-Based Image Embedding Extraction ([A.3](https://arxiv.org/html/2503.22241v3#A1.SS3)): Details the process of generating user-interest-biased image embeddings by extracting hidden states associated with the <embedding> token.
–
MLLM-Based Image Embedding Training ([A.4](https://arxiv.org/html/2503.22241v3#A1.SS4)): Explains how image embeddings are fine-tuned using GPT-4-generated captions and similarity labels, supervised with cross-entropy and contrastive loss.
–
MLLM-Embedding-Based Graph Construction ([A.5](https://arxiv.org/html/2503.22241v3#A1.SS5)): Outlines the construction of a sparse relational graph using MLLM embeddings and a similarity threshold, to guide agent traversal.
–
Agent-Based Node Membership & Cluster Merge Assessment ([A.6](https://arxiv.org/html/2503.22241v3#A1.SS6)): Presents the use of MLLM-generated textual conclusions to assess candidate node membership and determine whether clusters should be merged.
–
Agent-Centric Graph Traversal ([A.7](https://arxiv.org/html/2503.22241v3#A1.SS7)): Describes how agents iteratively expand clusters within each connected component based on MLLM-guided evaluations of neighboring nodes.
–
Agent-Centric Cluster Merge ([A.8](https://arxiv.org/html/2503.22241v3#A1.SS8)): Details the global refinement stage in which semantically redundant clusters are merged through agent-based inter-cluster similarity assessments.
- •
Appendix [B](https://arxiv.org/html/2503.22241v3#A2): Additional qualitative examples that highlight the interpretability and effectiveness of our agent-based clustering framework.
–
Examples of Agent-Based Node Membership Assessment ([B.1](https://arxiv.org/html/2503.22241v3#A2.SS1)): Showcases how agents assess candidate nodes across different clustering aspects (e.g., number, suits, color) by reasoning over visual features and user-defined criteria. Incorrect connections in the relational graph are progressively pruned through these assessments, improving clustering purity.
–
Examples of Agent-Based Cluster Merge Assessment ([B.2](https://arxiv.org/html/2503.22241v3#A2.SS2)): Demonstrates how agents evaluate whether semantically similar clusters should be merged. The examples illustrate how missing edges in the graph are recovered via agent-guided assessments, enhancing the cohesion and completeness of the final clustering results.

## Appendix A More Framework Details with Key Code Snippets

### A.1 Embedding & Agent Training Data Generation

The embedding and agent training data are generated dynamically during the training process. For embedding training, in each epoch, we first extract user-interest-biased image embeddings using the MLLM, as described in Sec.[A.3](https://arxiv.org/html/2503.22241v3#A1.SS3). We then perform hard negative mining by selecting the top $1,024$ image pairs with the highest embedding similarity entropy. Over $50$ training epochs, this results in approximately $50,000$ image pairs. As shown in the provided code snippet, GPT-4 is used to evaluate the semantic similarity of these image pairs based on user-defined criteria, from which binary pseudo labels are inferred. These labels supervise the pairwise embedding similarities using a binary cross-entropy loss, as detailed in Sec.[A.4](https://arxiv.org/html/2503.22241v3#A1.SS4). In addition, GPT-4 generates detailed captions for each image, appending a special <embedding> token at the end. These captions serve as targets for training the MLLM’s language generation via cross-entropy loss. The hidden state corresponding to the <embedding> token in the MLLM output is then projected to form the final image embedding, which captures user-interest-aligned semantics.

For agent training, we construct a relational graph in each epoch using embeddings extracted by the MLLM, as described in Sec.[A.5](https://arxiv.org/html/2503.22241v3#A1.SS5). From this graph, we uniformly sample a subgraph containing $1,024$ nodes. GPT-4 then traverses this subgraph following the procedure in Sec.[A.7](https://arxiv.org/html/2503.22241v3#A1.SS7), evaluating the membership of each neighboring node with respect to the current cluster based on user-defined interests. These membership assessments are collected at each traversal step and used as training samples. The membership assessment with GPT4 is similar to “generate_image_similarity_with_GPT4” in the following code snippet. During training, the MLLM is tasked with predicting whether each sampled neighboring node belongs to its associated cluster, conditioned on user interests, as described in Sec. [A.6](https://arxiv.org/html/2503.22241v3#A1.SS6). The model’s predictions are supervised using GPT-4’s decisions, optimized via cross-entropy loss.

Figure: Listing 1: Embedding Training Data Generation Code Snippet

### A.2 Multi-Modal Large Language Model

The Multi-Modal Large Language Model (MLLM) serves as a crucial component in the proposed agent-centric personalized multiple clustering framework. In this setup, the MLLM is used both as an embedding extractor and an agent, assisting in the traversal of relational graphs to search for clusters based on user-specific interests. As detailed in the provided code snippet, the model is initialized with a pre-trained version of LlavaQwenForCausalLM, incorporating a vision tower for multimodal capabilities, enabling it to generate embeddings that reflect both visual and textual data.

A distinctive feature of the approach is the fine-tuning of the language model using LoRA, which ensures that only specific layers, including the language model head and embedding layers, are optimized. This selective fine-tuning contributes to the model’s efficiency by focusing on task-relevant parts of the network. Moreover, the inclusion of an embedding token (the specialized token <embedding>) enables the model to transform the generated detailed image descriptions, aligned with user-defined criteria, into embeddings that are biased toward user interests. These embeddings form the backbone of the relational graph, where nodes represent data instances, and edges encode the similarities between them.

Figure: Listing 2: Multi-Modal Large Language Model Code Snippet

### A.3 MLLM-Based Image Embedding Extraction

The MLLM-Based Image Embedding Extraction process is crucial for generating user-interest-biased image representations in the agent-centric personalized clustering framework. As detailed in the provided code snippet, the model takes input data, including images and user-defined criteria, and generates detailed image descriptions aligned with those interests. The hidden states corresponding to the <embedding> token in the generated descriptions serve as image embeddings, representing the images in a compact form that reflects user preferences.

The embeddings are further refined through layer normalization and projection to align them with user-defined criteria during training. They are then normalized to facilitate similarity calculation and relational graph construction, supporting efficient agent-centric graph traversal for personalized clustering. In contrast to CLIP-based embeddings, which are constrained by coarse-grained modality alignment, this approach leverages the MLLM’s reasoning capabilities to generate contextually rich embeddings that focus on user-specified aspects and attend to relevant visual details, thereby improving clustering accuracy.

Figure: Listing 3: MLLM-Based Image Embedding Extraction Code Snippet

### A.4 MLLM-Based Image Embedding Training

The MLLM-Based Image Embedding Training process is designed to fine-tune image embeddings in alignment with user interests, thereby enabling more accurate and efficient clustering. As outlined in the provided code snippet, this process involves training the Multi-Modal Large Language Model (MLLM) on image pairs, where each image is associated with a caption (a detailed description based on user interests) generated by GPT-4, and each pair is assigned a binary similarity pseudo label inferred from GPT-4’s similarity assessment.

During training, the model takes each image and user-defined criteria as input, with GPT-4’s image caption (appended with an <embedding> token) serving as the target for the MLLM to regress. This is supervised using cross-entropy loss. The hidden states corresponding to the <embedding> token in the target are extracted as image embeddings, acting as the user-interest-biased representation of the image. These embeddings are refined through layer normalization and projection, then further normalized to ensure they are on the correct scale for similarity calculations. The similarities between the two image embeddings are computed using cosine similarity, scaled by the learnable logit_scale parameter, and supervised with GPT-4’s pseudo labels via binary cross-entropy loss. This contrastive-style supervision ensures that the image embeddings are fine-tuned to reflect the user interests, improving their effectiveness in downstream clustering tasks.

Figure: Listing 4: MLLM-Based Image Embedding Training Code Snippet

### A.5 MLLM-Embedding-Based Graph Construction

To enable efficient agent traversal and clustering, we construct a relational graph in which nodes represent input images and edges encode the pairwise similarity between their corresponding embeddings extracted by the MLLM. As shown in the provided code snippet, we first extract all valid embeddings and compute their cosine similarities, scaled by the logit_scale parameter to adjust the sharpness of similarity scores. A sigmoid function is then applied to map raw similarity values into the range $\left[0,1\right]$. To ensure graph sparsity and remove noisy or semantically weak connections, we filter out edges whose similarity scores fall below a predefined threshold.

A relational graph is then constructed by adding each image (identified by its sample_id) as a node. High-confidence edges—those with similarity scores exceeding the threshold—are added between node pairs, with edge weights corresponding to their computed similarity. This results in a sparse, high-quality relational graph that preserves meaningful semantic relationships while eliminating spurious connections. By retaining only a compact set of high-quality neighbors, the graph significantly reduces the number of traversal steps required by agents to evaluate them, thereby improving clustering efficiency.

Figure: Listing 5: MLLM-Embedding-Based Graph Construction Code Snippet

### A.6 Agent-Based Node Membership & Cluster Merge Assessment

To ensure that clustering decisions reflect fine-grained, user-specific semantics, we incorporate an agent-based assessment mechanism during both node membership evaluation and cluster merging. This process leverages the reasoning capabilities of the MLLM to guide decisions based on visual content and user-defined criteria.

For node membership assessment, each candidate node and its associated cluster centroids are encoded as visual inputs and passed to the MLLM, along with a prompt designed to elicit a binary decision. The model generates a textual response containing a structured <CONCLUSION> tag indicating whether the candidate should be included in the cluster. A response of “yes” results in a positive assignment (similarity = $1.0$), while “no” leads to rejection (similarity = $0.0$). Unrecognized or missing conclusions (similarity = $-1$) are ignored during agent graph traversal.

Similarly, cluster merge assessment involves presenting two sets of cluster centroids as inputs. The MLLM evaluates whether the two clusters represent redundant or semantically similar concepts. The decision, extracted from the <CONCLUSION> tag, determines whether the clusters should be merged or remain separate. These assessments are stored in dictionaries that guide subsequent relational graph updates and cluster refinement during the agent traversal process.

This agent-based assessment framework introduces semantic interpretability and user-interest alignment into the clustering pipeline, enabling more accurate and personalized outcomes.

Figure: Listing 6: Agent-Based Node Membership & Cluster Merge Assessment Code Snippet

### A.7 Agent-Centric Graph Traversal

Once the relational graph is constructed, we employ agents to traverse the graph to discover clusters aligned with user interests. As illustrated in the following code snippet, each connected component of the graph is treated as an independent subgraph. For each component, we assign an agent to it and initialize a cluster by selecting the node with the highest degree (weighted by edge similarity), which serves as the initial centroid (i.e., representative node). The agents then iteratively expand their respective clusters by evaluating the membership of neighboring nodes.

Candidate neighbors are selected from the neighborhood of the current cluster within its component. An MLLM-based agent evaluates each candidate node against the cluster’s centroids based on user-defined criteria to determine whether it should be merged into the cluster. Positive candidates are added to the cluster, and the centroids of the cluster are updated by selecting the top-degree nodes. In contrast, negative candidates lead to the removal of their connecting edges to the cluster, reducing noise and potential interference in future traversal.

This iterative process continues until no more valid neighbors remain. If a cluster is completed, it is pushed to the global list, and the agent initializes a new cluster within the remaining subgraph. This method ensures that the traversal is both efficient and semantically meaningful, as decisions are guided by MLLM-based assessments. The agent-centric traversal strategy enables the discovery of fine-grained, user-interest-aligned clusters while dynamically refining the graph structure to suppress noisy or irrelevant connections.

Figure: Listing 7: Agent-Centric Graph Traversal Code Snippet

### A.8 Agent-Centric Cluster Merge

After local clusters are formed through agent-centric graph traversal with each connected component, we perform a global refinement step to merge semantically redundant clusters. As shown in the provided code snippet, each cluster is represented by a set of centroids (i.e., representative nodes), and the model identifies neighboring clusters as merge candidates by evaluating inter-cluster connectivity. For each candidate cluster pair, an MLLM-based agent assesses whether the two clusters should be merged based on user-defined criteria. If deemed redundant, the clusters are merged; otherwise, connecting edges are removed to reinforce separation.

To maintain index consistency, cluster pairs selected for merging are processed in reverse order of their indices. Once merged, the new cluster’s centroids are updated by selecting the top-degree nodes within the merged subgraph. This merging process reduces redundancy and corrects connectivity errors that may have occurred during local graph traversal. By leveraging the MLLM’s reasoning capabilities in the global stage, the framework produces clusters that are both well-separated and structurally coherent, while remaining closely aligned with fine-grained user interests.

Figure: Listing 8: Agent-Centric Cluster Merge Code Snippet

## Appendix B Additional Examples of Model Inputs and Outputs

### B.1 Examples of Agent-Based Node Membership Assessment

Table [4](https://arxiv.org/html/2503.22241v3#A2.T4) presents additional qualitative examples illustrating the interpretability of our agent-based node membership assessment across different clustering aspects, including number, suits, and color. Each example shows a set of cluster centroids and a candidate node, along with the corresponding prompt and MLLM-generated output. The agent evaluates the candidate’s membership by reasoning over visual cues relevant to the specified aspect. For instance, in the “Number” and “Suits” aspects of playing cards, the agent focuses on rank or suit similarities, respectively, while ignoring irrelevant attributes. In the “Color” aspect of fruit images, the agent attends to subtle hue variations. These examples demonstrate the MLLM’s ability to generate interpretable decisions aligned with user-defined criteria, validating the semantic alignment and explainability of our clustering process.

It is worth noting that incorrect edges in the relational graph are corrected through agents’ graph traversal, as negative assessments lead to edge removals, progressively refining the graph structure and improving clustering purity.

### B.2 Examples of Agent-Based Cluster Merge Assessment

Table [5](https://arxiv.org/html/2503.22241v3#A2.T5) presents qualitative examples of agent-based cluster merge assessment across different clustering aspects, including number, suits, and color. Each case includes a set of cluster centroids, candidate centroids, a task-specific prompt, and the corresponding MLLM-generated decision. The agent evaluates whether the two clusters should be merged based on user-defined clustering preference. For instance, in the “Number” and “Suits” aspects, the agent compares rank or suit consistency across clusters, while in the “Color” aspect, it focuses on visual color alignment. The structured output includes a human-interpretable explanation and a final decision marked with a <CONCLUSION> tag. These examples demonstrate the model’s ability to make semantically meaningful merge decisions and reinforce the interpretability of our agent-based framework.

It is worth noting that missing edges in the relational graph are corrected through agents’ cluster merging, as positive assessments trigger the integration of semantically redundant clusters, enhancing the cohesion and completeness of the clustering structure.

**Table 4: Examples of Agent-Based Node Membership Assessment.**
| Aspect | Prompt | Cluster Centroids | Candidate | MLLM Generation |
| --- | --- | --- | --- | --- |
| Number | Determine whether the candidate playing card should be included in the existing playing card cluster based on rank. Ignore the suits and focus only on rank comparison. Respond with <CONCLUSION> YES </CONCLUSION> or <CONCLUSION> NO </CONCLUSION>. |  |  | The current cluster consists of cards with the rank 9. The candidate card is ranked 6. Based on rank difference, it should not be included. The answer is <CONCLUSION> NO </CONCLUSION>. |
|  |  |  | The cluster contains cards of rank 3. The candidate card is also of rank 3. Based on rank similarity, it should be included. The answer is <CONCLUSION> YES </CONCLUSION>. |  |
|  |  |  | The cluster contains cards of rank 4. The candidate card is also of rank 4. Based on rank similarity, it should be included. The answer is <CONCLUSION> YES </CONCLUSION>. |  |
| Suits | Determine whether the candidate playing card should be included in the existing playing card cluster based on suits. Ignore the rank and focus only on suits comparison. Respond with <CONCLUSION> YES </CONCLUSION> or <CONCLUSION> NO </CONCLUSION>. |  |  | The cluster contains cards of the club suit. The candidate card is also a club. Based on suit similarity, it should be included. The answer is <CONCLUSION> YES </CONCLUSION>. |
|  |  |  | The cluster contains cards of the diamond suit. The candidate card is also a diamond. Based on suit similarity, it should be included. The answer is <CONCLUSION> YES </CONCLUSION>. |  |
|  |  |  | The cluster contains cards of the heart suit. The candidate card is also a heart. Based on suit similarity, it should be included. The answer is <CONCLUSION> YES </CONCLUSION>. |  |
| Color | Determine whether the candidate fruit should be included in the existing fruit cluster based on color<br>similarity. Ignore the species and focus only on color comparison. Respond with <CONCLUSION> YES </CONCLUSION>, or <CONCLUSION> NO </CONCLUSION>. |  |  | The cluster consists of yellow-colored fruits. The candidate fruit also appears yellow. Based on color similarity, it should be included.<br>The answer is <CONCLUSION> YES </CONCLUSION>. |
|  |  |  | The cluster consists of green-colored fruits. The candidate fruit is yellow-green, which differs from the cluster’s dominant green tone. Based on color difference, it should not be included. The answer is <CONCLUSION> NO </CONCLUSION>. |  |
|  |  |  | The cluster consists of red-colored fruits. The candidate fruit is also red, though darker in tone. Based on color similarity, it should be included. The answer is <CONCLUSION> YES </CONCLUSION>. |  |

**Table 5: Examples of Agent-Based Cluster Merge Assessment.**
| Aspect | Prompt | Cluster Centroids | Candidate Centroids | MLLM Generation |
| --- | --- | --- | --- | --- |
| Number | Determine whether the two playing card clusters should be merged based on rank. Ignore the suits and focus only on rank comparison. Respond with <CONCLUSION> YES </CONCLUSION> or <CONCLUSION> NO </CONCLUSION>. |  |  | The cluster centroids contain cards of rank 9. The candidate centroids consist of cards of rank 6. Since the ranks differ, the clusters should not be merged. The answer is <CONCLUSION> NO </CONCLUSION>. |
|  |  |  | The cluster centroids consist of cards ranked 3. The candidate centroids consist of cards ranked 8. Due to the rank difference, the clusters should not be merged. The answer is <CONCLUSION> NO </CONCLUSION>. |  |
|  |  |  | Both cluster and candidate centroids consist of cards ranked 4. Based on rank similarity, the clusters should be merged. The answer is <CONCLUSION> YES </CONCLUSION>. |  |
| Suits | Determine whether the two playing card clusters should be merged based on suits. Ignore the rank and focus only on suits comparison. Respond with <CONCLUSION> YES </CONCLUSION> or <CONCLUSION> NO </CONCLUSION>. |  |  | The cluster centroids consist of cards from the club suit, while the candidate centroids consist of spade suit cards. As the suits differ, the clusters should not be merged. The answer is <CONCLUSION> NO </CONCLUSION>. |
|  |  |  | Both the cluster and candidate centroids consist of cards from the diamond suit. As the suits match, the clusters should be merged. The answer is <CONCLUSION> YES </CONCLUSION>. |  |
|  |  |  | The cluster centroids consist of cards from the heart suit, while the candidate centroids consist of cards from the spade suit. Due to suit difference, the clusters should not be merged. The answer is <CONCLUSION> NO </CONCLUSION>. |  |
| Color | Determine whether the two fruit clusters should be merged based on color<br>similarity. Ignore the species and focus only on color comparison. Respond with <CONCLUSION> YES </CONCLUSION>, or <CONCLUSION> NO </CONCLUSION>. |  |  | The cluster centroids consist of yellow-colored fruits, while the candidate centroids consist of green-colored fruits. Due to the color difference, the clusters should not be merged. The answer is <CONCLUSION> NO </CONCLUSION>. |
|  |  |  | Both the cluster and candidate centroids consist of red-colored fruits. As the colors are consistent, the clusters should be merged. The answer is <CONCLUSION> YES </CONCLUSION>. |  |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: Specific questions can be found in the above weakness section.
soundness: 2
strengths: 1) Multiple clustering according to user's preference is an important and interesting problem. It is also sound to take the benefits of the MLLM to gather interest-based embeddings for personalized clustering. Similarity-graph based cluster search using the agent is also interesting and reasonable.

2) The proposed method provides better performance (sometimes significantly) over various multiple clustering datasets. The contribution of the MLLM embeddings and the graph search way are both demonstrated in the ablation study.
summary: This paper aims to generate image clustering results according to user-specified interest by taking benefit of the rich information in multi-modal large language models (MLLM). Specifically, according to the user's interest, a MLLM will be fine-tuned using pairs of images pseudo labeled by GPT with the user's interest specified. Then, the trained MLLM can be used to extract the corresponding image embedding. Those embeddings would be used to measure the similarity between data points to build graphs. Thereafter, each agent would be assigned to each connected component in the graph to search clusters using the similarities based on the embeddings. Moreover, these agents are fine-tuned by cluster vs candidate membership using GPT's assessment. The proposed method provides better performance no most multiple clustering datasets compared to existing methods.
weaknesses: 1) Some technical details are missing in the presentation. For example, the similarity between embeddings are calculated with a learnable logit scale. It is not clear how this is learned and in which step? Secondly, clusters with similar semantic will be merged in the end. However, this merging step was not clearly discussed or evaluated.

2) For the proposed method, it seems that both the embedding extractor and the membership assessment agent need fine-tuning for different use interest on different datasets. Thereafter, the additional training cost is significant. Moreover, compared to end-to-end deep multiple clustering or decoupled clustering using k-means or so, the cost of additional graph search using agents is also way significant. The discussion on this is necessary. 

3) For the above point in 2), it is necessary to conduct experiment to demonstrate that fine-tuning of those two models are necessary. For example, without fine-tuning, what is the performance. Similarly, without the membership agent, how the performance is degenerated?

4) In Table 1, some baselines like MCV or DDMC were not discussed at all without any references. What are these?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can you provide a more detailed theoretical justification for why MLLMs, specifically, are well-suited for clustering tasks over other methods like CLIP?
- How sensitive is the algorithm to the graph density threshold?
- The paper claims to improve on prior methods by 140%, but the reported NMI and RI scores do not show such drastic improvements in many cases. Can you provide more context to understand the significance of these improvements?
soundness: 2
strengths: - The framework introduces an interesting agent-based approach to multiple clustering. Leveraging MLLMs for embedding generation and graph traversal is an innovative step to incorporate context-aware embeddings.
- The results are promising.
summary: This paper introduces an agent-centric personalized multiple clustering framework that leverages multi-modal large language models (MLLMs) to traverse a relational graph and identify clusters based on user preferences. The proposed method aims to overcome the limitations of CLIP-based embeddings by using MLLMs to provide a more context-aware and user-interest-driven approach to clustering. The paper demonstrates the framework on various datasets, reporting improvements in performance metrics like NMI and RI, surpassing state-of-the-art models.
weaknesses: - The theoretical foundation for using MLLMs for clustering is weak. I recommend authors to provide a deeper analysis of why MLLMs are particularly suited for this task compared to other methods.
- Why is the threshold for graph density set at 0.6, and what are the effects of different thresholds on the performance of the method?
- The method’s explanation could be more structured, and the steps in the algorithm require further clarification. In particular, the paper does not explain how the clustering performance improves with the agent-centric approach versus traditional methods in a manner that ties back to the user-defined preferences.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Q1- In line 304, "The generated descriptions are aligned with GPT-4's outputs using cross-entropy loss." How does it correspond to Figure 3b? What does it mean when "Embedding: <embedding>" and "Generation" both point to "Append"? Whose Embedding is "Embedding: <embedding>"?

Q2- Similarly, in Figure 4, is the “Supervision” trained by entering the entire text or a binary label of“No”?

Q3- It is necessary to discuss whether fine-tuning the MLLM is necessary. Inputting images into the existing MLLM webpage-based model seems to be able to distinguish the categories of images very well.
soundness: 3
strengths: S1- The proposed model seems simple and easy to implement, and has relatively high flexibility for real-world scenarios.

S2- As far as I know, this is a relatively new paradigm, which is different from the previous representation learning-based methods, because there is an upper performance limit for moving the representation distribution from a universal feature space to a user-specific feature space.

S3- The performance on many datasets has been significantly improved compared to existing methods.
summary: This paper liberates multiple clustering from the CLIP-based representation model and transforms it into a retrieval and query task, avoiding the inherent problem of mismatch between representations and downstream tasks. This is the most essential difference between this method and the existing ones.
weaknesses: W1- The writing of this paper before the fifth page is clear. From the sixth page on, some expressions are difficult to understand. I will clarify this point in the questions, and hope that the author will further clarify it in the rebuttal stage.

W2- The definition of connected components in Eq. 3 should be unclear; a more accurate description would be the largest connected component; The $w(u, v)$ in Eq. 2 is confusing to read. It is not until I continue reading Eq. 8 that I know its definition.

W3- The author's code links are empty, but the appendix provides code examples for each module.

W4- The processes in Figures 3b and 4b and the corresponding main text descriptions are not clear. It took me some time to understand them. I suggest improving this part of the content.

W5- I think the author did not clearly explain the essential drawbacks of CLIP-based models, why CLIP's alignment mode hinders the learning ability of fine-grained, user specific features.

W6- How about the performance of an MLLM without fine-tuning? This point requires supplementary experiments to prove.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Real-World Applications: Where can this personalized clustering method be applied in real-world scenarios? Could the authors provide specific examples or domains where user interest-based clustering is critical?
2. Learning Without Supervisory Signals from Large Models: For tasks where large models cannot provide supervision signals (e.g., determining whether a portrait is beautiful), how should the learning process be designed?
3. Supervised Classification vs. Clustering: Since GPT-4 can provide supervisory signals, why not directly label some images with GPT-4 and use supervised classification methods to achieve the same goal, rather than constructing clusters?
4. Future of Vision Models: Incidentally, does the author believe that all image-related tasks (e.g., classification, clustering, segmentation) eventually will be solved by large vision models, similar to how large language models have addressed many language-related tasks?
soundness: 2
strengths: 1. Fine-Grained User Interest Capture: The method leverages Multimodal Large Language Models (MLLMs) to supplant conventional CLIP-based alignment, enabling a deeper, reasoning-based understanding of context. This is achieved through customized prompts that elicit fine-grained semantic embeddings, thereby capturing nuanced user preferences.

2. Sparse Relational Graph Design: To mitigate the computational overhead from excessive agent traversal in fully-connected relational graphs—a consequence of redundant weak edges—the proposed approach constructs a sparse graph. This is done by computing edge weights from MLLM-generated user interest embeddings and subsequently pruning weak edges using a fixed threshold (τ = 0.6).
summary: This paper focuses on personalized multiple clustering and proposes a framework with multi-modal large language models (MLLMs): it first extracts user interest-biased embeddings via MLLMs to construct a weak edge-filtered sparse relational graph, then deploys multiple MLLMs for parallel graph traversal to search interest-aligned clusters, and finally uses a global MLLM to merge redundant clusters; with MLLMs fine-tuned lightweight via LoRA and GPT-4 generating supervised pseudo-labels, the framework achieves good performance on some multiple clustering datasets.While the method demonstrates strengths in fine-grained user interest modeling, efficiency optimization via sparse graph structures, and rigorous experimental validation, it is constrained by several limitations, including high computational costs, limited coverage of datasets and baselines, and insufficient algorithmic innovation.
weaknesses: 1. High Computational Cost: High Computational Cost: The use of Qwen2-7B (7B parameters) requires 4 NVIDIA A100 GPUs for 3 days, limiting practicality for resource-constrained teams. But in practice, the improvement on the dataset is not significant.

2. Limited Dataset & Baseline Coverage: Experiments focus on high-quality datasets but lack tests on real-world or cross-domain tasks (e.g., datasets of different art styles and those on food ripeness levels). Comparisons to low-cost concurrent deep clustering methods are also missing. Several recent clustering works are very relevant to the proposed method. The authors are encouraged to include them in the related works, and if possible, make some comparisons with them.
	Interactive Deep Clustering via Value Mining, NeurIPS 2024
	Personlized Clustering via Targeted Representation Learning, AAAI 2025
	Image Clustering with External Guidance, ICML 2024

3. Lack of Algorithmic Innovation: The method relies heavily on supervisory signals from GPT-4 and the feature extraction capabilities of the MLLM rather than achieving good performance through innovative algorithm design. The algorithm itself lacks novel contributions, as its core components build on existing paradigms.


Decision: Reject