Interactive Deep Clustering via Value Mining
Authors: Honglin Liu
Peng Hu
Changqing Zhang
Yunfan Li
Xi Peng
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Clustering; Interactive Clustering
Primary area: human-AI_interaction
TL;DR: We propose incorporating external user interaction to tackle hard samples at cluster boundaries, which existing deep clustering methods fail to discriminate internally from the data itself.

Abstract: In the absence of class priors, recent deep clustering methods resort to data augmentation and pseudo-labeling strategies to generate supervision signals. Though achieved remarkable success, existing works struggle to discriminate hard samples at cluster boundaries, mining which is particularly challenging due to their unreliable cluster assignments. To break such a performance bottleneck, we propose incorporating user interaction to facilitate clustering instead of exhaustively mining semantics from the data itself. To be exact, we present Interactive Deep Clustering (IDC), a plug-and-play method designed to boost the performance of pre-trained clustering models with minimal interaction overhead. More specifically, IDC first quantitatively evaluates sample values based on hardness, representativeness, and diversity, where the representativeness avoids selecting outliers and the diversity prevents the selected samples from collapsing into a small number of clusters. IDC then queries the cluster affiliations of high-value samples in a user-friendly manner. Finally, it utilizes the user feedback to finetune the pre-trained clustering model. Extensive experiments demonstrate that IDC could remarkably improve the performance of various pre-trained clustering models, at the expense of low user interaction costs. The code could be accessed at pengxi.me.

Interactive Deep Clustering via Value Mining
Honglin Liu', Peng Hu', Changqing Zhang Yunfan Li"; Xi Peng'.4
'College of Computer Science, Sichuan University, Chengdu, China Coolege of Intelligence and Computing, Tianjin University, Tianjin, China Tianjjn Key Lab of Machine Learning, Tianjin, China *State Key Laboratory of Hydraulics and Mountain River Engineering, Sichuan University, Chengdu, China
(tristanliuhl, penghu.ml, ml, yunfanli gm, pengx.gm/ @gmail com,
zhangchanggingāttju edu.cn
Abstract
In the absence of class priors, recent deep clustering methods resort to data aug- mentation and pscudo-labeling strategics to generate supervision signals. Though achieved remarkable success, existing works struggle to discriminate hard samples at cluster boundaries, mining which is particularly challenging due to their unre- liable cluster assignments. To break such performance bottleneck, propose incorporating user interaction to facilitate clustering instead of exhaustively mining semanties from the data itself. To be exact, present Interactive Deep Clustering (IDC), a plug-and-play method designed to boost the performance of pre-trained clustering models with minimal interaction overhead. More specifically, IDC first quantitatively evaluates sample values based on hardness, representativeness, and diversity, where the representativeness avoids selecting outliers and the diversity prevents the selected samples from collapsing into small number of clusters. IDC then queries the cluster affiliations of high-value samples user-friendly manner. Finally, it utilizes the user feedback to finetune the pre-trained clustering model. Extensive experiments demonstrate that IDC could remarkably improve the performance of various pre- traineddd clustering models, at the expense of low user interaction costs. The code could be accessed pengxi.me
Introduction
Clustering aims at partitioning samples into semantically distinct groups. recent years, deep
clustering methods [3) [3], powered by the feature extraction ability of neural networks, have excelled in handling large-scale and high-dimensional data across various domains, including image segmentation
anomaly detection 255 medical analysis [1. bioinformatics
20), and so on.
To discover the semantical data partitions, the of deep clustering lies in designing supervision signals to extract discriminative information from data. To this end, early efforts reformulate the self-representatioon property (31), hierarchical structure (33. or assignment distribution prior [42] into differentiable objectives for model optimization. Recently, inspired by the success of contrastive learning 66 I.I the community has shifted towards constructing self-supervision signals via data
augmentations, thus promoting the contrastive clustering paradigm [18 46 14] The latest research
indicates that pseudo-labels could further enhance the clustering performance 38
21
29 22).
Despite these merits, almost all deep clustering methods suffer from the performance ceiling due to the limited information inherent in the data [19. Particularly, this limitation is reflected in the poor discrinination of hard boundary samples as shown in Fig. Ta Consequently, it has great chance to
"Corresponding Authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input Images
Input Images
Whin to lolviing images belorge right mmgge?
Clustering Model
Clustering Model
x
thee magg theonge tw glvan
Clustering Results
Top-10 Neighbors of Hard Sample
Clustering Results
User Interaction
(a) Existing Deep Clustering Pipeline
(b) Our Interactive Deep Clustering Pipeline
Figure 1: Our key idea. (a) Existing deep clustering methods suffer from poor discrimination of hard boundary samples. As a showcase, we highlight one hard sample (red circle) whose neighborhood includes visually similar but semantically different neighbors (red boxes), leading bottleneck. (b) Instead of exhaustively mining internal semantics from data, we propose incorpo- rating external user interaction to address the hard sample problem. In brief, we select high-value samples and query their cluster affiliations, which improves the clustering performance remarkably as visualized in the T-SNE plots.
performance
improve the overall performance remarkably through mining hard samples. Current pseudo-labeling strategies, however, focus on easy samples with high-confident cluster predictions, while failing to handle hard boundary samples with unreliable predictions. To tackle hard samples, recent effort attempts to mitigate their impact by neglecting them when constructing neighborhoods [45). Nevertheless, this approach, akin an ostrich avoidance policy, essentially sidesteps the core problem rather than solving it, ultimately leaving hard samples inseparable.
Acknowledging limitations in tackling hard samples internally, we present straightforward approach by incorporating external user interaction, as illustrated in Fig. Tb In brief, given an arbitrary pre- trained clustering model, aim to correct its cluster assignments of hard samples with minimal user interaction overhead. To achieve this, we confront two challenges: i) constructing an efficient and user- friendly interaction interface, and if) effectively utilizing user feedback. To tackle the first challenge, we present a novel strategy to mine valuable samples based on hardness, representativeness, and diversity for user inquiries with mathematical formulations. Here, the representativeness is designed to avoid selecting outliers and the diversity is used to prevent the selected samples from collapsing into small number of clusters. For user convenience, instead of directly requesting class labels. we inquire about the affiliation of each selected sample with its nearest cluster centers. For the second challenge, we design two new losses to finetune the pre-traincd model using both positive and negative user feedback. Specifically, positive feedback indicates the semantic alignment w.r.t. the selected cluster, while negative feedback denies all candidate clusters as semantically inconsistent. Additionally, we propose a regularization loss to preserve the overall cluster boundary of the original model, preventing it from overfitting the inquired samples. Notably, our method is a plug-in that can be effortlessly integrated into existing clustering methods, thereby enhancing their performance.
The major contributions of this work could be summarized as follows:
We propose incorporating external user interaction to break through the performance ceil- ing of existing deep clustering methods, specifically correcting the hard samples that are indistinguishable internally.
To reduce interaction costs, we present value-mining strategy to select hard, representative, and diverse samples for user inquiry. To simplify the interaction, we design user-friendly interface to ask for cluster affiliations of these selected valuable samples.
The proposed IDC could be easily integrated into any pre-trained deep clustering model. Extensive experiments demonstrate that our IDC could significantly boost the performance of various state-of-thee the-art deep clustering methods with negligible user interaction costs.

---

2
Related Work
In this section, we briefly review two fields related to this work, namely, deep clustering and hard sample mining.
2.1 Deep Clustering
Thanks to the powerful feature extraction ability, deep clustering methods have shown promising
results complex real-world data and advanced rapidly in past years [42 43 [ [3]. Recently, the success of self-supervised learning [6 [1 [0] gives rise to a series of contrastive clustering
methods [18 46
21
14] However, even enhanced by data augmentation and pseudo-labeling
strategies 38 29
4. the performance of existing deep clustering methods is inherently upper-
bounded by the limited internal supervision signals. Instead of exhaustively mining semantics from the data, a recent work attempts to leverage external data and models to facilitate clustering [19]. Another branch of study focuses on integrating prior class labels 16 or pairwise constraints into the clustering process to boost the performance.
Different from existing studies that pursue overall performance improvements, this work aims to address the specific hard sample problem. Notably, the performance bottleneck of existing methods lies in the poor discrimination for hard samples the cluster boundaries. Given the difficulty of internally correcting cluster assignments for hard boundary samples, we propose incorporating external user interaction as a straightforward solution. By inquiring about the cluster affiliations of representative and diverse hard samples, our method could significantly boost the performance of pre-trained clustering models with low interaction overhead.
2.2 Hard Sample Mining
Hard samples refer to data points that are difficult to recognize and understand due to their ambiguous or weak semantics, which widely exist various tasks such as face recognition [33], person re- identification [44], image segmentation (28) object detection (2), and cross-modal retrieval 23 On the one hand, the model is likely to make wrong predictions for these samples. On the other hand, mining these samples could significantly improve the model performance. Notably. hard sample mining is usually conducted in a supervised manner. For clustering, it is daunting to correct the assignments of hard samples cluster boundaries by the model itself due to the absence of class label priors. As an attempt, SeCu [2) recently proposes assigning larger weights to hard samples when computing cluster centers for better cluster discriminability. However, the improvement is limited due to the unreliable cluster assignments of hard samples.
The differences between this work and previous hard sample mining methods are twofold. On the one hand, most existing works focus on enclosed supervised learning, while we explore hard sample mining for unsupervised clustering by incorporating user interaction. On the other hand, unlike previous works that solcly pursue sample hardness, we further consider the representativeness and diversity of hard samples, resulting in comprehensive evaluation of data value. Such a value mining strategy helps to improve the cluster model with interaction costs as low as possible.
3 Method
In this section, we introduce our novel Interactive Deep Clustering method (IDC). As illustrated in Fig. [2] IDC consists of two primary stages: user interaction and model optimization. Initially, IDC solicits the user to determine the cluster affiliation of highly valuable samples, which are strategically selected based their hardness, representativeness, and diversity. Subsequently, during the model optimization stage, IDC refines the cluster assignments of these samples according to the user feedback, while preserving the overall decision boundary of the pre-trained model. The two stages are further detailed in Sections and and 1.nd3.7 3.2
3.1 User Interaction
To boost the pre-trained clustering model with minimal user interaction cost, we select the most valuable samples for user inquiries. The value of each sample is appraised based on three proposed
3

---

Input
User Interaction
Model Optimization
Hardness
Posttive Feedback
Postlive Loss
What babvenng
craggry
Iepresentativeress
Negatlive Loss
coleglly
Pre-trained Clustering Model
Negative Feedback
Faggs takegg
given image?
Diversity
Regularization
No the compgey
Selection
User Inquiry
Confident Thresholding
The overall framework of IDC consists of two stages: user interaction and model op- timization. In the user interaction stage. given a pre-trained clustering model, IDC first selects high-value samples based on hardness, representativeness, and diversity. Then it inquires the user about the affiliations of the selected samples relative their nearest cluster centers. In the model optimization stage, IDC utilizes both positive and negative user feedback to finetune the trainre model with positive and negative losses for cluster performance improvement. Meanwhile, IDC adopts regularization loss on high-confident predictions to prevent overfitting inquired samples.
Figure 2:
criteria: hardness h, representativenes: r, and diversity d, encapsulated by the equation:
8/2 h4 + Ta + di.
(I)
where Us denotes the importance the i-th sample. We elaborate on the three metrics as follows:
Hardness. Typically, a pre-trained clustering model could accurately assign clusters to easy samples near cluster centers. However, it may fail hard samples situated cluster peripheries. In other words, identifying these boundary samples is pivotal for boosting the clustering performance. Therefore, quantify the hardness of the i-th sample by its proximity to cluster centers:
h log(1
Cg1 + Cgg),
where 2; is the L2-normalized feature of the i-th sample, and Co. Caz denote the closest and second- closet cluster centers to 24, respectively. A higher h; score indicates greater uncertainty in cluster assignment for the i-th sample.
Representativeness, While correcting hard samples is beneficial, focusing solely hardness may lead to suboptimal results, as the most challenging samples could be outliers that negatively impact the model's generalization ability. To tackle this problem, we prefer samples reside in dense regions, where inquiring about a single sample could correct the cluster assignments of numerous adjacent ones. Formally, define the representativeness of the i-th sample by the density of its K nearest. neighbors as follows:
K
T; = - -log Σ 2
210)
1=1
where Z(() refers to the j-th nearest neighbor of 2; and K is the number of nearest neighbors empirically set to 20. A higher r; score suggests a more compact local structure, indicating that the i-th sample is more representative.
Diversity. In practice, we discover that pursuing hardness and representativeness may result in an unbalanced sample distribution, heavily collapsing into small number of clusters as shown in Fig. 3 To avoid this, we present the "diversity" metric to ensure sufficient dispersion of the selected samples. Different from hardness and representativeness which are independent of the selection,

---

Algorithm 1 Valuable Sample Selection
Input: Sample features Z {Z1,-. 2N). number of samples to be selected M
Output: Selected sample indices
{81,-.... 8as)
1: Initialize the selected indices S = ( and the remaining indices R {I....,, N)
2: Compute cluster centers of by k-means
3: for i E [1, .V] do
4: Compute the hardness score h; by Eq.
5: Compute the representativeness score r; by Eq.
6: Initialize the diversity score d; = 0, since no sample has been selected
7: Compute the value score U, by Eq. (I)
8: end for
9: for [1, .M do
10:
Select the 8y-th sample with the highest value from ZR.
11: S U {, R R (sy)
12: Update the diversity score d for ZR by Eq.
Update the value score v for ZR by Eq.
14: end for
the diversity of a given sample is measured by its deviation from previously selected samples. In our implementation, the sample with the highest e; score selected iteratively until M samples are selected. In each iteration, the diversity of the i-th sample is computed according to the already selected samples:
d; min jeS
log(1 1 =2),
where S represents the indices of the selected samples.
For user interaction, we select the top M = 500 valuable samples with the highest scores in our experiments. The selection process outlined Algorithm I According Theorem proved below, IDC could select the most valuable samples to minimize the user interaction cost.
Theorem 1. The value of the selected sample decreases as the selection progresses, i.e.,
1/, 2 v.t1 33411 Vj E M
(5)
where 8j denotes the index of the j-th selected sample, and of denotes the i-th sample's 's value in the j-th selection.
Proof. We denote S the of selected sample indices and R the set of remaining sample indices after the j-th selection. Further, the i-th sample's hardness, representativeness, and diversity in the j-th selection (i.e., i E R/-l) are denoted as hi r/. and di, respectively. By the definition of sample value in Eq. ( we have
of= 1d + r! + d/
(6)
Notably, since choose sj instead of 8j+1 in the j-th selection, there must be
m y
8,
By the definition of diversity Eq. we have
d min log(1 2az41 2;) min log(1 28s41 2,) dd+1 oj+1
8
VESS
EES
where the inequality holds since $ = ( ) and thus S 1 C St. Furthermore, as hardness and representativeness scores are irrelevant to the selection process (i.e., hg 1 2 rá 23 = r ))
we have
+ r/
+ d s++
h++1 r1+1
+ di+1 rtt1
(9)
8941
8541
8541
Finally, by combining Eq. and Eq.
9 we arrive at
2 2
2
#+1 1
(10)
which completes the proof of Theorem [
5

---

Upon selecting the most valuable samples, we inquire about their cluster affiliations relative to the nearest cluster centers. For each selected sample, we provide T 5 nearest cluster center candidates [ and then request the user to determine which candidate shares the same semantics with Notably, such an inquiry strategy more user-friendly than directly asking about the pair-wise correlation between two samples, by aiding users in grasping cluster semantics and partitioning criteria. User feedback could be either positive (selecting a candidate) or negative (rejecting all candidates) to the given sample, which serves the subsequent model finetuning strategy introduced in the next section.
the anchor as illustrated in Fig. 2
3.2 Model Optimization
Based the user feedback, we present a positive loss Lposs negative loss Lengs and regularization loss Lrey to finetune the clustering model:
Lpos + Lung + freg-
(11)
The three loss terms designed to utilize positive feedback. negative feedback, and to prevent overfitting the queried samples, respectively, with details provided below.
Positive Loss. Positive user feedback refers to identifying the cluster centroid sharing the same semantics with the inquiry sample. To exploit this feedback, we draw the sample and the cluster centroid closer by the following positive loss:
Mpne
Epss
Vij log Pijs
(12)
MpS
where Mpss denotes the count of positive feedback, C' the number of clusters, Pt refers to probability of sample i belonging to cluster j, and Vij E {0, 1} is an indicator that equals one iff the j-th cluster is selected by user.
Negative Loss. Negative user feedback indicates that no candidates match the semantics of the inquiry sample. To leverage the feedback, we enforce the sample apart from all candidate clusters using the following negative loss:
Mang C
1 Mneg
Lueg
Uij log(1 Pij);
(13)
where Mneg is the count of negative feedback, and ūi is an indicator that equals one if the j-th cluster is the randomly chosen candidate, and zero otherwise.
Regularization Loss. To reduce the interaction cost, only small amount of samples are selected for user interaction. However, exclusively finetuning the model with the above two losses risks overfitting to the inquiry samples, potentially compromising previously correct cluster predictions. To tackle this problem, we propose preserving the overall cluster boundary by retaining confident predictions. Formally, the regularization loss is defined follows:
Creg
Ipg 7] logp.3 argmax pa
(14)
where N is the count of all samples, T 0.99 is the confidence threshold, [[cond] {0, 13 is an indicator that equals one iff the condition cond holds.
The above three losses applied to optimize the pre-trained clustering model for performance improvement. After finetuning, we could directly obtain the improved cluster assignments from the model's 's cluster head
In practice, each cluster center represented by nearest sample. We append a cluster head for pre-trained clustering models that do not have one. More details are provided in Section 4.2

---

Experiments
In this section, we first apply the proposed IDC to two state-of-the-art deep clustering methods, and evaluate the performance five widely used image clustering benchmarks. Then conduct ablation studies and parameter analyses to validate the robustness and effectiveness of IDC.
4.1 Datasets and Evaluation Metrics
We evaluate IDC on five widely used image clustering datasets, including CIFAR- 10 177, STL-10 (8. ImageNet-Il 10 [3] and ImageNet-Dogg Dogs [5), detailed in Table
17,, CIFAR-20
Three widely used clustering metries are adopted for performance evaluation, including Normalized Mutual Information (NMI), Accuracy (ACC), and Adjusted Rand Index (ARI). Higher scores signify superior clustering results.
Table I: A summary of the used datasets.
Dataset
Split
Samples
Classes
CIFAR-10
Train+Test
60000
10
CIFAR-20
Train+Test
60000
20
STL-10
Train+Test
13000
10
ImageNet-10
Train
13000
10
ImageNet-Dogs
Train
19500
15
4.2 Implementation Details
Without loss of generality, we apply the proposed IDC on two recent methods TCL 21 and ProPos [14], on behalf of deep clustering models with and without cluster head, respectively. Notably, for clustering models without cluster head like ProPos, we append a randomly initialized two-layer fully connected network as an alternative. In the model optimization stage, we finetune the pre-trained clustering model for 100 epochs. For ProPos, we warm up the cluster head with the positive and negative loss in Eq. 12 and 13 in the first 50 epochs, since the prediction confidences are unreliable initially. To balance the effect of user feedback and model regularization, we use two independent data loaders for the inquiry and confident samples, with batch sizes of 100 and 500, respectively. All images are augmented consistently with the pre-trained clustering model for finetuning, while the original images are used for value evaluation and pseudo-labeling. All experiments are conducted on a single Nvidia RTX 3090 GPU on the Ubuntu 20.04 platform with CUDA 12.0.
4.3
Comparisons with State of the Arts
We first compare the proposed IDC with 14 recent deep clustering methods, including CC
SCAN [88 NMM 19. MICE B77] BYOL [10], GCC [46), SPICE [29]. IDFD [3). TCC
34
DivClust 26 SeCu 32 CONR 45 TCL 21). and ProPos [4. In addition, we include two repre-
sentative semi-supervised classification and clustering baselines FixMatch (35) and Cop-KKeann 44)
for benchmarking. For FixMatch, we use the ResNet-34 [12 as the backbone and annotate the inquiry images with positive user feedback for fair comparisons. For Cop-Kmeane we use the TCL. image features as the input and transform the user feedback as must- and cannot-link constraints.
As shown in Table 2 IDC gains consistent performance improvement, especially on more challenging datasets. Specifically, IDC boosts the clustering accuracy TCL/ProPos by 16.3% 19.2% and 14.4% 2% on CIFAR-20 and ImageNet-Dogs, respectively. Besides, the results show that solely correcting the cluster assignments of 500 samples brings marginal performance improvement, since they are only small portion of the data. Notably, IDC also outperforms the semi-supervised baseline FixMatch. Such a result could attributed to customized valuable sample selection strategy. Namely, the selected inquiry samples are catered to the pre-trained clustering model, which may not suit the general semi- supervise classification. Moreover, the superior performance of IDC compared with Cop- Kmeans demonstrates its stronger ability to utilize user feedback for model optimization.
7

---

Table Clustering performance comparison with the state-of-the methods on five benchmarks. The performance of PDCPis is unavailable as the code of ProPos on STL-10 has not been released. To make clear comparison, we add a baseline by manually correcting the cluster assignments of 500 query samples, denoted by TCL and ProPos
CIFAR-10
CIFAR-20
STL-10
ImageNet- 10
ImageNet-
Method
NMI
ACC
ARI
NMI
ACC
ARI
NMI
ACC
ARI
NMI ACC ARI NMI ACC ARD
CC 18
70.5
79.0
63.7
43.1
42.9
26.6
76.4
85.0
72.6
85.9 89.3 82.2
44.5
42.9
27.4
SCAN 38
79.7
88.3
77.2
48.6
50.7
33.3
69.8
80.9
64.6
NMM 9
74.8
84.3
70.9
48.4
47.7
31.6
69.4
80.8
65.0
MICE 37
73.7
83.5
69.8
43.6
44.0
28.0
75.2
42.3
43.9
BYOL 10
81.7
89.4
79.0
55.9
56.9
39.3
71.3
82.5
65.7
86.6
93.9
87.2
63.5
69.4
54.8
GCC 46
76.4
85.6
72.8
47.2
47.2
30.5
68.4
78.8
63.1
84.2
90.1
82.2
49.0
52.6
36.2
SPICE (29
73.4
83.8
70.5
44.8
46.8
29.4
81.7
90.8
81.2
82.8
92.1
83.6
57.2
64.6
47.9
IDFD 36
71.1
81.5
66.3
42.6
42.5
26.4
64.3
75.6
57.5
89.8
95.4
90.1
54.6
59.1
41.3
TCC
79.0
90.6
73.3
47.9
49.1
31.2
73.2
81.4
68.9
84.8
89.7
82.5
55.4
59.5
41.7
DivClust [26
72.4
81.9
68.1
44.0
43.7
28.3
93.6
87.8
51.6
52.9
37.6
SeCu
93.0
85.7
55.1
55.2
39.7
73.3
83.6
69.3
CoNR 45
87.1
93.3
86.5
60.3
59.0
44.8
84.6
92.2
83.8
89.8
95.8
90.9
74.2
80.2
67.6
FixMatch
35
86.8
92.8
85.4
57.2
67.2
47.3
68.6
49.2
92.5
Cop- Cop-Kmeans 40
82.3
89.0
78.6
52.2
52.4
34.7
78.1
85.4
73.1
85.5
88.6
81.0
61.5
63.5
49.7
TCL [2
81.9
88.7
78.0
52.9
53.1
35.7
79.9
86.8
75.7
87.5
89.5
83.7
62.3
64.4
51.6
TCL
82.2
88.9
78.4
53.2
53.5
36.1
82.0
88.6
78.5
88.6
90.4
85.0
62.8
65.6
52.3
IDCrCL (Ours)
84.4
92.7
84.8
58.1
69.4
48.7
85.3
92.7
84.6
93.2
97.2
93.9
69.1
78.8
63.6
ProPos 14
87.7
93.6
87.1
59.1
59.1
43.6
75.8
86.7
73.7
88.9
95.2
89.6
73.0
76.9
66.9
ProPos
87.9
93.7
87.3
59.3
59.4
43.8
73.8
67.8
ProPos (Ours)
90.5
95.7
90.9
69.2
78.3
61.4
94.1
77.6
(a) Hard
(b) Hard+Rop
(c) Hard+Rep+Div
Figure 3: T-SNE visualizations of samples selected by different strategies among all data points, on the ImageNet-Dogs dataset where selected samples are highlighted by red dots.
4.4 Ablation Study and Parameter Analysis
To prove the robustness and effectiveness IDC, we conduct ablation studies and parameter analyses on the TCL-based model. Specifically, for the user interaction stage, study the effectiveness of the valuable sample selection strategy, as well as the impact the number of selected samples M and candidate cluster centers T. For the model optimization stage, we investigate the effectiveness of the three loss terms.
starting with the clustering hardness, we additionally consider representativeness and diversity to quantify the value of each sample. Here, to provide intuitive understanding of the three criteria, we visualize the selected samples among all data points in Fig. 3 can seen, solely considering hardness would select most boundary samples, which are not representative enough and thus sub-optimal for reducing
Effectiveness of the valuable sample selection strategy. As detailed in Section 3.1

---

Table 3: Performance with different sample selection strategies on CIFAR-20 and ImageNet-Doge
CIFAR-20
ImageNet-Dogs
Selection Strategy
NMI
ACC
ARI
NMI
ACC
ARI
None (Pre-trained Model)
52.2
52.6
34.9
61.8
64.1
50.9
Hard
51.3
57.7
37.0
68.2
75.9
60.2
Rard+Rep
35.8
36.3
12.6
59.7
65.3
49.4
Hard+Rep+Div
58.1
69.4
48.7
69.1
78.8
63.6
racdom
76
74
70-
58
68
55
CLEAR 20
Imagee Dop
61-
150
200
300
400
500
600
700
0
100
200
300
400
500
800
700
(a) M CIFAR-20
(b) M on ImageNet-Dogs
(c) T both datasets
Figure 4: Influence of different numbers of selected samples and candidates T. (a)-(b) Clustering accuracy with different M on CIFAR-20 and ImageNet-Dogs respectively, compared with the random selection baseline. (c) Clustering accuracy with different T on both datasets.
the interaction cost. When additionally considering the representativeness, however, the selected samples would collapse into dense subsets, leading to significant performance drop as shown in Table Finally, further integrating the diversity results in samples simultancously comprising the three expected characteristics, which gives the best clustering performance.
Impact of the number of selected samples and candidates. For interaction cost reduction, select M = 500 most valuable samples for the user inquiry. and T nearest cluster centers as the candidates. Here, investigate how different numbers of M and T influence the final clustering performance. As depicted in Fig. 4 (a) and (b), the performance of IDC improves as the number of selected samples increases. Notably, the improvement grows rapidly the start but gradually levels off as more samples are selected. Moreover, valuable samples selected by IDC consistently outperform the random selection baseline. These results not only demonstrate the effectiveness of our valuable sample selection strategy, but also verify the monotonously decreased sample value as proved Theorem T For the number of candidates, Fig. (c) shows that comparing the inquiry sample with the five nearest centers strikes the best balance between performance and interaction cost.
Table 4: Performance with different combinations of loss terms CIFAR-22 20 and ImageNet-Dogg
CIFAR-20
ImageNet-Doge
yyss
Lieg
Lyeg
NMI
ACC
ARI
NMI
ACC
ARI
56.8
65.8
46.2
68.3
76.9
62.5
7.0
9.6
2.4
26.8
24.1
3.1
50.9
52.9
35.2
61.7
64.9
51.8
55.0
66.2
44.1
67.9
77.5
61.6
58.7
67.6
47.8
68.8
77.5
62.7
36.7
39.2
17.3
53.1
59.7
34.8
58.1
69.4
48.7
69.1
78.8
63.6
Effectiveness of the loss terms. To prove the effectiveness of the positive loss Loos in Eq. (22) evaluate different
the negative loss Lneg in Eq. (13) and the regularization loss in Eq. (14).

---

combination of the three losses and the results are shown in Fig. 4 On the one hand, no single loss is adequate to yield promising clustering results. In particular, solely leveraging the negative loss would deny all the Top-5 predictions and thus severely damage the decision boundary of the pre-trained clustering model, leading to the model collapse. On the other hand, each loss indispensable during the model optimization. Notably, the positive loss brings the most substantial performance improvement, as offers the most direct clustering guidance to the model.
5
Conclusion
Instead of mining semantics from internal data, we propose an interactive deep clustering method IDC, which incorporates user interaction to address the hard sample problem. By mathematically measuring the sample value defined on hardness, representativeness, and diversity, IDC selects the highest-value samples and inquiries about their cluster affiliations through a user-friendly interaction interface. By fine-tuning the pre- tre-trained clustering model leveraging user feedback, IDC remarkably improves the performance of various offte-of-the-ar deep clustering methods. For future studies, one potential direction is to consider the mistakes in user feedback, and correspondingly improve the robustness of IDC. Another possible direction is to design a more advanced interaction pipeline, for aligning clustering results with the user's personalized partition criterion. In general, we hope this work could provide novel insight to the community, attracting more attention to the interactive clustering paradigm which is promising and less explored area.
Acknowledgements
This work was supported in part by NSFC under Grant 62176171, U21B2040, 623B2075, 62472295; in part by the Fundamental Research Funds for the Central Universities under Grant CJ202303; and in part by Sichuan Science and Technology Planning Project under Grant 24NSFTD0130.
References
[1] Henrique Aguiar, Mauro Santos, Peter Watkinson, and Tingting Zhu. Learning of cluster-hased feature importance for electronic health record time-series. In International conference on machine learning, pages 161-179. PMLR, 2022.
[2] Edward Ayers, Jonathan Sadeghi, John Redford, Romain Mueller, and Puncet K Dokania. Query-based hard-image retrieval for object detection at test time. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14692-14700, 2023.
[3] Sugato Basu, Arindam Banerjee, and Raymond J. Mooney. Semi-supervised clustering by seeding. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, page 27-34, San Francisco, CA. USA, 2002. Morgan Kaufmann Publishers Inc.
[4] Shaotian Cai, Liping Qiu. Xiaojun Chen, Qin Zhang, and Longteng Chen. Semantic-enhance image clustering. In Proceedings of the AAAI conference artificial intelligence, volume 37, pages 6869-686 2023.
[5] Jianlong Chang, Lingfeng Wang, Gaofeng Meng, Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In Proceedings of the IEFE international conference computer vision, pages 5879-5887, 2017.
[6] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16794 16804, 2021.
[8] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-laye networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics. pages JMLR Workshop and Conference Proceedings, 2011.
10

---

[9] Zhiyuan Dang, Cheng Deng, Xu Yang. Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 13693-13702, 2021.
[10] Jean-Bastier Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-2123 2020.
[11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738, 2020.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Jiabo Huang, Shaogang Gong, and Xiatian Zhu. Deep semantic clustering by partition confidence maximisation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8849-8855, 2020.
[14] Zhizhong Huang, Jie Chen, Junping Zhang, and Hongming Shan. Learning representation for clustering via prototype scattering and positive sampling, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[15] Xu Ji, Jono F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEF/CVF international conference on computer vision. pages 9865 9874, 2019.
[16] Zhen Jiang, Yongzhao Zhan, Qirong Mao, and Yang Du. Semi supervuuer clustering under "compact- cluster" assumption. IEEE Transactions on Knowledge and Data Engineering, 35(5):5244 5256, 2022.
[17] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
[18] Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng. Joey Tianyi Zhou, and Peng. Contrastive clustering. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages
2021.
[19] Yunfan Li, Peng Hu, Dezhong Peng, Jiancheng Lv, Jianping Fan, and Xi Peng. Image clustering with external guidance. arXiv preprint arXiv:2310. 1989, 2023.
[20] Yunfan Li, Yijie Lin, Peng Hu, Dezhong Peng. Han Luo, and Xi Peng. Single-cell rna-seq debiased clustering via batch effect disentanglement. IEEE Transactions on Neural Networks and Learning Systems, pages 1 11, 2023.
[21] Yunfan Li, Mouxing Yang, Dezhong Peng, Taihao Li, Jiantao Huang, and Xi Peng. Twin contrastive learning for online clustering. International Journal of Computer Vision. 1300))2205-2221, 2022.
[22] Sihang Liu, Wenming Cao, Ruigang Fu, Kaixiang Yang, and Zhiwen Yu. Rpse: Robust pseudo-labeling for semantic clustering. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14008-14016, 2024.
[23] Xinran Ma, Mouxing Yang. Yunfan Li, Peng Hu, Jiancheng Lv. and Peng- Cross-modal retrieval with noisy correspondence via consistency refining and mining. IEEE transactions on image processing, 2024.
[24] Laura Manduchi, Kieran Chin ChennC, Holger Michel, Sven Wellmann, and Julia Vogt. Deep conditional gaussian mixture model for constrained clustering. Advonces in Neural Information Processing Systems, 34:1 11:303--1314, 2021.
[25] Amir Markovitz, Gilad Sharir, Itamar Friedman, Lihi Zelnik-Manor, and Shai Avidan. Graph embedded pose clustering for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages
2020.
[26] Joannis Maniadis Metaxas, Georgios Tzimiropoulos, and loannis Patras. Divclust: Controlling diversity in deep clustering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3418-3428, 2023.
[27] Tri Nguyen, Shahana Ibrahim, and Xiao Fu. Deep clustering with incomplete noisy pairwise annotations: A geometric regularization approach. In International Conference on Machine Learning, pages 25980-26007. PMLR, 2023.

---

[28] Dong Nie. Li Wang, Lei Xiang. Sihang Zhou, Ehsan Adeli, and Dinggang Shen. Difficulty- aware attention network with confidence learning for medical image segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1085-1092, 2019.
[29] Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions on Image Processing. 31:7264-721 -278, 2022.
[30] Xi Peng, Jiashi Feng, Shijie Xiao, Wei-Yun Yau, Joey Tianyi Zhou, and Songfan Yang. Structured autoencoders for subspace clustering. JEEE Transactions on Image Processing, 27(10):5076 5086, 2018.
[31] Xi Peng. Shijie Xiao, Jiashi Feng. Wei-Yun Yau, and Zhang Yi. Deep subspace clustering with sparsity prior. In IJCAL. pages 1925-1931, 2016.
[32] Qi Qian. Stable cluster discrimination for deep clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16645 (6645-16654, 2023.
[33] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[34] Yuming Shen, Ziyi Shen, Menghan Wang, Jie Qin, Philip Tomr and Ling Shao. You never cluster alone. Advances in Neural Information Processing Systems, 34:27734-27746, 2021.
[35] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li, Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-600 2020.
[36] Yaling Tao, Kentaro Takagi, and Kouta Nakata. Clustering- representation learning via instance discrimination and feature decorrelation. arXiv preprint arXiv:2106.0013)) 2021.
[37] Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image clustering. In International conference on learning representations, 2020.
[38] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European conference computer vision, pages 268 285. Springer, 2020.
[39] Kiri Wagstaff' and Claire Cardie. Clustering with instance-level constraints. AAAI/AAL, 1097:577-584, 2000.
[40] Kiri Wagstaff, Claire Cardie, Seth Rogers, Stefan Schrödl,, al. Constrained k-means clustering with background knowledge. In Ieml, volume 1, pages 577-584, 2001.
[41] Jianlong Wu, Keyu Long, Fei Wang. Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8150-8159, 2019.
[42] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pages 478-487. PMLR, 2016
[43] Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5157-5156, 2016.
[44] Mouxing Yang, Zhenyu Huang, Peng Hu, Taihao Li, Jiancheng Lv, Xi Peng. Learning with twin noisy labels for visible-infrared person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14308-14317, 2022.
[45] Chunlin Yu, Ye Shi, and Jingya Wang. Contextually affinitive neighborhood refinery for deep clustering. Advancex in Neural Information Processing Systems, 36, 2024.
[46] Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, and Shenn Hua. Graph contrastive clustering. In Proceedings of the IEEE/CVF international conference computer vision, pages 9224 222329923,222 2021.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
The paper provides the full set of assumptions and complete (and correct)
proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data is widely used and is available to everyone. We are now organizing our code and will release it soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Previous deep clustering works didn't 't report the error bars. Therefore, it's meaningless report the error bars since there's comparison since don't 't have enough computer resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We introduce user interaction and the time of interaction is uncontrollable. Besides, the time for fine-tuning up the pre-trained model.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conduct in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "4CtAeVwRXG", "reviewer": "Reviewer_MjhJ", "summary": "In this work, the authors propose a plug-and-play module to boost the clustering performance of existing methods through user interaction. A sample value evaluation criterion is designed to propose valuable user queries of a high performance-to-cost ratio. Experiments show that the proposed module could significantly improve the clustering performance with low user interaction costs.", "questions": "1) What is the time complexity of the sample selection process? Does it scale to large datasets?\n2) Why are the choices of $\\lambda_1$ and $\\lambda_2$ different for two pre-trained clustering models? \n3) Querying about 500 samples is non-negligible. Have you evaluated how much time it takes for a person to give feedback?\n4) In implementation details, the authors write \"For ProPos, we warm up the cluster head with the positive and negative loss in Eq. 12 and 13 in the first 50 epochs, since the prediction confidences are unreliable initially\". A recent clustering work (Towards Calibrated Deep Clustering Network, arXiv 2024) shows that initializing the two-layer cluster head with k-means is effective. I wonder if the initialization strategy works in the IDC framework.", "limitations": "No significant limitations are found.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "While most existing deep clustering works focus on improving feature discriminability, this work improves the clustering performance from a new perspective. Though introducing user interaction seems to violate the unsupervised setting of clustering, it is meaningful in real-world applications. In addition to the interesting topic, the proposed method itself is also technically sound and solid. The authors propose to reduce the interaction costs by selecting high-value samples and designing an easy interaction form, which improves the application prospect of the method.", "weaknesses": "1) In Table 1, I suggest the authors include a performance baseline by considering the 500 query samples are correctly clustered. As a result, the improvement brought by the proposed method would be clearer.\n2) The hyper-parameters are tuned on different datasets, and no parameter analyses are provided.\n3) Since this work is more application-oriented, I encourage the authors to carefully design the interaction interface to further improve the interaction efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and detailed suggestions for our work. Below are the point-by-point responses to your concerns.\n\n**Weakness 1: The relative performance improvement**\n\nAccording to your comments, we added a baseline by manually correcting the cluster assignments of the 500 query samples, denoted by TCL$^{\\dagger}$ and ProPos$^{\\dagger}$ in the table below.\n\n|  |   | CIFAR-10 |  |      | CIFAR-20 |      |      | STL-10 |      |      | ImageNet-10 |      |      | ImageNet-Dogs |      |\n| :---------------------: | :--: | :------: | :--: | :--: | :------: | :--: | :--: | :----: | :--: | :--: | :---------: | :--: | :--: | :-----------: | :--: |\n|  | NMI  |   ACC    | ARI  | NMI  |   ACC    | ARI  | NMI  |  ACC   | ARI  | NMI  |     ACC     | ARI  | NMI  |      ACC      | ARI  |\n|           TCL           | 81.9 |   88.7   | 78.0 | 52.9 |   53.1   | 35.7 | 79.9 |  86.8  | 75.7 | 87.5 |    89.5     | 83.7 | 62.3 |     64.4      | 51.6 |\n|     TCL$^{\\dagger}$     | 82.2 |   88.9   | 78.4 | 53.2 |   53.5   | 36.1 | 82.0 |  88.6  | 78.5 | 88.6 |    90.4     | 85.0 | 62.8 |     65.6      | 52.3 |\n|  IDC$_{\\textrm{TCL}}$   | 83.9 |   91.6   | 82.8 | 59.4 |   69.3   | 49.5 | 85.8 |  92.7  | 84.7 | 92.9 |    97.0     | 93.5 | 67.8 |     78.0      | 62.3 |\n|         ProPos          | 87.8 |   93.6   | 87.1 | 59.1 |   59.1   | 43.6 |  -   |   -    |  -   | 88.9 |    95.2     | 89.6 | 73.0 |     76.9      | 66.9 |\n|   ProPos$^{\\dagger}$    | 87.9 |   93.7   | 87.3 | 59.3 |   59.4   | 43.8 |  -   |   -    |  -   | 89.6 |    95.5     | 90.3 | 73.8 |     77.8      | 67.8 |\n| IDC$_{\\textrm{ProPos}}$ | 90.5 |   95.7   | 90.9 | 69.2 |   78.3   | 61.4 |  -   |   -    |  -   | 93.2 |    97.3     | 94.1 | 77.6 |     86.1      | 74.8 |\n\nThe results show that solely correcting the cluster assignments of 500 samples brings marginal performance improvement since they are only a small portion of the data. On the contrary, our IDC effectively utilizes user feedback on 500 query samples, which leads to much better performance. We will add the baseline to the revised manuscript.\n\n**Weakness 2 & Question 2: Inconsistent $\\lambda_1, \\lambda_2$ for different clustering models**\n\nIn the previous version, we roughly tuned $\\lambda_1, \\lambda_2$ to produce relatively cluster-balanced inquiry samples. According to your suggestion, we applied our IDC to the pre-trained TCL clustering model with $\\lambda_1=1, \\lambda_2=1$​ (the same configuration for ProPos). It turns out that IDC works well with the default hyper-parameters.\n\n|             |             |      | CIFAR-10 |      |      | CIFAR-20 |      |      | STL-10 |      |      | ImageNet-10 |      |      | ImageNet-Dogs |      |\n| :---------: | :---------: | :--: | :------: | :--: | :--: | :------: | :--: | :--: | :----: | :--: | :--: | :---------: | :--: | :--: | :-----------: | :--: |\n| $\\lambda_1$ | $\\lambda_2$ | NMI  |   ACC    | ARI  | NMI  |   ACC    | ARI  | NMI  |  ACC   | ARI  | NMI  |     ACC     | ARI  | NMI  |      ACC      | ARI  |\n|     2.5     |     0.5     | 83.9 |   91.6   | 82.8 | 59.4 |   69.3   | 49.5 | 85.8 |  92.7  | 84.7 | 92.9 |    97.0     | 93.5 | 67.8 |     78.0      | 62.3 |\n|     1.0     |     1.0     | 84.4 |   92.7   | 84.8 | 58.1 |   69.4   | 48.7 | 85.3 |  92.7  | 84.6 | 93.2 |    97.2     | 93.9 | 69.1 |     78.8      | 63.6 |\n\nAs a result, we decide to remove the two tunable parameters $\\lambda_1, \\lambda_2$ in the next version.\n\n**Weakness 3: Design of interaction interface**\n\nThanks for the suggestion, we will provide a user-friendly interaction interface to aid real-world applications.\n\n**Question 1: Time complexity and scalability**\n\nThe proposed sample selection process consists of computing three metrics (hardness, representativeness, and diversity) and selecting the most valuable sample. Denote the number of samples as $N$, the number of clusters as $C$, and the number of query samples as $M$, the time complexity of these steps is analyzed as follows.\n\nFor the hardness metric, we compute the cosine distance to the two nearest centers for each sample, which is of $O(NC)$ time complexity.\n\nFor the representativeness metric, we compute the sum of Euclidean distance of $K$ nearest neighbors for each sample, which is of $O(NK)$ time complexity. Note that the $K$ nearest neighbors search is of $O(N+KlogK)$ time complexity, which could be omitted.\n\nFor the diversity metric, we compute the cosine distance to samples previously selected for each sample, which is of $O(NM)$ time complexity.\n\nFinally, selecting the most valuable sample selection requires $O(M)$ time complexity.\n\nIn summary, as the number of query samples $M$ are usually larger than the number of clusters as $C$ and the number of nearest neighbors $K$, the overall time complexity of the sample selection process is  $O(NM)$. Since only a small portion of samples are selected for query (i.e., $M<<N$), our sample selection strategy scales to large datasets.\n\n**Question 3: Interaction time cost**\n\nWe asked three people to answer the 500 queries on CIFAR-20 and ImageNet-Dogs. On average, it took about 6 seconds to decide the cluster affiliation relative to the nearest cluster centers for each sample, and querying 500 samples requires about 50 minutes. Nevertheless, querying 100 samples, which takes only 10 minutes, already achieves half the performance improvement brought by 500 samples as shown in Fig. 4. In practice, the user could flexibly decide the number of queries based on the demand, making a trade-off between efficiency and performance.\n\n**Question 4: Cluster head initialization**\n\nThanks for pointing out the cluster head initialization strategy proposed in the mentioned work. However, the authors did not release the code. We tried initializing the cluster head following the instructions in the paper but failed to reproduce the results. We will keep paying attention to the mentioned work and update the cluster head initialization strategy when applicable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwGXxDzjZ9", "reviewer": "Reviewer_uVM4", "summary": "This work proposes an interactive deep clustering framework IDC, which could be integrated with existing deep clustering methods. The key idea is to adjust the decision boundary by querying the cluster affiliations of high-value samples. The authors applied IDC to two pre-trained clustering models on five datasets. Experimental results prove the effectiveness of the proposed method.", "questions": "Please refer to the weakness section.", "limitations": "In this work, the authors assume that the user feedback is always correct. It is unknown how IDC performs when there are mistakes in the user feedback.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is interesting to investigate whether introducing weak supervision to unsupervised clustering would be helpful. In real-world applications, I think most people are willing to improve the clustering performance through low-cost interaction with the clustering algorithm.  \n- The proposed IDC is technically sound. The authors prove that the progressive sample selection strategy could select the most valuable samples to reduce the interaction cost.\n- The authors show that IDC outperforms semi-supervised classification and clustering baselines, which proves the effectiveness and superiority of IDC.", "weaknesses": "- In the evaluation, the authors assume the user feedback is always correct, which may not hold in real-world applications. It is worth exploring how IDC behaves when there are mistakes in the user feedback. Alternatively, some examples could be supplied to show that the interaction task is easy enough for the user to correctly predict the cluster affiliations of those hardest samples.\n- Table 1 shows that the previous semi-supervised clustering method Cop-Kmeans even achieves worse results than applying Kmeans on original features. Explanations of the results need to be supplied.\n- Figure 4 shows that the superiority of the proposed sample selection strategy against random selection is marginal on ImageNet-Dogs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and kind suggestions for our work. Below are the point-by-point responses to your concerns mentioned in the weaknesses and limitations sections. \n\n**Weakness 1 & Limitation 1: Imperfect user feedback**\n\nAccording to your advice, we asked three people to answer the 500 queries on CIFAR-20 and ImageNet-Dogs. We counted the number of correct user feedback and used the feedback to finetune the TCL clustering model. The results are shown in the table below. As requested, we also supplied more examples of user queries in Figure 3 in the attached PDF file.\n\n|                  |      |      | CIFAR-20 |                                 |      |      | ImageNet-Dogs |                                 |\n| :--------------: | :--: | :--: | :------: | :-----------------------------: | :--: | :--: | :-----------: | :-----------------------------: |\n|                  | NMI  | ACC  |   ARI    | Number of correct user feedback | NMI  | ACC  |      ARI      | Number of correct user feedback |\n| Pre-trained TCL  | 52.9 | 53.1 |   35.7   |                -                | 62.3 | 64.4 |     51.6      |                -                |\n| Perfect feedback | 59.4 | 69.3 |   49.5   |               500               | 67.8 | 78.0 |     62.3      |               500               |\n| User 1 feedback  | 57.5 | 66.4 |   46.4   |               452               | 66.7 | 75.3 |     59.9      |               460               |\n| User 2 feedback  | 58.0 | 68.0 |   48.2   |               453               | 66.4 | 74.6 |     59.2      |               458               |\n| User 3 feedback  | 55.7 | 64.5 |   44.1   |               431               | 65.5 | 74.1 |     58.9      |               447               |\n\nThe above results demonstrate that: i) it is not difficult for the user to correctly predict the cluster affiliations of the query samples, and ii) IDC is robust to the mistakes in the user feedback, which suits real-world applications.\n\n**Weakness 2: Explanations on Cop-Kmeans results**\n\nCop-Kmeans is a semi-supervised clustering method built upon the vanilla K-means. Given the pairwise constraints (i.e., must-link and cannot-link), Cop-Kmeans forces sample pairs with must-link constraints to be assigned to the same cluster and vice versa. In practice, the pairwise constraints often conflict with the cluster assignments made by the vanilla K-means. As a result, the cluster centers would deviate from the underlying data distribution due to \"outliers\" introduced by the pairwise constraints, leading to inferior clustering performance. Such a limitation of Cop-Kmeans could be attributed to the fixed features during the Kmeans process. In other words, Cop-Kmeans fails when the input features are intrinsically inseparable by assigning each sample to the nearest cluster center. In comparison, the proposed IDC could adjust the features by optimizing the cluster head, overcoming the intrinsic limitation of Cop-Kmeans.\n\nWe will add the above discussion to the revised manuscript.\n\n**Weakness 3: Marginal improvements against random selection on ImageNet-Dogs**\n\nAs you pointed out, the superiority of our sample selection strategy against the random selection baseline on ImageNet-Dogs is less significant than that on CIFAR-20. Such a result is in fact reasonable since ImageNet-Dogs contains only 1/3 samples compared with CIFAR-20 (see dataset summary table below). \n\n|Dataset|Split|Samples|Classes|\n|:--:|:--:|:--:|:--:|\n|CIFAR-10|Train+Test|60000|10|\n|CIFAR-20|Train+Test|60000|20|\n|STL-10|Train+Test|13000|10|\n|ImageNet-10|Train|13000|10|\n|ImageNet-Dogs|Train|19500|15|\n\nIn other words, for the same number of query images, its proportion relative to the entire dataset on ImageNet-Dogs is larger than that on CIFAR-20. As the proportion increases, the performance gap between different sample selection strategies will be less significant, which explains the relatively marginal performance improvement against random selection on ImageNet-Dogs. For a more reasonable validation, to keep the proportion consistent, we investigate the selected sample number $M$ in the range 0-200 with an interval of 25 for ImageNet-Dogs. The results are provided in Figure 1 in the attached PDF file. As can be seen, when querying for 50 samples, our sample selection strategy outperforms random selection by 3.88 in terms of clustering ACC, larger than the gap of 2.3/0.42 ACC when querying 200/700 samples. The above results demonstrate the superiority of our sample selection strategy, especially when only a small portion of samples are queried.\n\nWe will add the above results and discussions in the Appendix in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCvNX5ui04", "reviewer": "Reviewer_2n5n", "summary": "This paper proposes a way to integrate user-feedback to fine-tune clustering models and improve their performance on hard samples, i.e. samples near cluster boundaries. In particular, the authors propose a score to value and select samples for user-interaction. This score is a combination of *hardness*, *representativeness* and *diversity*, each one defined with a mathematical formulation. Samples are ranked based on their value and the top M samples are selected for user-interaction. Then a user gives feedback on each selected sample by indicating whether this sample belongs to one of the nearest cluster centers (positive feedback), or it does not belong to any of them (negative feedback). Once this user-feedback is gathered, the model is fine-tuned on this information, and a regularization term is included to ensure the model still retains its predictions on high-confidence samples. While the *hardness* term in the score encourages hard samples to be selected, *representativeness* encourages samples from dense regions to be selected, and finally *diversity* ensures samples are not selected only from a handful of small regions. The authors apply their method to two recent pre-trained clustering models, and compare their results with alternative methods on five image datasets.", "questions": "Some questions for the authors and suggestions for additional results to show are in the Weaknesses section. \nBelow are suggested minor changes for quality of the text.  \n- Line 53: change \"model-irrelevant\" to \"model-agnostic\"\n- Line 217: change \"state of the arts\" to \"state-of-the-art\"", "limitations": "While in the checklist the authors state that they discuss the limitations in the conclusion section, I do not find that this is the case, and I think it would be important to explicitly mention important limitations of the proposed approach. Examples of such limitations are the fact that this method relies on the user being able to give accurate feedback on the clustering, which in many real-world settings is not reasonable to assume, and the fact that the method is only tested on image data (and not other modalities where user feedback might be less effective or more costly to get).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper deals with the important problem of improving model clustering performance on hard samples. \n- The idea is simple and well-presented, and looking at the reported metrics it leads to improved results. This however might not be an extensive evaluation (see Weaknesses below). \n- On the experimental settings considered by the authors, many state-of-the-art methods are reported for comparison.", "weaknesses": "- One intrinsic and crucial weakness of this approach is the assumption that a user is able to give meaningful and accurate feedback on the clustering. In many real world settings and with complex data types (e.g. genomics) this is an unreasonable assumption to make, hence the method cannot be applied. In other settings (e.g. medical applications) such a feedback might require domain experts and hence be very costly. This intrinsic limitation significantly impacts the practical relevance of the method. \n- While the results for the presented method look positive compared to state-of-the-art, important information is missing to really assess if the clustering performance is enhanced, and specifically if it is actually enhanced on hard samples over which the model does not receive user-feedback. This is particularly important since the authors claim \"Different from existing studies that pursue overall performance improvements, this work aims to address the specific hard sample problem.\" In detail, how is the performance after fine-tuning on user-feedback calculated? Is there a hold-out test set? If the same data used for user-feedback and fine-tuning is then used to test model performance (e.g. when computing results in Table 1), then I think the metrics reported are not strong evidence. In fact the improved performance might be due mainly to the model clustering better the samples over which it receives user-feedback. What would be important evidence for the value of the proposed method is showing that after fine-tuning the model performs better specifically on hard samples **over which it did not receive user-feedback nor was fine-tuned on**. Unfortunately this seems to be missing. \n- There is no Appendix to the paper, and I think it is a pity, since important details regarding e.g. datasets or training/test splits and experiments are missing. How big is the number of samples M=500 selected for user-feedback relative to the total size of each dataset? Also I think it would be important to give more details on user-interaction (e.g. mentioning how many users participated, how were they selected, describing the interface used to give feedback). \n- Figure 4 shows that for the number of samples M=500 selected for user-feedback in the experiments the impact of the proposed sample selection strategy is marginal, especially for ImageNet-Dogs. In particular, since the accuracies of the proposed strategy and the random one are close, it would be good to show also the same ablation results for ARI and NMI to see if the selection strategy yields a concrete improvement on these metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of the problem this work aims to address, as well as our simple yet effective idea. We sincerely appreciate your insightful and constructive comments. Below are the point-by-point responses to your concerns raised in weaknesses, questions, and limitations.\n\n**Weakness 1 & Limitation: Imperfect user feedback**\n\nWe agree with you that providing accurate user feedback would be more costly when handling more complex data types. However, we would like to highlight the motivation of our work, namely, introducing user interaction to address the hard sample problem suffered by fully unsupervised clustering methods. If a data sample is so hard that even humans cannot distinguish it, then machine learning methods are more unlikely to correctly cluster it solely based on the data itself. In other words, by selecting the most valuable samples for user queries, our method provides a chance of grouping samples undistinguishable by clustering methods, with additional user interaction costs as low as possible. We believe such a solution caters to practical needs.\n\nMoreover, in real-world genomic and medical applications, the users are very likely to be domain experts since not everyone has the demand for clustering genomic data. Besides, as we later show in the response to Weakness 3, our method is robust to the potential mistakes in the user feedback, which suits real-world applications.\n\n**Weakness 2: More evidence of performance improvement**\n\nAccording to your comments, we further assessed the experimental results and concluded that the performance improvement of our IDC on hard samples is not incremental. Specifically, following the common understanding, we treated samples with cluster assignment confidence lower than 0.9 as hard samples (we did not use the hardness metric defined in the paper as the logarithmic value is less intuitive). Note that the sample confidence for ProPos was computed through a cluster head initialized with Kmeans cluster centers. Then, to better understand the performance gain brought by IDC on hard samples, we partitioned hard samples into two subgroups, namely, i) hard-in, consisting of hard samples selected for query, and ii) hard-out, consisting of hard samples not selected for query. We reported the proportion of hard-in samples among all hard samples, and the clustering accuracy of hard-in/out samples before/after the model finetuning. Below are the results of IDC applied to the TCL and ProPos models, respectively.\n\n|IDC$_{\\textrm{TCL}}$|CIFAR-10|CIFAR-20|STL-10|ImageNet-10|ImageNet-Dogs|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|hard-in ratio|4.6%|2.3%|35.1%|22.9%|7.9%|\n|hard-in ACC (before)|47.9|28.2|38.8|50.0|29.6|\n|hard-in ACC (after)|99.4 (↑51.5)|81.7 (↑53.5)|98.9 (↑60.1)|94.3 (↑44.3)|89.8 (↑60.2)|\n|hard-out ACC (before)|48.5|23.9|43.5|49.7|27.1|\n|hard-out ACC (after)|60.4 (↑11.9)|37.4 (↑13.5)|62.5 (↑19.0)|70.5 (↑20.8)|55.1 (↑20.0)|\n\n|IDC$_{\\textrm{ProPos}}$|CIFAR-10|CIFAR-20|ImageNet-10|ImageNet-Dogs|\n|:-:|:-:|:-:|:-:|:-:|\n|hard-in ratio|3.0%|1.5%|8.0%|7.4%|\n|hard-in ACC (before)|58.6|40.2|48.6|38.9|\n|hard-in ACC (after)|99.1 (↑40.5)|96.0 (↑55.8)|97.2 (↑48.6)|82.2 (↑43.3)|\n|hard-out ACC (before)|58.5|29.2|53.1|38.3|\n|hard-out ACC (after)|81.5 (↑23.0)|63.0 (↑33.8)|77.8 (↑24.7)|58.6 (↑20.3)|\n\nFrom the above results, one could arrive at three conclusions: i) the selected query samples only take a small percentage among all hard samples. In other words, simply correcting the cluster assignments of selected samples could only bring small improvements on hard samples; ii) the clustering accuracy of selected hard samples improves significantly, which is natural since these samples are used for finetuning; and iii) the clustering accuracy of hard samples not selected for query also greatly increases after finetuning, which demonstrates that the user feedback benefits the held-out data as well.\n\nTo further show that the user feedback also boosts the clustering performance of samples not selected for interaction, we added a baseline by manually correcting the cluster assignments of the 500 query samples, denoted by TCL$^{\\dagger}$ and ProPos$^{\\dagger}$ in the table below.\n\n|||CIFAR-10|||CIFAR-20|||STL-10|||ImageNet-10|||ImageNet-Dogs||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||NMI|ACC|ARI|NMI|ACC|ARI|NMI|ACC|ARI|NMI|ACC|ARI|NMI|ACC|ARI|\n|TCL|81.9|88.7|78.0|52.9|53.1|35.7|79.9|86.8|75.7|87.5|89.5|83.7|62.3|64.4|51.6|\n|TCL$^{\\dagger}$|82.2|88.9|78.4|53.2|53.5|36.1|82.0|88.6|78.5|88.6|90.4|85.0|62.8|65.6|52.3|\n|IDC$_{\\textrm{TCL}}$|83.9|91.6|82.8|59.4|69.3|49.5|85.8|92.7|84.7|92.9|97.0|93.5|67.8|78.0|62.3|\n|ProPos|87.8|93.6|87.1|59.1|59.1|43.6|-|-|-|88.9|95.2|89.6|73.0|76.9|66.9|\n|ProPos$^{\\dagger}$|87.9|93.7|87.3|59.3|59.4|43.8|-|-|-|89.6|95.5|90.3|73.8|77.8|67.8|\n|IDC$_{\\textrm{ProPos}}$|90.5|95.7|90.9|69.2|78.3|61.4|-|-|-|93.2|97.3|94.1|77.6|86.1|74.8|\n\nThe results show that solely correcting the cluster assignments of 500 samples brings marginal performance improvement since they are only a small portion of the data. On the contrary, our IDC effectively utilizes user feedback on 500 query samples, which leads to much better performance.\n\nTo sum up, the above results demonstrate that our user interaction strategy could greatly improve the clustering performance on samples, no matter whether they are selected for the user query. We will add the above results and discussions to the revised manuscript.\n\n**Due to the space limitation, we put the remaining part of the responses in the global author response above.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xGI96iDZX8", "reviewer": "Reviewer_N8tX", "summary": "This paper proposes to incorporate user interactions to deal with the hard samples in deep clustering. By asking the user for the cluster affiliations of boundary samples, the proposed framework improves the performance of existing clustering methods.", "questions": "See weaknesses.", "limitations": "The authors do not discuss the limitations of this work. I think that the cost of selecting 500 images for interaction is non-negligible, which should be included in the manuscript as a potential limitation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The hard boundary samples are indeed the performance bottleneck of most existing clustering methods, since they cannot be easily distinguished in a fully unsupervised manner. This work proposes to incorporate user interaction to guide the clustering process, which is a natural thought and a practical solution.\n2. Experiments demonstrate the effectiveness of the proposed IDC framework in boosting clustering performance with reasonable interaction costs.\n3. The paper is well-structured and easy to follow in general.", "weaknesses": "1. I noticed that the hyper-parameters are different when applying the proposed IDC to TCL and ProPos. However, it is daunting to tune the hyper-parameters for unsupervised tasks. The authors need to provide detailed instructions on how these two parameters should be tuned in practice. Besides, parameter analyses on $\\lambda_1$ and $\\lambda_2$ are expected to investigate the robustness of IDC.\n2. As shown in Table 2, selecting samples based on \"Hard+Rep\" leads to even inferior performance than the pre-trained baseline. More explanations are needed on the unusual phenomenon.\n3. The current utilization of user feedback is relatively simple. More advanced and thorough utilization strategies are worth exploring to reduce the interaction cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback and suggestions. Below we provide the point-by-point response to your concerns.\n\n**Weakness 1:  Inconsistent $\\lambda_1, \\lambda_2$ for different clustering models**\n\nWe agree that it is daunting to tune the hyper-parameters for unsupervised tasks. In the previous version, we roughly tuned $\\lambda_1, \\lambda_2$ to produce relatively cluster-balanced inquiry samples. According to your suggestion, we applied our IDC to the pre-trained TCL clustering model with $\\lambda_1=1, \\lambda_2=1$ (the same configuration for ProPos). It turns out that IDC works well with the default hyper-parameters.\n\n|             |             |      | CIFAR-10 |      |      | CIFAR-20 |      |      | STL-10 |      |      | ImageNet-10 |      |      | ImageNet-Dogs |      |\n| :---------: | :---------: | :--: | :------: | :--: | :--: | :------: | :--: | :--: | :----: | :--: | :--: | :---------: | :--: | :--: | :-----------: | :--: |\n| $\\lambda_1$ | $\\lambda_2$ | NMI  |   ACC    | ARI  | NMI  |   ACC    | ARI  | NMI  |  ACC   | ARI  | NMI  |     ACC     | ARI  | NMI  |      ACC      | ARI  |\n|     2.5     |     0.5     | 83.9 |   91.6   | 82.8 | 59.4 |   69.3   | 49.5 | 85.8 |  92.7  | 84.7 | 92.9 |    97.0     | 93.5 | 67.8 |     78.0      | 62.3 |\n|     1.0     |     1.0     | 84.4 |   92.7   | 84.8 | 58.1 |   69.4   | 48.7 | 85.3 |  92.7  | 84.6 | 93.2 |    97.2     | 93.9 | 69.1 |     78.8      | 63.6 |\n\nAs a result, we decide to remove the two tunable parameters $\\lambda_1, \\lambda_2$ in the next version.\n\n**Weakness 2:  Explanations of ablation study results**\n\nThe representativeness metric encourages the model to select samples in high-density regions. As shown in Figure 3(b), with the \"Hard+Rep\" criterion, the selected samples will be heavily imbalanced across clusters. As a result, during the model finetuning, the hard samples would collapse into a few clusters (easy samples will be restricted by the regularization loss), leading to inferior performance even worse than the pre-trained baseline. Such a result is exactly our motivation for designing the diversity metric during sample selection.\n\nWe will supply the above explanations in the revised manuscript. \n\n**Weakness 3: More advanced utilization strategies**\n\nThanks for the suggestion. We agree that more advanced strategies could utilize user feedback more thoroughly. Some potential directions would be introducing mixup augmentations, supervised contrastive loss, or focal loss to improve the model finetuning. We will explore these improvements in future works.\n\n**Limitation: Interaction cost**\n\nAccording to your comments, we asked three people to answer the 500 queries on CIFAR-20 and ImageNet-Dogs. On average, it took about 6 seconds to decide the cluster affiliation relative to the nearest cluster centers for each sample, and querying 500 samples requires about 50 minutes. Nevertheless, querying 100 samples, which takes only 10 minutes, already achieves half the performance improvement brought by 500 samples as shown in Fig. 4. In practice, the user could flexibly decide the number of queries based on the demand, making a trade-off between efficiency and performance.\n\nWe will add the above discussions to the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the reviewers for their insightful and constructive comments. We supply a PDF containing the supplementary figures and provide the point-by-point responses below.

---
*Due to the space limitation, we put part of the responses to Reviewer 2n5n in this section:*

**Weakness 3: Dataset and user interaction details**

The experimental configs and details are mentioned in Section 4.2. According to your comments, we supply the summary of datasets used in our experiments in the following table. The 500 selected samples accounted for 0.8%-3.8% of the total size of each dataset.

|Dataset|Split|Samples|Classes|
|:-:|:-:|:-:|:-:|
|CIFAR-10|Train+Test|60000|10|
|CIFAR-20|Train+Test|60000|20|
|STL-10|Train+Test|13000|10|
|ImageNet-10|Train|13000|10|
|ImageNet-Dogs|Train|19500|15|

For user interaction, in the current version, we assumed that the user gives perfect responses to 500 queries. According to your comments, we asked three colleagues to answer the 500 queries on CIFAR-20 and ImageNet-Dogs. We counted the number of correct user feedback and used the feedback to finetune the TCL clustering model. The results are shown in the table below. We also supplied more examples of the interaction interface in Figure 3 in the attached PDF file.

||||CIFAR-20||||ImageNet-Dogs||
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
||NMI|ACC|ARI|Number of correct user feedback|NMI|ACC|ARI|Number of correct user feedback|
|Pre-trained TCL|52.9|53.1|35.7|-|62.3|64.4|51.6|-|
|Perfect feedback|59.4|69.3|49.5|500|67.8|78.0|62.3|500|
|User 1 feedback|57.5|66.4|46.4|452|66.7|75.3|59.9|460|
|User 2 feedback|58.0|68.0|48.2|453|66.4|74.6|59.2|458|
|User 3 feedback|55.7|64.5|44.1|431|65.5|74.1|58.9|447|

The above results demonstrate that: i) it is not difficult for the user to correctly predict the cluster affiliations of the query samples, and ii) IDC is robust to the mistakes in the user feedback, which suits real-world applications.

We will add the above additional information and results to the Appendix in the next version.

**Weakness 4: Marginal improvements against random selection on ImageNet-Dogs**

As you pointed out, the superiority of our sample selection strategy against the random selection baseline on ImageNet-Dogs is less significant than that on CIFAR-20, especially when more images are queried. Such a result is in fact reasonable since ImageNet-Dogs contains only 1/3 samples compared with CIFAR-20 (see dataset summary table above). In other words, for the same number of query images, its proportion relative to the entire dataset on ImageNet-Dogs is larger than that on CIFAR-20. As the proportion increases, the performance gap between different sample selection strategies will be less significant, which explains the relatively marginal performance improvement against random selection on ImageNet-Dogs. For a more reasonable validation, to keep the proportion consistent, we investigate the selected sample number $M$​ in the range 0-200 with an interval of 25 for ImageNet-Dogs. The results are provided in Figure 1 in the attached PDF file. As can be seen, when querying for 50 samples, our sample selection strategy outperforms random selection by 3.88 in terms of clustering ACC, larger than the gap of 2.3/0.42 ACC when querying 200/700 samples. According to your comments, we supplied figures of ARI and NMI metrics, which show a consistent tendency. The above results demonstrate the superiority of our sample selection strategy, especially when only a small portion of samples are queried.

We will add the above results and discussions to the Appendix in the next version.

**Questions: Typo**

We sincerely appreciate your meticulous review. We will correct these typos in the next version.

Meta-review: This paper introduces an interactive deep clustering framework (IDC) designed to enhance existing clustering methods through minimal user interaction. 

All reviewers agree that this work is highly interesting and the proposed method itself is technically sound and solid and leads to improved results.  User interactions  to guide the clustering process can be important in real-world applications. I vote for accepting this work, as this paper looks at the clustering performance from a new perspective, and I think that this work is interesting to the community. For a clear positioning of the work, it would be good if the authors could discuss their work in the light of this work https://arxiv.org/abs/1805.11571 in the final version.
Decision: Accept (poster)