Towards Calibrated Deep Clustering Network

## Abstract

Abstract Deep clustering has exhibited remarkable performance; however, the over-confidence problem, i.e., the estimated confidence for a sample belonging to a particular cluster greatly exceeds its actual prediction accuracy, has been overlooked in prior research. To tackle this critical issue, we pioneer the development of a calibrated deep clustering framework. Specifically, we propose a novel dual-head (calibration head and clustering head) deep clustering model that can effectively calibrate the estimated confidence and the actual accuracy. The calibration head adjusts the overconfident predictions of the clustering head, generating prediction confidence that match the model learning status. Then, the clustering head dynamically select reliable high-confidence samples estimated by the calibration head for pseudo-label self-training. Additionally, we introduce an effective network initialization strategy that enhances both training speed and network robustness. The effectiveness of the proposed calibration approach and initialization strategy are both endorsed with solid theoretical guarantees. Extensive experiments demonstrate the proposed calibrated deep clustering model not only surpasses state-of-the-art deep clustering methods by 10 times in terms of expected calibration error but also significantly outperforms them in terms of clustering accuracy.

## 1 Introduction

Clustering aims to categorize the input samples into distinct groups, where samples within the same group exhibit greater similarity than those in other groups, without utilizing any ground-truth supervisory information. Recently, deep learning-based clustering methods, a.k.a. deep clustering, have showcased remarkable clustering performance owing to the powerful feature representation capability of deep neural networks (DNN) [, ]. However, training a deep clustering network is challenging as no supervisory information is available.
Existing methods solve the training dilemma by introducing some side tasks in the training process. For example, DEC [] introduces the auto-encoder (AE) during the training, and [] adopts a generative adversarial network.

The pseudo-labeling-based methods first extract features by self-supervised learning (like MoCo []), then add a clustering head to produce the clustering prediction, and further uses the model’s own output as pseudo labels to guide the unsupervised clustering network training, have become mainstream in deep clustering [, , ]. Although those methods have achieved remarkable clustering performance, they have two drawbacks. First, they generally use a fixed threshold to select the pseudo labels, which ignores the learning status of the model, i.e., in the early training stages, a higher threshold will select fewer and class-imbalanced training samples, which depress the convergence speed, while in the later training stages, the fixed threshold is more likely to introduce noisy pseudo-labels due to the increasing predicted confidence. More critically, those methods all implicitly assume that the output confidence of the model can reflect the true probability of a sample belonging to a certain cluster. However, those methods all face a serious overconfidence problem, i.e., the predicted confidence of the model’s output will largely exceed the actual prediction accuracy. As shown in Fig. 1, the overconfidence problem of the state-of-the-art (SOTA) deep clustering methods like SPICE [] and SCAN [] is even worse than the supervised learning model, as they rely on the pseudo-labels to train the network, and the inevitable noise in the pseudo-labels aggravates the overconfidence problem.
Accurate confidence estimation is important to a machine learning model, especially in building trustworthy decision-making systems like medical diagnosis [], autonomous vehicles [], and financial trading [], which suggests when we can trust the prediction of a learning model. However, confidence calibration has not been studied under deep clustering networks, and the common calibration methods like Temperature Scaling [, , ] rely on a labeled validation set, which is not accessible in deep clustering. Moreover, as shown in Fig. 1-(d), the regularization-based calibration methods like Label Smoothing (LS) [] will over-penalize the high-reliable samples, making the prediction of reliable and unreliable samples indistinguishable.
Second, the current methods initialize the clustering head randomly, which is unstable and will destroy the feature learned by self-supervised learning, i.e., as shown in Fig. 1-(f), the clustering accuracy (ACC) of MoCo equipped with K-means is $50.7\%$, while its performance will degrade to $10.4\%$ after random initialization.

Our Contributions. For the first time, we investigate the construction of a well-calibrated deep clustering network (CDC) in this paper. Specifically, we propose a dual-head network structure comprising a clustering head and a calibration head. Within the calibration head, we propose a novel calibration approach that calibrates the confidence of predictions generated by the clustering head for all samples. This enables the output of the calibration head to accurately estimate the actual confidence level of the predictions.
Meanwhile, within the clustering head, we utilize the calibrated confidence provided by the calibration head to assess the learning progress of each class and selectively choose high-confidence samples for effective pseudo-labeling. These two heads mutually reinforce each other by leveraging information from one another.
Additionally, we present an effective initialization approach to initialize both the clustering head and the calibration head.
As shown in Fig. 1-(f), the clustering performance of our method is quite high only after the proposed initialization strategy (CDC Init.). The effectiveness of both the proposed calibration approach and initialization strategy are endorsed with solid theoretical guarantees. Extensive experimental evaluation on six benchmark datasets shows that our model surpasses SOTA deep clustering methods by approximately 10 times in terms of expected calibration error (ECE) and significantly outperforms them regarding clustering accuracy.

The rest of this paper is organized as follows: In Section 2, we review the related works regarding deep clustering and confidence calibration. Then, in Section 3, we introduce the proposed model in detail. Afterward, we evaluate our model and compare it with the SOTA deep clustering models in Section 4. Finally, we conclude this paper in Section 5.

Figure: (a) Supervised

## 2 Related Works

Deep Clustering.
Deep clustering methods can be broadly classified into two categories: (I) Clustering based on representation learning which first learns a representation of the data and then applies a clustering model like K-means [, ] and Spectral Clustering [, ] to obtain the final clustering, (II) Iterative deep clustering with self-supervision which aim to learn the data representation and performs clustering simultaneously under the supervision of self-training [, ], self-labeling [, , ] and contrastive information [, ]. Recently, self-labeling methods have demonstrated superior performance by selecting high-confidence instances through thresholding soft assignment probabilities and updating the entire network by minimizing the cross-entropy loss of the selected instances. However, they often use a fixed threshold to determine the assignment of pseudo-labels, which does not take into account the learning status of the model.
How to determine a suitable threshold is still a challenging task.

Confidence Calibration.
Capturing the accurate confidence estimation is important to real-world decision-making systems, like self-driving car [] and disease diagnosis [], as in those applications, prediction systems not only should be accurate but also should suggest when they are likely to be not correct. Accurate confidence estimation is also important to the interpretability neural networks.
However, modern neural networks usually face the overconfidence problem, i.e., the estimated confidence by a neural network usually exceeds its real prediction accuracy []. Many calibration methods were proposed to solve this problem. Roughly speaking, the existing calibration methods can be divided into two categories:
(I) Post-calibration methods calibrate the model’s output after model training. For example, Guo et al. [] proposed Temperature Scaling that learns the temperature in the softmax function by a labeled validation set.
(II) Regularization-based methods penalize the confidence of model’s predictions during the training process via some regularization techniques [, , ]. For example, Label Smoothing [] uses soft labels to avoid overfitting the network’s predictions. Focal Loss [] is designed to fit hard samples and prevents easy samples from dominating the training process, implicitly penalizing overconfident predictions. The $L_{1}$ Norm [] directly serves as an additional regularization term to the cross-entropy to control the confidence level.

As demonstrated in Fig. 1-(b)(c), deep clustering methods face a more severe overconfidence problem than the supervised learning models. The reason is that the SOTA deep clustering methods rely on pseudo-labeling, which will unavoidably aggravate the overconfidence problem.
Unfortunately, this problem has been overlooked in prior researches. Moreover, the previous calibration methods can not be directly applied to solve the overconfidence problem in unsupervised clustering, as the post-calibration method needs a labeled validation set to turn its hyper-parameters, while the regularization methods will depress the quality of the selected pseudo labels and accordingly harm the clustering performance. Therefore, in this paper, we study how to build a calibrated deep clustering neural network to achieve both high clustering performance and accurate confidence estimation.

## 3 Proposal Method

Notation. Denote by $\mathcal{D}_{u}=\{\boldsymbol{x}_{i}:i\in\{1,2,...,N\}\}$ a training set of $N$ unlabeled samples. Weak and strong augmentation strategies from [] are used to obtain $\boldsymbol{x}_{i}^{w}$ and $\boldsymbol{x}_{i}^{s}$ for each sample $\boldsymbol{x}_{i}$. Let $f(\varTheta;\boldsymbol{x}_{i})$ be the feature extractor, $g(\theta;\cdot)$ be the classifier, $\sigma\left(\cdot\right)$ be the softmax function, and $C$ be the predetermined classes. Then, the output probabilities are $\boldsymbol{p}_{i}^{w}=\sigma\left(g\left(\theta;f\left(\varTheta;\boldsymbol{
x}_{i}^{w}\right)\right)\right)$ and $\boldsymbol{p}_{i}^{s}=\sigma\left(g\left(\theta;f\left(\varTheta;\boldsymbol{
x}_{i}^{s}\right)\right)\right)$.
In each training batch, we select $B$ samples from $\mathcal{D}_{u}$. In the remainder of this paper, all the operators are performed in a batch without special mention.

Overview. As shown in Fig. 2, the proposed model is a dual-head network containing a clustering head and a calibration head.
We first pre-train the backbone network by MoCo-v2 [] and use the proposed feature prototype-based initialization strategy to enhance the initial discriminative capacity of the two heads.
Then, we train the clustering head by selecting high-confidence pseudo-labels through a cross-entropy loss, where the confidence level of samples is provided by the calibration head. At the same time, we train the calibration head with a novel calibration approach to perceive and estimate the output confidence of the samples from the clustering head.
Finally, the clustering and calibration heads are optimized simultaneously to achieve mutual boosting.

### 3.1 Initializing the Clustering and Calibration Heads

After pre-training the backbone with MoCo-v2, we need to add a clustering head, usually constructed by a multilayer perceptron (MLP), to generate the clustering prediction.
As shown in Fig. 1-(f), the backbone pre-trained by MoCo-v2 can produce a quite discriminative representation, e.g., if we apply K-means on the pre-trained features of CIFAR-20, the clustering accuracy is about $50.7\%$, while the randomly initialized head clustering on the pre-trained backbone only produces a clustering accuracy of $10.4\%$. This means the discriminative representation of the pre-trained backbone is not directly passed to the clustering head.
As the clustering head need to generate pseudo-labels for self-supervision, the random initialization strategy adopted in the previous methods would produce low-quality pseudo-labels, leading to unsatisfied clustering performance.

Therefore, we propose a feature prototype-based initialization strategy to initialize the clustering head (also the calibration head in our method), trying to transform the discriminative representation ability of the pre-trained backbone to the clustering and calibration heads.
Specifically, as shown in the lower right corner of Fig. 2, we use a three-layer MLP as the clustering head. For the MLP with $D$ input units, $H$ hidden units, and $C$ output units,
we have the input variable $\boldsymbol{z}\in\mathbb{R}^{N\times D}$, the intermediate variable $\boldsymbol{h}\in\mathbb{R}^{N\times H}$, the output variable $\boldsymbol{o}\in\mathbb{R}^{N\times C}$, and two linear layer weights $\boldsymbol{W}^{\left(1\right)}\in\mathbb{R}^{H\times D}$ and $\boldsymbol{W}^{\left(2\right)}\in\mathbb{R}^{C\times H}$.
The first linear layer weight $\boldsymbol{W}^{\left(1\right)}$ connects the input variable $\boldsymbol{z}$ and the intermediate variable $\boldsymbol{h}$.
To initialize the first linear layer $\boldsymbol{W}^{\left(1\right)}$,
we perform K-means clustering on the input variable $\boldsymbol{z}$, and split it into $H$ mini-clusters. Then we use each clustering center (prototype) to initialize $\boldsymbol{W}^{\left(1\right)}$, i.e.,

$$ $\boldsymbol{W}^{\left(1\right)}=\texttt{Kmeans}_{H}\left(\boldsymbol{z}\right),$ (1) $$

where $\texttt{Kmeans}_{H}\left(\boldsymbol{z}\right)$ returns $H$ prototypes of $\boldsymbol{z}$ from the K-means clustering.
Note that we set the bias of the linear layer to 0.
By the above setting, one neuron of the output will denote a prototype,
and this neuron will yield large values (resp. small values) for samples belonging to (resp. not belonging) the mini-cluster related to that prototype, marking it have the discriminative ability.
Besides, the MLP also embeds a BN layer and a ReLU layer.
For the second layer weight $\boldsymbol{W}^{\left(2\right)}$, we apply the same strategy that performs K-means clustering on the intermediate variable $\boldsymbol{h}$ to initialize $\boldsymbol{W}^{\left(2\right)}$, i.e., $\boldsymbol{W}^{\left(2\right)}=\texttt{Kmeans}_{C}\left(\boldsymbol{h}\right)$.
After the initialization with feature prototypes, we further orthogonalize the weights to improve the discriminative ability of the network.

Figure: Figure 2: Illustration of the proposed CDC framework. The calibration head penalizes the overconfident predictions from the clustering head. The clustering head, in turn, uses the calibrated confidence provided by the calibration head to select high-confidence samples for training. Note that the calibration head has the same structure as the clustering head.
Refer to caption: x7.png

Property 1 Assuming $z^{*}$ as the prototype obtained by applying K-means to the features $z$, when the linear layer weights $W=z^{*}$, the cluster assignment of the output $Wz$ aligns well with $z^{*}$.

Theoretical Analysis. The proof of Property 1 is provided in A.1, demonstrating that the proposed strategy can effectively propagate well-defined discriminability from the feature space to the output space, thereby outperforming random initialization.

### 3.2 Calibration Head

The calibration head serves the purpose of aligning the output confidence of the model with its true accuracy. However, the well-known calibration methods are not directly applicable here: (I) Post-calibration methods like Temperature Scaling requires a labeled validation set to tune its hyper-parameter, which is not accessible in unsupervised clustering tasks. (II) Regularization-based methods like Label Smoothing (LS), Focal Loss (FL), and $L_{1}$ Norm (L1) penalize the prediction confidence of all the samples, which are harmful for failure detection, i.e., misclassification errors cannot be detected by filtering out low-confidence predictions. As indicated in Fig. 1-(e), although LS promotes the ECE performance, it will also reduce the high-confidence predictions.

Therefore, we aim to penalize the confidence of samples located in the unreliable region while not penalizing the confidence of samples in the reliable region. To this end, we use the K-means algorithm to divide the embeddings matrix (before the clustering head and calibration head) into $K$ mini-clusters and record the average prediction of each mini-cluster in the clustering head as $\hat{\boldsymbol{q}}_{k}=\frac{\sum_{\boldsymbol{x}_{i}\in Q_{k}}{\boldsymbol{
p}_{i}^{clu}}}{|Q_{k}|}$, where $Q_{k}$ denotes the samples belonging to the $k$-th mini-cluster, $|Q_{k}|$ is the total number of samples in this mini-cluster, and $\boldsymbol{p}_{i}^{clu}$ is the prediction of $\boldsymbol{x}_{i}$ by the clustering head. Then, we take the average prediction of each mini-cluster by the clustering head as the target distribution of the samples in that mini-cluster of the calibration head, i.e.,

$$ $\displaystyle\mathcal{L}_{cal}=-\frac{1}{B}\sum_{k}\sum_{\boldsymbol{x}_{i}\in Q _{k}}{\hat{\boldsymbol{q}}_{k}\log\left(\boldsymbol{p}_{i}^{cal}\right)},$ (2) $$

where $\boldsymbol{p}_{i}^{cal}$ denotes the output of $\boldsymbol{x}_{i}$ predicted by the calibration head. The above calibration method aligns the feature representation of the samples with their output in the clustering head.

Furthermore, we add a negative entropy loss to make the predicted class distribution more uniform, avoiding all samples being divided into the same cluster:

$$ $\displaystyle\mathcal{L}_{en}=\frac{1}{C}\sum_{j=1}^{C}{\boldsymbol{p}_{:,j}^{ cal}\mathrm{log}~{}\boldsymbol{p}_{:,j}^{cal}},$ (3) $$

where $\boldsymbol{p}_{:,j}^{cal}$ denotes the $j$-th column of the calibration head for a batch of samples.
Combining Eqs. (2) and (3), the overall loss of the calibration head is:

$$ $\displaystyle\mathcal{L}=\mathcal{L}_{cal}+w_{en}\mathcal{L}_{en},$ (4) $$

where $w_{en}\geq 0$ is a hyper-parameter.

Theoretical Analysis.
Assuming K-means divides features into $(a+b)$ regions, where $a$ regions
are Reliable Regions that do not cross clustering decision boundaries, and the set of samples in the reliable regions are denoted as $T$, and $b$ regions are Unreliable Regions that cross the clustering decision boundaries with samples in the unreliable regions denoted as $F$.

Theorem 1 (Region-aware Penalty) The average confidence of the predictions before and after calibration are denoted as $Conf^{clu}$ and $Conf^{cal}$. Then, confidence penalty occurs only in unreliable regions with $\underset{T}{\mathbb{E}}\left[Conf^{cal}\right]\leq\underset{T}{\mathbb{E}}
\left[Conf^{clu}\right]$ while not in reliable regions with $\underset{F}{\mathbb{E}}\left[Conf^{cal}\right]=\underset{F}{\mathbb{E}}\left[
Conf^{clu}\right]$.

Theorem 2 (Improve Calibration) The calibration errors given by the clustering head and the calibration head are denoted as $ECE^{clu}$ and $ECE^{cal}$. Under some mild conditions, we have $ECE^{cal}\leq ECE^{clu}$.

Theorem 1 indicates that the proposed calibration method only penalizes the confidence of the unreliable samples, which avoids the prediction confusion brought by the previous regularization-based calibration methods. Theorem 2 indicates the proposed calibration method can promote calibration theoretically. Please see the detailed proofs of those two theorems in A.3 and A.4.

Remark. Based on the aforementioned characteristics of the calibration head, we make two specific designs: 1) Stop gradient. As the calibration head also include samples in the unreliable regions, we avoid loss optimization for the overall network and restrict it to optimizing only the calibration head. 2) Dual-head decoupling. The calibrated predictions conflict with overconfident predictions in the same head, therefore, we employ a dual-head framework to decouple them.

### 3.3 Clustering Head

Due to the lack of ground-truth information, the most crucial issue in training the clustering head is generating highly reliable pseudo labels. Previous methods [, , ] usually adopt a fixed and predefined threshold to select samples as the pseudo labels. However, such a fashion overlooks the fact that the model’s output confidence continuously increases during the training process.
Specifically, in the early training strategy, the threshold is typically set too high, resulting in the selection of only a limited number of samples and slowing down the training speed. As training progresses, the confidence of all samples increases, leading to the selection of more samples as pseudo labels. Consequently, more incorrect pseudo labels are included, negatively impacting the clustering performance.
Furthermore, as previously analyzed, deep clustering networks face a significant overconfidence problem, which hinders the model’s ability to perceive its learning status, further complicating the determination of an appropriate threshold. Additionally, previous methods utilize a single global threshold for all classes, which biases the model towards selecting more samples from easy classes (those with higher confidence) while disregarding hard classes (those with lower confidence).

To address these challenges, we use the confidence of the calibration head estimates to determine the threshold dynamically for different classes. This approach allows a more accurate estimation of the model’s learning status and helps mitigate the discrepancies in data utilization across categories.
Specifically, we first calculate the average confidence of each cluster in the calibration head, i.e.,

$$ $\boldsymbol{p}_{avg_c}=\sum_{i\in S_{c}^{cal}}\frac{\boldsymbol{p}_{i}^{w_ cal}}{|S_{c}^{cal}|},\forall c=1,2,\cdots C,$ (5) $$

where $\boldsymbol{p}_{i}^{w_cal}$ denotes the confidence of the weak augmentation of $\boldsymbol{x}_{i}$ estimated by the calibration head, $S_{c}^{cal}$ is the set of samples being categorized to the $c$-th cluster by the calibration head, $|S_{c}^{cal}|$ indicates the total number of samples belong to the $c$-th cluster, and $\boldsymbol{p}_{avg_c}$ denotes the average confidence of the samples belonging to the $c$-th cluster estimated by the calibration head.

As the calibration head is well-calibrated, the average confidence of each cluster estimated by the calibration head can well capture the learning status of each cluster, i.e., if $\boldsymbol{p}_{avg_c}$ is large (resp. small), the model is more (resp. less) confident about the prediction for the samples in the $c$-th cluster; accordingly, we should select more (resp. less) samples from this cluster as the pseudo-labels. Therefore, we determine the number of samples selected as the pseudo-labels for the $c$-th cluster via

$$ $M_{c}=\boldsymbol{p}_{avg_c}\times\left\lfloor B/C\right\rfloor,$ (6) $$

where $\lfloor\cdot\rfloor$ denotes the lower approximation operator. Finally, we select the top $M_{c}$ samples from each cluster to constitute the final pseudo-labeled samples: $S=\{(\boldsymbol{x}_{i},y_{i})\}$,
where $y_{i}=\underset{c}{\mathrm{argmax}}~{}\boldsymbol{p}_{i}^{w_cal}$ returns the cluster index producing the largest prediction as the pseudo label for $\boldsymbol{x}_{i}$.

After selecting the pseudo labels, we use the cross-entropy loss to optimize the parameters of the feature extractor and the clustering head:

$$ $\displaystyle\mathcal{L}_{clu}=-\frac{1}{|S|}\sum_{\boldsymbol{x}_{i}\in S}{y_ {i}\mathrm{log}~{}\boldsymbol{p}_{i}^{s_clu}},$ (7) $$

where $|S|$ denotes the total number of selected pseudo-labeled samples, $\boldsymbol{p}_{i}^{s_clu}$ denotes the confidence of the strong augmentation of $\boldsymbol{x}_{i}$ estimated by the clustering head.

### 3.4 Joint Training and Final Prediction

Joint Optimization. After the pre-training by MoCo-V2, and the initialization of the clustering head and the calibration head,
we optimize the clustering head and feature extractor by minimizing Eq. (7) and optimize the calibration head by minimizing Eq. (4). The overall training process of our model is summarized in Algorithm 1 (shown in B.2).

Final Prediction. The clustering head achieves clustering result similar to that of the calibration head but with a worse ECE. Therefore, we use the calibration head to make the final prediction.

## 4 Experiments

Datasets and Backbones. We conducted experiments on six widely used benchmark datasets, CIFAR-10 [], CIFAR-20 [], STL-10 [], ImageNet-10 [], ImageNet-Dogs [], and Tiny-ImageNet [].
Similar to [, ], we used ResNet-34 and MLP (512d-BN []-ReLU []-$C$d) for all experiments.

Implementation Details. The comparison methods can be divided into two types: methods that first learn an embedding by a network and then achieve clustering by using K-means []: MoCo-v2 [], SimSiam [], BYOL [], TCL [], TCC [], DMICC [], ProPos [] and CoNR [], and the methods that can achieve clustering and feature learning simultaneously: DivClust [], CC [], SeCu [], SCAN [], and SPICE []. More details are shown in B.1.
For CDC, we trained the model for 100 epochs using the Adam optimizer with a learning rate of 5e-5 for the encoder, 1e-4 for the MLP. The learning rate of the encoder on CIFAR-20 and Tiny-ImageNet was adjusted to 1e-5 for better learning of noisy pseudo labels. We set $B=1,000$ for CIFAR-10, CIFAR-20, and STL-10, $B=500$ for ImageNet-10 and ImageNet-Dogs, $B=5,000$ for Tiny-ImageNet. We set $K=500$ for CIFAR-10, $K=40$ for CIFAR-20 and ImageNet-Dogs, $K=150$ for STL-10 and ImageNet-10, and $K=1,000$ for Tiny-ImageNet. Besides, we set $w_{en}=1.0$ for all experiments. For simplicity, the prediction of the clustering head is denoted as CDC-Clu, and the prediction of the calibration head is denoted as CDC-Cal.

Evaluation Metrics.
We used three common clustering metrics: clustering Accuracy (ACC) [], Normalized Mutual Information (NMI) [], and Adjusted Rand Index (ARI) [] to evaluate the clustering performance, and used the Expected Calibration Error (ECE) [] to evaluate the calibration performance. Moreover, we applied the failure rejection ability metrics AUROC, AURC, and FPR95 recommended in [, ] to evaluate the separation quality of low-confidence and high-confidence predictions.

**Table 1: The clustering performance (ACC, ARI) and calibration error ECE on six image benchmarks. The best results are shown in bold, while the second best results are underlined. $\uparrow$ ($\downarrow$) means the higher (resp. lower), the better.**
| Method | CIFAR-10 | CIFAR-20 | STL-10 | ImageNet-10 | ImageNet-Dogs | Tiny-ImageNet |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ACC$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ |
| K-means [] | 22.9 | 4.9 | N/A | 13.0 | 2.8 | N/A | 19.2 | 6.1 | N/A | 24.1 | 5.7 | N/A | 10.5 | 2.0 | N/A | 2.5 | 0.5 | N/A |
| MoCo-v2 [] | 82.9 | 64.9 | N/A | 50.7 | 26.2 | N/A | 68.8 | 45.5 | N/A | 56.7 | 30.9 | N/A | 62.8 | 48.1 | N/A | 25.2 | 11.0 | N/A |
| Simsiam [] | 70.7 | 53.1 | N/A | 33.0 | 16.2 | N/A | 49.4 | 34.9 | N/A | 78.4 | 68.8 | N/A | 44.2 | 27.3 | N/A | 19.0 | 8.4 | N/A |
| BYOL [] | 57.0 | 47.6 | N/A | 34.7 | 21.2 | N/A | 56.3 | 38.6 | N/A | 71.5 | 54.1 | N/A | 58.2 | 44.2 | N/A | 11.2 | 4.6 | N/A |
| TCL [] | 88.7 | 78.0 | N/A | 53.1 | 35.7 | N/A | 86.8 | 75.7 | N/A | 89.5 | 83.7 | N/A | 64.4 | 51.6 | N/A | - | - | - |
| TCC [] | 90.6 | 73.3 | N/A | 49.1 | 31.2 | N/A | 81.4 | 68.9 | N/A | 89.7 | 82.5 | N/A | 59.5 | 41.7 | N/A | - | - | - |
| DMICC [] | 82.8 | 69.0 | N/A | 46.8 | 29.1 | N/A | 80.0 | 62.5 | N/A | 96.2 | 91.6 | N/A | 58.7 | 43.8 | N/A | - | - | - |
| ProPos [] | 94.3 | 88.4 | N/A | 61.4 | 45.1 | N/A | 86.7 | 73.7 | N/A | 96.2 | 91.8 | N/A | 77.5 | 67.5 | N/A | 29.4 | 17.9 | N/A |
| CoNR [] | 93.2 | 86.1 | N/A | 60.4 | 44.3 | N/A | 92.6 | 84.6 | N/A | 96.4 | 92.2 | N/A | 79.4 | 66.7 | N/A | 30.8 | 18.4 | N/A |
| DivClust [] | 81.9 | 68.1 | - | 43.7 | 28.3 | - | - | - | - | 93.6 | 87.8 | - | 52.9 | 37.6 | - | - | - | - |
| CC [] | 85.2 | 72.8 | 6.2 | 42.4 | 28.4 | 29.7 | 80.0 | 67.7 | 11.9 | 90.6 | 85.3 | 8.1 | 69.6 | 56.0 | 19.3 | 12.1 | 5.7 | 3.2 |
| SeCu-Size [] | 90.0 | 81.5 | 8.1 | 52.9 | 38.4 | 13.1 | 80.2 | 63.1 | 9.9 | - | - | - | - | - | - | - | - | - |
| SeCu [] | 92.6 | 85.4 | 4.9 | 52.7 | 39.7 | 41.8 | 83.6 | 69.3 | 6.5 | - | - | - | - | - | - | - | - | - |
| SCAN-2 [] | 84.1 | 74.1 | 10.9 | 50.0 | 34.7 | 37.1 | 87.0 | 75.6 | 7.4 | 95.1 | 89.4 | 2.7 | 63.3 | 49.6 | 26.4 | 27.6 | 15.3 | 27.4 |
| SCAN-3 [] | 90.3 | 80.8 | 6.7 | 51.2 | 35.6 | 39.0 | 91.4 | 82.5 | 6.6 | 97.0 | 93.6 | 1.5 | 72.2 | 58.7 | 19.5 | 25.8 | 13.4 | 48.8 |
| SPICE-2 [] | 84.4 | 70.9 | 15.4 | 47.6 | 30.3 | 52.3 | 89.6 | 79.2 | 10.1 | 92.1 | 83.6 | 7.8 | 64.6 | 47.7 | 35.3 | 30.5 | 16.3 | 48.5 |
| SPICE-3 [] | 91.5 | 83.4 | 7.8 | 58.4 | 42.2 | 40.6 | 93.0 | 85.5 | 6.3 | 95.9 | 91.2 | 4.1 | 67.5 | 52.6 | 32.5 | 29.1 | 14.7 | N/A |
| CDC-Clu (Ours) | 94.9 | 89.4 | 1.4 | 61.9 | 46.7 | 28.0 | 93.1 | 85.8 | 4.8 | 97.2 | 94.0 | 1.8 | 79.3 | 70.3 | 17.1 | 34.0 | 20.0 | 37.8 |
| CDC-Cal (Ours) | 94.9 | 89.5 | 1.1 | 61.7 | 46.6 | 4.9 | 93.0 | 85.6 | 0.9 | 97.3 | 94.1 | 0.8 | 79.2 | 70.0 | 7.7 | 33.9 | 19.9 | 11.0 |
| Supervised | 89.7 | 78.9 | 4.0 | 71.7 | 50.2 | 11.0 | 80.4 | 62.2 | 10.0 | 99.2 | 98.3 | 0.9 | 93.1 | 85.7 | 0.9 | 47.7 | 24.3 | 5.1 |
| +MoCo-v2 | 94.1 | 87.5 | 2.4 | 83.2 | 68.4 | 6.7 | 90.5 | 80.7 | 3.5 | 99.9 | 99.8 | 0.4 | 99.5 | 99.0 | 0.9 | 53.8 | 30.9 | 8.4 |

### 4.1 Main Results

Superior Clustering Ability.
As shown in Tab. 1, applying K-means to the pre-trained features significantly outperforms directly applying K-means to the raw features, demonstrating the importance of deep representation learning in clustering. Along this line, ProPos and CoNR have continuously improved the clustering performance but cannot provide confidence prediction. For self-labeling-based methods, CDC-Cal outperforms DivClust, CC, SeCu, SCAN, and SPICE on almost all the datasets, validating the importance of dynamic threshold setting. On CIFAR-10 and STL-10, CDC-Cal with the same pre-trained model (MoCo-v2) increases accuracy by 0.8% and 2.5% than supervised learning. On large-scale datasets, CDC-Cal has achieved improvements of 1.4%, 11.7%, and 4.8% on ImageNet-10, ImageNet-Dogs, and Tiny-ImageNet. Meanwhile, the results of the clustering head CDC-Clu are close to those of the calibration head CDC-Cal. In summary, our method ranks first in 11 out of 12 cases in the clustering metrics, suggesting its superior clustering ability.

Excellent Calibration Performance.
The calibration error of CDC-Cal is far superior to that of CDC-Clu and all the compared methods.
CC does not apply the pseudo-labeling strategy, so its ECE is relatively low. This explains that overfitting to the wrong pseudo-labels is a reason for the high calibration error.
SPICE-2 uses dual-softmax losses to encourage sharper predictions, which leads to more severe overconfidence than SCAN, while our calibration head significantly reduces the calibration error on CIFAR-10 (15.4%$\xrightarrow{}$1.1%), CIFAR-20 (52.3%$\xrightarrow{}$40.6%), and STL-10 (7.8%$\xrightarrow{}$0.8%) compared to SPICE-2.
Fig. 1 provides a visual comparison of the ECE on CIFAR-20, where our method achieves well-aligned predictions and better clustering performance.

Competitive Failure Rejection Ability.
Fig. 3 compares different regularization-based calibration methods. CDC-Cal improves AUROC by 8.6%, decreases ARPC by 10.6%, and decreases FPR95 by 6.5% compared to the second-best method, demonstrating its effective error rejection capability. The second line further demonstrates its ability to separate correct and misclassified samples, which is beneficial for selecting reliable pseudo-labels for feedback into the network.
Although Focal Loss (FL) may produce a lower ECE than our method, its clustering ACC is much worse. Besides, our method outperforms FL on all the failure rejection metrics (AUROC, ARPC, and FPR95), suggesting our method can better separate the correct prediction and wrong prediction.

Figure: (a) ACC (%) $\uparrow$
Refer to caption: x8.png

Figure: (a) CIFAR-10
Refer to caption: x18.png

### 4.2 Ablation Study

I. Initialization. The training process in Fig. 4 provides a intuitive display of the advantages of the initialization strategy in terms of training speed and stability. Our method can transform the discriminative ability of the pre-trained features to the clustering head directly. Tab. 2-I quantitatively demonstrates that prototype-based initialization leads to a notable enhancement in clustering performance. Specifically, when we exclude the initialization stage from our method (w/o Init.+CDC), the clustering performance experiences a significant decline, showing the efficacy of the initialization strategy.

II. Confidence-Aware Selection.
Tab. 2-II shows that if we fix the threshold at certain values, the clustering performance will drop significantly, suggesting the effectiveness of the proposed confidence-aware thresholding strategy.

III. Single-head Setting. Tab. 2-III shows in a single-head setting, the network employs its own overconfident predictions for confidence-aware sample selection, which may lead to the selection of numerous incorrect pseudo-labels, resulting in performance degradation including both the clustering performance and the calibration performance.

IV. Dual-head Decoupling. Tab. 2-IV shows that combining the conflicting losses of encouraging overconfidence and penalizing confidence in a single head (Clu+Cal) will decrease both the calibration performance and the clustering performance especially on CIFAR-20.

V. Stop Gradient for the Calibrating Head.
If we do not stop gradient for the calibration head, the clustering performance will degrade significantly especially on CIFAR-20 and STL-10. This is because the calibration loss relies on unreliable regions to provide penalties, which will include unreliable information in the feature extractor if we do not stop gradient.

**Table 2: The ablation results of the proposed model.**
| Type | Settings | CIFAR-10 | CIFAR-20 | STL-10 |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ACC$\uparrow$ | NMI$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | NMI$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | NMI$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ |  |
| I | After Randomly Init. | 19.1 | 7.6 | 3.1 | 8.5 | 10.4 | 5.7 | 1.0 | 4.9 | 19.2 | 6.0 | 2.4 | 8.5 |
| After Proposed Init. | 87.2 | 79.8 | 76.1 | 1.0 | 56.4 | 56.9 | 41.2 | 5.2 | 89.8 | 80.9 | 79.3 | 2.7 |  |
| w/o Init.+CDC | 89.4 | 86.5 | 83.5 | 3.3 | 44.4 | 52.3 | 31.0 | 11.9 | 73.3 | 70.0 | 60.6 | 17.9 |  |
| II | Fixed Thre. (0.99) | 80.6 | 69.8 | 65.8 | 6.5 | 54.9 | 55.9 | 37.1 | 15.1 | 89.6 | 81.0 | 79.3 | 1.0 |
| Fixed Thre. (0.95) | 91.9 | 85.2 | 83.9 | 3.5 | 50.8 | 49.2 | 30.5 | 12.6 | 91.8 | 84.0 | 83.3 | 1.1 |  |
| Fixed Thre. (0.90) | 92.7 | 86.5 | 85.3 | 3.2 | 43.3 | 43.2 | 27.3 | 4.1 | 93.0 | 86.1 | 85.7 | 1.0 |  |
| Fixed Thre. (0.80) | 93.6 | 87.5 | 86.9 | 1.7 | 49.9 | 50.5 | 33.6 | 3.9 | 93.0 | 86.1 | 85.7 | 2.0 |  |
|  | CDC-Cal (Ours) | 94.9 | 89.3 | 89.5 | 1.1 | 61.7 | 60.9 | 46.6 | 4.9 | 93.0 | 85.8 | 85.6 | 0.9 |
| III | Single-head (Clu) | 93.9 | 88.0 | 87.5 | 2.3 | 59.7 | 61.3 | 45.3 | 31.6 | 92.6 | 85.3 | 84.9 | 5.1 |
| IV | Single-head (Clu+Cal) | 94.8 | 89.0 | 89.1 | 1.8 | 57.8 | 58.7 | 43.1 | 21.2 | 93.0 | 85.7 | 85.5 | 3.0 |
| V | Cal (w/o Stop Gradient) | 93.0 | 86.0 | 85.7 | 2.0 | 49.6 | 52.1 | 34.2 | 12.4 | 86.7 | 76.3 | 63.9 | 2.5 |

## 5 Conclusion

Summary. In this paper, we have introduced a novel calibrated deep clustering model that outperforms SOTA deep clustering models in terms of clustering performance. Notably, our model is the first deep clustering method capable of calibrating the output confidence. In particular, the expected calibration error of our is 10 times better than certain compared methods. To achieve these results, we proposed a dual-head clustering network consisting of a clustering head and a calibration head. The clustering head utilizes the calibrated confidence estimated by the calibration head to select highly reliable pseudo-labels. The calibration head, on the other hand, calibrates the output confidence of all samples using a novel regularization loss. Additionally, we introduced an effective initialization strategy for both the clustering head and the calibration head, significantly promoting the training stability.
Through our calibrated deep clustering neural network, the output confidence of the model is well aligned with its true accuracy, which means we can trustfully judge when we can rely on the prediction of the model. Moreover, as we do not need any label to train the network, the above benefit can be gained quite easily.

Limitations. The proposed method assumes the number of clusters is known in advance. We will try to infer the appropriate cluster number in the future.

## Appendix A Appendix / supplemental material

### A.1 Proof of Property 1

Property 1 Assuming $z^{*}$ as the prototype obtained by applying K-means to the features $z$, when the linear layer weights $W=z^{*}$, the cluster assignment of the output $Wz$ aligns well with $z^{*}$.

proof
When conducting K-means on feature $z$ to derive prototypes $z^{*}$, we formulate an optimization function as follows:

$$ $\displaystyle min_{z^{*}}||z-z^{*}||^{2}$ $\displaystyle=min_{z^{*}}-2z^{*}z+||z||^{2}+||z^{*}||^{2}$ (8) $\displaystyle=min_{z^{*}}-z^{*}z=max_{z^{*}}z^{*}z\xlongequal{W=z^{*}}max_{W}Wz$ $$

The above process demonstrates that the optimal clustering assignment in the feature space similarly applies in the output space, indicating consistency between the feature space and the output space.

### A.2 Intuitive explanation

Recalling the setup in Sec. 3.2, we use K-means to divide the features into $(a+b)$ regions, where $a$ regions are Reliable Regions that do not cross clustering decision boundaries and $b$ regions are Unreliable Regions that cross the clustering decision boundaries.

In the reliable region (the left half of Fig. 5), the mean predicted value for four samples within this region is $[0.97,0.03]$, which is consistent with the average maximum confidence of $0.97$ for each sample. In the unreliable region (the right half of Fig. 5), the mean value voted by four samples within this region is $[0.45,0.55]$, while the average maximum confidence of these samples is $0.70$. This inconsistency is the source of confidence penalty.

Figure: Figure 5: Calibration Example
Refer to caption: x20.png

Therefore, by identifying regions with similar features in the feature space, maintaining predictions in reliable regions, and penalizing predictions in unreliable regions, confidence calibration capable of detecting failure rejections can be achieved.

### A.3 Proof of Theorem 1

Theorem 1 (Region-aware Penalty) The average confidence of the predictions before and after calibration are denoted as $Conf^{clu}$ and $Conf^{cal}$. Then, confidence penalty occurs only in unreliable regions with $\underset{T}{\mathbb{E}}\left[Conf^{cal}\right]\leq\underset{T}{\mathbb{E}}
\left[Conf^{clu}\right]$ while not in reliable regions with $\underset{F}{\mathbb{E}}\left[Conf^{cal}\right]=\underset{F}{\mathbb{E}}\left[
Conf^{clu}\right]$.

proof

Considering a binary classification problem like [], let’s assume that the features of the data $\left(\boldsymbol{z},y\right)\in\mathbb{R}^{m}\times\left\{-1,1\right\}$ are generated from a mixture of two Gaussian distributions, where each class corresponds to one Gaussian distribution. Thus, the following conditional distribution holds: $\boldsymbol{z}|y~{}N\left(y\boldsymbol{\theta}^{*},\sigma^{2}\boldsymbol{I}\right)$. Assuming the classifier is $C\left(\boldsymbol{z}\right)=sgn\left(\hat{\boldsymbol{\theta}}^{T}\boldsymbol
{z}\right)$, where $\hat{\boldsymbol{\theta}}=\sum_{i=1}^{n}{\frac{\boldsymbol{z}_{i}y_{i}}{n}}$. The confidence vector is denoted as $\left[p_{+1}\left(\boldsymbol{z}\right),p_{-1}\left(\boldsymbol{z}\right)
\right]^{T}$, with $p_{c}\left(\boldsymbol{z}\right)=\frac{1}{\exp\left(-2c\hat{\boldsymbol{\theta
}}^{T}\boldsymbol{z}_{i}/\sigma^{2}\right)+1}$, where $c\in\left\{-1,+1\right\}$.

In reliable region, we have

$\underset{T}{\mathbb{E}}\left[Conf^{cal}\right]=\underset{\boldsymbol{z}\in T}
{\mathbb{E}}\left[p_{c}^{cal}\left(\boldsymbol{z}\right)\right]=\underset{
\boldsymbol{z}\in T}{\mathbb{E}}\left[\bar{p}_{c}^{clu}\left(\boldsymbol{z}
\right)\right]=\underset{\boldsymbol{z}\in T}{\mathbb{E}}\left[p_{c}^{clu}
\left(\boldsymbol{z}\right)\right]=\underset{T}{\mathbb{E}}\left[Conf^{clu}\right]$.

In unreliable region, we have

$\begin{aligned} \underset{F}{\mathbb{E}}\left[Conf^{cal}\right]&=\mathbb{I}
\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{+1}^{cal}\left(
\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\underset{\boldsymbol{z}\in F
}{\mathbb{E}}\left(p_{+1}^{cal}\left(\boldsymbol{z}\right)\right)+\mathbb{I}
\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{-1}^{cal}\left(
\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\underset{\boldsymbol{z}\in F
}{\mathbb{E}}\left(p_{-1}^{cal}\left(\boldsymbol{z}\right)\right)\\
&=\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{+1}^{cal}
\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\left(\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0}{\mathbb{E}}
\left(p_{+1}^{cal}\left(\boldsymbol{z}\right)\right)+\underset{\boldsymbol{z}
\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0}{\mathbb{E}}\left(p_{+
1}^{cal}\left(\boldsymbol{z}\right)\right)\right)\\
&\quad+\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{-1}^
{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\left(\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0}{\mathbb{E}}
\left(p_{-1}^{cal}\left(\boldsymbol{z}\right)\right)+\underset{\boldsymbol{z}
\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0}{\mathbb{E}}\left(p_{-
1}^{cal}\left(\boldsymbol{z}\right)\right)\right)\\
&=\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{+1}^{cal}
\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\left(\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0}{\mathbb{E}}
\left[p_{-1}^{cal}\left(\boldsymbol{z}\right)-\left(p_{-1}^{cal}\left(
\boldsymbol{z}\right)-p_{+1}^{cal}\left(\boldsymbol{z}\right)\right)\right]
\right)\\
&\quad+\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{+1}^
{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0}{\mathbb
{E}}\left(p_{+1}^{cal}\left(\boldsymbol{z}\right)\right)\\
&\quad+\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{-1}^
{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0}{\mathbb{E}}
\left(p_{-1}^{cal}\left(\boldsymbol{z}\right)\right)\\
&\quad+\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{-1}^
{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\left(\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0}{\mathbb
{E}}\left[p_{+1}^{cal}\left(\boldsymbol{z}\right)-\left(p_{+1}^{cal}\left(
\boldsymbol{z}\right)-p_{-1}^{cal}\left(\boldsymbol{z}\right)\right)\right]
\right)\\
&=\left[\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{+1}
^{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)+\mathbb{I}\left
(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{-1}^{cal}\left(\boldsymbol
{z}\right)\right]\geq\frac{1}{2}\right)\right]\underset{\boldsymbol{z}\in F,
\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0}{\mathbb{E}}\left(p_{-1}^{cal}
\left(\boldsymbol{z}\right)\right)\\
&\quad+\left[\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p
_{+1}^{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)+\mathbb{I}
\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{-1}^{cal}\left(
\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\right]\underset{\boldsymbol
{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0}{\mathbb{E}}\left(p
_{+1}^{cal}\left(\boldsymbol{z}\right)\right)\\
&\quad-\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{+1}^
{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0}{\mathbb{E}}
\left[p_{-1}^{cal}\left(\boldsymbol{z}\right)-p_{+1}^{cal}\left(\boldsymbol{z}
\right)\right]\,\,\left(\textbf{I}\right)\\
&\quad-\mathbb{I}\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{-1}^
{cal}\left(\boldsymbol{z}\right)\right]\geq\frac{1}{2}\right)\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0}{\mathbb
{E}}\left[p_{+1}^{cal}\left(\boldsymbol{z}\right)-p_{-1}^{cal}\left(
\boldsymbol{z}\right)\right]\,\,\left(\textbf{II}\right)\\
&=\underset{\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0}
{\mathbb{E}}\left(p_{-1}^{cal}\left(\boldsymbol{z}\right)\right)+\underset{
\boldsymbol{z}\in F,\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0}{\mathbb
{E}}\left(p_{+1}^{cal}\left(\boldsymbol{z}\right)\right)-\left(\textbf{I}
\right)-\left(\textbf{II}\right)\\
&=\underset{F}{\mathbb{E}}\left[Conf^{clu}\right]-\left(\textbf{I}\right)-
\left(\textbf{II}\right)\\
\end{aligned}$

As $\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}<0$ , we have $p_{-1}^{cal}\left(\boldsymbol{z}\right)-p_{+1}^{cal}\left(\boldsymbol{z}\right
)=p_{-1}^{clu}\left(\boldsymbol{z}\right)-p_{+1}^{clu}\left(\boldsymbol{z}
\right)\geq 0$.

$\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}\geq 0$ , we have $p_{+1}^{cal}\left(\boldsymbol{z}\right)-p_{-1}^{cal}\left(\boldsymbol{z}\right
)=p_{+1}^{clu}\left(\boldsymbol{z}\right)-p_{-1}^{clu}\left(\boldsymbol{z}
\right)\geq 0$.

Hence, $\underset{F}{\mathbb{E}}\left[Conf^{clu}\right]-\underset{F}{\mathbb{E}}\left[
Conf^{cal}\right]=\left(\textbf{I}\right)+\left(\textbf{II}\right)\geq 0$ and calibration methods can penalty overconfidence.

### A.4 Proof of Theorem 2

Theorem 2 (Improve Calibration) The calibration errors given by the clustering head and the calibration head are denoted as $ECE^{clu}$ and $ECE^{cal}$. Under some mild conditions, we have $ECE^{cal}\leq ECE^{clu}$.

proof

$ECE=\underset{\boldsymbol{v}=\hat{\boldsymbol{\theta}}^{T}\boldsymbol{z}}{
\mathbb{E}}|ACC\left(\boldsymbol{v}\right)-p_{c}\left(\boldsymbol{z}\right)|$

In reliable region, we have

$ECE_{T}^{clu}-ECE_{T}^{cal}=\underset{\boldsymbol{v}_{a}=\hat{\boldsymbol{
\theta}}^{T}\boldsymbol{z},\boldsymbol{z}\in T}{\mathbb{E}}\left[|ACC^{clu}
\left(\boldsymbol{v}_{a}\right)-p_{c}^{clu}\left(\boldsymbol{z}\right)|-|ACC^{
cal}\left(\boldsymbol{v}_{a}\right)-p_{c}^{cal}\left(\boldsymbol{z}\right)|\right]$.

As the two heads have the same clustering assignments, we have

$\underset{T}{\mathbb{E}}\left[ACC^{clu}\right]=\underset{T}{\mathbb{E}}\left[
ACC^{cal}\right]$, $\underset{T}{\,\,\mathbb{E}}\left[Conf^{clu}\right]=\underset{T}{\mathbb{E}}
\left[Conf^{cal}\right]$,

then

$ECE_{T}^{clu}-ECE_{T}^{cal}=0$.

In unreliable region, we have

$ECE_{F}^{clu}-ECE_{F}^{cal}=\underset{\boldsymbol{v}_{b}=\hat{\boldsymbol{
\theta}}^{T}\boldsymbol{z},\boldsymbol{z}\in F}{\mathbb{E}}\left[|ACC^{clu}
\left(\boldsymbol{v}_{b}\right)-p_{c}^{clu}\left(\boldsymbol{z}\right)|-|ACC^{
cal}\left(\boldsymbol{v}_{b}\right)-p_{c}^{cal}\left(\boldsymbol{z}\right)|\right]$.

As the calibration head uses the predictions of the clustering head, resulting in similar clustering assignment outcomes for both, thus we have

$ACC^{clu}\left(\boldsymbol{v}_{b}\right)\approx ACC^{cal}\left(\boldsymbol{v}_
{b}\right)=ACC$.

$\left(i\right)$ If $p_{c}^{clu}\left(\boldsymbol{z}\right)>ACC$, $p_{c}^{cal}\left(\boldsymbol{z}\right)>ACC$, the model faces overconfidence, we have

$\begin{aligned} \@add@centering\centering ECE_{F}^{clu}-ECE_{F}^{cal}&=
\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{c}^{clu}\left(\boldsymbol{z
}\right)-p_{c}^{cal}\left(\boldsymbol{z}\right)\right]\\
&=\left(\mathbf{I}\right)+\left(\mathbf{II}\right)\geq 0\end{aligned}$

$\left(ii\right)$ If $p_{c}^{clu}\left(\boldsymbol{z}\right)>ACC$, $p_{c}^{cal}\left(\boldsymbol{z}\right)<ACC$, and
$\left(\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{c}^{clu}\left(
\boldsymbol{z}\right)\right]+\underset{\boldsymbol{z}\in F}{\mathbb{E}}\left[p
_{c}^{cal}\left(\boldsymbol{z}\right)\right]\right)\geq\underset{F}{\mathbb{E}
}ACC$
holds, we have

$\begin{aligned} ECE_{F}^{clu}-ECE_{F}^{cal}=\underset{\boldsymbol{z}\in F}{
\mathbb{E}}\left[p_{c}^{clu}\left(\boldsymbol{z}\right)\right]+\underset{
\boldsymbol{z}\in F}{\mathbb{E}}\left[p_{c}^{cal}\left(\boldsymbol{z}\right)
\right]-2\underset{F}{\mathbb{E}}ACC\geq 0\end{aligned}$.

$\left(iii\right)$ If $p_{c}^{clu}\left(\boldsymbol{z}\right)<ACC$, $p_{c}^{cal}\left(\boldsymbol{z}\right)<ACC$, the model faces underconfidence. We do not consider this situation as deep networks are more prone to overconfidence.

$\left(iv\right)$ If $p_{c}^{clu}\left(\boldsymbol{z}\right)<ACC$, $p_{c}^{cal}\left(\boldsymbol{z}\right)>ACC$, due to $p_{c}^{clu}\left(\boldsymbol{z}\right)\geq p_{c}^{cal}\left(\boldsymbol{z}\right)$ always holds in Theorem 1, there is no solution.

In summary, we can get

$ECE^{cal}\leq ECE^{clu}$.

## Appendix B Experiment Details and Results

### B.1 Implementation Details

**Table 3: A summary of the datasets**
| Dataset | Split | #Samples | #Classes | Image Size |
| --- | --- | --- | --- | --- |
| CIFAR-10 [] | Train+Test | 60,000 | 10 | 32×32 |
| CIFAR-20 [] | Train+Test | 60,000 | 20 | 32×32 |
| STL-10 [] | Train+Test | 13,000 | 10 | 96×96 |
| ImageNet-10 [] | Train | 13,000 | 10 | 224×224 |
| ImageNet-Dogs [] | Train | 19,500 | 15 | 224×224 |
| Tiny-ImageNet [] | Train | 100,000 | 200 | 64×64 |

Datasets. As shown in Tab. 3, we conducted experiments on six widely used benchmark datasets, CIFAR-10 [], CIFAR-20 [], STL-10 [], ImageNet-10 [], ImageNet-Dogs [], and Tiny-ImageNet []. Similar to [], we used 20 superclasses of the CIFAR-100 dataset to construct the CIFAR-20, and 10, 15, and 200 subclasses of the ImageNet-1k [] dataset to extract the ImageNet-10, ImageNet-Dogs, and Tiny-ImageNet. Meanwhile, we extended STL-10 by 100,000 relevant unlabeled data during the pretext training and removed them afterwards. We used the dataset from ImageNet-10, ImageNet-Dogs, and Tiny-ImageNet datasets for both training and testing, while the rest datasets were trained and tested on the merged datasets.

Backbones. To facilitate training on small datasets such as CIFAR-10 and CIFAR-20, we followed [, ] and modified the first convolution layer’s kernel of the ResNet-34 with with a kernel size of 3 × 3, padding of 2 and stride of 1, and removed the first max-pooling layer. Moreover, All experiments are conducted on an NVIDIA RTX 3090 GPU.

Representation Learning.
For MoCo-v2, SimSiam and BYOL, according to [, , , ], we apply the same augmentation strategy for all datasets including a random ResizedCrop with a image size reported in Tab. 3 and a random HorizontalFlip, followed by a random ColorJitter and a random Grayscale. Moreover, we use GaussianBlur except for CIFAR-10 and CIFAR-20 due to the size of image. We adopt the ResNet-34 as backbone and a 2-layer MLP head (hidden layer 4096-d, with BN and ReLU, output layer 256-d) as projector. Besides, BYOL use another similar 2-layer MLP head as predictor. For opitimizer, MoCo-v2 and SimSiam adopt SGD optimizer, while BYOL adopt LARS [] optimizer. we train all these methods over 1,000 epochs with base learning rate 0.5 and batch size 256 on the datasets shown in Tab. 3. We update the learning rate with a cosine decay learning rate schedule [] and start with a warmup [] for 50 epochs. For the momentum hyperparameter $\tau$, we set 0.99 and 0.996 for MoCo-v2 and BYOL. For other hyperparameters, we set the temperature and weight decay as 0.2 and 0.0001 for all methods, and queue size 32768 for MoCo-v2.
For ProPos, due to the difference of backbone, we re-implemented the results on Tiny-ImageNet dataset using ResNet-34. For other datasets, since the backbone, data augmentation and dataset partition scheme are the same as ours, we cite the results of []. For TCL [], TCC [], DMICC [], and CoNR [], we directly cited the results of the paper.

Comparison Method.
Under the unified data augmentation and dataset partition scheme, we use ResNet-34 and MoCo-v2 to re-implement CC, SCAN and SeCu. For CC, we use a smaller initial learning rate of 5e-5 to fine-tune the model, and the rest of the parameters are consistent with []. For SCAN and SeCu, we use the same hyperparameters as the [, ]. For SPICE, except for SPICE-3 use WideResNet-28-2 for Cifar10, WRN-28-8 for CIFAR-20, and WRN-37-2 for STL-10, the rest of the experiments use ResNet-34, and these results are from the official model library(^1^11https://github.com/niuchuangnn/SPICE). Since SPICE-3 is not release the model on Tiny-ImageNet dataset, we cite the results and are not test the model’s ECE. For DivClust [], we directly cited the results of the paper.

Supervised Baseline.
We train on the training datasets of CIFAR-10, CIFAR-20, STL10 and Tiny-ImageNet, and evaluate the results on the test or validation datasets. Due to a lack of the designated test datasets on ImageNet-10 and ImageNet-Dogs, we report the results on the training datasets. We use Adam optimizer with weight decay of 0.0001 for all the experiments, and the batch size is set to 256. We adopt cross entropy on strong augmented samples to train a baseline model for 500 epochs with an initial learning rate of 0.001, and fine-tune the MoCo-v2 model 50 epochs with an initial learning rate of 0.0001 to obtain a better baseline.

Sample Selection Strategy. While the clustering head selects only high-confident samples for training, the calibration network utilizes all samples during the training process. This is done to obtain a more accurate estimation of the overall confidence and enhance the model’s robustness.

Augmentation Strategy. Unlike the clustering head, which employs weakly augmented samples to generate pseudo-labels and considers strongly augmented samples as the distribution to be learned, the samples learned by the calibration head undergo no augmentation at all, eliminating any representation differences caused by sample transformations.

### B.2 Training Process

Figure: Algorithm 1 The training of Calibrated Deep Clustering

### B.3 Clustering Performance: NMI

In addition to the ACC and ARI results shown in Sec. 4.1, our method also achieved 5 out of 6 best results on NMI (Tab. 4).

**Table 4: The clustering performance NMI on six image benchmarks. The best results are shown in bold, while the second best results are underlined.**
| Method | CIFAR-10 | CIFAR-20 | STL-10 | ImageNet-10 | ImageNet-Dogs | Tiny-ImageNet |
| --- | --- | --- | --- | --- | --- | --- |
| K-means[] | 8.7 | 8.4 | 12.5 | 11.9 | 5.5 | 6.5 |
| MoCo-v2 [] | 77.7 | 56.4 | 69.3 | 54.1 | 62.8 | 43.0 |
| Simsiam [] | 67.7 | 32.6 | 61.8 | 81.7 | 45.9 | 36.4 |
| BYOL [] | 65.5 | 43.9 | 67.0 | 70.3 | 63.4 | 29.6 |
| TCL [] | 81.9 | 52.9 | 79.9 | 87.5 | 62.3 | - |
| TCC [] | 79.0 | 47.9 | 73.2 | 84.8 | 55.4 | - |
| DMICC [] | 74.0 | 45.2 | 68.9 | 91.7 | 58.1 | - |
| ProPos [] | 88.6 | 60.6 | 75.8 | 90.8 | 73.7 | 46.0 |
| CoNR [] | 86.7 | 60.4 | 85.2 | 91.1 | 74.4 | 46.2 |
| DivClust [] | 72.4 | 44.0 | - | 89.1 | 51.6 | - |
| CC [] | 76.9 | 47.1 | 72.7 | 88.5 | 65.4 | 32.5 |
| SeCu-Size [] | 79.3 | 51.6 | 69.4 | - | - | - |
| SeCu [] | 86.1 | 55.1 | 73.3 | - | - | - |
| SCAN-2 [] | 79.3 | 52.2 | 78.8 | 88.9 | 60.1 | 43.1 |
| SCAN-3 [] | 82.5 | 51.2 | 83.6 | 92.5 | 67.4 | 40.7 |
| SPICE-2 [] | 73.5 | 44.3 | 80.4 | 82.8 | 56.9 | 44.9 |
| SPICE-3 [] | 85.4 | 57.6 | 86.1 | 90.2 | 62.6 | 42.7 |
| CDC-Clu (Ours) | 89.3 | 60.6 | 86.0 | 93.1 | 76.7 | 47.2 |
| CDC-Cal (Ours) | 89.3 | 60.9 | 85.8 | 93.2 | 76.5 | 47.5 |
| Supervised | 78.8 | 59.8 | 67.3 | 97.7 | 86.6 | 59.3 |
| +MoCo-v2 | 86.9 | 72.9 | 81.8 | 99.7 | 98.7 | 64.0 |

### B.4 Reliability diagrams

The reliability diagrams for all models trained on CIFAR-20 are shown in Fig. 6. Our proposed method demonstrates lower calibration error when the overall average confidence and clustering accuracy are comparable.

Figure: (a) Sup.
Refer to caption: x21.png

Figure: (a) CDC-LS
Refer to caption: x32.png

### B.5 Ablation Study

Effect of Calibration Head Loss.
We provide more detailed reliability diagrams on CIFAR-20, where the calibration head is applied with different regularization losses. Although these methods can reduce calibration errors, their performance does not significantly improve. This is related to the penalty imposed on high-confidence samples by these methods: over-penalizing high-confidence samples, which is evident in the reliability diagram in Fig. 7. Since samples with similar features are more likely to have consistent outputs, our method imposes a smaller penalty on high-confidence samples than on low-confidence samples, which effectively maintains the output of reliable samples.

Effect of the Clustering Head Learning on Reliable Region and the Calibration Head Learning on Overall Region. As shown in Tab. 5, we have (i) vs Clu (All) +Cal (All) and Clu (All)+Cal (part). Due to learning erroneous information, the model exhibits poor performance and calibration error. (ii) vs Clu (Part)+Cal (Part). The model does not learn from low-confidence regions, thus achieving performance closest to the proposed method, but with a certain disparity in calibration error.

Architecture and Hyperparameter. As shown in Tab. 6, more MLP layers may not necessarily bring better performance, as they may introduce more parameters and complexity. Moreover, the proposed model is robust to $w_{en}$, so we fixed it as $w_{en}=1$ in the experiments.

**Table 5: The ablation results of initialization strategy and dual-head interaction schema.**
| Settings | CIFAR-10 | CIFAR-20 | STL-10 |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ACC$\uparrow$ | NMI$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | NMI$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ | ACC$\uparrow$ | NMI$\uparrow$ | ARI$\uparrow$ | ECE$\downarrow$ |
| Clu (All)+Cal (All) | 93.7 | 87.8 | 87.0 | 1.6 | 58.1 | 59.7 | 43.4 | 7.4 | 92.1 | 84.7 | 83.8 | 3.3 |
| Clu (All)+Cal (Part) | 93.6 | 87.7 | 86.9 | 1.8 | 56.4 | 57.5 | 41.6 | 8.6 | 92.5 | 85.1 | 84.5 | 3.1 |
| Clu (Part)+Cal (Part) | 94.7 | 89.0 | 89.1 | 1.3 | 59.2 | 58.2 | 43.4 | 5.1 | 92.9 | 85.6 | 85.3 | 1.2 |
| CDC (Ours) | 94.9 | 89.3 | 89.5 | 1.1 | 61.7 | 60.9 | 46.6 | 4.9 | 93.0 | 85.8 | 85.6 | 0.9 |

**Table 6: Ablation results of the # MLP layers and $w_{en}$ on STL-10.**
| Layer Number | 2 | 3 | 4 | 5 | $w_{en}$ | 0.1 | 0.5 | 1 | 5 | 10 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ACC | 92.8 | 93.0 | 92.2 | 91.9 | ACC | 93.0 | 92.8 | 93.0 | 93.1 | 93.2 |
| NMI | 85.5 | 85.8 | 84.7 | 84.2 | NMI | 85.9 | 85.6 | 85.8 | 85.9 | 86.1 |
| ARI | 85.0 | 85.6 | 84.0 | 83.5 | ARI | 85.6 | 85.2 | 85.6 | 85.7 | 85.9 |
| ECE | 2.8 | 0.9 | 1.3 | 1.7 | ECE | 1.0 | 1.3 | 0.9 | 0.9 | 1.0 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 2
strengths: - Novel Approach to Calibration: The introduction of a dual-head model that effectively calibrates confidence estimates addresses a critical gap in deep clustering, enhancing the reliability of predictions.

- Improved Self-Training Mechanism: By leveraging high-confidence samples for pseudo-label self-training, the proposed method improves the quality of the training data, which is crucial for achieving better clustering performance.

- Robust Experimental Validation: The extensive experiments conducted demonstrate the effectiveness of the calibrated deep clustering model, showcasing significant improvements in both expected calibration error and clustering accuracy compared to existing methods, which adds credibility to the proposed approach.
summary: This paper addresses the over-confidence problem in deep clustering, where the predicted confidence for a sample significantly exceeds its actual accuracy. The authors propose a novel calibrated deep clustering framework that consists of a dual-head model, including a calibration head and a clustering head. The calibration head corrects the overconfident predictions from the clustering head, aligning estimated confidence with the model's learning status. The clustering head then selects reliable high-confidence samples for pseudo-label self-training. Additionally, the paper introduces a new network initialization strategy that improves training speed and robustness. The proposed model demonstrates significant improvements over state-of-the-art methods, achieving a 10× reduction in expected calibration error and enhanced clustering accuracy, supported by solid theoretical guarantees.
weaknesses: - This paper would benefit from a more comprehensive literature review. Specifically, it overlooks some closely related works, such as [a,b], which address the over-confidence issue for standard supervised learning or unsupervised domain adaptation (i.e., unlabeled target data clustering with the help of labeled source data). I recommend that the authors include a discussion of these works and its connections to their research. This will help to contextualize their contributions within the existing body of knowledge.

[a] Revisiting the Calibration of Modern Neural Networks. NeurIPS, 2021.

[b] Transferable Calibration with Lower Bias and Variance in Domain Adaptation. NeurIPS, 2020.

- The authors mention that the calibration head adjusts the overconfident predictions of the clustering head to generate prediction confidence that aligns with the model's learning status. However, in the context of completely unsupervised learning and clustering, it is not clear how the model's learning status is determined without any labeled data. I recommend that the authors provide a more detailed explanation of the methodology used to assess the model's learning state in the absence of supervision. This clarification is essential for understanding the effectiveness of the calibration head and its role in improving prediction confidence.

- The authors state that the clustering head dynamically selects reliable high-confidence samples estimated by the calibration head for pseudo-label self-training. However, it would be beneficial to explore whether insights from existing semi-supervised learning (SSL) methods, such as those discussed in [c], could be incorporated to further enhance the self-training process.

Additionally, a critical analysis of current self-training methods is warranted to determine whether they contribute to poor calibration of predictions. It would be valuable to quantify the extent of this calibration issue and discuss its implications on model performance. By addressing these points, the authors could strengthen their argument and provide a more comprehensive understanding of the calibration dynamics in their approach.

[c] Towards Discovering the Effectiveness of Moderately Confident Samples for Semi-Supervised Learning. CVPR, 2022.

- The authors note that existing approaches rely on a fixed threshold to select pseudo labels, which does not account for the model's learning status. This can lead to issues during different training stages: a higher threshold in the early stages may result in selecting fewer and class-imbalanced samples, hindering convergence speed, while in later stages, a fixed threshold may introduce noisy pseudo-labels due to increased predicted confidence.

It would be beneficial for the authors to consider incorporating insights from existing SSL works, such as [d] and its subsequent studies, which have proposed dynamic thresholds and class-wise thresholds. These methods could potentially enhance the adaptability of pseudo-label selection throughout the training process, improving both the quality of the training data and the overall performance of the model. A discussion on how these approaches could be integrated or compared with the proposed methodology would strengthen this work.

[d] FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling. NeurIPS, 2021.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - What exactly is reported in Table 2? Are these the best or average results over multiple runs? If yes, how many runs were conducted and what are the standard deviations across runs? See also the corresponding weakness described above.

- Hyperparameter sensitivity analysis w.r.t. K in Figure 8 shows that CDC is stable for different K for CIFAR-20 and STL-10, but it is set quite different for each data set.  Is there a heuristic to set K or does K need tuning w.r.t. the ground truth labels for each data set? See also the corresponding weakness described above.

- Is the MoCo-v2 pretraining used for CC, SCAN and SeCu as well or just for your method? Based on your description it is not entirely clear whether you just use the same ResNet-34 backbone for these methods or if you use MoCo-v2 pretraining for them as well. 

- Please clarify how you conducted the MoCo-v2 pretraining and which hyperparameters you used for each data set.

- Based on my understanding your method could be used as a dedicated self-labeling stage on top of any deep clustering method, similar to how SCAN's self-labeling is currently used to improve clustering performance after the joint deep clustering stage. I would be interested to see how your method would improve, e.g., SeCu after its deep clustering stage. This would further illustrate the general applicability of your method and increase its significance for the deep clustering community.
soundness: 3
strengths: **Originality**
- The authors identify and investigate the issue of overconfidence in deep clustering methods. Based on their findings they propose a novel Calibrated Deep Clustering (CDC) method.  This is a quite an interesting angle to improve deep clustering methods and connects the field to confidence calibration methods from (semi)-supervised learning. 

**Quality**
- The authors provide extensive experiments across different benchmark data sets and deep clustering methods. Further, they made the extra effort of reimplementing SeCu, SCAN and CC to obtain a unified benchmarking environment increasing the fairness of the comparison. I would highly encourage the authors to release their code with the full benchmarking environment, as this will benefit the deep clustering community. Additionally, the authors provide ablation studies and a hyperparameter analysis for their algorithm CDC.

**Clarity**
- The paper is well written, and all design decisions are clearly motivated

**Significance**
- Their method CDC combines SCAN’s self-labeling procedure with a calibration head.  As self-labeling is widely used and their method does not make specific assumptions about the underlying deep clustering method, I believe this method can impact many deep clustering methods going forward.
summary: The authors identify the problem of overconfident predictions in deep clustering methods. To overcome this problem, they propose a dual head design with a dedicated calibration head aside the usual clustering head to improve deep clustering performance and calibration. Further, they propose an initialization strategy for the dual heads that improves stability when transitioning between the pretraining and deep clustering phases.
weaknesses: - Unclear reporting of experimental results.  The authors fail to mention if they report average performance results in Table 2 and other figures, or if they report the results of a single (best) run. As many deep clustering methods have high variability across different random seeds it would be important to report standard deviations as well, e.g., SCAN reports standard deviations of up to 3% for CIFAR-20.  Currently, it is not possible to judge whether the proposed method CDC is stable across multiple runs, because no standard deviations are reported and it is not mentioned if several runs are conducted at all.

- The hyperparameter for mini-clusters K needs to be specified quite differently for each data set ranging from 40 to 1000. Further, K does not seem to be related to the ground truth number of clusters of each data set and the authors do not provide any unsupervised heuristic that can be used to set K. The same holds true for the batch size B.

- The authors do not specify how they tuned their MoCo-v2 backbone. Further, no hyperparameters for the pretraining are reported. This is an important aspect that needs to be reported for reproducibility.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could the authors comment on the sensitivity with respect to the hyperparameters as well as on the variability in results over multiple runs?

Should the subscripts be switched in Theorem 1? So $E_F[Conf^{cal}]$ <= $E_F[Conf^{clu}]$ and  $E_T[Conf^{cal}]$ = $E_T[Conf^{clu}]$?
soundness: 3
strengths: - The paper is overall well written and the contributions presented in a clear and well-motivated manner. 

- Considering calibration in deep clustering is growing more important as state-of-the-art methods rely increasingly on pseudo-label self-training and has been not been addressed previously to the reviewers knowledge.

- The proposed approaches are intuitive and effective as indicated by the empirical results and the ablation study. 

- Empirical results are supported by theoretic insights and analysis.
summary: The paper considers calibration in the context of deep clustering, more specifically pseudo-label self-training based deep clustering. The authors propose an architecture consisting of a backbone (trained in a self-supervised manner) and two heads, where one is the common clustering head, while the other is a calibration head. The calibration head obtains calibrated confidences by overclustering the data, computing the average prediction for each of the clusters, and then aligning the predictions of the calibration head for members of the clusters to the clusters average prediction. The better calibrated predictions can then be leveraged for the pseudo-label selection. Beyond the calibration head, the authors propose a new initialization strategy for clustering (and calibration) heads that does ensure initialization that resembles the k-means performance on the backbone features as well as a mechanism to dynamically select the threshold for pseudo-label selection for each class. The proposed approach is evaluated on six datasets, obtaining promising performance and lower expected calibration error compared to prior deep clustering baselines.
weaknesses: Hyperparameter selection is challenging in a clustering setting when no labeled data is available. It is therefore beneficial to have approaches that are robust with respect to hyperparameters. Currently, the sensitivity to the added hyperparameters is not discussed and relative specific hyperparameters are defined for different datasets (B, K) or not provided (H). How were these hyperparameters selected and how sensitive is the model to these choices. Do the authors have insights on how these should be chosen in the typical clustering setting where labeled data is unavailable? 

One weakness of the work is that results are currently reported over single runs and without information about variability. Deep clustering approaches tend to be less stable than their supervised counterparts and some more information on the robustness of these results would therefore be beneficial.

The claim in the abstract regarding surpassing state-of-the-art deep clustering methods by 10x in term of expected calibration error should be revised to clarify that this is compared to only some of the baselines and on some of the considered datasets.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: No
soundness: 2
strengths: This paper proposes a dual head deep clustering model that can effectively calibrate the estimated confidence and the actual accuracy.
summary: This paper proposes a dual head deep clustering model that can effectively calibrate the estimated confidence and the actual accuracy.
weaknesses: According to the discussion between the authors and other reviewers, I altered my rating.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. The 'classifier $g(\theta, \cdot)$' is mentioned only in the Notation and Algorithm 1. However its role in the proposed pipeline remains unclear to me. Can you clarify its specific function within the pipeline? Is this term meant to generally refer to both the clustering head and calibration head?
2. Can you provide a detailed algorithm outlining how prediction is done?
3. In Equation 4, is the total loss simply the sum of the two individual losses? Is there a hyperparameter that controls the weighting between them? Are the two losses on similar scales? Additionally, is the use of the negative entropy loss inspired by prior works? It would be helpful to see an ablation study that removes this term and compare the results.
4. How does the algorithm perform in the presence of out-of-distribution (OOD) samples? Specifically, could you provide a Density-confidence plot as in figure 3, when OOD samples are introduced in the dataset similar? One possible way to generate OOD samples is by adding random Gaussian noise to selected samples or by mixing data from irrelevant datasets.
soundness: 3
strengths: 1. The authors provide a comprehensive comparison of their method against 14 prior works. They have also tested the method on 6 datasets, which is a sufficient number to validate their approach. The experimental results clearly demonstrate that CDC achieves significantly better performance than previous methods. The experimental design, including both the main experiments and ablation studies, is rigorous and well-justified.
2. The figures are well-designed and effectively highlight the advantages of the proposed approach. Both the presentation of the results and the visuals are compelling and easy to interpret.
3. The authors offer sound theoretical proof showing that the proposed method achieves a lower expected calibration error and penalizes only unreliable regions, adding to the method’s credibility.
4. The motivation behind the work is well-justified. Both the introduction and abstract are clearly written and provide a strong foundation for the paper.
summary: This paper proposes calibrated deep clustering, a method to calibrate the confidence of deep clustering networks. Specifically, it proposes a separate calibration head alongside the traditional clustering head to optimize the learning and prediction confidence of clusters. This approach leads to better-calibrated models as it ensures that the calibration head consistently achieves lower errors compared to the clustering head. The authors provide a theoretical proof to demonstrate that the calibration error from the calibration head is less than or equal to that of the clustering head, supporting the validity of their method.
weaknesses: 1. The authors should reconsider the structure of the paper. In my view, the methods section should first focus on the network and training design, followed by secondary aspects like initialization strategies. While initialization techniques can improve performance and speed up convergence, they are supplementary rather than a primary contribution. Therefore, dedicating excessive space to this, even with theoretical justifications, is unnecessary. Additionally, the experimental results section is too brief. I recommend condensing the methods description and moving a more detailed analysis of results from the appendix into the main body of the paper.

2. The notation is inconsistent and confusing. Several terms are introduced without adequate mathematical explanation. For example, $p_i^{clu}$ is first mentioned in Line 230 as "the prediction of $x_i$ by the clustering head" but lacks a formal mathematical definition. I suggest including an equation, such as defining the clustering head as $g(\theta_{clu},\cdot)$ and expressing $p_i^{clu} = g(\theta_{clu}, x_i)$. A similar issue with $p_i^{cal}$ in Line 235. When introducing the proposed pipeline, the authors should clearly guide the reader through the entire process, from input sample $x$ to the final loss function, in a step-by-step manner in the methods section.

3. The title of Section 3 should be 'Proposed Method'.

Overall, the paper presents promising results and good analysis. However, the presentation of the methodology is lacking, which makes the paper difficult to follow. Revising the organization and clarity of the exposition is neccessary.

Meta-review: This paper presents a work that addresses the over-confidence problem in deep clustering. A novel calibration method is proposed. A new network initialization strategy is also introduced to improve training efficiency. Both theoretical analysis and empirical evaluations are provided, which verify the superiority of the algorithm. The potential weakness of the work is the selection of hyperparameters. This paper provides useful insights for clustering. Reviewers and AC are overall satisfied with the rebuttal and revised manuscript. AC therefore recommends accepting this paper.

Decision: Accept (Poster)