Clustering-friendly Representation Learning via Instance Discrimination and Feature Decorrelation
Authors: Yaling Tao
Kentaro Takagi
Kouta Nakata
Venue: ICLR 2021 Poster (2021)
Keywords: clustering
representation learning
deep embedding

Abstract: Clustering is one of the most fundamental tasks in machine learning. Recently, deep clustering has become a major trend in clustering techniques. Representation learning often plays an important role in the effectiveness of deep clustering, and thus can be a principal cause of performance degradation. In this paper, we propose a clustering-friendly representation learning method using instance discrimination and feature decorrelation. Our deep-learning-based representation learning method is motivated by the properties of classical spectral clustering. Instance discrimination learns similarities among data and feature decorrelation removes redundant correlation among features. We utilize an instance discrimination method in which learning individual instance classes leads to learning similarity among instances. Through detailed experiments and examination, we show that the approach can be adapted to learning a latent space for clustering. We design novel softmax-formulated decorrelation constraints for learning. In evaluations of image clustering using CIFAR-10 and ImageNet-10, our method achieves accuracy of 81.5% and 95.4%, respectively. We also show that the softmax-formulated constraints are compatible with various neural networks.

idea of spectral Published as a conference paper at ICLR 2021
CLUSTERING-FRIENDDI REPRESENTATION LEARN- ING VIA INSTANCE DISCRIMINATION AND FEATURE
DECORRELATION
Yaling Tao, Kentaro Takagi Kouta Nakata
Corporate R&D Center, Toshiba Corporation
1, Komukai Toshiba-cho, Saiwai-ku, Kawasaki, Kanagawa, Japan (yalingl tao, kentarol takagi, kouta nakata @tt.aaaaaatooho.oooooooooo
ABSTRACT
Clustering is one of the most fundamental tasks machine learning. Recently, deep clustering has become major trend in clustering techniques. Representa- tion learning often plays an important role in the effectiveness deep clustering, and thus can be principal cause of performance degradation. In this paper, we propose a cestering--fiennde crimination and feature decorrelation. Our deep-learning- based representation learning method is motivated by the properties of classical spectral clustering. Instance discrimination learns similarities among data and feature decorrelation moves redundant correlation among features. We utilize an instance discrimination method in which learning individual instance classes leads learning similarity among instances. Through detailed experiments and examination, we show that the approach can be adapted learning a latent space for clustering. We design decorrelation constraints for learning. In evaluations of image clustering using CIFAR- 10 and ImageNet-10, 10, our method achieves accuracy of 81.5% and 95.4%, respectively. We also show that the softmax-formull constraints are compatible with various neural networks.
representation learning method using instance dis-
novel
INTRODUCTION
Clustering is one of the most fundamental tasks in machine learning. Recently, deep clustering has become a major trend in clustering techniques. fundamental form, autoencoders are used for feature extraction, and classical clustering techniques such as k-means are serially applied to the features. Recent deep clustering techniques integrate learning processes feature extraction and clustering, yielding high performance for large-scale datasets such handwritten digits Hu al.] Shaham
Shahaa et al. (2018): : Xie et al. (2016); Tao et al.) (2018). However, those methods have complex, in the case of real-world photograph dataset CIFAR-10 Krizhevsky et al. (2009). Several works report powerful representation learning leads to improvement of clustering performance on complex datasets Chang al. (2017): Wu al. (2019).
fallen short when targets become
Learning representation is key challenge to unsupervised clustering.
In order learn representations for clustering, recent works utilize metric learning which auto- matically learns similarity functions from data Chang et al. (2017); Wu et al. (2019). They assign pseudo-labels or pseudo-graphy to unlabeled data by similarity measures in latent space, and learn discriminative representations to cluster data. These works improve clustering performance on real world images such as CIFAR-10 and ImageNet-10. and indicate the impact of representation learning on clustering. Although features from learned similarity function and pseudo-labels work well for clustering, algorithms still seem to be heuristic; we design novel algorithm which based on knowledge from established clustering techniques. In this work, exploit clustering which uses eigenvectors derived from similarities.
Spectral clustering has been theoretically and experimentally investigated, and known to outperform other traditional clustering methods Von Luxburg (2007). The algorithm involves similarity matrix construction, transformation from similarity matrix Laplacian, and eigendecomposition. Based on

---

Published as a conference paper at ICLR 2021
eigenvectors, data points mapped into a lower dimensional representation which carries informa- tion of similarities and is preferable for clustering. We bring this idea of eigenvector representation into deep representation learning.
We design the representation learning with two aims: 1) learning similarities among instances; and 2) reducing correlations within features. The first corresponds to Laplacian, and the second corresponds to feature orthogonality constrains the spectral clustering algorithm. Learning process integrating both is relevant to eigendecomposition of Laplacian matrix in the spectral clustering.
For the first aim, we adopt the instance discrimination method presented in Wu et al. (2018), where each unlabeled instance is treated as its own distinct class, and discriminative representations learned to distinguish between individual instance classes. This numerous-class discriminative learn- ing enables learning partial but important features, such small foreground objects natural images. Wu et al. 2018) showed that the representation features retain apparent similarity among images and improve the performance of image classification by the nearest neighbor method. We extend their work to the clustering tasks. We clarify their softmax formulation works like similarity matrix in spectral clustering under the condition that temperature parameter т, which was underexplored in Wu et al. 2018), is set to be a larger value.
For the second aim, we introduce constraints which have the effect of making latent features or- thogonal. Orthogonality is often essential idea in dimension reduction methods such as principal components analysis, and it is preferable for latent features to be independent to ensure that redundant information is reduced. Orthogonality is also essential to connection between proposed method and spectral clustering, as stated in Section 3.4 In addition to a simple soft orthogonal constraint, we design novel formuu--orruuatedd decorrelation constraint. Our softmax constraint is "softer" than the soft orthogonal constraint for learning independent feature spaces, but realizes stable improvement of clustering performance.
Finally, we combine instance discrimination and feature decorrelation into learning representation to improve the performance of complex image clustering. For the CIFAR-IO and ImageNet- 10 datasets, our method achieves accuracy of 81.5% and 95.4%. respectively, Our Torch Paszke et al. (2019) implementation of IDFD is available https:: /ttea:://gttauat wttp:///w//..o///////tteeeaaaaaa fri friendly_ hetrlyereereeeeeteeeeeeee ion learning
Our main contributions are as follows:
We propose a clustering-friende representation learning method combining instance dis- crimination and feature decorrelation based on spectral clustering properties.
We adapt deep representation learning by instance discrimination to clustering and clarify the essential properties of the temperature parameter.
We design roftmax-formuland orthogonal constraint for learning latent features and realize stable improvement of clustering performance.
Our representation learning method achieves performance comparable to state-of-the-art levels for image clustering tasks with simple k-means.
2 RELATED WORK
Deep clustering methods offer state-of-the-arr performance in various fields. Most early deep clustering methods, such as Vincent al.] (2010); Tian et al.) (2014), are two-stage methods that apply clustering after learning low-dimensional representations of data in a nonlinear latent space. The autoencoder method proposed in Hinton & Salakhutdinoy (2006) is one of the most effective methods for learning representations. Recent works have simultaneously performed representation learning and clustering Song et al. (2013); Xie et al. (2016); Yang et al. (2017); Guo al. (0117); Tao et al.) (2018). Several methods based on generative models have also been proposed Jiang et al. (2016); Dilokthanakul et al. (2016). These methods outperform conventional methods, and sometimes offer performance comparable that of supervised learning for simple datasets. Deep-learning- based unsupervised image clustering is also being developed Chang et al. (2017); Wu et al. (2019); Ji et al. (2019); Gupta al. (2020); Van Gansbeke et al. (2020).
2

---

Published as a conference paper at ICLR 2021
Several approaches focus on learning discriminative representations via deep learning. Booaanoski & Joulin (2017) found a mapping between images on a uniformly discretized target space, and enforced their representations resemble distribution of pairwise relationships. Caron et al. (2018) applicd pseudo-labels to output as supervision by k-means and then trained a deep neural network, Donahue al. (0066) proposed bidirectional generative adversarial networks for learning generative models that map simple latent distributions to complex real distributions, in order for generators to capture semantic representations, [jellm et al. (2018) proposed deep infomax to maximize mutual information between the input and output of an encoder. [W et al. (2018) was motivated by observations in supervised learning that the probabilities of similar image classes become simultaneously high. They showed that discriminating individual instance classes leads to learning representations that retain similarities among data.
ICC et al. (019) and SCAN Van Gansbeke et al. (2200) are two recent works focusing on image clustering and obtained high performance. Ji al. (2019) directly learns semantic labels without learning representations based on mutual information between image pairs. SCAN
Van Gansbeke al. (2220) focuses on the clustering phase and largely improved performance based given pre-designed representation learning. By contrast, focus on learning clustering- friendly representation space where objects can be simply clustered.
Our method exploits the idea of spectral clustering (Shi & Malik (2000): Meila & Shi (200)):
Von Luxburg (2007); Ng et al.] (2002). From one perspective, spectral clustering finds a low dimen- sional embedding of data in the eigenspace of the Laplacian matrix, which is derived from pairwise similarities between data. By using the embedded representations, we can proceed to cluster the data by the k-means algorithm in the low-dimensional space. Spectral clustering often outperforms earlier algorithms such as k-means once pair similarities are properly calculated. |Shaham et al. (2018) incorporated the concept of spectral clustering into deep a neural network structure. Similarities were calculated by learning a Siamese net (Shaham & Lederman (2018) where the input positive and negative pairs were constructed according to the Fuclidean distance.
PROPOSED METHOD
Given unlabeled dataset X |= and predefined number of clusters k. where z; denotes the ith sample, we perform the clustering task in two phases, namely, representation learning and clustering. This work focuses the first phase, which aims to learn embedding function mapping data x to representation v so that v is preferable for clustering. fo is modeled as a deep neural network with parameter O. We use V ) to denote the whole representation set.
fo (x)
3.1 INSTANCE DISCRIMINATION
We apply the instance discrimination method proposed by Wu et al. (2018) to learn clustering-friendly representations that capture similarity between instances. The objective function formulated based on the softmax criterion. Each instance is assumed to represent a distinct class. For given data T1, nn the corresponding representations are UI...., Un and data o; is classified into the ith class. Accordingly, the weight vector for the ith class can be approximated by a vector etz The probability of representation e being assigned into the ith class
exp(v. v/T)
P((lu) Σn ビ exp (ep(vuJu v/T)
where of o measures how well v matches the jith class, 7 is temperature parameter that controls the concentration of the distribution Hinton al. 2015), and e is normalized to ||인|| =
The objective maximizes the joint probability I (ol as
L = Σ log P(ilfo(x))) = - log( Σ exp(
exp( Uz
(2)
Wu et al. (2018) shows that features obtained by minimizing the objective retain similarity between image instances and improve the performance of nearest neighbor classification. For clustering, note that the parameter t, which is underexplored in Wu et al. (2018), has a large impact on clustering performance. The effect of is discussed later and experimental results are shown in 4.2.1
3

---

Published as a conference paper at ICLR 2021
(x)
X [x
CNN backbone
Py(1 LJ n)
(vd).
Low dim
12 norm
2nd Image
Instance discriminate Softmax
image
d-Dim
d-Dim
(st image
transpose
Image
fflla
Amm
Clustering e.g. K. means
norm
2nd dimension of feature
Feature-
ndependent Softmax
Ith dimensson of fenture
Y predd
(ff/)th dimcasion feature
Figure Pipeline of our method.
3.2 FEATURE DECORRELATION
We define of latent feature vectors f and use denote the /th feature vector. Transposition of latent vectors V coincides with {fik where is the dimensionality of representations.
The simple constraint for orthogonal features is,
LFO IVVV 7|
1)
Σ 5T Ø=1
Our novel constraint is based on softmax formulation of exp(Si f/T2)
Q(I
Σm exp(ff
Q(I is analogous to p)) (( measures how correlated feature vector is to itself and how dissimilar is others. 72 the temperature parameter. We formulate the feature decorrelation constraint
LF Σ log Q(f))
Σ
fT/T2 log Σ explf fi/ f1/Ta) )
(5)
Both constrains in Eq. and Eq. 5 aim to construct independent features. Conventionally, is preferable for features to be independent to ensure that redundant information is reduced, and orthogonality is a common technique. Compare Eq. (3) and Eq. (5), we can see that minimizing and 1f
LF and Lro can result in a similar effect, ff decorrelate latent features.
or 0(1 + j). and both try to
Our softmax constraint in Eq. (5) shows practical advantages in flexibility and stability. Eq. (3) is called soft orthogonal constraint, but is still strict enough to force the features to be orthogonal. If larger than underlying structures that are hidden and unknown, all features are forcibly orthogonalized and the resultant features may not be appropriate. Softmax formulation allows off-diagonal elements to be non-zero and alleviates the problem of strict orthogonality.
Partial derivatives of Lr and LFo with respect to 25t f3 ft are calculated
aLg = 1-115 5,1
dzzt
1xp(x31/2) and 8Lro 25 22yt, where 55t an indicator function. Since the derivatives 8zj1
T2 Σj exp(e)1/T2)
4

---

Published as a conference paper at ICLR 2021
nearly equal zero due to 3n1 in the case of = we focus the case of + L. When j # L, the ranges of partial derivatives are 0 v 0Lg v
ōzyt - and -2 v úLfo 2. The monotonicity of Ly can
0zzt
lead to more stable convergence. The advantages of Lp are confirmed by experiments in section
3.3 OBJECTIVE FUNCTION AND LEARNING MODEL
Combining instance discrimination and feature decorrelation learning, we formulate our objective function LIDED as follows:
LIDED L1 aLF,
Where is weight that balances the contributions of two terms T, and L.-
Figure shows the learning process for the motif of image clustering. Input images X are converted into feature representations in a lower d-dimensional latent space, via nonlinear mapping with deep neural networks such as ResNet He et al. (2016). The d-dimensional vectors are simultaneously learned through instance discrimination and feature decorrelation. clustering method, such as classical k-means clustering, is then used on the learned representations to obtain the clustering results.
Optimization can be performed by mini-batch training. To compute the probability P(i]v) in Eq. (1). {ey} is needed for all images. Like Wu et al. (2018); Xiao et al. (2017), we maintain feature
memory bank for storing them. For Q in Eq. (4) all [Jmm } of d dimensions in the current mini-batch can be obtained, we simply calculate the ( within the mini-batches.
We combine 1, and LFo formulate an alternative loss LIDFO in E.q.
LIDFO Lr aLFO.
We refer to representation learning using LIDED. LIDFO, and L loss instance discrimination and feature decorrelation (IDDD), instance discrimination and feature orthogonalization (IDFO), and instance discrimination (ID), respectively.
3.4 CONNECTION WITH SPECTRAL CLUSTERING
We explain the connection between IDFD and spectral clustering. We consider fully connected graph consisting of all representation points, and the similarity matrix W and degree matrix D can be written as Wi; sj exp(x? ///T) and D5 Σn exp(1 Um/T). The loss function of spectral clustering Shaham et al. (018)) can be reformulated as
Lsp = (TT) 2 ΣΣ
wy U - 831 =
exp
D U/
||u
1y/1?
(8)
k
where L Laplacian matrix. f are feature vectors. Spectral clustering performed by minimizing Lsr subject to orthogonal condition of f. and when Lgp takes minimum value f become eigen- vectors of Laplacian L. According to Section 3.2 minimizing Lp can approximate the orthogonal condition. Under this condition, minimizing I, can approximate the minimizing Lsr, which is explained as follows.
According to [2. minimizing loss L, means maximizing of v; and minimizing of dj. When i j. we have |0 minimizing T uj. As v are normalized, Lsp can be rewritten using cosine metric as
2 = O. Lsp becomes zero. We need consider only the influence on Lsp from
cos 0
LSP =
exp
sin 2
(9)
2
aLeg can be caleulated then 30
ÖLsp
cos 0
sin 0(T + cos exp
(10)
00
5

---

Published as a conference paper at ICLR 2021
According to Eq. we get Uhse 2 0 when 2 2. This means Lşp monotonically decreases 20
when we minimize of Vz. Therefore, the impact from minimizing 1z U3 is good for minimizing Lsp-- Even if 7 is little smaller than 2, because 7 controls the scale of derivatives and the range of 0 where the derivative is negative, large decreases the scale and narrows the range, resulting influence on the total loss. From this viewpoint, the effectiveness of minimizing Lı using large is approximately the same as that of Lgp. By adding feature decorrelation constraints, IDFD becomes analogous to spectral clustering.
small
4 EXPERIMENTS
We conducted experiments using five datasets: CIFAR-10
Krizhevsky et al. (2009) CIFAR-100
Krizhevsky et al. ) STL-10 Coates et al. (201T), IetgeNet-10 10 Deng al. 2009). and ImageNet-Dog Deng et al. (2009). We adopted ResNet18 8 He et al. (2016) as the neural network architecture in our main experiments. The same architecture is used for all datasets. Our experimental settings are in accordance with that Wu et al. (0188). Data augmentation strategies often used on images are also adopted in experiments. Details about datasets and experimental setup are given in Appendix A
For IDFD, the weight is simply fixed at 1. Orthogonality constraint weights for IDFO were a 10 on CIFAR-10 10 and CIFAR-100, and 0.5 on STL-100 and ImageNet subsets. The weight was set according to the orders of magnitudes of losses. In the main experiments, we set temperature parameter 1 for IDFO and IDFD, and 72 2 for IDFD. In order to fully investigate work, we also constructed two versions of instance discrimination (ID) that uses only L1 loss, ID(original) with small 0.07 and ID(tuned) with large 1.
We compared D(tunedd), IDFO, and IDFD with ID(original) and six other competitive methods, clus- tering with an autoencoder (AE) Hinton & Salakhutdinov (2006). deep embedded clustering (DEC) Xie et al. (2066), deep adaptive image clustering (DAC) Chang et al. (2017), deep comprehensive correaaion mining (DCCM) /Wu et al. (2019), invariant information clustering (IIC) Ji al. (2019), and semantic clustering by adopting nearest neighbors (SCAN) Van Gansbeke et al. (2020) We use three metrics measure clustering performance: standard clustering accuracy (ACC), normalized
mutual information (NMI), and adjusted rand index (ARI). These metries give values in [0, 1]. with higher scores indicating more accurate clustering assignments.
4.1 MAIN RESULTS
Table T lists the best performances for each method. The results for the four methods AE, DEC, DAC, and DCCM are cited from Wu et al.] (2019), and results for two methods IIC and SCAN are cited from Van Gansbeke et al. (2020). Comparing these results, we conclude that ID(tuned), IDFO, and IDFD, clearly outperform these methods excluding SCAN for all datasets, according to the metrics ACC. NMI, and ARI. For dataset CIFAR-10. 10, ID(tuned), IDFO, and IDFD yielded ACC values of 77.6%, 82.8%, and 81 81.5%, respectively. For dataset ImageNet-10, ID(tuned), IDFO, and IDFD achieved ACC values of 93.7%, 94.2%, and 95.4%. The high performance is comparable with that of supervised and semi-supervised methods. Gaps between the results of ID(tuned) and those of IDFO and IDFD reflect the effect of the feature constraint term. The performance is improved for all datasets by introducing feature orthogonalizatton and decorrelation. Impressively, ID(tuned) significantly outperformed (D(original) on all datasets, showing strong impact of temperature parameter. This will be discussed separately in section 4.2.1
In addition, we note that IDFD differs from SCAN in that IDFD focuses on the representation leaning while SCAN focuses on clustering by given representation learning. Both SCAN and IDFD demonstrate significant improvement on performance compared with other methods. Results of IDFD and SCAN showed effecctiveness of efforts on both representation learning and clustering phases of deep clustering.
We also examine the learning stability of ID(tuned), IDFO, and IDFD, Figure 2 illustrates the accuracy on CIFAR- 10 running each of D(tuned), IDFO, and IDFD. We can see that both IDFO and IDFD obtained higher peak ACC values than ID(tuned). In particular, IDFD yielded higher performance than ID over the entire learning process. IDFO performed better than the other two methods and obtained the highest ACC value in earlier epochs. However, the ACC widely fluctuated

---

Published as a conference paper at ICLR 2021
Table I: Clustering results (%) of various methods five datasets.
Dutnset
CIFAR-10
CIFAR-100
STL-10
ImageNet-10 NMI
ImageNet-Dog
Metric
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ART
ACC
ARI
ACC
NMI
ART
AE
31.4
23.9
16.9
16.5
10.0
4.8
30.3
25.0
16.1
31.7
21.0
15.2
18.5
10.4
7.3
DEC
30.1
25.7
18.5
13.6
5.0
35.9
27.6
38.1
20.3
12.2
7.9
DAC
52.2
39.6
30.6
23.8
18.5
8.8
47.0
36.6
25.7
52.7
39.4
30.2
27.5
21.9
11.1
DCCM
62.3
49.6
40.8
32.7
28.5
17.3
48.2
37.6
26.2
71.0
60.8
55.5
38,3
32.1
18.2
DDooriginal)
44.0
30,9
22.1
26.7
22.1
10.8
51.4
36,2
28.5
63.2
47.8
42.0
36.5
24.8
17.2
ICC
61.7
51.1
41.1
25.7
22.5
11.7
59.6
49,6
39.7
SCAN
88.3
79.7
77.2
50.7
48.6
33.3
80.9
69.8
64.6
([tundd)
77.6
68.2
61.6
40.9
39.2
24.3
72.6
64.0
$2.6
93.7
86.7
86.5
47.6
47.0
33.5
IDFO
82.8
71.4
67.9
42.5
43.2
24.4
75.6
63.6
56.9
94.2
87.6
61.2
57.9
41.4
IDFD
81.5
71.1
66.3
42.5
42.6
26.4
75.6
64,3
57.5
95.4
89.8
90.1
59.1
54.6
41.3
over the learning process and dropped in later epochs. As analyzed in performance higher than ID and more stable than IDFO.
3.2 our proposed IDFD makes
4.2 DISCUSSION
4.2.1
ANALYSIS ON TEMPERATURE PARAMETER
Gaps between results of (Doriginal)) and ID(tuned) in Table 1 show strong impact of temperature parameter. We theoretically and intuitively analyze the essential change caused by the temperature parameter in this subsection.
First, we consider why instance-level discrimination works and under what conditions. Difference in the performance of (D(oiiginal) and ID(uuned) suggests optimal distribution in latent space changes with the magnitude т. According to empirical investigation and theoretical analysis, we find that large t in Ls encourages data points to follow compact distribution when minimizing the loss, while a small 7 drives them to follow a uniform distribution. This means minimizing L, with a large reach good cesstering-friendde solution. This property explained by demonstrating examples and calculation, details are given in Appendix B.
In the definition of P(ilv) in Eq. [ when is small, compute softmax larger logits, resulting in higher prediction, and obtain a more confident model. From this viewpoint, we can leverage a small 7 to decrease class entanglement if we can learn accurate class-weight vector. In the general classification problem. since the weight of each class can be learned according to the real labels, is preferable for models be more confident. Most works therefore recommend setting a small value, such as = 0.07 Wu et al. (2018). In clustering, however, instance- discrimination is used learn similarity among samples, with only one sample in each class. Because the model is highly confident, each sample tends to be completely independent from each other. Similarity among samples is seemingly encouraged to approach close to zero, even for samples from the same class. This clearly deviates from the original intent of adopting instance-level discrimination to learn sample entanglements under the condition that each sample can be discriminative. A larger 7 than that used for classification is thus needed.
More experiments over different temperature settings on ID and IDFD were conducted on CIFAR- 10. Figure 3 shows the accuracy of ID for (0.07, 0.2, 0.5, 0.8. 1, 5, 10). We calculated the mean and standard deviation of ACC values over the last 500 epochs for each experiment. From the results, we can that ID suffer significant performance degradation when is too small too large. This agrees with our analysis above. We also investigate the impact of 72 by fixing T = 1. Figure shows the accuracy the IDFD for T2 {0.1, 0.5, 1, 2, 3, 4, 5, 10}. Experimental results show that IDFD is relatively robust to the parameter and enables stable representation learning.
4.2.2 REPRESENTATION DISTRIBUTION AND FEATURE BEHAVIOR
Figure visualizes the results of representations learned in four experiments: (a) ID(original), (b) ID(uueed), (c) IDFO with r = and a = 10, and (d) IDFD with 5 = 1, T2 = 2, and a = on CIFAR- 10. di-eimnnnoonn representations were embedded into two dimensions by t-SNE (t-distributed stochastic neighbor embedding) |aaten & Hinton (2008). Colors indicate ground truth classes. The distributions for the (Dorigiiaal) and IIX(te)d) again show the significant difference between
7

---

Published as a conference paper at ICLR 2021
107
Figure 2: ACC values over learn- Figure 3: Accuracy of ID for var- var-Figure 4: Accuracy of IDFD for various T2 settings.
ing process.
ious settings.
them. Data distribution when = 1 is apparently more cuustering-friendee than when = 0.07. Furthermore, compared with (D(uned)) IDFO and IDFD can separate samples from different classes with certain margins. IDFO tended to construct a patch-like distribution within one class. In contrast, IDFD maintained tighter connection among samples of the same class and more distinct borders between different classes.
(original)
(b) (mased)
(6)
Figure Distribution feature representations on CIFAR-1O.
Figure 6 shows distribution feature representations on ImageNet- 10 learned by IDFD. We can see that representations of ImageNet- 10 siaterrinn-rrendly and even better than that of CIFAR-10. This is consistent with the results in Table [ by metrics ACC, NMI, and ARI. In addition to that, we also plot sample images corresponding to points lying near the border between clusters. We can see that these samples are certainly similar in appearance.
Figure 6: Distribution of feature representations on ImageNet-10 learned by IDFD and samples corresponding to points in some areas.
We investigate the effects of orthogonal and decorrelation constraints LFO and LF. iilustratts the feature correlations of ID(tuned), IDFO, and IDFD on dataset CIFAR-10. We see that IDFO clearly decorrelates features and IDFD retains moderate level of feature correlation between II) and IDFD. Taken together with Figure 2 these results suggest that the softmax formulation of IDFD alleviates the problem of strict orthogonality and enables stable representation learning.
4.2.3 INVESTIGATION FOR PRACTICAL USE
We investigate the dependencies of our method on networks through experiments other networks: ConvNet Wu et al. (2019), VGG16 Simonyan & Zisserman (2014). and ResNet34 et al. (2016). Performance evaluated using the CIFAR-10 dataset. Results listed in Table show that IDFD
8

---

Published as a conference paper at ICLR 2021
DDO
DFD
1.00
20
20
0.75
0.50
80
80
-0.25
-0.50
100
100
100
-1.00
25
75
100
125
25
75
100
125
25
50
75
100
125
Figure Feature correlation matrix on CIFAR-10 with ResNet18
can work on various networks. IDFD outperforms ID(uuned), and FD term shows more obvious effect on these networks. We also confirm the effect of cooperation between L, and L- from the viewpoint of spectral clustering, combinations of AE and LF were evaluated in terms of clustering performance. found that AE cannot benefit from LF as L, did. This result verified that LF has deep relation with Lı, and IDFD is not simple combination. We also investigate the importance of data augmentation in performance through experiments. Due to the page limit, our extended experiments are given in Appendix C
Table 2: Clustering results (%) on various network architectures.
Network Metric
ConvNet
VGG16
ResNet18
ResNet34
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
ID(tuned)
26.8
15.0
8,9
39.3
31.6
20.9
77.6
68.2
61.6
80.2
71.1
64.6
IDFD
42.0
32.7
23.2
56.8
46.7
36.5
81.5
71.1
66.3
82.7
73.4
68.4
5 CONCLUSION
We present
representation learning method combining instance discrimination and feature decorrelation based on spectral clustering properties. Instance discrimination learns similarities among data and feature decorrelation removes redundant correlation among features. We analyzed why instance discrimination works for clustering and clarified the conditions. We designed formullaed feature decorrelation constraint for learning the latent space to realize stable improvement of clustering performance. We also explained the connection between our method and spectral clustering. The proposed representation learning method achieves accuracies comparable to state-of-the-ar values the CIFAR- 10 and ImageNet- 10 datasets with simple k-means. We also verified IDFD loss works on multiple neural network structures, and our method expected to be effective for various kinds of problems.
REFERENCES
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In Proceedings of the 34th International Conference Machine Learning-Voll Volume 70, pp. 517-526. JMLR. org, 2017.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng. Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In The IEEF International Conference Computer Vision (ICCV), Oct 2017.
Ting Chen. Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05/7 05709, 2020.

---

Published as a conference paper at ICLR 2021
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 21122112220 22---22,222 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. pp. 248-255. leee, 2009.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint 161 (.06611..244, 2016.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1rXiv:1605.09772, 2016.
Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. In International Joint Conference Artificial Intelligence, pp. 1753-1759, 2017.
Divam Gupta, Ramachandran Ramjee, Nipun Kwatra, and Muthian Sivathanu. Unsupervised cluster- ing using proudd--eeiisupprviisee learning. In International Conference on Learning Representa- lions, 2020. htttp://:////
net/
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:/503.0223 02531,22312220 2015.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-50D) 2006.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchile Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 06670, 2018.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eicchi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing augmenueetted training. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 1558-1567. JMLR. org, 2017.
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9865-9874 2019.
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embed- ding: An unsupervised and generative approach to clustering. arXiv preprint arXiv: 1e:::06111000 1.05148, 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 97509)25579- 2008.
Marina Meila and Jianbo Shi. Learning segmentation by random walks. Advances neural information processing systems, pp. 873-879, 2001.
Andrew Y Ng. Michael I Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In Advances neural information processing systems, pp. 849-856. 2002.
10

---

Published as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers..nuzziss.ccc
9015 pytorch an an- co--imperatiii style high performaeeee deep learning Library.
pdf
Uri Shaham and Roy R Lederman. Learning by coincidence: Siamese networks and common variable learning. Pattern Recognition, 74:52-63, 2018.
Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. arXiv preprint arXiv:/801.01530 1.1587,7 2018.
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(8):888- 900888990000000 2000.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Chunfeng Song, Feng Liu, Yongzhen Huang, Liang Wang. and Tieniu Tan. Auto-encoder based data clustering. In Iheroamerican Congress on Paterrn Recognition, pp. 117-124. Springer, 2013.
Yaling Tao, Kentaro Takagi, and Kouta Nakata. Rdec: Integrating regularization into deep embedded clustering for imbalanced datasets. In Asian Conference on Machine Learning, pp. 49-64, 2018.
Fei Tian, Bin Gao, Qing Cui, Enhong Chen, and Tie- Yan Liu. Learning deep representations for graph clustering. In Twenty-Fighth AAAI Conference on Actificial Intelligence, 2014.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc
Van Gool. Scan: Learning to classify images without labels. In Proceedings of the European Conference on Computer Vision, 2020.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. Journal of machine learning research, 1 11(Dec):3371-344 2010.
Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, (((4):395446 2007.
Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 15508159, 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. Joint detection and identifi- cation feature learning for person scarch. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3415-3424 2017.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In Proceedings of the 33rd International Conference on International Conference on Machine Learning- Learning-Volume Volume 48, pp. 478-487, 2016.
Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards amaans-friendl spaces: Simultaneous deep learning and clustering. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3861-3870 JMLR. org, 2017.
11

---

Published as a conference paper at ICLR 2021
APPENDICES
A DATASETS AND EXPERIMENTAL SETUP
Five datasets were used to conduct experiments: CIFAR-10 Krizhevsky et al. (2009). CIFAR- 100 Krizhevsky et al. (2009), STL-10 Coates et al. (20IT), ImageNet-10 Deng et al. (2009), and ImageNet-Dogg Deng et al. 2209). Table lists the numbers of images, number of clusters, and image sizes of these datasets. Specifically, the training and testing sets of dataset STL-110 were jointly used in our experiments. Images from the three ImageNet subsets were resized to 96 x 96 x 3.
Table 3: Image datasets used in experiments.
Dataset
Images
Clusters
Image size
CIFAR-10 Krizhevsky et 2009
50,000
10
32 x x
CIFAR- FHEAR-I0KKnzhesskk Krizhessky el al.
2009
50,000
20
32 x 32 x 3
STL-10 et al. 200
13,000
10
96 x 96 x 3
Imagenet- TO al. (2009)
13,000
10
96 x 96 x 3
Imagenet- Deng et al. 2009
19,500
15
96 x 96 x 3
We adopted ResNet He et al. (016) the neural network architecture in main experiments. For simplicity, we used ResNet18, which according to our preliminary experiments yields sufficiently high performance. The same architecture was used for all datasets except the input layer. In accordance with the experimental settings of Wu et al. (2188) the dimension of latent feature vectors was set to d = 128, and stochastic gradient descent optimizer with momentum / 0.9 used. The learning rate Ir was initialized to 0.03, then gradually scaled down after the first 600 epochs using coefficient of 0.1 every 350 epochs. The total number of epochs was
2000, and the batch size was set to B = 128. Orthogonality constraint weights for IDFO were a = 10 for CIFAR- 10 and CIFAR-100 and a 0.5 for the STL-10 and ImageNet subsets. The weight for IDFO a was set according to the orders of magnitudes of the two losses L, and Lo. For IDFD, the weight was simply fixed at In the main experiments, we set the default temperature parameter value 7 1 for ID(uuned), IDFO, and IDFD, and = 2 for IDFD.
B OPTIMAL SOLUTIONS OF CLUSTERING AND INSTANCE DISCRIMINATION
In Section 4.2.1 concluded that minimizing L, under the condition that is large can reach a clustering-friende solution. Details about the analysis and calculation was demonstrated by two-dimensional toy model as follows.
Empirically, we observe that visually similar images tend to get similar assignment probabilities. Similar images can thus be projected to close locations the latent space. This also motivated ID Wu et al. (2018). In the case of ID, similar images z, and xy yield respective highest probabilities pii and Pij. and also receive relatively high Piy and Py; values. This property can retain over the process of approximation to the optimal solution. Because instance-level discrimination tries to maximally scatter embedded features of instances over the unit sphere Wu et al. (2018), all representations are thus uniformly spread over the latent space with each representation relatively similar to its surroundings, we call this uniform case. We also consider another case that yields an optimal clustering solution where all samples from the same class are compacted to one point and & clusters are uniformly spread over the space. We call this compact case. Figure shows the representation distributions in the two cases. Because we normalize e, two-dimensional representations form circle.
In the uniform case, n representations uniformly located on circle with an angular interval of 0 = 2T /n, and the inner product between two neighboring representations is cos 8. Without loss of generality, we can start with an arbitrary point e; and orderly mark all samples as 144j. The cosine similarity between e; and Uitj can then be calculated by o 2/ cos cos Accordingly, the loss

---

Published as a conference paper at ICLR 2021
uniform
compact
y=
1.0
0.5
05
-1.0
-1.0
-0.5
0.0
05
10
-1.0
-0.5
0.0
05
10
Figure 8: Two extreme cases representation distribu-
tions over two-dimensional space. Left: uniform. Right: Figure 9: exp(cos 0/T) with differ- compact.
ent r settings.
contributed by sample in the uniform case can be calculated
exp(1/T) 2m= exp(cos m8/T) log 1 Σ exp(cosm0/7)
1 cxp(1/T)
Tunisfrm
log
(11)
Similarly, in the compact case, n/k data from the same class are exactly compacted to point and i corresponding points located on a circle an angular interval of 6' 2r/k. The inner product between an arbitrary start sample v; and the j-th sample can be calculated as of Ui+j cos 10', where I = mod n/k: The probability of assigning to the cluster with j becomes Pij exp(cos 0' /T)
Ee exp(eos /T) Accordingly, the loss contributed by sample in the compact can be calculated
exp(1/T) I exp(cos
= log exp(LI/T)
L Leompact log
n
(12)
n
exp(cos cl' /T) c
Comparing Eq. and 12
we see that the difference between Luniform and compact comes only from the denominator part of the logarithm. These are two discrete forms of the same integral I exp(cos 0 dd. Clearly, L L'niform equals Leompact when k, + +o0. We therefore need to consider only the general case where n sufficiently large and k < n.
the domain 0 E [0, [0.2.]. We can see that the curve becomes flatter as increases. A flat function f means that for an arbitrary (0, 0') pair in its domain of definition,
Figure 9 shows plot of function values exp( (0)) with different settings
have f(0) 2 f(0).. In this situation even k < n, the difference between the summations of these two discrete functions is not large. Accordingly, we can say Lecompact approximate to foriform for large т. In other words, minimizing I, approach the compact situation where same-class samples assemble and differing samples separate. Learning instance-level discrimination for clustering is therefore reasonable.
C EXTENDED EXPERIMENTS
In Section 4.2.3 we have reported some investigations of our method for practical use. Details about several important experiments are supplemented follows.
C. IMPACT OF NETWORK ARCHITECTURE
As Table shows, IDFD can be applied to various networks, and the performance gaps between IDFD and ID(urnedd) on networks like ConvNet Wu et al. 2019) and VGG16 (imonyan Zisserman (2014) are more significant than ResNet He et al. (2016). We added the feature correlation matrix of VGG16 in Figure 10 IDFD on VGGI6 obtained sparse correlations similar to the case of
ResNet18 in Figure 7 while ID on VGG16 obtained denser and stronger correlations than ResNet1 88 presumably constructing redundant features that degraded clustering. In the case of VGG16, the feature decorrelation term I.r exhibits larger effect clustering performance than that of ResNet.
2

---

Published as a conference paper at ICLR 2021
Our proposed losses work all network architectures, and expect introduce the losses to various networks that are suitable for individual problems.
IDF
IDFD
0.75
0.50
-0.25
100
100
100
00.50
-0.75
120
120
-1.00
50
75
100
125
25
50
75
100
125
25
50
75
100
125
Figure 10: Feature correlation matrix learned by VGG16 on CIFAR-10.
C.2 COMBINATION OF AUTOENCODER AND FEATURE DECORRELATION
In order to further confirm the cooperation effect of instance discrimination and feature decorrelation from the viewpoint of spectral clustering, a combination autoencoder and feature decorrelation was evaluated in terms of clustering performance. Autoencoder has been verified by datasets such as handwritten digits to be an effective method for deep clustering. In this experiment, we used ConvNet Wu al. (2019) for the autoencoder architecture and trained it the CIFAR-10 dataset. We applied k-means to representations learned from autoencoder only and autoencoder combined with feature decorrelation, which called AE and AEFD, respectively. According to our experiments, the ACC value of AE was 26.0%, and the ACC value of AEFD was 22.4%. Compared to the improvement from ID to IDFD (from 26.8% 42.0% as shown in Table 2). we see that AE cannot benefit from FD as ID. This result again indicates that FD has a deep relation with ID as we analyzed in Section
C.3 IMPACT OF DATA AUGMENTATION
For reproduction of our results and practical use, we note that data augmentation (DA) has strong impact the performance. DA is known to have impact on image classification and representation learning. Like in Wu et al. (2018), several generic and accepted techniques, such as cropping and grayscale, were used for data augmenting this work. The details of the augmentation in the original code can be linked to Wu et al. (2018). In order to investigate the impact of DA, we conducted experiments on five datasets with and without DA and compared their clustering results. Table shows the results. We can see that methods without DA suffered significant performance degradations for clustering, as well as for classification Chen et al. (2020). This reminds us not to ignore the effects of DA in practical use.
Table 4: Clustering results (%) with or without data augmentation on five datasets.
Dataset
CIFAR-10
CIFAR-100
STL-10
ImageNet- 10
ImageNet-Dog NMI
NMI
NMI
ACC
NMI
ARI
ACC
NMI
ACC
ID W/O
18.7
9.5
4.1
1488
3.7
23.6
12.7
1.9
IDFD WIO DA
23.6
12.1
6.0
16.2
11.6
4.4
24.8
17.6
8.3
37.2
23,8
15.6
15.5
5.5
2.5
ID With DA
76.6
65.7
58.3
36.7
35.7
21.9
57.1
49.0
36.8
85.8
79.1
70.5
29.4
16.0
28.5
IDED With DA
81.5
71.1
66.3
42.5
42.6
26.4
75.6
64.3
57.5
95.4
89.8
90.1
59.1
54.6
41.3
To further find out main factors affecting the performance, we also executed experiments by removing each technique used for DA. Take the example of CIFAR-10, techniques used for data augmentation include: ColorJitter, RandomResizeddCryp RandomGrayscale, and Random!HorizomullFiig All these techniques are generic and easy to be implemented. They have been integrated into general deep learning frameworks such PyTorch. According to experimental results as shown in Figure [1] we find that RandomResizedCrop. RandomGrayscale, and ColorJitter have strong effect on image clustering.
3

---

Published as a conference paper at ICLR 2021
DNCe musict
mRaaneelll il p.reeiC $1 Feskctt.......
29 US3 USM 200
Figure Effect each technique used for DA on CIFAR-IO.
For practice we also applied IDFD to our private images produced by manufacturing process. Generic DA like above were used these images. IDFD showed good performance on these images according to our experiments. This indicates that our method can be simply applied to practical images. For other types of data such as text and time series, corresponding data augmentation techniques are needed to cooperate with our method.
4


Reviews:
[{"review_id": "ZVDdaaQyBpl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed an improved deep-learning-based representation learning method that provides more efficient features for clustering analysis. \n(1) According to the comparison experiments on several widely used datasets,  the integration of a softmax-formulated orthogonal constraint is able to provide more stable latent feature representation. \n(2) As far as know, the widely-used deep clustering methods used to alternatively optimize the feature representation model parameters and update the anchors that provided by clustering method such as k-means, I am wondering if the proposed method in this study could integrate the two steps in a real end-to-end fashion. \n(3) I was deeply impressed by the far above state-of-the-art values of evaluation metric of this proposed representation learning method. Although the authors provide some distribution illustrations of latent features on CIFAR-10 dataset, what about the visualization on the ImageNet-10? Besides, adding some 'real' visualization results existing in the original image space rather than the latent space could help to illustrate if the proposed method could mine visually meaningful concepts from the view of visual contents. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "XI8l6irhAbr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "One of the main contributions is this idea of feature decorrelation where they encourage the representation features to be independent / orthogonal.\nThe other is instance discrimination. This aims to capture the similarity between individual data points. Both of these are interesting contributions to the field of 'deep clustering'.\n\n\nBesides the stated contributions, I thought there were a number of other positive aspects of this. \n\tA) I thought that the spectral clustering connection was nice and I am glad the authors included it.\n\tB) The evaluation is fairly detailed. I particularly appreciate the fact that the authors used datasets that are somewhat larger than often used in the literature (MNIST and CIFAR-10 vs CIFAR-100 and ImageNet-10). The inclusion of the study of the temperature parameter also helped clarify a few questions I had when reading it. \n\tC) Finally, the evaluation clearly shows the benefit of their contributions in terms of performance. \n\t\nThere are a number of questions I have with the work as is.\n\tA) Given the two methods proposed,  IDFO, IDFD, neither of which outperforms the other on all tasks, and given this is unsupervised learning, how does one know which method to use?\n\tB) Why was the alpha parameter set to 1 for IDFD? How does one know what to set this to for different datasets? If it's always 1, why is it included at all?  This is particularly important to understand in unsupervised settings. \n\tC) The impact of data augmentation is discussed in the supplementary but this is stated as being extremely important to the performance of the model. It is unclear to me whether the results in the main text include the augmentation process?  If so, then given this, I think it should be stated in the main text as it has an effect on both instance discrimination and feature decorrelation considering the addition of augmented images.  The results in supplementary Table 4 include KNN and don't match up with the main results in the main text which further confused me.\n\tD) I was left wondering how well this method works on non-image data? Other works in the literature have explored this. \n\tE) For Fig. 2 is this ACC calculated on the validation set or test set?\n\tF) What were the effects of resizing the ImageNet images? Can this model handle larger images, and if so, how does this effect performance?\n\nMinor\n\tA) References are badly formatted in Table 3.\n\t\n\nOverall, my questions above notwithstanding, I think this is an interesting contribution which shows the benefit of instance discrimination and feature decorrelation for deep clustering.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DOs_-Cjr11r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a clustering-friendly representation learning method using instance discrimination and feature decorrelation. Instance discrimination loss and feature decorrelation loss are combined to optimize the network. The paper is well qritten and experimental results are good. I have some questions about this paper:\n1. There is no ablation analysis about the two loss terms in Eq.(6). What about the contributions of the two loss terms? \n2. What is the motivation of Eq.(3)? I.e., why the \"=\" holds between the second and third expressions?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received mostly positive reviews. The reviewers praised the strong performance when compared with previous work.
Also, the evaluation clearly shows the benefit of the proposed contributions in terms of performance.
Most concerns raised by reviewers were properly addressed in the rebuttal.

Lack of comparison to several previous works has been noted in a comment, but the authors clarified this concern, stating that the current work is a “large deviation from prior works”. The authors promised to include the missing references into the comparison.

Given the reviews, comments, and author's answers, I suggest acceptance.
Decision: Accept (Poster)