Subspace Clustering via Robust Self-Supervised Convolutional Neural Network
Authors: Dario Sitnik
Ivica Kopriva
Venue:  (2021)
Keywords: deep subspace clustering
convolutional neural networks
self-supervised learning
correntropy
generalization
robustness

Abstract: Subspace clustering (SC) approaches based on the self-representation model achieved encouraging performance when compared with the clustering algorithms that rely on proximity measures between data points. However, they still face serious limitations in real-world applications. One limitation relates to the linearity assumption of the self-representation model. The reason is that, usually, samples lie in non-linear manifolds, e.g. face images acquired under non-uniform illumination and different poses. Another limitation relates to errors that can be random or sample-specific (outliers), whereas in real-world applications their origin is mostly unknown. Furthermore, the majority of existing algorithms use external clustering validation methods to measure clustering quality, and that requires access to ground-truth (GT) labels. Hence, it is practically important to develop a deep SC method that jointly learns self-expressive (SE) feature representation and handles data corruptions of unknown origin, and estimates clustering error using the internal clustering validation method, i.e. without access to the GT. Mostly, the base of the recently developed deep SC networks is convolutional autoencoder. It is an end-to-end fully convolutional network that is based on the minimization of the reconstruction error. Together, the autoencoder and an additional SE module are forming a Deep SC network (DSCNet). Hence, the total loss function of DSCNet is composed of reconstruction loss and SE loss. That is, during the learning process, the quality of clustering is not taken into account. Self-supervised convolutional SC network ($S^2$ConvSCN) addressed this issue through the addition of a fully connected layer (FC) module and a spectral clustering module that, respectively, generate soft- and pseudo-labels.

While inheriting the architecture of the $S^2$ConvSCN, this paper proposes a robust SE loss and an early self-stopping criterion for training. Robustness to arbitrary (unknown) types of data corruptions is achieved by using the correntropy induced metric (CIM) of the error of the SE model. By mapping the input data space to a reproducible kernel Hilbert space (RKHS), correntropy defines an $\ell_2$ distance in RKHS and creates nonlinear distance measure in the original input data space. Hence, correntropy is the optimal measure for error with the arbitrary non-Gaussian distribution as opposed to the MSE that implies Gaussian distribution of errors. Thus, because it is estimated directly from data, CIM can handle data corruption of unknown origin. As opposed to the DSCNet and $S^2$ConvSCN, the self-stopping criterion of the proposed algorithm is achieved by reaching a plateau in the change of loss value.

Although the architecture of $S^2$ConvSCN contains the FC module, which is capable of handling out-of-sample data by using a softmax classifier, its performance has not been tested on unseen data. The importance of the FC module is, however, emphasized through self-supervision. In a truly unsupervised setting, pseudo-labels generated from the spectral clustering module guide the learning process through the induced classification error at the end of the FC module. They also enhance the goodness of the self-representation matrix through penalizing incorrectly connected elements. Adding these two loss functions to the total loss makes pseudo-labels an especially important component in label-constrained applications. Such dual self-supervision, in theory, enables deep networks to learn representation from available unlabeled data and to use the small number of labeled data to validate the trained model.

In addition to the main contributions of the paper, this study has three side-contributions. It aimed to set up a more transparent way for the proper performance estimation of deep SC models and to integrate block-diagonal regularization into the gradient descent learning process. The majority of SC studies optimize hyperparameters using external clustering validation methodology, whereas the same labels are used for tuning hyperparameters and evaluating the final clustering performance. Furthermore, models like $S^2$ConvSCN suffer from an early commitment problem as they depend on weight-transfer from pretrained models that have “seen” the GT already (e.g. DSCNet transferred to $S^2$ConvSCN). From the machine learning perspective, data leakage and hyperparameter tuning based on the test GT are unacceptable. Measured clustering performances of $S^2$ConvSCN and DSCNet on the out-of-sample (unseen) data, thus, differ significantly from the optimistic one presented in the original papers. Also, post-processing of self-representation matrix is reduced to a significant extent.  Robust $S^2$ConvSCN outperforms its baseline version by a significant amount for both seen and unseen data on four well-known datasets. To the best of our knowledge, such an ablation study on unseen data has not been reported previously in SC studies.

Under review as a conference paper at ICLR 2021
SUBSPACE CLUSTERING
VIA ROBUST SELF-
SUPERVISED CONVOLUTIONAL NEURAL NETWORK
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep subspace clustering (SC) algorithms recently gained attention due their ability to successfully handle nonlinearities in data. However, the insufficient capability of existing SC methods to deal with data corruption of unknown (ar- bitrary) origin hinders their generalization ability and capability to address real- world data clustering problems. This paper proposes the robust formulation of the self-supervised convolutional subspace clustering network (S CSnvSNN) that in- corporates the fully connected (FC) layer and, with an additional spectral cluster- ing module, is capable of estimating the clustering error without using the ground truth. Robustness to data corruptions is achieved by using the correntropy induced metric (CIM) of the error that also enhanced the generalization capability of the network. The experimental finding showed that CIM reduces sensitivity to over- fitting during the learning process and yields better clustering results. In a truly unsupervised training environment, Robust S*ConvSCN outperforms its baseline version by a sigmificant amount for both seen and unseen data on four well-known datasets.
INTRODUCTION
Subspace clustering approaches have achieved encouraging performance when compared with the clustering algorithms that rely on proximity measures between data points. The main idea behind the subspace model is that the data can be drawn from low Oow-dimensional subspaces which are embedded in nigh-ddimnnsionl ambient space (Lodhi & Bajwal 2018). Grouping such data associated with re- spective subspaces is known as the subspace clustering (Vidal 1011). That is, each low-dimensional subspace corresponds to a class or category. Up to now, two main approaches for recovering low- dimensional subspaces are developed: models that are based on the self- self-representation property. and non-linear generalization of subspace clustering called union of subspaces (UoS) (Lodhi & Bajwal 2018 Lu & Do 2008 Wu & Bajwa, 2014 2015). UoS algorithms are out of the scope of this work. Self-representation subspace clustering is achieved two steps: (i) learning representation matrix from data X and building corresponding affinity matrix A |C| |C (ii) clustering the data into clusters by grouping the cigenvectors of the graph Laplacian matrix that correspond with the lead- ing l eigenvalues. This second step is known spectral clustering (Ng al. 2002; Von Luxburg. 2007]. Owning to the presumed subspace structure, the data points obey the sel self-expressiveness or self-representation property (Elhamifar & Vidal 2013) Peng et al., 2016b] Liu et al.] 2012; Li & Vidal 2016 Favaro et al.) 2011). In other words, each data point can be represented as a linear combination of other points a dataset: X-XC.
The self-representation approach is facing serious limitations regarding real-world datasets. One limitation relates the linearity assumption because in a wide range of applications samples lie in nonlinear subspaces, c.g. face images acquired under non-uniform illumination and different poses Ji et al., (017). Standard practice for handling data from nonlinear manifolds to use the kernel trick on samples mapped implicitly into high dimensional space. Therein, samples better conform to linear subspaces (Patel et al.) 20133] Patel & Vidal 2014; Xiao al.) 2015) Brbić & Kopriva, appropriate kernel function for given data set is quite difficult task (Zhang et al. 2019b). The second limitation of existing deep SC methods relates to their assumption that the origin of data corrupiion is known, in which the proper error model can be employed. In worl applications origin of data corruption is unknown. That can severely harm the algorithm's learnnng process if the non-robust loss function is used. Furthermore, validation
2018), However, identifying

---

Under review as a conference paper at ICLR 2021
(i.e. stopping of the learning process) in most of the deep SC methods often requires access to the ground-truth labels. That stands for violation of the basic principle of unsupervised machine learning and yields the overly-optimistic results. Dataset size is also a limitation when it comes to memory requirements. Since the self-representation subspace clustering is based on building the affinity matrix, memory complexity increases as the square of the dataset size. However, the latter limitation is not in the main focus of this work.
Motivated by the exceptional ability of deep neural networks to capture complex underlying struc- tures of data and learn discriminative features for clustering (Hinton & Salakhutdinov, 2006 Dilok-
thanakul al. 2016 Ghasedi Dizaji et al. 2017 Tian al. 2014; Xie et al. (0166) deep subspace
clustering approaches emerged recently (Ji et al. 2017; Abavisani & Patel 220 Peng et al.
2016a
Yang et al. 2019; /Zhou et al.] 2018 Ji et al. 2019b; Peng et al.] 2018 2017 7 Zhou et al. 2019
Zhang et al. |019a| (heeiKKddissaaarrr et al.] 2020). In particular, it is shown that convolutional neural networks (CNNs), when applied to images of different classes, can learn features that lie in a UoS [Lzaama al. 2018 Mostly, the base of the recently developed deep subspace-clusteringg networks is convolutional autoencoder. It is an end-to-end fully convolutional network that is based on the minimization of the reconstruction error. Together, the autoencoder and additional exfeexxprsssoon (SE) module are forming a Deep subspace clustering network (DSCNet) (Ji et al.) 2017). Hence, the total loss function of DSCNet is composed of reconstruction loss and SE model loss. That is, during the learning process the clustering quality is not taken into account. Self-supervised convolutional SC network (S CCN) (hang et al. 2019a) addressed this issue through the addition of a fully connected layer (FC) module and a spectral clustering module that, respectively, generate soft- and pseudo-labels. Dual self-supervision achieved by forcing these two modules to converge towards consensus. Related accumulated loss, therefore, participates in enhancing the self- self-representation matrix and the quality of features extracted in the encoder layer. The architecture of SoCnSSCN has possibility of direct classification once the learning process is completed. A trained encoder and the FC module can make a new network that can directly classify unseen data, also known as an out-of-sample problem. However, while this network can be validated and compared with other algorithms on a separate data set, such an ablation study was not completed. Furthermore, the main disadvantage of the DSCNet architecture, and indirectly S S*ConvSCN, is that the network training is stopped when the accuracy is highest (Ji et al. 2019a). First, it is a direct violation of the unsuper- vised learning principle the ground-truth labels exposed. Second, the reported performance (hhng et al. 2019a Ji et al. 2017) is overly-optimistic and can not be compared to other algo- rithms. Also, as mentioned (Haettele et al.) 2020), most self-expressive based deep subspace clustering models suffer from the need of post-processing the self- self-representation matrix. Compared to the baseline model, we significantly reduced the post-processing while maintaining the frese-free matrix.
Mentioned research problems led three main contributions of proposed Robust S SCCoNSCN:
robustness to errors of the unknown (arbitrary) origin is achieved by using the correntropy induced metric (CIM) in the self-expression loss,
the network is trained using the early-stopping method while monitoring only the accumu- lated loss,
thanks correntropy based loss function the training process is less sensitive to data cor- ruptions which enables the network to generalize hetter.
This study has, also, three side-contributions:
the performance of models is estimated using the unseen (out-of-sample) data,
block-diagonal regularization of self-representation matrix is integrated into the gradient descent learning process,
post-proesssing of self-representation matrix is reduced to significant extent.
A complete head to head comparison of the baseline S²ConvSCN model and our robust approach can be seen in Figure
2

---

Under review as a conference paper at ICLR 2021
Self-expressive layer
Early stopping
Evaluation
Robustness
Highest accuracy requires access to mifir lntwits
On in-se lobels used for yyyerpa anreler upamication
adcitive white Geussion noise
approach:
MSE &
CIM f2 +
Our approach:
[[C-A]P AIP
bessee an label free relotve eitaaion
presiously
out out-ol-sammen sa---llalpplle labels
model
To data corruption of unknown (orbitrary) origin
Figure 1: Key differences between the baseline SCConVSCN and proposed robust version. Unlike the baseline SoConySCN, robust version learns in truly unsupervised (label-fred manner. The last regularization in the self-expressive layer stands for symmetric loss introduced in Equation (IO).
BACKGROUND AND RELATED WORK
2.1 MAIN NOTATIONS DEFINITIONS
Throughout this paper, matrices are represented with bold capital symbols and vectors with bold lower-case symbols. X E RdxN represents data matrix comprised of N data samples with dimen-
sionality d. (H m())
represent feature maps produced at the output of layer 1-1 Thus, H X
and (4) X. X represents the output of the decoder and represents number of convolutional
(1) m
layers in the autoencoder. ww; (1) stand for set of filters with associated biases { {b.² :-11
1-1 that form a convolutional layer = 1,...,,, L. 2n = h (1/2) (t)... mLl/2) (+) E dR stands for (1/2)
feature vector comprised of vectorized and concatenated feature maps, with d extracted features, in the top layer L. (encoder output) representing input sample x,, n = =,...., N. C RNXN stands for representation matrix in self-ex expressvee model Z = ZC. A = [C| |CT] is the affinity matrix and L D AD is corresponding graph Laplacian matrix. D is diagonal degree matrix such that D;; Eド= Aij- | =V V =2, is the Frobenius norm of matrix X. (x(x) ||x| (E 3/,, x S the é, norm x. lo(x) = ||x||o = #1x, + 0, i I,...,d); where # denotes the cardinality function, is εa quasi norm of x. The Sp. 0 < p < 1, Schatten norms of matrix X are defined as the corresponding l, norms the vector singular values of X, i.. (X) |Xo(X)]l, ll where (X) stands for the vector of singular values of X. Depending the context, 0 represents matrix/vector of all zeros and 1 represents matrix/vector of all ones.
2
Grouping the data according to the linear subspaces they are drawn from is known as subspace clustering Vidal 2011). The problem is formally defined in:
Definition 1. Let X in RR - {S.). of dimensions d, min [d, for I. Let X, be collection of N; samples drawn from subspace S,, N > N,. The problem of subspace clustering is to X.....,
X²] be set of sample vectors drawn from a union of k subspaces
segment samples into the subspaces they are drawn from. Throughout this paper, as it is the case in the majority of other papers, we have assumed that number of clusters is known a priori.
2.2 APPROACHES TO SUBSPACE CLUSTERING
Usually, processes that operate in different modes generate data in real-world scenarios. Each mode models such data as lying on a subspace, while the whole process, thus, generates data lying on a union of subspaces (UoS) (Lodhi & Bajwa 2018). The alternative to the UoS model is the self- representation based subspace model. It implies that every sample from the dataset can rep- resented linear combination of other samples from the same cluster. While shallow models
3

---

Under review as a conference paper at ICLR 2021
directly optimize such self-represnntttioon matrix, their deep counterparts train the whole network to better extract features from the raw data and achieve representation linearity.
Many approaches to deep subspace clustering are based the introduction of the self-representation in the feature space (Abavisani & Patel 2018: Ji et al. 2017: Peng ct al. 2016a Zhou et al.] 2018
2019 Zhang al., 2019a Kheirandishfard al. 2020, Zhang et al. 2020p. However, one weakness of self-expressive deep subspace clustering models is that their perfomance mainly depends on the self-representation matrix. Thus, elimination of the noise is done by post-processing (Haeffele al, 2020). It appears in many cases that from the final performance point of view the post-processing matters more than depth the network. By the virtue of rerf-sepresnnttiion property, improvements of the shallow subspace clustering methods are of direct relevance to their deep counterparts. The subspace clustering task accomplished through (i) learning the representation matrix C from data X, and (ii) clustering the data into & clusters by grouping the eigenvectors of the graph Laplacian matrix L that correspond with the leading eigenvalues. This second step known as spectral clustering (Ng et al. 2002 Von Luxburg 2007). Low-rank (Liu et al.) 2012: Favaro et al. 2011 and sparse models (Elhamifar & Vidal 2013) are one of the commonly used algorithms to solve SC clustering problem. They aim to learn the low-rank and sparse representation matrix by solving the following optimization problem (Li & Vidal 2016):
min x ||lls + s.t. Z = ZC, diag( C) = c
(I)
where À and nonnegative regularization constants. If number of layers L 0 problem is related to shallow subspace clustering. Constraint dCag(C) = O is necessary to prevent sparseness regularized optimization algorithms converge towards trivial solution where each data point rep- resents itself. This constraint is not necessary for problem constrained only by low-rank. When data samples are contaminated with additive white Gaussian noise (AWGN) problem (T) becomes:
min C
E + AIC
+ |[CI, s.t. diag(C) = 0
where E stands for the modelling error (noise): = Z ZC.
(3)
Alternatively, square of the Frobenius norm of C is used for regularization (Lu al. 2012):
min E + +
(4)
Objective (4 is used also in the self-expression module of the S*ConvSCN in (Zhang et al. 201 As seen from ( and (4) the MSE measure for discrepancy between Z and its self-ropresentation ZC is justified only for the contamination by the AWGN. For sample-specific corruptions (outiers) the proper norm is ||E|lg,1 while for large random corruptions the proper choice is [E]]; (Liu et al. 2012). However, errors in real world data have different origins and magnitude and may not follow specific probabilistic model. Sometimes, it is hard know the true origin of corruptions present in data. Thus, to obtain method robust to arbitrary corruption we propose to introduce the CIM of the error. Rationale behind introduction of any regularization on of block-diagonality. Even though |Clls, and ||C||p 0 S S 1 in principle satisfy the enforced block-diagonality condition, their approximation of the BD structure of C is indirect (Lu et al. 2018). Hence, for comparison, this study proposes introduction of loss function with gradient- BD regularization representation matrix C.
to reflect its structural property
2.3 BASELINE MODEL DESCRIPTION
The base of the DSCNet (Ji et al.) 2017) is fully convolutional autoencoder. Following the flat- tened latent code of the autoencoder is an additional self-expression module which forms the full architecture of the DSCNet. While it is producing representation matrix C, the clustering algorithm can only cluster the in-sample data. Thus, clustering error is not part of the learning process and, therefore, does not influence the quality of learned features and representation matrix. According
4

---

Under review as a conference paper at ICLR 2021
the training procedure given in (Ji et al. 2019a), the performance of the DSCNet strongly depends on observing true labels every epoch. Although presented as an unsupervised method, DSCNet is using the minimum of clustering eπror an early stopping criterion which is in contradiction with the principles unsupervised learning and arguably leads to the overfitting and overly-optimistic performance estimation. As be in Figure [ much better representation matrix C can be learned from a model by ignoring loss and stopping when the accuracy is highest. Another prob- lematic part is that the DSCNet algorithm post-processing matrix with hyper-parameter optimized using the ground truth labels. Post-processing of C, as done in DSCNet (Ji et al. 201 9a), has three steps. First, is thresholded by keeping only d largest, in terms of magnitude, elements in row. The accepted number of largest elements depends on their sum Ss. If the sum Ss exceeds EE Cjú, i E {I,...,Ay N). where À stands for empirically set thresholding constant and N represents the number elements in a row, other elements (not included in the sum Sg) are set to zero. This first step is skipped in robust model. Second, knowing the dimensionality of the dataset d, after SVD decomposition of thresholded representation matrix C only d largest eigenvalues with related eigenvectors were kept meaning the remaining eigenvectors span the noise subspace. Third, resulting values of Laplacian matrix are further suppressed by an exponentiation. The reconstructed representation matrix serves an input to the spectral clustering algorithm which produces pseudo- labels.
Furthermore, DSCNet architecture is base for the SCConvSCN model Zhang al. is why S SCCNNSCCN suffers from the majority of problems as DSCNet.
2019a). which
4.25
log (loss)
0.9
ACC
4.00
0.8
3.75
0.7
log(loss)
3,50
0.6
3.25
0.5
0.4
0.3
2.75
0.2
2.50
0.1
0
20 40
60
80
100
120
140
Epoch
Figure 2: The figure illustrates the change of accuracy and loss over epochs for the original DSCNet model with MSE in the self-expressive layer. Model was trained and tested on full COIL20 dataset with & regularization of representation matrix C (see Equation (6)). It can be seen that accuracy is oscillating even though the loss is decreasing. Thus, selecting the model requires access to ground truth labels and makes unfair practice.
A novelty that is presented in S2ConvSCN is an FC layer, attached to the latent code, that is up- grading the DSCNet model. On its output, FC has softmax which can directly classify the input data. Upon finishing the learning procedure, the encoder and FC layer form a new model that can be used for evaluating the learning procedure the independent unseen set. While authors of the S*ConySCN model (Zhang et al., 2019a) provide a tool for dealing with the out-of-sample data, the actual performance on unseen data has not been tested. The second novelty of S*ConvSCN is training the FC layer from pseudo-labels. When training the network, the spectral clustering module clusters the data according the affinity matrix every epochs and updates the pseudo-labels. is important to note that the mentioned affinity matrix is constructed from the matrix C learned in the self-expression layer and it changing every epoch during the training. Pseudo-labels assigned to the data are used in two ways. The first is to train the FC layer for soft classification, and the second is to suppress the matrix values for samples that do not belong to the same cluster. Both ways are known as self-supervision. Keeping in mind that: (i) the S*ConNSCN is upgraded DSCNet model, (ii) it is using the pretrained DSCNet parts of the network, and (iii) it has not been tested on the in- dependent dataset, it is reasonable conclude that the label-leakage also appeared in S S2ConVSCN. Regardless of having the self-supervision modules, the first pseudo-labels generated from DSCNet contain leaked knowledge about the group affiliations of the data.
5

---

Under review as a conference paper at ICLR 2021
Authors of both (Ji al. 2017) and (Zhang al. 2019a) suggest that converge. parts of the model should be pretrained before training the whole model. However, there is no guarantee that the model will keep converging after attaching final layers modules. For example, if pretrained DSCNet yields reasonably good pseudo-lahels, it is possible that after attaching the FC layer and performing self-supervision in C*ConvSC,, the matrix C will get worse and the whole model will diverge from the optimal solution. Thus, it is important to tune constants associated with the loss functions of the mentioned modules. As the SoConvSCN has many loss functions regarding different layers or modules, a smaller learning rate could be beneficial in the sense that the overall loss can reach better minimum in the error space. However, decreasing the learning rate on plateau impacts the less-contributing losses. That can spoil the goodness of representation matrix C while the training process tries reach the loss minimum.
ROBUST SELF-SUPERVISED CONVOLUTIONAL SUBSPACE CLUSTERING NETWORK
Motivated by discussion in previous section, we propose two new objective functions LCTM for the self-expression module of the S ConnSCN architecture:
min CTM + ||
min CIM2 (E) + ||C||2
(6)
where is defined in Eq. (3) [C][j]] denotes BD regularization and ||C||, denotes & regular-
ization of representation matrix represents a trade-off constant. The CIM loss is defined in Appendix A.2 Objectives (5) and 6 ensure smooth decrease of the loss function that enables the use of label-free stopping criterion. For the sake of completeness, other loss functions of Robust S'ConvSCN are stated. Auto-encoder reconstruction loss, where X represents the input data and X represents the output of the decoder is defined as:
CEEC = 2N
X ==1
x1,12
X
2.V
Pseudo-labels form a matrix Q where element 9s is set to if samples i and j have the same pseudo-label. Otherwise, Rÿ is set to O. When Q is calculated, ||C loss is defined as:
||q 9 = |Cll
Lcq
Σ 2.j
Ciyl
where qi and qj represent one-hot encoded pseudo-labels for i-th and j-th sample. |cyl stands for the absolute representation value for the corresponding samples in the representation matrix C. Cross-entropy and center loss of FC layer is:
"mn( In(1 Sa.), LONT 1 N Σ y, 1 LCE N
1
(9)
where y/ stands for a softmax normalized output and yj represents the logits of the FC layer for j-th sample. f represents a centroid of cluster и taken from the spectral clustering output for given sample j (Zhang al. 2019a). Additionally, in order to yield stable and unique solution, representation matrix C is forced to be symmetrical (Lu et al. 2018). A novelty introduced in this robust version of the network is symmetric loss which is defined as:
1 Σ 2
CSYM M
cc a IC-All
(10)

---

Under review as a conference paper at ICLR 2021
where A represents symmetric affinity matrix. Every loss function has assigned trade-off regu- larization constant A1 to As to regulate its importance. Thus, the total loss function, with novelties bolded, is defined follows:
Cr LREC + XLCIM X2ccq + AsCCE + A&CONT + AsCSYM-
(11)
Furthermore, to address the overfitting issue, we present results both on the train sets and on the independent test sets. The latter will give better insights into the capability of used algorithms. Also, matrix post-processing is reduced by skipping the first out of three steps described in Sub- section 2.3 A more informative illustration of the complete architecture (shared by the baseline S*ConvSCN and robust model) is shown in Figure
FC
--Joct+ |
Z
zc
[L/21
MSE loss (asseline)
CIM loss Viouust
(L/242)
H
(1))
H(L-1)
=X)
HIL
Figure 3: Self-supervised robust (and baseline) S'ConvSCN architecture showing all layers and modules. Encoder and decoder layers are denoted by H()) to HLL) Together, flattened encoder features Z and representation matrix weights C construct self-expression layer ZC where optimal C learned. Spectral clustering (SC) module receives affinity matrix A as an input. Pseudo-labels from SC module serve target labels for the FC-softmax classifier. The matrix Q is used regularizer for the representation matrix C to learn a better representation.
EXPERIMENTAL RESULTS
In this section, we compare the clustering performance of Robust S*ConvSCN with state-of-the-ar self-supervised 2 S2ConNSCN model on four well-known datasets. Because such comparison would be unfair to the Robust S²ConvSCN model, performance is not compared to the performances of shallow and recent deep subspace clustering methods reported the literature. Shallow models use labels to optimize hyperparameters and then evaluate the performance based on these labels. Our approach used only the in-sample labels for enterparameterttunnngg while out-of-sample labels were kept for evaluation. A worth-noting alternative approach of hyperparameter optimization that does not require labels was presented in (Lipor & Balzano| 2020). However, their approach is not based raw data but rather extracting features using scattering network. Regarding the label-leakage issue from pretrained parts of the network (see Figure 2 this study avoids this problem by learning from seratch within defined standards. The performance is evaluated in term of accuracy (Acc):

---

Under review as a conference paper at ICLR 2021
Ac = max felly
Σ { = rs
(12)
N
where II stands for the permutation space of [k] defined as all possible orderings of the k-element set {1, .1.2.....k. kj. We compare the performance of Robust ConnSSC with state-of-the-art self- supervised deep subspace clustering algorithm (Zhang et al. 2019a). ADAM optimizer (Kingma & (014)) an extension to stochastic gradient descent used the proposed learning procedure, For full implementation details, see Appendix B Hyperparameter settings and regularization constants
are shown Tables and in Appendix
4.1 COMPARISON ON FULL DATASETS
As previously discussed, learning and evaluating on the same data yields an overly optimistic esti- mation of the model's performance. Due the fair comparison, model to the different versions of the SCConSCC model in the same manner as reported in (Zhang al. 2019a). However, we did not use accuracy as stopping criterion. Instead, fixed number of epochs, learning rate decay, and early stopping based on loss decrease only were used. Same settings are used for the baseline and robust version (see Table 3 in Appendix B}. Thus, the obtained results differ significantly from (Zhang et al. 201 report. For S ConvSCN, optimal regularization constants were transferred from (Zhang et al. 2019a). In Table a comparison
implemented and compared our
S'ConvSCN and Robust SoConVSCN with different C matrix regularization strategies can be seen. When comparing Figure 2 and Figure 4 can be seen that CIM prevents overtraining of the network and that loss can be used for early stopping criterion.
Table 1: Accuracy comparison of CoConSCC and Robust S*ConNSCN trained and tested full COIL20, COIL100, and Extended YaleB datasets. BD and L2 stand for block diagonal and &g regularization of representation matrix, respectively.
COIL20
COIL100
EYaleB
ConnSSCN + BD
0.81111
0.30375
0.56867
S*ConvSCN + L2 (Zhang al. (0119a)
0.63333
0.55319
0.75247
RoSCCn + BD
0.76250
0.50528
0.41159
R.S CoSySON + L.2
0.88403
0.68805
0.81661
5.0
0.9
0.8
4.5
0.7
(og(loss)
0.6
4.0
log(loss)
0.5
ACC
3.5
0.4
0.3
3.0
0.2
0.1
20
40
60
80
100
120
140
Epoch
Figure 4: The figure illustrates the change of accuracy and loss over epochs for Robust DSCNet model with CIM in the self-expressive model. In comparison with Figure 2 it can be seen that Robust DSCNet can use the change in loss CIM prevents overfitting.
8

---

Under review as a conference paper at ICLR 2021
4.2 CLASSIFICATION PERFORMANCE ON UNSEEN DATA
Having several independent observations is crucial for the model's performance validation. Thus, according Table B in Appendix B datasets were split to stratified folds. For MNIST, there is enough data, each fold was additionally divided into a train and test set, 70% and 30%, respectively. For other datasets, the remaining folds were all used for testing as these datasets have very limited number of data per label, i.c. additional splitting as for MNIST could not be possible. In the context of (Li et al,, 2020), the obtained results Table support the claim that the network is robust to noise when early-stopping methods are applied.
Table 2: Accuracy comparison of Connn and Robust SCConvSCN trained and tested on in- dependent folds for each dataset. BD and L2 stand for block diagonal and Eg regularization of representation matrix, respectively. The best mean accuracy for each dataset is bolded.
MNIST
COIL20
COIL100
EaaleB
CoCnNSCN BD
mean
0.18721
0.71686
0.33762
0.49187
stddev
0.06258
0.07814
0.04736
0.04261
SoConvSCN L2 Zhang et al. 2019a
mean
0.12659
0.67486
0.52099
0.63735
stddev
0,02569
0.05917
0.02295
0,07062
mean 0.50417
0.79824
0.44882
0,50017
R.S + BD
stddev 0.05329
0.02984
0.03845
0.02695
R.S ConySCN + 1.2
mean
0.42979
0.82547
0.55461
0.75239
stddev
0.04878
0.02537
0.01454
0,03346
5 DISCUSSION AND CONCLUSION
As it could be seen in Table 2 Robust SCConvSCN outperforms S²ConvSCC on all datasets re- gardless of regularization imposed on representation matrix C. Moreover, in the case of the MNIST dataset, Robust S SCCNnVSCN with BD regularization leads to the best results. The CIM loss, indeed, handles the corruptions in the data better than MSE, especially the possible amplification of corru-- tions due to their propagation through deep layers. The noise robustness can be explained by Chen al. (0166) where is shown that learning using the correntropy-loss function generalizes better. However, training of deep learning algorithms is an especially time-consuming task. Thus, settings in Appendix Table JableBaand and regularization constants could possibly be improved further testing. As discussed in earlier sections, we assume that (Zhang et al. 2019a) approach leads to overly op- timistic estimation of the model's performance. Nevertheless, Robust S C*ConySCN outperformed CCC showing its superiority.
Using datasets for which the dimensionality is unknown could be a challenging task. Without that information, it is difficult to post-process the representation matrix to eliminate the noise (Haeffele et al. 2020). Also, there is a problem with pseudo-labels refinement frequency. If it oceurs too often it could lead to divergence of the whole model. If the refinement occurs too rarely, the model could get stuck in the poor minimum. Finding a more robust pscudo-labe, refinement strategy and elimination of the need for the prior knowledge of d is still open question. Also, towards identifying which loss has higher significance during the training is needed as there are many non-equally important loss functions in Robust S*ConNSCN, In sense of memory constraints, mini- batch training of deep subspace clustering models could cross the barrier of memory complexity and offer more approachable learning.
research
To conclude, among finding a more robust model with better early stopping criterion, this ablation study aimed set up new, more transparent way for the evaluation of deep subspace clustering models. As baseline algorithms experience early commitment problem, i.c. they depend on weight- transfer from pretrained models, true labels are leaked during the training process. For that, we propose an evaluation of the algorithm on independent data to have a proper estimate of model performance, As can be seen from Table [ measured performances of S"ConvSCN significantly differ from the optimistic one presented in (Zhang et al.) 2019a). The combination of gradient- based learning with CIM loss and early-stopping strategy (Chen et al. 2016 Li et al.) 2020) did, indeed, improve the robustness the unknown errors in data. Additionally, the presented model which incorporates label-free learning and the robust correntropy loss can easily be extended to multi-modal and multi-view data. We aim address that in our further research.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz. Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens. Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https: /wwp://www..... tensorferrrrro low org/ Software available from tensorflow.org.
Mahdi Abavisani and Vishal M Patel. Deep multimodal subspace clustering networks. IEEE Journal of Selected Topics in Signal Processing, 12(6):1601-1614, 2018.
Maria Brbić and Ivica Kopriva. Multi-view low-rank sparse subspace clustering, Pattern Recogni- tion, 73:247-258, 2018.
Liangjun Chen, Hua Qu Jihong Zhao, Badong Chen, and Jose C Principe. Efficient and robust deep learning with correntropy-induced loss function. Neural Computing and Applications, 27 (4):1019-1031, 2016.
François Chollet et al. Keras. httts:://eras. 2015.
Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders, arXiv preprint arXiv:1611.026445 2016.
Ehsan Elhamifar and Rene Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(11):2755-281,, 2013.
Paolo Favaro, René Vidal, and Avinash Ravichandran. closed form solution to robust subspace estimation and clustering. In IEEE Conference Computer Vision and Pattern Recognition, pp-
1801-1807. IEEE, 2011.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5736-5745, 2017.
Benjamin D. Haeffele, Chong You, and René Vidal. A critique of self-expressive deep subspace clustering, 2020.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science. 313(5786):504 507, 2006.
Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. Deep subspace clustering networks. In Advances Neural Information Processing Systems, pp. 24 24-33, 2017.
Pan Ji, Tong Zhang. Hongdong Li, Mathieu Salzmann, and Ian Reid. Depp-subspace-cusserig ltttps://githbb
networks, commit: 35662233444T77770066666444441413 pan 1ia9911900/0 Sesi1199/Deeeee subspace- networkk 2019a.
com/
Xu Ji, João Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In Proceedings of the IEEE International Conference Computer Vision, pp. 9865-9874, 2019b.
Mohsen Kheirandishfard, Fariba Zohrizadeh, and Farhad Kamangar. Multi-level representation learning for deep subspace clustering. The IEEE Winter Conference on Applications of Com- puter Vision, pp. 2039-2043 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
10

---

Under review as a conference paper at ICLR 2021
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http://yamn.leca www.uanm.eeu....mmm www.ee..w..../////e/...o./ com/exxb//nnist, 2, 2010.
Kuang-Chih Lee, Jeffrey Ho, and David J Kriegman. Acquiring linear subspaces for face recognition under variable lighting. IEEE Transactions Pattern Analysis and Machine Intelligence, 27(5): 684-698, 2005.
José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding- a plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8109-8118 2018.
Chun-Guang Li and Rene Vidal. A structured sparse plus structured low-rank framework for sub- space clustering and completion. IEEE Transactions on Signal Processing, 64(24))6557-6570, 2016.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 4313-4324. PMLR, 2020.
John Lipor and Laura Balzano. Clustering quality metrics for subspace clustering. Pattern Recog- nition, pp. 107328, 2020.
Guangean Liu, Zhouchen Lin, Shuicheng Yan. Sun, Yong Yu, and Yi Ma. Robust recovery of subspace structures by low-rank representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):171-184, 2012.
Weifeng Liu, Puskal Pokharel, and José Príncipe. Correntropy: Properties and applications in non-gaussian signal processing. IEEE Transactions Signal Processing, 55(11):5286-52)) 5298, 2007.
Muhammad Asad Lodhi and Waheed U Bajwa. Detection theory for union of subspaces. 1EEE Transactions on Signal Processing. 66(24):6347-6360 2018
Can-Y Yi Lu, Hai Min, Zhong- Biong-Qiu Zhao, Lin Zhu, De-Shuang Huang, and Shuicheng Yan. Robust and efficient subspace segmentation via least squares regression. In European Conference Computer Vision, pp. 347-360. Springer, 2012.
Canyi Lu, Jiashi Feng, Zhouchen Lin, Tao Mei, and Shuicheng Yan. Subspace clustering by block diagonal representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2): 487-501, 2018.
Yue M Lu and Minh N Do. A theory for sampling signals from union of subspaces. transactions on signal processing, 56(6):2334-235 2345, 2008
IEEE
Nene, Shree K Nayar, and Hiroshi Murase. Columbia object image library (coil-100)(tcch. rep.). Columbia University, 1996a.
Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20). 1996b.
Andrew Y Ng, Michael I Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In Advances Neural Information Processing Systems, pp. 849-856, 2002.
Vishal M Patel and René Vidal. Kernel sparse subspace clustering. IEEE International Conference IEEE, 2014.
Image Processing, pp.
Vishal M Patel, Hien Van Nguyen, and René Vidal. Latent space sparse subspace clustering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 225-232, 2013.
Xi Peng, Shijie Xiao, Jiashi Feng, Wei-Yun Yau, and Zhang Yi. Deep subspace clustering with sparsity prior. In International Joint Conferences on Artificial Intelligence, pp. 1925-1931, 2016a.
2016,
Xi Peng, Zhiding Yu, Zhang Yi. and Huajin Tang. Constructing the 12-graph for robust subspace learning and subspace clustering. IEFE Transactions Cybernetics, 47(4): 1053-16655-16, 2016b.

---

Under review as a conference paper at ICLR 2021
Xi Peng, Jiashi Feng, Jiwen Lu, Yum Yau, and Zhang Yi. Cascade subspace clustering. In Thirty-Fite Seirtt-First AAAI Conference on Artificial Intelligence, 2017.
Xi Peng, Jiashi Feng, Shijie Xiao, Wei-Yun Yun Yau, Joey Tianyi Zhou, and Songfan Yang. Structured autoencoders for subspace clustering. IEEE Transactions on Image Processing, 27(10)50076- 5086, 2018.
Fei Tian, Bin Gao, Qing Cui, Enhong Chen, and Tie- Liu. Learning deep representations for graph clustering. In Twenty-Fighth AAAI Conference Artificial Intelligence, 2014.
René Vidal. Subspace clustering. JEEE Signal Processing Magazine, (8(2))52-68, 2011.
Ulrike Von Luxburg. A tutorial spectral clustering. Statistics and Computing, 17(4):395-416 2007.
Tong Wu and Waheed U Bajwa. Revisiting robustness of the union-of-subspaces model for data- adaptive learning of nonlinear signal models. In 2014 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp. 3390-3394, IEEE, 2014.
Tong Wu and Waheed Bajwa. Metric-constrained kernel union of subspaces. In 2015 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5778-5782 IEEE, 2015.
Shijie Xiao, Mingkui Tan, Dong Xu, and Zhao Yang Dong. Robust kernel low-rank representation. IEEE Transactions on Neural Networks and Learning Systems, 27(1 (7(11))2268-221 2015.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International Conference on Machine Learning, pp. 478-487, 2016.
Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, and Wei Liu. Deep spectral clustering using dual autoencoder network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4066-4075, 2019.
Junjian Zhang, Chun-Guang Li, Chong You, Xianbiao Qi, Honggang Zhang, Jun Guo, and Zhouchen Lin. Self-supervised convolutional subspace clustering network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5473-5482, 2019a.
Junjian Zhang, Chun-Guang Li, Tianming Du Honggang Zhang, and Jun Guo. Convolutional subspace clustering network with block diagonal prior. IEEE Access, 8(201):5723-57732 2020.
Tong Zhang, Pan Ji, Mehrtash Harandi, Wenbing Huang, and Hongdong Li. Neural collaborative subspace clustering, arXiv preprint wrXiv:1904.10596 2019b.
Lei Zhou, Bai Xiao, Xianglong Liu, Jun Zhou, Edwin R Hancock, Latent distribution preserv- ing deep subspace clustering. In 28th International Joint Conference on Artificial Intelligence. York, 2019.
Pan Zhou, Yunqing Hou, and Jiashi Feng. Deep adversarial subspace clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1596-1604, 2018.
A APPENDIX PRELIMINARIES
A. BLOCK DIAGONAL REGULARIZATION
To introduce BD regularization we state the proposition from (Von Luxburg 2007) for the graph Laplacian matrix L.
12

---

Under review as a conference paper at ICLR 2021
Proposition 1. (Proposition 4 in (Von Luxburg 2007 Number of connected components and spectra of L). Let G an undirected graph with nonnegative weights. Then the multiplicity & of the eigenvalues 0 of L equals the number of connected components in the graph. Thus, based Proposition we define the BD regularization of C as the sum of the smallest eigenvalues of L:
||
Σ
ALL
(13)
i= k+1
where Ae( À 2 A2 2 ... 1(N- 2 2..2 Av. stands for the i-th eigenvalue L.
A.2 CORRENTROPY
Here briefly introduce the correntropy and its properties that qualify loss function robust to data corruptions. Let S = sww] E dx N and 'I = [ti,..., ty] E R be two realizations of the corresponding random variables. The empirical correntropy estimated from data as:
E (s t(), Ko (st, t.) exp (- ||s, t;
V(S. T)
(14)
20²
where (s; t;) is the Gaussian kernel. Herein, present from (Liu et al. properties correntropy that justify its use as robust error measure.
2007 two (out of ten)
Property 1. (Property 3 in (Liu et al. 2007 ). Correntropy involves all the even moments of the random variable & S:
(-1)a
(S
T) 2n
(S
E
(15)
2 m=0
2"n!
2n
where denotes mathematical expectation. As a increases, the high-order moments decay faster; so, the second-order moment tends to dominate, and the correntropy approaches correlation.
While the MSE involves only second-order moments and is, thus, optimal measure for error dis- tributed normally, correntropy is the optimal measure for error with the arbitrary (non-Gaussian) distribution. Furthermore, as can be seen from Equation (14). correntropy is data-driven.
Property 2 (Property 8 in (Luu et al. 2007)). The function:
(x 0) V(S,T))
CTM T)
(16)
defines CIM in sample space. Since its limes is final, the CIM function is robust to large random errors.
B APPENDIX - EXPERIMENTAL SETUP
Robust SoConVSCN model is implemented in Keras (Chollet et al. 2015) and Tensorflow (Abadi al., 2015). First, the random seed is fixed for reproducibility. To have reliable performance estimate on the test set and multiple independent train and test folds for each dataset, a stratified k- fold splitting was performed. Depending
the dataset constraints, the train-test split was performed on every fold or one fold served as a train set while the rest k folds served as a test set. Hence, independent & observations of algorithms' performances for each dataset have been produced.
As discussed in Section 3 instead monitoring accuracy every epoch, and consequently supervis- ing the learning using ground truth labels, only the loss for reducing learning rate and early
13

---

Under review as a conference paper at ICLR 2021
stopping. The training is stopped either after reaching the early stopping criterion after reaching the maximum number of epochs. As in (Zang et al. 2019a), pretraining was also applied. Firstly, the autoencoder is pretrained to replicate the imput. Secondly, the self-expression layer together with the pretrained autoencoder (DSCNet (Ji et al. 2017 was trained to reach limited number of epochs the early stopping criterion. After that, the FC layer and self-supervision modules were added to the pretrained DSCNet. This procedure forms a starting point for both SCConySCN and Robust SCConYSCN, Values of syyer-parameters A1 ... A5 found in Eq. are shown in Table
During the warming-up phase, pseudo-labels were not refined because the newly introduced layers and modules could affect the convergence process (see Section for the discussion). For the sake of fair comparison, different versions of SoConvSCN were trained using the same settings as Robust S²ConySCN. 3 shows shows settings for MINST (LeCun at. 2010). 2010), COIL--20 (Nene et al. 1996b), COII-1100 (Nene et al. 1996a), and Extended Yale B (Lee et al. 2005). Detailed settings for the baseline and robust model can seen in Table 3
Table 3: Settings of the baseline ConnSSCN and Robust S*ConvSCN per each dataset. T-max represents maximum number epochs and TO stands for the number of epochs after which pseudo- labels are refined. Warm-up represents the number of epochs without the refinement of pseudo- labels, LR stands for starting learning rate, and Min LR represents the minimum learning rate.
MNIST
COIL20
COIL100
EYaleB
data splits
T-max
9000
4000
4000
9000
TO
30
50
40
30
Warm-up
50
100
80
50
LR
le-3
le-4
le-4
le-4
Min LR
le-6
le-6
le-7
le-6
Table 4: S'ConvSCN and CoNvSCN hyperparameters. \ to As are regularization constants from total loss Eq. 11 while the trade-off constant y depends usage of BD (Eq. (5) La (Eq. (6)
regularization.
Model
X2
X3
A5
? Eq. (5)
7 Eq. (6)
ConvSCN
30
8
6
3e-2
3e-2
Coil20
RS RS* ConvSCN
300
6
3e-3
3e-3
S ConnSSCN
3.5
3e-2
3e-2
Coil100
RS ConvSCN
600
3.5
1e-3
1e-3
S ConvSCN
6.3
72
36
2
3e-2
3e-2
EYaleB
R$ ConvSCN
6.3
72
36
2
3e-2
3e-2
Sde ConvSCN
72
36
3e-2
3e-2
MNIST
RS ConnSCN
30
72
36
3e-2
3e-2
C APPENDIX COMPLEXITY AND CONVERGENCE
Dataset size and choice of representation matrix regularization have direct impact on time and memory complexity. As the model must learn in batch mode, the number of samples in the dataset N determines the number of parameters in the ceff-expression layer 2 Thus, dataset size impacts memory and time complexity. Also, regularization imposed on the representation matrix affects the computational time of the algorithm. the sum of & smallest eigenvalues of Laplacian matrix L. Thus. if the BD regularization is chosen, the model will have higher time complexity due to the usage of the singular value decomposition algorithm. Although is not directly associated with the time complexity per epoch, the spectral clustering module refines pseudo-labels every TO epochs which adds to overall training time.
presented in Section A BD regularization is defined as
To properly emphasize individual loss functions, all regularization constants in the overall loss func- tion must be tuned (discussed in Section 3). As a good starting point, we used regularization con- stants from (Zhang al. 2019a) and refined only those associated with the error measure term in
14

---

Under review as a conference paper at ICLR 2021
the self-expression layer and representation matrix C. Depending the fold size and regularization used, training of one fold on one Nvidia Quadro P6000 GPU takes 2-6 hours. Testing on one fold is within several seconds.
15


Reviews:
[{"review_id": "j97wq0newWC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to deep subspace clustering based on minimizing the correntropy induced metric (CIM), with the goal of establishing when training should be stopped and generalizing to unseen data. The main contribution over the existing S2ConfSCN method is a change from squared error loss to CIM when optimizing over the affinity matrix. A key benefit of CIM as a loss is that it does not decrease arbitrarily with training epochs, so it provides a means of estimating when training should cease without needing ground truth labels. The authors argue that CIM \"ensures a smooth decrease of the loss function that enables the use of label-free stopping criterion.\" However, this claim is only justified through a minimal empirical evaluation. The authors also include a means of enforcing block diagonal structure in the learned affinity matrix.\n\nWhile the ideas presented in the paper are important for the area of deep subspace clustering, the overall contribution of this paper is quite limited. Further, the results in Table 1 do not improve on the results of shallow subspace clustering methods such as SSC and EnSC. The authors argue that these methods are tuned using ground truth labels, but even tuning with the KSS Cost (as in Lipor and Balzano 2020) results in better performance on the given datasets. Given that shallow subspace clustering methods maintain strong theoretical guarantees, the empirical results presented here do not justify the use of deep methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VjLUppqKm0D", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to add a correntropy induced metric (CIM) loss term to improve the robustness of the self-supervised convolutional subspace clustering network (${\\rm S}^2$ConvSCN) to data corruption. The authors show that in a truly unsupervised training environment, the proposed robust ${\\rm S}^2$ConvSCN method outperforms the original ${\\rm S}^2$ConvSCN in [1].\n\nPros:\n1. The authors point out a very interesting fact that many deep subspace clustering methods often require access to the ground-truth labels to decide the stopping of the learning process and optimize hyperparameters using external clustering validation methodology, whereas the same labels are used for tuning hyperparameters and evaluating the final clustering performance. \n\nCons:\n1. The presentation style of this paper needs improvement. For example:\n    1.1 As this paper is based on the comparison of [1], the differences in Figure 1 should be clearly stated. Currently, this comparison is mixed in the contribution part, which is really confusing. The reviewer would recommend that before showing the results, the authors elaborator the differences between the proposed method and the method in [1] point by point. Then clearly state which part of the method in [1] is modified for \"fair\" comparison. This is very important for evaluating the contributions of this paper. \n    1.2 Define $\\bf{E}$ in (1).\n    1.3 Please define CIM near (4) and (5), or at least explicitly give the reference to your Appendix. Same for $||\\bf{C}||_{[k]}$. The reviewer believes that this paper is modified from [2], but the presentation flow is not carefully addressed. The Arxiv version is more readable.\n    1.4 Define $\\bf{Q}$ shown between (6) and (7).\n    1.5 The reviewer recommends that the authors include their model structure.\n\n2. The authors mention \"unfair\" comparison in the paper. I think a possible fair way to compare the performance of ${\\rm S}^2$ConvSCN method and robust ${\\rm S}^2$ConvSCN method on an \"unseen\" dataset. \n\n3. Please give your hyper-parameters in the paper.\n\n4. Please define the permutation space of $[k]$ in (11).\n\n5. For \"fair\" comparison, the author modified the method in [1]. However, in 4.1, only \"fixed number of epochs, learning rate, decay, and early stopping based on loss decrease only were used\". These should be explicitly stated. Are they the same as those in Table 3?\n\n6. The authors also mention the training time in the paper. However, for the device, only the word \"GPU\" is mentioned in the last paragraph without any details.\n\nIn summary, I believe this paper can be interesting if the presenting issue is well-addressed, and the comparison is well-stated, but the current version is not really good for acceptance.\n\n\nReference:\n[`1] Junjian Zhang, Chun-Guang Li, Chong You, Xianbiao Qi, Honggang Zhang, Jun Guo, and Zhouchen Lin. Self-supervised convolutional subspace clustering network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5473-5482, 2019a.\n[2] https://arxiv.org/abs/2004.03375\n\n----\nAfter rebuttal:\n\nThank the authors for providing a revised version. The paper is more readable, however, I still hesitate to give 6 for the current version.  I would say that the fact pointed in this paper is really interesting, but this paper may need further improvement in the presentation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8cz22Zh96v-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the robust formulation of the self-supervised convolutional subspace clustering network. In order to handle the data corruptions, the correntropy induced metric (CIM) of the error is embedded into the loss function. I have some concerns about this paper:\n1. The major issue of this work is the limited novrlty. In its current form, the only novel point is the CIM loss of the representation error of Z in Eq.(4) and (5). Other loss terms in Eq.(10) are all used in previous works.\n2. There are too many parameters in Eq.(10). However, no ablation analysis for different loss terms are given.\n3. Experimental comparison is not sufficient. Only one method is used for comparison. However, there are many deep learning based clustering methods, they should be used for comparison.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a robust formulation of Deep Subspace Clustering (DSC) based on the correntropy induced metric (CIM) of the error. All three reviewers recommend rejection. Their major critiques are limited novelty, insufficient experiments and similar performance to non-deep methods. The rebuttal highlights that the novelty is not DSC or CIM, but rather that the formulation does not require knowing the labels. I agree with the reviewers that the paper's novelty is very limited and didn't find the author's response compelling enough to overturn the reviewer's opinions. 
Decision: Reject