Subspace Clustering via Robust Self-Supervised Convolutional Neural Network

1 INTRODUCTION . Subspace clustering approaches have achieved encouraging performance when compared with the clustering algorithms that rely on proximity measures between data points . The main idea behind the subspace model is that the data can be drawn from low-dimensional subspaces which are embedded in a high-dimensional ambient space ( Lodhi & Bajwa , 2018 ) . Grouping such data associated with respective subspaces is known as the subspace clustering ( Vidal , 2011 ) . That is , each low-dimensional subspace corresponds to a class or category . Up to now , two main approaches for recovering lowdimensional subspaces are developed : models that are based on the self-representation property , and non-linear generalization of subspace clustering called union of subspaces ( UoS ) ( Lodhi & Bajwa , 2018 ; Lu & Do , 2008 ; Wu & Bajwa , 2014 ; 2015 ) . UoS algorithms are out of the scope of this work . Self-representation subspace clustering is achieved in two steps : ( i ) learning representation matrix C from data X and building corresponding affinity matrix A = |C|+ |CT | ; ( ii ) clustering the data into k clusters by grouping the eigenvectors of the graph Laplacian matrix that correspond with the leading k eigenvalues . This second step is known as spectral clustering ( Ng et al. , 2002 ; Von Luxburg , 2007 ) . Owning to the presumed subspace structure , the data points obey the self-expressiveness or self-representation property ( Elhamifar & Vidal , 2013 ; Peng et al. , 2016b ; Liu et al. , 2012 ; Li & Vidal , 2016 ; Favaro et al. , 2011 ) . In other words , each data point can be represented as a linear combination of other points in a dataset : X=XC . The self-representation approach is facing serious limitations regarding real-world datasets . One limitation relates to the linearity assumption because in a wide range of applications samples lie in nonlinear subspaces , e.g . face images acquired under non-uniform illumination and different poses ( Ji et al. , 2017 ) . Standard practice for handling data from nonlinear manifolds is to use the kernel trick on samples mapped implicitly into high dimensional space . Therein , samples better conform to linear subspaces ( Patel et al. , 2013 ; Patel & Vidal , 2014 ; Xiao et al. , 2015 ; Brbić & Kopriva , 2018 ) . However , identifying an appropriate kernel function for a given data set is quite a difficult task ( Zhang et al. , 2019b ) . The second limitation of existing deep SC methods relates to their assumption that the origin of data corruption is known , in which case the proper error model can be employed . In real-word applications origin of data corruption is unknown . That can severely harm the algorithm ’ s learning process if the non-robust loss function is used . Furthermore , validation ( i.e . stopping of the learning process ) in most of the deep SC methods often requires access to the ground-truth labels . That stands for violation of the basic principle of unsupervised machine learning and yields the overly-optimistic results . Dataset size is also a limitation when it comes to memory requirements . Since the self-representation subspace clustering is based on building the affinity matrix , memory complexity increases as the square of the dataset size . However , the latter limitation is not in the main focus of this work . Motivated by the exceptional ability of deep neural networks to capture complex underlying structures of data and learn discriminative features for clustering ( Hinton & Salakhutdinov , 2006 ; Dilokthanakul et al. , 2016 ; Ghasedi Dizaji et al. , 2017 ; Tian et al. , 2014 ; Xie et al. , 2016 ) , deep subspace clustering approaches emerged recently ( Ji et al. , 2017 ; Abavisani & Patel , 2018 ; Peng et al. , 2016a ; Yang et al. , 2019 ; Zhou et al. , 2018 ; Ji et al. , 2019b ; Peng et al. , 2018 ; 2017 ; Zhou et al. , 2019 ; Zhang et al. , 2019a ; Kheirandishfard et al. , 2020 ) . In particular , it is shown that convolutional neural networks ( CNNs ) , when applied to images of different classes , can learn features that lie in a UoS ( Lezama et al. , 2018 ) . Mostly , the base of the recently developed deep subspace-clustering networks is convolutional autoencoder . It is an end-to-end fully convolutional network that is based on the minimization of the reconstruction error . Together , the autoencoder and an additional self-expression ( SE ) module are forming a Deep subspace clustering network ( DSCNet ) ( Ji et al. , 2017 ) . Hence , the total loss function of DSCNet is composed of reconstruction loss and SE model loss . That is , during the learning process the clustering quality is not taken into account . Self-supervised convolutional SC network ( S2ConvSCN ) ( Zhang et al. , 2019a ) addressed this issue through the addition of a fully connected layer ( FC ) module and a spectral clustering module that , respectively , generate soft- and pseudo-labels . Dual self-supervision is achieved by forcing these two modules to converge towards consensus . Related accumulated loss , therefore , participates in enhancing the self-representation matrix and the quality of features extracted in the encoder layer . The architecture of S2ConvSCN has a possibility of direct classification once the learning process is completed . A trained encoder and the FC module can make a new network that can directly classify unseen data , also known as an out-of-sample problem . However , while this network can be validated and compared with other algorithms on a separate data set , such an ablation study was not completed . Furthermore , the main disadvantage of the DSCNet architecture , and indirectly S2ConvSCN , is that the network training is stopped when the accuracy is highest ( Ji et al. , 2019a ) . First , it is a direct violation of the unsupervised learning principle as the ground-truth labels are exposed . Second , the reported performance ( Zhang et al. , 2019a ; Ji et al. , 2017 ) is overly-optimistic and can not be compared to other algorithms . Also , as mentioned in ( Haeffele et al. , 2020 ) , most self-expressive based deep subspace clustering models suffer from the need of post-processing the self-representation matrix . Compared to the baseline model , we significantly reduced the post-processing while maintaining the noise-free matrix . Mentioned research problems led to three main contributions of proposed Robust S2ConvSCN : • robustness to errors of the unknown ( arbitrary ) origin is achieved by using the correntropy induced metric ( CIM ) in the self-expression loss , • the network is trained using the early-stopping method while monitoring only the accumulated loss , • thanks to correntropy based loss function the training process is less sensitive to data corruptions which enables the network to generalize better . This study has , also , three side-contributions : • the performance of models is estimated using the unseen ( out-of-sample ) data , • block-diagonal regularization of self-representation matrix is integrated into the gradient descent learning process , • post-processing of self-representation matrix is reduced to a significant extent . A complete head to head comparison of the baseline S2ConvSCN model and our robust approach can be seen in Figure 1 . 2 BACKGROUND AND RELATED WORK . 2.1 MAIN NOTATIONS AND DEFINITIONS . Throughout this paper , matrices are represented with bold capital symbols and vectors with bold lower-case symbols . X ∈ Rd×N represents data matrix comprised of N data samples with dimen- sionality d. { H ( l ) i } m ( l ) i=1 represent feature maps produced at the output of layer l−1 . Thus , H ( 0 ) = X and H ( L ) = X̂ . X̂ represents the output of the decoder and L represents number of convolutional layers in the autoencoder . { w ( l ) i } m ( l ) i=1 stand for a set of filters with associated biases { b ( l ) i } m ( l ) i=1 that form a convolutional layer l = 1 , . . . , L. zn = [ h ( L/2 ) 1 ( : ) . . .h ( L/2 ) m ( L/2 ) ( : ) ] T ∈ Rd̂×1 stands for feature vector comprised of vectorized and concatenated feature maps , with d̂ extracted features , in the top layer L2 ( encoder output ) representing input sample xn , n = 1 , . . . , N . C ∈ R N×N stands for representation matrix in self-expressive model Z = ZC . A = |C|+ |CT | is the affinity matrix and L = D− 1 2AD 1 2 is corresponding graph Laplacian matrix . D is diagonal degree matrix such that Dii = ∑N j=1 Aij . ‖X‖F = √∑N i , j=1 x 2 ij is the Frobenius norm of matrix X . ` p ( x ) = ‖x‖p = ( ∑d i=1 ‖xi‖ p ) 1/p , 0 < p ≤ 1 is the ` p norm of x . ` 0 ( x ) = ‖x‖0 = # { xi 6= 0 , i = 1 , . . . , d } , where # denotes the cardinality function , is ` 0 quasi norm of x . The Sp , 0 < p ≤ 1 , Schatten norms of matrix X are defined as the corresponding ` p norms of the vector of singular values of X , i.e . Sp ( X ) = ‖σ ( X ) ‖p where σ ( X ) stands for the vector of singular values of X . Depending on the context , 0 represents matrix/vector of all zeros and 1 represents matrix/vector of all ones . Grouping the data according to the linear subspaces they are drawn from is known as subspace clustering ( Vidal , 2011 ) . The problem is formally defined in : Definition 1 . Let X = [ X1 , . . . , Xk ] be a set of sample vectors drawn from a union of k subspaces in Rd , ∪ki=1 { Si } , of dimensions di min { d , N } , for i = 1 , . . . , k. Let Xi be a collection of Ni samples drawn from subspace Si , N = ∑k i=1Ni . The problem of subspace clustering is to segment samples into the subspaces they are drawn from . Throughout this paper , as it is the case in the majority of other papers , we have assumed that number of clusters k is known a priori . 2.2 APPROACHES TO SUBSPACE CLUSTERING . Usually , processes that operate in different modes generate data in real-world scenarios . Each mode models such data as lying on a subspace , while the whole process , thus , generates data lying on a union of subspaces ( UoS ) ( Lodhi & Bajwa , 2018 ) . The alternative to the UoS model is the selfrepresentation based subspace model . It implies that every sample from the dataset can be represented as a linear combination of other samples from the same cluster . While shallow models directly optimize such a self-representation matrix , their deep counterparts train the whole network to better extract features from the raw data and achieve representation linearity . Many approaches to deep subspace clustering are based on the introduction of the self-representation in the feature space ( Abavisani & Patel , 2018 ; Ji et al. , 2017 ; Peng et al. , 2016a ; Zhou et al. , 2018 ; 2019 ; Zhang et al. , 2019a ; Kheirandishfard et al. , 2020 ; Zhang et al. , 2020 ) . However , one weakness of self-expressive deep subspace clustering models is that their perfomance mainly depends on the self-representation matrix . Thus , elimination of the noise is done by post-processing ( Haeffele et al. , 2020 ) . It appears in many cases that from the final performance point of view the post-processing matters more than depth of the network . By the virtue of self-representation property , improvements of the shallow subspace clustering methods are of direct relevance to their deep counterparts . The subspace clustering task is accomplished through ( i ) learning the representation matrix C from data X , and ( ii ) clustering the data into k clusters by grouping the eigenvectors of the graph Laplacian matrix L that correspond with the k leading eigenvalues . This second step is known as spectral clustering ( Ng et al. , 2002 ; Von Luxburg , 2007 ) . Low-rank ( Liu et al. , 2012 ; Favaro et al. , 2011 ) and sparse models ( Elhamifar & Vidal , 2013 ) are one of the commonly used algorithms to solve SC clustering problem . They aim to learn the low-rank and sparse representation matrix by solving the following optimization problem ( Li & Vidal , 2016 ) : min C λ ‖C‖pSp + τ ‖C‖ p p s.t . Z = ZC , diag ( C ) = 0 ( 1 ) where λ and τ are nonnegative regularization constants . If number of layers L = 0 problem ( 1 ) is related to shallow subspace clustering . Constraint diag ( C ) = 0 is necessary to prevent sparseness regularized optimization algorithms to converge towards trivial solution where each data point represents itself . This constraint is not necessary for problem constrained only by low-rank . When data samples are contaminated with additive white Gaussian noise ( AWGN ) problem ( 1 ) becomes : min C ‖E‖2F + λ ‖C‖ p Sp + τ ‖C‖pp s.t . diag ( C ) = 0 ( 2 ) where E stands for the modelling error ( noise ) : E = Z− ZC . ( 3 ) Alternatively , square of the Frobenius norm of C is used for regularization ( Lu et al. , 2012 ) : min C ‖E‖2F + λ ‖C‖ 2 F ( 4 ) Objective ( 4 ) is used also in the self-expression module of the S2ConvSCN in ( Zhang et al. , 2019a ) . As seen from ( 2 ) and ( 4 ) , the MSE measure for discrepancy between Z and its self-representation ZC is justified only for the contamination by the AWGN . For sample-specific corruptions ( outliers ) the proper norm is ‖E‖2,1 while for large random corruptions the proper choice is ‖E‖1 ( Liu et al. , 2012 ) . However , errors in real world data have different origins and magnitude and may not follow specific probabilistic model . Sometimes , it is hard to know the true origin of corruptions present in data . Thus , to obtain method robust to arbitrary corruption we propose to introduce the CIM of the error . Rationale behind introduction of any regularization on C is to reflect its structural property of block-diagonality . Even though ‖C‖Sp and ‖C‖p , 0 ≤ p ≤ 1 in principle satisfy the enforced block-diagonality condition , their approximation of the BD structure of C is indirect ( Lu et al. , 2018 ) . Hence , for comparison , this study proposes introduction of loss function with gradient-based BD regularization on representation matrix C .

Summary: This paper presents an approach to deep subspace clustering based on minimizing the correntropy induced metric (CIM), with the goal of establishing when training should be stopped and generalizing to unseen data. The main contribution over the existing S2ConfSCN method is a change from squared error loss to CIM when optimizing over the affinity matrix. A key benefit of CIM as a loss is that it does not decrease arbitrarily with training epochs, so it provides a means of estimating when training should cease without needing ground truth labels. The authors argue that CIM "ensures a smooth decrease of the loss function that enables the use of label-free stopping criterion." However, this claim is only justified through a minimal empirical evaluation. The authors also include a means of enforcing block diagonal structure in the learned affinity matrix.