A Critique of Self-Expressive Deep Subspace Clustering

1 INTRODUCTION AND BACKGROUND . Subspace clustering is a classical unsupervised learning problem , where one wishes to segment a given dataset into a prescribed number of clusters , and each cluster is defined as a linear ( or affine ) subspace with dimension lower than the ambient space . There have been a wide variety of approaches proposed in the literature to solve this problem ( Vidal et al. , 2016 ) , but a large family of state-of-the-art approaches are based on exploiting the self-expressive property of linear subspaces . That is , if a point lies in a linear subspace , then it can be represented as a linear combination of other points within the subspace . Based on this fact , a wide variety of methods have been proposed which , given a dataset Z ∈ Rd×N of N d-dimensional points , find a matrix of coefficients C ∈ RN×N by solving the problem : min C∈RN×N { F ( Z , C ) ≡ 1 2 ‖ZC− Z‖2F + λθ ( C ) = 1 2 〈Z > Z , ( C− I ) ( C− I ) > 〉+ λθ ( C ) } . ( 1 ) Here , the first term ‖ZC − C‖2F captures the self-expressive property by requiring every datapoint to represent itself as an approximate linear combination of other points , i.e. , Zi ≈ ZCi , where Zi and Ci are the ith columns of Z and C , respectively . The second term , θ ( C ) , is some regularization function designed to encourage each data point to only select other points within the correct subspace in its representation and to avoid trivial solutions ( such as C = I ) . Once the C matrix has been solved for , one can then define a graph affinity between data points , typically based on the magnitudes of the entries of C , and use an appropriate graph-based clustering method ( e.g. , spectral clustering ( von Luxburg , 2007 ) ) to produce the final clustering of the data points . One of the first methods to utilize this approach was Sparse Subspace Clustering ( SSC ) ( Elhamifar & Vidal , 2009 ; 2013 ) , where θ takes the form θSSC ( C ) = ‖C‖1 + δ ( diag ( C ) = 0 ) , with ‖ · ‖1 denoting the ` 1 norm and δ an indicator function which takes value∞ if an element of the diagonal of C is non-zero and 0 otherwise . By regularizing C to be sparse , a point must represent itself using the smallest number of other points within the dataset , which in turn ideally requires a point to only select other points within its own subspace in the representation . Likewise other variants , with Low-Rank Representation ( LRR ) ( Liu et al. , 2013 ) , Low-Rank Subspace Clustering ( LRSC ) ( Vidal & Favaro , 2014 ) and Elastic-net Subspace Clustering ( EnSC ) ( You et al. , 2016 ) being wellknown examples , take the same form as ( 1 ) with different choices of regularization . For example , θLRR ( C ) = ‖C‖∗ and θEnSC ( C ) = ‖C‖1 + τ‖C‖2F + δ ( diag ( C ) = 0 ) , where ‖ · ‖∗ denotes the nuclear norm ( sum of the singular values ) . A significant advantage of the majority of these methods is that it can be proven ( typically subject to some technical assumptions regarding the angles between the underlying subspaces and the distribution of the sampled data points within the subspaces ) that the optimal C matrix in ( 1 ) will be “ correct ” in the sense that if Ci , j is non-zero then the ith and jth columns of Z lie in the same linear subspace ( Soltanolkotabi & Candès , 2012 ; Lu et al. , 2012 ; Elhamifar & Vidal , 2013 ; Soltanolkotabi et al. , 2014 ; Wang et al. , 2015 ; Wang & Xu , 2016 ; You & Vidal , 2015a ; b ; Yang et al. , 2016 ; Tsakiris & Vidal , 2018 ; Li et al. , 2018 ; You et al. , 2019 ; Robinson et al. , 2019 ) , which has led to these methods achieving state-of-the-art performance in many applications . 1.1 SELF-EXPRESSIVE DEEP SUBSPACE CLUSTERING . Although subspace clustering techniques based on self-expression display strong empirical performance and provide theoretical guarantees , a significant limitation of these techniques is the requirement that the underlying dataset needs to be approximately supported on a union of linear subspaces . This has led to a strong motivation to extend these techniques to more general datasets , such as data supported on a union of non-linear low-dimensional manifolds . From inspection of the right side of ( 1 ) , one can observe that the only dependence on the data Z comes in the form of the Gram matrix Z > Z . As a result , self-expressive subspace clustering techniques are amendable to the “ kerneltrick ” , where instead of taking an inner product kernel between data points , one can instead use a general kernel κ ( · , · ) ( Patel & Vidal , 2014 ) . Of course , such an approach comes with the traditional challenge of how to select an appropriate kernel so that the embedding of the data in the Hilbert space associated with the choice of kernel results in a union of linear subspaces . The first approach to propose learning an appropriate embedding of an initial dataset X ∈ Rdx×N ( which does not necessarily have a union of subspaces structure ) was given by Patel et al . ( 2013 ; 2015 ) who proposed first projecting the data into a lower dimensional space via a learned linear projector , Z = PlX , where Pl ∈ Rd×dx ( d < dx ) is also optimized over in addition to C in ( 1 ) . To ensure that sufficient information about the original data X is preserved in the low-dimensional embedding Z , the authors further required that the linear projector satisfy the constraint that PlP > l = I and added an additional term to the objective with form ‖X−P > l PlX‖2F . However , since the projector is linear , the approach is not well suited for nonlinear manifolds , unless it is augmented with a kernel embedding , which again requires choosing a suitable kernel . More recently , given the success of deep neural networks , a large number of studies Peng et al . ( 2017 ) ; Ji et al . ( 2017 ) ; Zeng et al . ( 2019b ; a ) ; Xie et al . ( 2020 ) ; Sun et al . ( 2019 ) ; Li et al . ( 2019 ) ; Yang et al . ( 2019 ) ; Jiang et al . ( 2019 ) ; Tang et al . ( 2018 ) ; Kheirandishfard et al . ( 2020b ) ; Zhou et al . ( 2019 ) ; Jiang et al . ( 2018 ) ; Abavisani & Patel ( 2018 ) ; Zhou et al . ( 2018 ) ; Zhang et al . ( 2018 ; 2019b ; a ) ; Kheirandishfard et al . ( 2020a ) have attempted to learn an appropriate embedding of the data ( which ideally would have a union of linear subspaces structure ) via a neural network , ΦE ( X , We ) , whereWe denotes the parameters of a network mapping defined by ΦE , which takes a dataset X ∈ Rdx×N as input . In an attempt to encourage the embedding of the data , ΦE ( X , We ) , to have this union of subspaces structure , these approaches minimize a self-expressive loss term , with form given in ( 1 ) , on the embedded data , and a large majority of these proposed techniques can be described by the general form : min We , C γF ( Z , C ) + g ( Z , X , C ) subject to Z = ΦE ( X , We ) ( 2 ) where g is some function designed to discourage trivial solutions ( for example ΦE ( X , We ) = 0 ) and γ > 0 is some hyper-parameter to balance the terms . Several different choices of g have been proposed in the literature . The first is to place some form of normalization directly on Z . For example , Peng et al . ( 2017 ) propose an Instance Normalization regularization , g ( Z , X , C ) = ∑N i=1 ( Z > i Zi − 1 ) 2 , which attempts to constrain the norm of the embedded data points to be 1 . Likewise , one could also consider Dataset Normalization schemes , which bound the norm of the entire embedded representation ‖Z‖2F ≥ τ or Channel/Batch Normalization schemes , which bound the norm of a channel of the embedded representation ( i.e. , a row of Z ) , ‖Zi‖2F ≥ τ , ∀i . We note that this is quite similar to the common Batch Norm operator ( Ioffe & Szegedy , 2015 ) used in neural network training which attempts to constrain each row of Z to have zero mean and constant norm . Another popular form of g is to also train a decoding network ΦD ( · , Wd ) with parameters Wd to map the self-expressive representation , ΦE ( X , We ) C , back to the original data to ensure that sufficient information is preserved in the self-expressive representation to recover the original data . We will refer to this as Autoencoder Regularization . This idea is essentially a generalization of the previously discussed work , which considered constrained linear encoder/decoder mappings ( Patel et al. , 2013 ; Patel & Vidal , 2014 ; Patel et al. , 2015 ) , to non-linear autoencodering neural networks and was first proposed by the authors of Ji et al . ( 2017 ) . The problem takes the general form : min We , Wd , C γF ( Z , C ) + ` ( X , ΦD ( ZC , Wd ) ) subject to Z = ΦE ( X , We ) , ( 3 ) where the first term is the standard self-expressive subspace clustering loss applied to the embedded representation , and the second term is a standard auto-encoder loss , with ` typically chosen to be the squared loss . Note that here both the encoding/decoding network and the optimal self-expression encoding , C , are trained jointly , and once problem ( 3 ) is solved one can use the recovered C matrix directly for clustering . Using the general formulation in ( 2 ) and the popular specific case in ( 3 ) , Self-Expressive Deep Subspace Clustering ( SEDSC ) has been applied to a variety of applications , but there is relatively little that it known about it from a theoretical standpoint . Initial formulations for SEDSC were guided by the intuition that if the dataset is drawn from a union of linear subspaces , then solving problem ( 1 ) is known to induce desirable properties in C for clustering . By extension one might assume that if one also optimizes over the geometry of the learned embedding ( Z ) this objective might induce a desirable geometry in the embedded space ( e.g. , a union of linear subspaces ) . However , a vast majority of the prior theoretical analysis for problems of the form in ( 1 ) only considers the case where the data is held fixed and analyzes the properties of the optimal C matrix . Due to the well-known fact that neural networks are capable of producing highly-expressive mapping functions ( and hence a network could produce many potential values for Z ) , the use of a model such as ( 2 ) / ( 3 ) is essentially using ( 1 ) as a regularization function on Z to encourage a union of subspaces geometry . To date , however , models such as ( 2 ) / ( 3 ) have been guided largely by intuition and significant questions remain regarding what type of data geometry is encouraged by F ( Z , C ) when one optimizes over both the encoding matrix , C , and the network producing the embedded data representation , Z .

Summary: This paper critiques the commonly-used self-expressive cost function used to learn embeddings for deep subspace clustering. The authors point out that the empirical improvements obtained by deep self-expressive subspace clustering may be artifacts of post processing on the learned affinity matrix. They then theoretically characterize the optimal solutions to a variety of cost functions/normalization procedures used within the deep subspace clustering literature, showing that these encourage points to be mapped to a singleton set up to a sign change.