Neurally boosted supervised spectral clustering

1 INTRODUCTION . Spectral clustering is a classical unsupervised algorithm for clustering in graphs . It gained prominence in data mining and machine learning approximately two decades ago ( Shi & Malik , 2000 ; Ng et al. , 2001 ) and it has since found many applications in multiple domains . While originally conceived as an unsupervised method , spectral clustering has also been employed in transductive learning , and in particular in semi-supervised node classification that is the focus of this work . Background . The work of Tang & Liu ( 2011 ) was likely what initiated a long line of work on multi-label classification with experimental focus on social network benchmarks . They described a general framework for node classification consisting of two steps : ( i ) computing a network embedding , ( ii ) using the embedding as input to a standard classifier . They instantiated step ( i ) with a spectral embedding , giving thus rise to what can be viewed as a supervised spectral clustering algorithm . Albeit simple , the algorithm proved to be superior relative to the then-available baselines . The thread continued with the very prominent DEEPWALK embedding algorithm ( Perozzi et al. , 2014 ) , that claimed superiority over spectral clustering which however remained competitive . The success of DEEPWALK unleashed a wave of research works on network embeddings , including ( Grover & Leskovec , 2016 ; Tang et al. , 2015 ; Tsitsulin et al. , 2018 ; Qiu et al. , 2018a ) , and a race for higher classification accuracy 1 . More recently , network embedding methods were succeeded by graph neural networks ( GNNs ) ( e.g . see ( Kipf & Welling , 2017 ; Velickovic et al. , 2018 ) ) that have dominated the competition . Since the publication of DEEPWALK , supervised spectral clustering was at-best considered a baseline , or entirely vanished from consideration . However it has recently made a comeback with the work of Huang et al . ( Huang et al. , 2021 ) that described a simple ‘ correction and smooth ’ ( C & S ) procedure that can enhance the output of other models . Some of the state-of-the-art results in in the Open 1A very detailed list can be found in https : //github.com/chihming/awesome-network-embedding Graph Benchmark leaderboard ( Hu et al. , 2021 ) are now based on variants of supervised spectral clustering in combination with the C & S process . These easily surpass previous embedding methods and even exceed methods based on GNNs using only a small fraction of their number of trainable parameters . A Critical Look . This work originates from the observation that previous literature has treated spectral clustering somewhat casually , and largely ignoring recent major advances in its theoretical understanding . As discussed above , supervised spectral clustering consists of two steps : ( i ) Computing a spectral embedding , and ( ii ) Giving the embedding as input to a multi-label classifier . In the literature , the term ‘ spectral clustering ’ is used for multiple different algorithms that implement steps ( i ) and ( ii ) in various and sometimes not adequately described ways . For example , for step ( i ) , ( Tang & Liu , 2011 ) and ( Perozzi et al. , 2014 ) use eigenvectors of the normalized graph Laplacian as proposed in ( Ng et al. , 2001 ) but appear to omit a normalization step from that algorithm , ( Huang et al. , 2021 ) also use the normalized graph Laplacian but with an additional ‘ rotation ’ transformation , while other works use the random walk matrix ( i.e . the non-symmetric normalization of the Laplacian ) which in fact is a more accurate relaxation of the underlying NCut problem ( Shi & Malik , 2000 ) . Classifiers used for step ( ii ) include an One-vs-All SVM-based model ( Tang & Liu , 2011 ) , an One-vs-Rest logistic regression model ( Perozzi et al. , 2014 ) , a softmax regression model , and a three-layered neural network ( Huang et al. , 2021 ) . Adding to the confusion , the computation of eigenvectors has been viewed both as computationally expensive to the point of preventing its application on larger networks ( Perozzi et al. , 2014 ) , or as able to scale up to very large networks via standard iterative eigensolvers whose fast convergence can not however be guaranteed ( Huang et al. , 2021 ) . Goal . These considerations provide the motivation of our work . Concretely , we seek to provide a theoryinformed implementation of supervised spectral clustering , maximize its performance with the use of a carefully designed neural model and study its practical performance in the context of multi-label classification on social networks . Contributions . Our RADIAL classifier can be conceptually viewed as a ‘ neural ’ implementation of the seminal theoretical work of Lee et al . ( 2014b ) on multi-way Cheeger inequalities for spectral clustering . These theoretical guarantees are based on a variant of spectral clustering that is not well-known or widely used , yet it can be theoretically analyzed without assumptions about the underlying graphs . The analysis of Lee et al . applies to unsupervised clustering . Thus a key contribution of our work is to transfer the geometric intuition developed in ( Lee et al. , 2014b ) to the supervised setting with the design of custom-made neural classifiers that respect and exploit the geometric facts . Indeed , RADIAL raises the performance of supervised spectral clustering on transductive node classification to previously unobserved levels , sometimes more than 15 % higher than vanilla classifiers . As our ablation study confirms , this performance is achieved by introducing certain modifications and enhancements to the basic unsupervised algorithm of ( Lee et al. , 2014b ) , and other training tricks . Within our experimental context , when only graphic data are taken as input , we show that when training nodes are scarce , supervised spectral clustering is directly competitive to prominent alternative methods that were thought to be vastly superior . On the other hand , we observe that non-parametric diffusion-based methods become progressively more effective when training points are abundant , thus offering an empirical explanation of the success of the C & S algorithm ( Huang et al. , 2021 ) . Moreover , based on intuition derived in section 3.4 , we develop a method of enhancing the performance of standard GCNs ( Kipf & Welling , 2017 ) . More specifically , in section 4 , we show that when node features are available , the performance of GCNs can be significantly enhanced by feeding as additional input features learned by our RADIAL classifier . 2 SPECTRAL CLUSTERING WITH SUPERVISION : THE ALGORITHM . 2.1 SPECTRAL EMBEDDING AND GEOMETRIC INTUITION . We begin with a review of the spectral embedding algorithm . Along it we give concrete details on how it can be implemented to run in near-linear time with strong convergence guarantees . In Section 2.1.3 we discuss the geometric intuition that underlies the theoretical analysis in ( Lee et al. , 2014b ) . We note that the material in this subsection is not new . We present it for completeness and in particular in order to discuss the geometric intuition which plays a key role in the construction of our classifiers . 2.1.1 DEFINITIONS AND TERMINOLOGY . Let G = ( V , E , w ) be a weighted undirected graph . Definition 1 ( Cut and Volume ) . If S ⊂ V , we define the cut of S to be cut ( S ) = ∑ i∈S , j 6∈S wi , j We also define the volume of S to be vol ( S ) = ∑ s∈S cut ( v ) . We will often use the term degree for the cut of a single node . Definition 2 ( Conductance ) . The conductance of a cluster S ⊂ V is defined by φ ( S ) = cap ( S , V − S ) min ( vol ( S ) , vol ( V − S ) ) We denote by D the diagonal matrix of the degrees , by d the vector of the node degrees , and by L = D−A the graph Laplacian , where A is the adjacency matrix of the graph . 2.1.2 SPECTRAL EMBEDDING AND THE UNSUPERVISED ALGORITHM . The combinatorial idea that underlies spectral clustering is that good clusters should have a relatively small volume to their exterior , or more specifically a low conductance . More concretely , the goal is to find k disjoint clusters { S1 , S2 , ... Sk } such that maxi φ ( Si ) is as small as possible . In this paper we will be looking for k clusters that also provide a partitioning of the vertex set , i.e . we have ⋃ i Si = V . The partitioning problem is less well-understood ( Louis et al . ) , but its simpler non-partitioning variant has been studied extensively from a theoretical perspective ( Lee et al. , 2014b ) , and approximation guarantees are known for it . The spectral clustering algorithm is broadly understood as a relaxation of the discrete clustering problem ( e.g . see ( Cucuringu et al. , 2016 ) for a complete discussion ) , and a number of variants have been proposed ( von Luxburg , 2007 ) . In its simplest form , the variant that is analyzed in ( Lee et al. , 2014b ) consists of three steps , summarized here : ( a ) Spectral Embedding . Compute the eigenvectors xj corresponding to the k smallest non-zero eigenvalues of the generalized problem Lx = λDx , under the constraint that xT d = 0 . The eigenvectors are normalized so that xTj Dxj = 1 . Let X ∈ Rn×k be the matrix whose columns are the k eigenvectors2 . ( b ) Orthogonal Projection . LetX ′ = XR , whereR′ ∈ Rk×k′a random orthogonal projection matrix , where k′ ≤ k. 2Spectral embeddings can be computed in nearly-linear time . We discuss their computation in Appendix C. ( c ) Radial Projection . Let Y be the matrix resulting by dividing each row of X by its Euclidean norm , i.e . by ‘ radially projecting ’ each point onto the unit sphere in Rk . Each row of Y becomes the embedding of the corresponding node . ( d ) Geometric Partitioning . Use some unsupervised geometric clustering algorithm on the embedding Y . ( e.g . k-means , or the provable algorithms presented in ( Lee et al. , 2014b ) ) . We remark that the two hyperparameters k and k′ are further specified in sections 2.2 and 3.1 . 2.1.3 THE GEOMETRIC INSIGHT . The radial projection step reflects the geometric intuition that the eigenvectors place the nodes into k directions in the embedding space . Consider the eigenvector matrix X ∈ Rn×k . These generalized eigenvectors are D-orthogonal ( Stewart & Sun , 1990 ) , i.e . we have xTi Dxj = 0 , for each i 6= j . Given that the eigenvectors are also normalized , we have XTDX = I . Using a well known identity we have trace ( XTDX ) = trace ( DXXT ) = k. Summing up the n diagonal entries of DXXT we thus get n∑ j=1 dj ||xj ||22 = k. ( 1 ) Now pick an arbitrary direction in Rk , i.e . a unit vector z . Let w = Xz . Since XTDX = I , and zT z = 1 we have zTXTDXz = wTDw = n∑ j=1 djw 2 j = 1 . ( 2 ) Let us now take a closer look at the above equations . From equation 1 we get that that the d-weighted mass of the n embedding points is k. On the other hand , note that w2j is the norm of the projection of X [ j ] onto z . Thus , equation 2 shows that the d-weighted mass of the projections of the n points onto z is always equal to 1 , for any direction z . A further interpretation of the above equations is that any given direction z captures only a 1/k fraction of the d-weighted sum of norms of the n embedding points . Hence the d-mass of the points must concentrate in at least k different directions . While finding such directions is not straightforward , ( Lee et al. , 2014b ) proves that radially projecting the points onto the unit sphere has the following property : if S is a set of points in Y that are in a ball of a small diameter in Rk , then S also can not concentrate a large fraction of the d-mass of the points , and thus the points in Y are expected to be in k well-separated spheres in Rk . This specific geometric arrangement facilitates the design of an unsupervised geometric clustering algorithm on the points of Y in ( Lee et al. , 2014b ) , but it will also guide the design of our supervised geometric clustering model .

Summary: This paper reviewed the application of spectral clustering embedding on node-level classification problems. It first revisited the theoretical foundation of spectral clustering, related to the graph NCut problems. Then it introduced neural network-based classifiers which utilize the embedding for classification tasks. Their benchmark results validated that the neural network framework showed certain advantages in accuracy. And the "correction & smooth" introduced by Huang et al (ICLR 2021) would also improve the performance in a relative big margin.