Neurally boosted supervised spectral clustering

1 INTRODUCTION . Spectral clustering is a classical unsupervised algorithm for clustering in graphs . It gained prominence in data mining and machine learning approximately two decades ago ( Shi & Malik , 2000 ; Ng et al. , 2001 ) and it has since found many applications in multiple domains . While originally conceived as an unsupervised method , spectral clustering has also been employed in transductive learning , and in particular in semi-supervised node classification that is the focus of this work . Background . The work of Tang & Liu ( 2011 ) was likely what initiated a long line of work on multi-label classification with experimental focus on social network benchmarks . They described a general framework for node classification consisting of two steps : ( i ) computing a network embedding , ( ii ) using the embedding as input to a standard classifier . They instantiated step ( i ) with a spectral embedding , giving thus rise to what can be viewed as a supervised spectral clustering algorithm . Albeit simple , the algorithm proved to be superior relative to the then-available baselines . The thread continued with the very prominent DEEPWALK embedding algorithm ( Perozzi et al. , 2014 ) , that claimed superiority over spectral clustering which however remained competitive . The success of DEEPWALK unleashed a wave of research works on network embeddings , including ( Grover & Leskovec , 2016 ; Tang et al. , 2015 ; Tsitsulin et al. , 2018 ; Qiu et al. , 2018a ) , and a race for higher classification accuracy 1 . More recently , network embedding methods were succeeded by graph neural networks ( GNNs ) ( e.g . see ( Kipf & Welling , 2017 ; Velickovic et al. , 2018 ) ) that have dominated the competition . Since the publication of DEEPWALK , supervised spectral clustering was at-best considered a baseline , or entirely vanished from consideration . However it has recently made a comeback with the work of Huang et al . ( Huang et al. , 2021 ) that described a simple ‘ correction and smooth ’ ( C & S ) procedure that can enhance the output of other models . Some of the state-of-the-art results in in the Open 1A very detailed list can be found in https : //github.com/chihming/awesome-network-embedding Graph Benchmark leaderboard ( Hu et al. , 2021 ) are now based on variants of supervised spectral clustering in combination with the C & S process . These easily surpass previous embedding methods and even exceed methods based on GNNs using only a small fraction of their number of trainable parameters . A Critical Look . This work originates from the observation that previous literature has treated spectral clustering somewhat casually , and largely ignoring recent major advances in its theoretical understanding . As discussed above , supervised spectral clustering consists of two steps : ( i ) Computing a spectral embedding , and ( ii ) Giving the embedding as input to a multi-label classifier . In the literature , the term ‘ spectral clustering ’ is used for multiple different algorithms that implement steps ( i ) and ( ii ) in various and sometimes not adequately described ways . For example , for step ( i ) , ( Tang & Liu , 2011 ) and ( Perozzi et al. , 2014 ) use eigenvectors of the normalized graph Laplacian as proposed in ( Ng et al. , 2001 ) but appear to omit a normalization step from that algorithm , ( Huang et al. , 2021 ) also use the normalized graph Laplacian but with an additional ‘ rotation ’ transformation , while other works use the random walk matrix ( i.e . the non-symmetric normalization of the Laplacian ) which in fact is a more accurate relaxation of the underlying NCut problem ( Shi & Malik , 2000 ) . Classifiers used for step ( ii ) include an One-vs-All SVM-based model ( Tang & Liu , 2011 ) , an One-vs-Rest logistic regression model ( Perozzi et al. , 2014 ) , a softmax regression model , and a three-layered neural network ( Huang et al. , 2021 ) . Adding to the confusion , the computation of eigenvectors has been viewed both as computationally expensive to the point of preventing its application on larger networks ( Perozzi et al. , 2014 ) , or as able to scale up to very large networks via standard iterative eigensolvers whose fast convergence can not however be guaranteed ( Huang et al. , 2021 ) . Goal . These considerations provide the motivation of our work . Concretely , we seek to provide a theoryinformed implementation of supervised spectral clustering , maximize its performance with the use of a carefully designed neural model and study its practical performance in the context of multi-label classification on social networks . Contributions . Our RADIAL classifier can be conceptually viewed as a ‘ neural ’ implementation of the seminal theoretical work of Lee et al . ( 2014b ) on multi-way Cheeger inequalities for spectral clustering . These theoretical guarantees are based on a variant of spectral clustering that is not well-known or widely used , yet it can be theoretically analyzed without assumptions about the underlying graphs . The analysis of Lee et al . applies to unsupervised clustering . Thus a key contribution of our work is to transfer the geometric intuition developed in ( Lee et al. , 2014b ) to the supervised setting with the design of custom-made neural classifiers that respect and exploit the geometric facts . Indeed , RADIAL raises the performance of supervised spectral clustering on transductive node classification to previously unobserved levels , sometimes more than 15 % higher than vanilla classifiers . As our ablation study confirms , this performance is achieved by introducing certain modifications and enhancements to the basic unsupervised algorithm of ( Lee et al. , 2014b ) , and other training tricks . Within our experimental context , when only graphic data are taken as input , we show that when training nodes are scarce , supervised spectral clustering is directly competitive to prominent alternative methods that were thought to be vastly superior . On the other hand , we observe that non-parametric diffusion-based methods become progressively more effective when training points are abundant , thus offering an empirical explanation of the success of the C & S algorithm ( Huang et al. , 2021 ) . Moreover , based on intuition derived in section 3.4 , we develop a method of enhancing the performance of standard GCNs ( Kipf & Welling , 2017 ) . More specifically , in section 4 , we show that when node features are available , the performance of GCNs can be significantly enhanced by feeding as additional input features learned by our RADIAL classifier . 2 SPECTRAL CLUSTERING WITH SUPERVISION : THE ALGORITHM . 2.1 SPECTRAL EMBEDDING AND GEOMETRIC INTUITION . We begin with a review of the spectral embedding algorithm . Along it we give concrete details on how it can be implemented to run in near-linear time with strong convergence guarantees . In Section 2.1.3 we discuss the geometric intuition that underlies the theoretical analysis in ( Lee et al. , 2014b ) . We note that the material in this subsection is not new . We present it for completeness and in particular in order to discuss the geometric intuition which plays a key role in the construction of our classifiers . 2.1.1 DEFINITIONS AND TERMINOLOGY . Let G = ( V , E , w ) be a weighted undirected graph . Definition 1 ( Cut and Volume ) . If S ⊂ V , we define the cut of S to be cut ( S ) = ∑ i∈S , j 6∈S wi , j We also define the volume of S to be vol ( S ) = ∑ s∈S cut ( v ) . We will often use the term degree for the cut of a single node . Definition 2 ( Conductance ) . The conductance of a cluster S ⊂ V is defined by φ ( S ) = cap ( S , V − S ) min ( vol ( S ) , vol ( V − S ) ) We denote by D the diagonal matrix of the degrees , by d the vector of the node degrees , and by L = D−A the graph Laplacian , where A is the adjacency matrix of the graph . 2.1.2 SPECTRAL EMBEDDING AND THE UNSUPERVISED ALGORITHM . The combinatorial idea that underlies spectral clustering is that good clusters should have a relatively small volume to their exterior , or more specifically a low conductance . More concretely , the goal is to find k disjoint clusters { S1 , S2 , ... Sk } such that maxi φ ( Si ) is as small as possible . In this paper we will be looking for k clusters that also provide a partitioning of the vertex set , i.e . we have ⋃ i Si = V . The partitioning problem is less well-understood ( Louis et al . ) , but its simpler non-partitioning variant has been studied extensively from a theoretical perspective ( Lee et al. , 2014b ) , and approximation guarantees are known for it . The spectral clustering algorithm is broadly understood as a relaxation of the discrete clustering problem ( e.g . see ( Cucuringu et al. , 2016 ) for a complete discussion ) , and a number of variants have been proposed ( von Luxburg , 2007 ) . In its simplest form , the variant that is analyzed in ( Lee et al. , 2014b ) consists of three steps , summarized here : ( a ) Spectral Embedding . Compute the eigenvectors xj corresponding to the k smallest non-zero eigenvalues of the generalized problem Lx = λDx , under the constraint that xT d = 0 . The eigenvectors are normalized so that xTj Dxj = 1 . Let X ∈ Rn×k be the matrix whose columns are the k eigenvectors2 . ( b ) Orthogonal Projection . LetX ′ = XR , whereR′ ∈ Rk×k′a random orthogonal projection matrix , where k′ ≤ k. 2Spectral embeddings can be computed in nearly-linear time . We discuss their computation in Appendix C. ( c ) Radial Projection . Let Y be the matrix resulting by dividing each row of X by its Euclidean norm , i.e . by ‘ radially projecting ’ each point onto the unit sphere in Rk . Each row of Y becomes the embedding of the corresponding node . ( d ) Geometric Partitioning . Use some unsupervised geometric clustering algorithm on the embedding Y . ( e.g . k-means , or the provable algorithms presented in ( Lee et al. , 2014b ) ) . We remark that the two hyperparameters k and k′ are further specified in sections 2.2 and 3.1 . 2.1.3 THE GEOMETRIC INSIGHT . The radial projection step reflects the geometric intuition that the eigenvectors place the nodes into k directions in the embedding space . Consider the eigenvector matrix X ∈ Rn×k . These generalized eigenvectors are D-orthogonal ( Stewart & Sun , 1990 ) , i.e . we have xTi Dxj = 0 , for each i 6= j . Given that the eigenvectors are also normalized , we have XTDX = I . Using a well known identity we have trace ( XTDX ) = trace ( DXXT ) = k. Summing up the n diagonal entries of DXXT we thus get n∑ j=1 dj ||xj ||22 = k. ( 1 ) Now pick an arbitrary direction in Rk , i.e . a unit vector z . Let w = Xz . Since XTDX = I , and zT z = 1 we have zTXTDXz = wTDw = n∑ j=1 djw 2 j = 1 . ( 2 ) Let us now take a closer look at the above equations . From equation 1 we get that that the d-weighted mass of the n embedding points is k. On the other hand , note that w2j is the norm of the projection of X [ j ] onto z . Thus , equation 2 shows that the d-weighted mass of the projections of the n points onto z is always equal to 1 , for any direction z . A further interpretation of the above equations is that any given direction z captures only a 1/k fraction of the d-weighted sum of norms of the n embedding points . Hence the d-mass of the points must concentrate in at least k different directions . While finding such directions is not straightforward , ( Lee et al. , 2014b ) proves that radially projecting the points onto the unit sphere has the following property : if S is a set of points in Y that are in a ball of a small diameter in Rk , then S also can not concentrate a large fraction of the d-mass of the points , and thus the points in Y are expected to be in k well-separated spheres in Rk . This specific geometric arrangement facilitates the design of an unsupervised geometric clustering algorithm on the points of Y in ( Lee et al. , 2014b ) , but it will also guide the design of our supervised geometric clustering model .

Summary: The paper argues for refocusing the efforts of the graph neural network community onto previous work on spectral clustering. The authors propose the addition of simple, but structured, classification methods to an existing unsupervised spectral clustering algorithm. A battery of empirical results on standard graph benchmarks demonstrate the effectiveness of such simple models and compare favourably to existing, more complex models.