Learning to Learn Graph Topologies

1 Introduction . Graphs are an effective modelling language for revealing relational structure in high-dimensional complex domains and may assist in a variety of machine learning tasks . However , in many practical scenarios an explicit graph structure may not be readily available or easy to define . Graph learning aims at learning a graph topology from observation on data entities and is therefore an important problem studied in the literature . Model-based graph learning from the perspectives of probabilistic graphical models [ 24 ] or graph signal processing [ 32 , 30 , 28 ] solves an optimisation problem over the space of graph candidates whose objective function reflects the inductive graph-data interactions . The data are referred as to the observations on nodes , node features or graph signals in literature . The convex objective function often contains a graph-data fidelity term and a structural regulariser . For example , by assuming that data follow a multivariate Gaussian distribution , the graphical Lasso [ 2 ] maximises the ` 1 regularised log-likelihood of the precision matrix which corresponds to a conditional independence graph . Based on the convex formulation , the problem of learning an optimal graph can be solved by many iterative algorithms , such as proximal gradient descent [ 15 ] , with convergence guarantees . These classical graph learning approaches , despite being effective , share several common limitations . Firstly , handcrafted convex regularisers may not be expressive enough for representing the ⇤Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rich topological and structural priors . In the literature , only a limited number of graph structural properties can be modelled explicitly using convex regularisers , e.g . the ` 1 penalty for enforcing sparsity . More complex structures such as scale-free and small-world properties have been largely overlooked due to the difficulty in imposing them via simple convex optimisation . Secondly , graph structural regularisers might not be differentiable despite being convex , which perplexed the design of optimisation algorithms . Thirdly , iterative algorithms might take long to converge , as the search space grows quadratically with the graph size . Fourthly , tuning penalty parameters in front of structural regularisers and the step size of iterative algorithms are laborious . To address the above limitations , we propose a novel functional learning framework to learn a mapping from node observations to the underlying graph topology with desired structural property . Our framework is inspired by the emerging field of learning to optimise ( L2O ) [ 8 , 26 ] . Specifically , as shown in Figure 1 , we first unroll an iterative algorithm for solving the aforementioned regularised graph learning objective . To further increase the flexibility and expressiveness , we design a topological difference variational autoencoder ( TopoDiffVAE ) to replace the proximal projector of structural regularisation functions in the original iterative steps . We train the model in an end-to-end fashion with pairs of data and graphs that share the same structural properties . Once trained , the model can be deployed to learn a graph topology that exhibits such structural properties . The proposed method learns topological priors from graph samples , which are more expressive in learning structured graphs compared to traditional methods using analytic regularisers . Compared to deep neural networks , unrolling an iterative algorithm that is originally designed for graph learning problem introduces a sensible inductive bias that makes our model highly interpretable . We test the effectiveness and robustness of the proposed method in learning graphs with diverse topological structures on both synthetic and real-world data . The main contributions are as follows . Firstly , we propose a novel and highly interpretable neural networks to learn a data-to-graph mapping based on algorithmic unrolling . Secondly , we propose an effective TopoDiffVAE module that replaces the proximal operators of structural regularisers . To the best of our knowledge , this constitutes the first attempt in addressing the difficult challenge of designing convex functions to represent complex topological priors , which in turn allows for improved flexibility and expressiveness . Finally , the proposed method improves the accuracy of graph learning by 80 % compared to traditional iterative solvers . 2 Problem Formulation . Model-based graph learning Let G = { V , E , W } be an undirected weighted graph , where V is the node set with |V| = m , E is the edge set and W is the weighted adjacency matrix whose ij-th entry embodies the similarity between node i and node j . The combinatorial graph Laplacian matrix is defined as L = D W , where D = diag ( W1 ) is a diagonal matrix of node degrees . In many scenarios , we have access to a structured data matrix , which is denoted as X = [ x1 , x2 , . . . , xm ] > 2 Rm⇥n , where each column xi can be considered as a signal on the graph G. The goal of graph learning is to infer such a graph G that is fully represented by L or W from X . Specifically , we solve a generic optimisation problem min L tr ( X > LX ) + ⌦ ( L ) , ( 1 ) where the first term is the so-called Laplacian quadratic form , measuring the variation of X on the graph , and ⌦ ( L ) is a convex regulariser on L to reflect structural priors . Since W is symmetric , we introduce the vector of edge weights w 2 Rm ( m 1 ) /2+ and reparameterise Eq . ( 1 ) based on tr ( X > LX ) = P i , j Wij ||xi xj ||22 = 2w > y , where y 2 R m ( m 1 ) /2 + is the half-vectorisation of the Euclidean distance matrix . Now , we optimise the objective w.r.t w such that min w 2w > y + I { w 0 } ( w ) + ⌦1 ( w ) + ⌦2 ( Dw ) , ( 2 ) where I is an indicator function such that IC ( w ) = 0 if w 2 C and IC ( w ) = 1 otherwise , and D is a linear operator that transforms w into the vector of node degrees such that Dw = W1 . The reparameterisation reduces the dimension of search space by a half and we do not need to take extra care of the positive semi-definiteness of L. In Eq . ( 2 ) , the regularisers ⌦ is split into ⌦1 and ⌦2 on edge weights and node degrees respectively . Both formulations can be connected to many previous works , where regularisers are mathematically handcrafted to reflect specific graph topologies . For example , the ` 1-regularised log-likelihood maximisation of the precision matrix with Laplacian constraints [ 23 , 22 ] can be seen as a special case of Eq . ( 1 ) , where ⌦ ( L ) = log det ( L+ 2I ) + P i 6=j |Lij | . The ` 1 norm on edge weights enforces the learned conditional independence graph to be sparse . The author in [ 16 ] considers the log-barrier on node degrees , i.e . ⌦2 ( Dw ) = 1 > log ( Dw ) , to prevent the learned graph with isolated nodes . Furthermore , ⌦1 ( w ) = ||w||22 is often added to force the edge weights to be smooth [ 11 , 16 ] . Learning to learn graph topologies In many real-world applications , the mathematically-designed topological priors in the above works might not be expressive enough . For one thing , many interesting priors are too complex to be abstracted as an explicit convex function , e.g . the small-world property [ 36 ] . Moreover , those constraints added to prevent trivial solutions might conflict with the underlying structure . For example , the aforementioned log barrier may encourage many nodes with a large degree , which contradicts the power-law assumption in scale-free graphs . The penalty parameters , treated as a part of regularisation , are manually appointed or tuned from inadequately granularity , which may exacerbate the lack of topological expressiveness . These motivate us to directly learn the topological priors from graphs . Formally , we aim at learning a neural network F✓ ( · ) that maps the data term y to a graph representation w that share the same topological properties and hence belong to the same graph family G. With training pairs { ( yi , wi ) } ni=1 , we consider a supervised learning framework to obtain optimal F⇤✓ such that ✓⇤ = argmin ✓ Ew⇠G [ L ( F✓ ( y ) , w ) ] , ( 3 ) where L ( bw , w ) is a loss surrogate between the estimation bw = F✓ ( y ) and the groundtruth . Once trained , the optimal F⇤✓ ( · ) implicitly carries topological priors and can be applied to estimate graphs from new observations . Therefore , our framework promotes learning to learn a graph topology . In the next section , we elaborate on the architecture of F✓ ( · ) and the design of L ( · , · ) . 3 Unrolled Networks with Topological Enhancement . In summary , the data-to-graph mapping F✓ ( · ) is modelled with unrolling layers ( Section 3.1 ) inherited from a primal-dual splitting algorithm that is originally proposed to solve the model-based graph learning . We replace the proximal operator with a trainable variational autoencoder ( i.e . TopoDiffVAE ) that refines the graph estimation with structural enhancement in Section 3.2 . The overall network is trained in an end-to-end fashion with the loss function introduced in Section 3.3 . 3.1 Unrolling primal-dual iterative algorithm . Our model leverages the inductive bias from model-based graph learning . Specifically , we consider a special case of the formulation in Eq . ( 2 ) such that min w 2w > y + I { w 0 } ( w ) ↵1 > log ( Dw ) + ||w||22 ( 4 ) where ⌦1 ( w ) = ||w||22 and ⌦2 ( Dw ) = ↵1 > log ( Dw ) , both of which are mathematically handcrafted topological priors . To solve this optimisation problem , we consider a forward-backwardforward primal-dual splitting algorithm ( PDS ) in Algorithm 1 . The algorithm is specifically derived from Algorithm 6 in [ 21 ] by the authors in [ 16 ] . In Algorithm 1 , Step 3 , 5 and 7 update the primal variable that corresponds to w , while Step 4 , 6 and 8 update the dual variable in relation to node degrees . The forward steps are equivalent to a gradient descent on the primal variable ( Step 3 and 7 ) and a gradient ascent on the the dual variable ( Step 4 and 8 ) , both of which are derived from the differentiable terms in Eq. ( 4 ) . The backward steps ( Step 5 and 6 ) are proximal projections that correspond to two non-differentiable topological regularisers , i.e . the non-negative constraints on w and the log barrier on node degrees in Eq. ( 4 ) . The detailed derivations are attached in Appendix A.2 . It should be note that ` 1 norm that promotes sparsity is implicitly included , i.e . 2w > y = 2w > ( y /2 ) + ||w||1 given w 0 . We choose to unroll PDS as the main network architecture of F✓ ( · ) , since the separate update steps of primal and dual variables allow us to replace of proximal operators that are derived from other priors . Algorithm 1 PDS Input : y , , ↵ and . 1 : Initialisation : w ( 0 ) = 0 , v ( 0 ) = 0 2 : while |w ( t ) w ( t 1 ) | > ✏ do 3 : r ( t ) 1 = w ( t ) ( 2 w ( t ) +2y+D > v ( t ) ) 4 : r ( t ) 2 = v ( t ) + Dw ( t ) 5 : p ( t ) 1 = prox , ⌦1 ( r ( t ) 1 ) = max { 0 , r ( t ) 1 } 6 : p ( t ) 2 = prox , ⌦2 ( r ( t ) 2 ) , where prox ( t ) , ⌦2 ( r2 ) i = r2 , i p r22 , i+4↵ 2 7 : q ( t ) 1 = p ( t ) 1 ( 2 p ( t ) 1 +2y+D > p ( t ) 2 ) 8 : q ( t ) 2 = p ( t ) 2 + Dp ( t ) 1 9 : w ( t+1 ) = w ( t ) r ( t ) 1 + q ( t ) 1 10 : v ( t+1 ) = v ( t ) r ( t ) 2 + q ( t ) 2 11 : end while 12 : return bw = w ( T ) Algorithm 2 Unrolling Net ( L2G ) Input : y , T , enhancement ( t ) 2 { TRUE , FALSE } 1 : Initialisation : w ( 0 ) = 0 , v ( 0 ) = 0 2 : for t = 0 , 1 , . . . , T do 3 : r ( t ) 1 = w ( t ) ( t ) ( 2 ( t ) w ( t ) +2y+D > v ( t ) ) 4 : r ( t ) 2 = v ( t ) + ( t ) Dw ( t ) 5 : if enhancement ( t ) =TRUE , p ( t ) 1 = TopoDiffVAE ( r ( t ) 1 ) ; else , p ( t ) 1 = prox , ⌦1 ( r ( t ) 1 ) = max { 0 , r ( t ) 1 } . 6 : p ( t ) 2 = prox ( t ) , ⌦2 ( r ( t ) 2 ) , where prox ( t ) , ⌦2 ( r2 ) i = r2 , i p r22 , i+4↵ ( t ) ( t ) 2 7 : q ( t ) 1 = p ( t ) 1 ( t ) ( 2 ( t ) p ( t ) 1 +2y+D > p ( t ) 2 ) 8 : q ( t ) 2 = p ( t ) 2 + ( t ) Dp ( t ) 1 9 : w ( t+1 ) = w ( t ) r ( t ) 1 + q ( t ) 1 10 : v ( t+1 ) = v ( t ) r ( t ) 2 + q ( t ) 2 11 : end for 12 : return w ( 1 ) , w ( 2 ) , . . . , w ( T ) = bw The proposed unrolling network is summarised in Algorithm 2 , which unrolls the updating rules of Algorithm1 for T times and is hypothetically equivalent to running the iteration steps for T times . In Algorithm 2 , we consider two major substitutions to enable more flexible and expressive learning . Firstly , we replace the hyperparameters ↵ , and the step size in the PDS by layer-wise trainable parameters ( highlighted in blue in Algorithm 2 ) , which can be updated through backpropagation . Note that we consider independent trainable parameters in each unrolling layer , as we found it can enhance the representation capacity and boost the performance compared to the shared unrolling scheme [ 14 , 26 ] . Secondly , we provide a layer-wise optional replacement for the primal proximal operator in Step 5 . When learning complex topological priors , a trainable neural network , i.e . the TopoDiffVAE ( highlighted in red in Algorithm 2 ) takes the place to enhance the learning ability . TopoDiffVAE will be defined in Section 3.2 . It should be noted that such a replacement at every unrolling layer might lead to overly complex model that is hard to train and might suffer from overfitting , which depends on the graph size and number of training samples . Empirical results suggest TopoDiffVAE takes effects mostly on the last few layers . Comprehensibly , at first several layers , L2G quickly search a near-optimal solution with the complete inductive bias from PDS , which provides a good initialisation for TopoDiffVAE that further promotes the topological properties . In addition , we do not consider a trainable function to replace the proximal operator in Step 6 of Algorithm 1 , as it projects dual variables that are associated to node degrees but do not draw a direct equality . It is thus difficult to find a loss surrogate and a network architecture to approximate such an association .

Summary: The authors study graph structure learning. The paper's main contribution is to improve the unrolling primal-dual iterative algorithm used to solve equation 4 in the paper. The authors propose a Topological difference variational autoencoder (TopoDiffVAE), which can learn a mapping between two graph domains where one graph domain consists of richer structure information.