Learning to Learn Graph Topologies

1 Introduction . Graphs are an effective modelling language for revealing relational structure in high-dimensional complex domains and may assist in a variety of machine learning tasks . However , in many practical scenarios an explicit graph structure may not be readily available or easy to define . Graph learning aims at learning a graph topology from observation on data entities and is therefore an important problem studied in the literature . Model-based graph learning from the perspectives of probabilistic graphical models [ 24 ] or graph signal processing [ 32 , 30 , 28 ] solves an optimisation problem over the space of graph candidates whose objective function reflects the inductive graph-data interactions . The data are referred as to the observations on nodes , node features or graph signals in literature . The convex objective function often contains a graph-data fidelity term and a structural regulariser . For example , by assuming that data follow a multivariate Gaussian distribution , the graphical Lasso [ 2 ] maximises the ` 1 regularised log-likelihood of the precision matrix which corresponds to a conditional independence graph . Based on the convex formulation , the problem of learning an optimal graph can be solved by many iterative algorithms , such as proximal gradient descent [ 15 ] , with convergence guarantees . These classical graph learning approaches , despite being effective , share several common limitations . Firstly , handcrafted convex regularisers may not be expressive enough for representing the ⇤Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rich topological and structural priors . In the literature , only a limited number of graph structural properties can be modelled explicitly using convex regularisers , e.g . the ` 1 penalty for enforcing sparsity . More complex structures such as scale-free and small-world properties have been largely overlooked due to the difficulty in imposing them via simple convex optimisation . Secondly , graph structural regularisers might not be differentiable despite being convex , which perplexed the design of optimisation algorithms . Thirdly , iterative algorithms might take long to converge , as the search space grows quadratically with the graph size . Fourthly , tuning penalty parameters in front of structural regularisers and the step size of iterative algorithms are laborious . To address the above limitations , we propose a novel functional learning framework to learn a mapping from node observations to the underlying graph topology with desired structural property . Our framework is inspired by the emerging field of learning to optimise ( L2O ) [ 8 , 26 ] . Specifically , as shown in Figure 1 , we first unroll an iterative algorithm for solving the aforementioned regularised graph learning objective . To further increase the flexibility and expressiveness , we design a topological difference variational autoencoder ( TopoDiffVAE ) to replace the proximal projector of structural regularisation functions in the original iterative steps . We train the model in an end-to-end fashion with pairs of data and graphs that share the same structural properties . Once trained , the model can be deployed to learn a graph topology that exhibits such structural properties . The proposed method learns topological priors from graph samples , which are more expressive in learning structured graphs compared to traditional methods using analytic regularisers . Compared to deep neural networks , unrolling an iterative algorithm that is originally designed for graph learning problem introduces a sensible inductive bias that makes our model highly interpretable . We test the effectiveness and robustness of the proposed method in learning graphs with diverse topological structures on both synthetic and real-world data . The main contributions are as follows . Firstly , we propose a novel and highly interpretable neural networks to learn a data-to-graph mapping based on algorithmic unrolling . Secondly , we propose an effective TopoDiffVAE module that replaces the proximal operators of structural regularisers . To the best of our knowledge , this constitutes the first attempt in addressing the difficult challenge of designing convex functions to represent complex topological priors , which in turn allows for improved flexibility and expressiveness . Finally , the proposed method improves the accuracy of graph learning by 80 % compared to traditional iterative solvers . 2 Problem Formulation . Model-based graph learning Let G = { V , E , W } be an undirected weighted graph , where V is the node set with |V| = m , E is the edge set and W is the weighted adjacency matrix whose ij-th entry embodies the similarity between node i and node j . The combinatorial graph Laplacian matrix is defined as L = D W , where D = diag ( W1 ) is a diagonal matrix of node degrees . In many scenarios , we have access to a structured data matrix , which is denoted as X = [ x1 , x2 , . . . , xm ] > 2 Rm⇥n , where each column xi can be considered as a signal on the graph G. The goal of graph learning is to infer such a graph G that is fully represented by L or W from X . Specifically , we solve a generic optimisation problem min L tr ( X > LX ) + ⌦ ( L ) , ( 1 ) where the first term is the so-called Laplacian quadratic form , measuring the variation of X on the graph , and ⌦ ( L ) is a convex regulariser on L to reflect structural priors . Since W is symmetric , we introduce the vector of edge weights w 2 Rm ( m 1 ) /2+ and reparameterise Eq . ( 1 ) based on tr ( X > LX ) = P i , j Wij ||xi xj ||22 = 2w > y , where y 2 R m ( m 1 ) /2 + is the half-vectorisation of the Euclidean distance matrix . Now , we optimise the objective w.r.t w such that min w 2w > y + I { w 0 } ( w ) + ⌦1 ( w ) + ⌦2 ( Dw ) , ( 2 ) where I is an indicator function such that IC ( w ) = 0 if w 2 C and IC ( w ) = 1 otherwise , and D is a linear operator that transforms w into the vector of node degrees such that Dw = W1 . The reparameterisation reduces the dimension of search space by a half and we do not need to take extra care of the positive semi-definiteness of L. In Eq . ( 2 ) , the regularisers ⌦ is split into ⌦1 and ⌦2 on edge weights and node degrees respectively . Both formulations can be connected to many previous works , where regularisers are mathematically handcrafted to reflect specific graph topologies . For example , the ` 1-regularised log-likelihood maximisation of the precision matrix with Laplacian constraints [ 23 , 22 ] can be seen as a special case of Eq . ( 1 ) , where ⌦ ( L ) = log det ( L+ 2I ) + P i 6=j |Lij | . The ` 1 norm on edge weights enforces the learned conditional independence graph to be sparse . The author in [ 16 ] considers the log-barrier on node degrees , i.e . ⌦2 ( Dw ) = 1 > log ( Dw ) , to prevent the learned graph with isolated nodes . Furthermore , ⌦1 ( w ) = ||w||22 is often added to force the edge weights to be smooth [ 11 , 16 ] . Learning to learn graph topologies In many real-world applications , the mathematically-designed topological priors in the above works might not be expressive enough . For one thing , many interesting priors are too complex to be abstracted as an explicit convex function , e.g . the small-world property [ 36 ] . Moreover , those constraints added to prevent trivial solutions might conflict with the underlying structure . For example , the aforementioned log barrier may encourage many nodes with a large degree , which contradicts the power-law assumption in scale-free graphs . The penalty parameters , treated as a part of regularisation , are manually appointed or tuned from inadequately granularity , which may exacerbate the lack of topological expressiveness . These motivate us to directly learn the topological priors from graphs . Formally , we aim at learning a neural network F✓ ( · ) that maps the data term y to a graph representation w that share the same topological properties and hence belong to the same graph family G. With training pairs { ( yi , wi ) } ni=1 , we consider a supervised learning framework to obtain optimal F⇤✓ such that ✓⇤ = argmin ✓ Ew⇠G [ L ( F✓ ( y ) , w ) ] , ( 3 ) where L ( bw , w ) is a loss surrogate between the estimation bw = F✓ ( y ) and the groundtruth . Once trained , the optimal F⇤✓ ( · ) implicitly carries topological priors and can be applied to estimate graphs from new observations . Therefore , our framework promotes learning to learn a graph topology . In the next section , we elaborate on the architecture of F✓ ( · ) and the design of L ( · , · ) . 3 Unrolled Networks with Topological Enhancement . In summary , the data-to-graph mapping F✓ ( · ) is modelled with unrolling layers ( Section 3.1 ) inherited from a primal-dual splitting algorithm that is originally proposed to solve the model-based graph learning . We replace the proximal operator with a trainable variational autoencoder ( i.e . TopoDiffVAE ) that refines the graph estimation with structural enhancement in Section 3.2 . The overall network is trained in an end-to-end fashion with the loss function introduced in Section 3.3 . 3.1 Unrolling primal-dual iterative algorithm . Our model leverages the inductive bias from model-based graph learning . Specifically , we consider a special case of the formulation in Eq . ( 2 ) such that min w 2w > y + I { w 0 } ( w ) ↵1 > log ( Dw ) + ||w||22 ( 4 ) where ⌦1 ( w ) = ||w||22 and ⌦2 ( Dw ) = ↵1 > log ( Dw ) , both of which are mathematically handcrafted topological priors . To solve this optimisation problem , we consider a forward-backwardforward primal-dual splitting algorithm ( PDS ) in Algorithm 1 . The algorithm is specifically derived from Algorithm 6 in [ 21 ] by the authors in [ 16 ] . In Algorithm 1 , Step 3 , 5 and 7 update the primal variable that corresponds to w , while Step 4 , 6 and 8 update the dual variable in relation to node degrees . The forward steps are equivalent to a gradient descent on the primal variable ( Step 3 and 7 ) and a gradient ascent on the the dual variable ( Step 4 and 8 ) , both of which are derived from the differentiable terms in Eq. ( 4 ) . The backward steps ( Step 5 and 6 ) are proximal projections that correspond to two non-differentiable topological regularisers , i.e . the non-negative constraints on w and the log barrier on node degrees in Eq. ( 4 ) . The detailed derivations are attached in Appendix A.2 . It should be note that ` 1 norm that promotes sparsity is implicitly included , i.e . 2w > y = 2w > ( y /2 ) + ||w||1 given w 0 . We choose to unroll PDS as the main network architecture of F✓ ( · ) , since the separate update steps of primal and dual variables allow us to replace of proximal operators that are derived from other priors . Algorithm 1 PDS Input : y , , ↵ and . 1 : Initialisation : w ( 0 ) = 0 , v ( 0 ) = 0 2 : while |w ( t ) w ( t 1 ) | > ✏ do 3 : r ( t ) 1 = w ( t ) ( 2 w ( t ) +2y+D > v ( t ) ) 4 : r ( t ) 2 = v ( t ) + Dw ( t ) 5 : p ( t ) 1 = prox , ⌦1 ( r ( t ) 1 ) = max { 0 , r ( t ) 1 } 6 : p ( t ) 2 = prox , ⌦2 ( r ( t ) 2 ) , where prox ( t ) , ⌦2 ( r2 ) i = r2 , i p r22 , i+4↵ 2 7 : q ( t ) 1 = p ( t ) 1 ( 2 p ( t ) 1 +2y+D > p ( t ) 2 ) 8 : q ( t ) 2 = p ( t ) 2 + Dp ( t ) 1 9 : w ( t+1 ) = w ( t ) r ( t ) 1 + q ( t ) 1 10 : v ( t+1 ) = v ( t ) r ( t ) 2 + q ( t ) 2 11 : end while 12 : return bw = w ( T ) Algorithm 2 Unrolling Net ( L2G ) Input : y , T , enhancement ( t ) 2 { TRUE , FALSE } 1 : Initialisation : w ( 0 ) = 0 , v ( 0 ) = 0 2 : for t = 0 , 1 , . . . , T do 3 : r ( t ) 1 = w ( t ) ( t ) ( 2 ( t ) w ( t ) +2y+D > v ( t ) ) 4 : r ( t ) 2 = v ( t ) + ( t ) Dw ( t ) 5 : if enhancement ( t ) =TRUE , p ( t ) 1 = TopoDiffVAE ( r ( t ) 1 ) ; else , p ( t ) 1 = prox , ⌦1 ( r ( t ) 1 ) = max { 0 , r ( t ) 1 } . 6 : p ( t ) 2 = prox ( t ) , ⌦2 ( r ( t ) 2 ) , where prox ( t ) , ⌦2 ( r2 ) i = r2 , i p r22 , i+4↵ ( t ) ( t ) 2 7 : q ( t ) 1 = p ( t ) 1 ( t ) ( 2 ( t ) p ( t ) 1 +2y+D > p ( t ) 2 ) 8 : q ( t ) 2 = p ( t ) 2 + ( t ) Dp ( t ) 1 9 : w ( t+1 ) = w ( t ) r ( t ) 1 + q ( t ) 1 10 : v ( t+1 ) = v ( t ) r ( t ) 2 + q ( t ) 2 11 : end for 12 : return w ( 1 ) , w ( 2 ) , . . . , w ( T ) = bw The proposed unrolling network is summarised in Algorithm 2 , which unrolls the updating rules of Algorithm1 for T times and is hypothetically equivalent to running the iteration steps for T times . In Algorithm 2 , we consider two major substitutions to enable more flexible and expressive learning . Firstly , we replace the hyperparameters ↵ , and the step size in the PDS by layer-wise trainable parameters ( highlighted in blue in Algorithm 2 ) , which can be updated through backpropagation . Note that we consider independent trainable parameters in each unrolling layer , as we found it can enhance the representation capacity and boost the performance compared to the shared unrolling scheme [ 14 , 26 ] . Secondly , we provide a layer-wise optional replacement for the primal proximal operator in Step 5 . When learning complex topological priors , a trainable neural network , i.e . the TopoDiffVAE ( highlighted in red in Algorithm 2 ) takes the place to enhance the learning ability . TopoDiffVAE will be defined in Section 3.2 . It should be noted that such a replacement at every unrolling layer might lead to overly complex model that is hard to train and might suffer from overfitting , which depends on the graph size and number of training samples . Empirical results suggest TopoDiffVAE takes effects mostly on the last few layers . Comprehensibly , at first several layers , L2G quickly search a near-optimal solution with the complete inductive bias from PDS , which provides a good initialisation for TopoDiffVAE that further promotes the topological properties . In addition , we do not consider a trainable function to replace the proximal operator in Step 6 of Algorithm 1 , as it projects dual variables that are associated to node degrees but do not draw a direct equality . It is thus difficult to find a loss surrogate and a network architecture to approximate such an association .

Summary: This paper presents a novel method to estimate a graph topology between nodes directly from observations on nodes. In the field of graph signal processing and graphical models, it has been shown that we can estimate the graph topology from data observations at nodes when we can assume that data values at nodes vary smoothly over a graph, in other words when data values observed at node A would be statistically close to the data values observed at node B if node A and B are located in close positions at the entire graph.    This assumption enables us to formulate the graph topology estimation as an optimization problem over the space of graph candidates. In this formulation, the objective function often is defined as a convex function having a form of "graph-data fidelity term" + "structural regularization term", which is then optimized by many iterative algorithms such as proximal gradient descent.  This paper aims to write these handcrafted convex regularizers by a differentiable module, and perform end-to-end learning in a more expressive and flexible manner. This can relax the strong assumption required for each specific regularization (such as L1 or L2 penalty), and extend the same framework flexibly to a wider range of graphs, scale-free graphs, or small-world graphs, which is usually difficult to explicitly write out the corresponding relevant regularization.   A very interesting technical point is, the paper's approach is different from just replacing the regularizer with a neural network module, rather enroll the optimization steps of proximal gradient descent as an iterative procedure, and only critical parts are redesigned by neural networks. By doing this, the method can leverage the effective inductive bias developed in the field not to over-reformulate that target problem.  The experimental evaluations on both synthetic and real datasets demonstrate that this idea actually worked nicely, also showing clear superiority over existing approaches such as GLAD and DeepGraph.