Convergent Graph Solvers

1 INTRODUCTION . Our world is replete with networked systems , where their overall properties emerge from complex interactions among the system entities . Such networked systems attain their unique properties from their stationary states ; hence , finding these stationary properties is a common goal for many problems that arise in the science and engineering field . Examples of such problems include the minimization of the molecule ’ s potential energy that finds the stationary positions of the atoms to compute the potential energy of the molecule ( Moloi & Ali , 2005 ) , the PageRank algorithm that finds the stationary importance of online web pages to compute recommendation scores ( Brin & Page , 1998 ) , and the network analysis of fluid flow in porous media that finds the stationary pressures in pore networks to compute the macroscopic properties of the media ( Gostick et al. , 2016 ) . In these network-analytic problems , the network is often represented as a graph , and the stationary states are often computed through a problem-specific iterative method derived analytically from the prior knowledge of the target system . By applying the iterative map repeatedly , these methods compute the stationary states ( i.e. , fixed points ) and the associated properties of the target system . However , analytically deriving such iterative maps for highly complex systems typically requires tremendous efforts and time . Instead of deriving/designing the problem-specific iterative methods , researchers employ deep learning approaches to learn the iterative methods ( Dai et al. , 2018 ; Gu et al. , 2020 ; Hsieh et al. , 2019 ; Huang et al. , 2020 ) . These approaches learn an iterative map directly without using domain-specific knowledge , but using only the input and output data . By applying the learned iterative map , these approaches ( approximately ) compute the fixed points and predict the properties of a target system . Graph Neural Networks ( GNNs ) have been widely used to construct iterative maps ( Dai et al. , 2018 ; Gu et al. , 2020 ; Scarselli et al. , 2008 ; Alet et al. , 2019 ) . However , when applying this approach , the existence of the fixed points is seldom guaranteed . As a result , the number of iterative steps is often required to be specified , as a hyperparameter , to ensure the termination of the iterations . This may cause premature termination or inefficient backward propagation if the number of iterations is inadequately small or too large . 1The code is available at https : //github.com/Junyoungpark/CGS . In this study , we propose a convergent graph solver ( CGS ) , a deep learning method that can predict the solution of a target graph analytical problem using only the input and output data , and without requiring the prior knowledge of existing solvers or intermediate solutions . The forward propagation of CGS is designed to proceed in the following three steps : • Constructing the input-dependent linear-contracting iterative maps . CGS uses the input graph , which dictates the specification of the target network-analytic problem , to construct a set of linear contracting maps . This procedure formulates/set up the internal problem to be solved by considering the problem conditions and contexts ( i.e. , boundary conditions or initial conditions in PDE domains – the physical network problems ) . Furthermore , the input-dependent linear map can produce any size of transition map flexibly depending on the input size graph ; thus helping the trained model to generalize over unseen problems with different sizes ( size transferability ) . • Computing the fixed points via iterative methods . CGS constructs a set of linear contracting maps , each of which is guaranteed to have a unique fixed point that embeds the important features for conducting various end tasks . Thus , CGS computes the unique solutions of the constructed linear maps via iterative methods ( or direct inversion ) with convergence guarantee . • Decoding the fixed points to estimate the properties . By using a separate decoder architecture , we compute the fixed points in the latent space while expecting them to be an effective representation that can improve the predictive performance of the model . This enables CGS to be used not only for finding the real fixed points ( or its transportation ) if they exist , but also for computing the `` virtual fixed point '' as a representation learning method in general prediction tasks . The parameters of CGS are optimized with the gradient computed based on the implicit function theorem , which requires O ( 1 ) memory usage when computing the gradient along with the iterative steps . CGS is different from the other studies that solve the constrained optimization ( with convergence guarantee ) ( Gu et al. , 2020 ; Scarselli et al. , 2008 ; Tiezzi et al. , 2020 ) in that it does not impose any restriction when optimizing the parameters . Instead , CGS is inherently structured to have the unique fixed points owing to the uses of the linear map . Note that the structural restriction is only in the form of an iterative map ; we can flexibly generate the coefficients of the linear map using any network ( i.e. , GNN ) and utilize multiple linear maps to boost the representability . We evaluate the performance of CGS using two types of paradigmatic network-analytic problems : physical diffusion in networks and the Markov decision process , where the true labels indeed exist and can be computed analytically using linear and non-linear iterative methods , respectively . We also employ CGS to solve various graph classification benchmark tasks to show that CGS can serve as a general ( implicit ) layer like other GNN networks when conducting typical graph classification tasks . In these experiments , we seek to compute a virtual fixed point that can serve as the best hidden representation of the input when predicting the output . The results show that CGS can be ( 1 ) an effective solver for graph network problems or ( 2 ) an effective general computational layer for processing graph-structured data . 2 RELATED WORK . Convergent neural models . Previous studies that have attempted to achieve the convergence property of neural network embedding ( e.g. , hidden vectors of MLP and hidden states of recurrent neural networks ) can be grouped into two categories : soft and hard approaches . Soft approaches typically attain the desired convergence properties by augmenting the loss functions ( Erichson et al. , 2019 ; Miyato et al. , 2018 ) . Although the soft approaches are network-architecture agnostic , these methods can not guarantee the convergence of the learned mappings . On the other hand , hard approaches seek to guarantee the convergence of the iterative maps by restricting their parameters in certain ranges ( Gu et al. , 2020 ; Tiezzi et al. , 2020 ; Miller & Hardt , 2018 ; Kolter & Manek , 2019 ) . This is achieved by projecting the parameters of the models into stable regions . However , such projection may lead to non-optimal performances since it is performed after the gradient update , i.e. , the projection is disentangled from training objective . Implicit deep models . The forward propagation of CGS , which solves fixed point iterations , is closely related to deep implicit models . Instead of defining the computational procedures ( e.g. , the depth of layer in neural network ) explicitly , these models use implicit layers , which accept inputdependent equations and compute the solutions of the input equations , for the forward propagation . For instance , neural ordinary differential equations ( NODE ) ( Chen et al. , 2018 ; Massaroli et al. , 2020 ) solve ODE until the solver tolerance is satisfied or the integration domain is covered , the optimization layers ( Gould et al. , 2016 ; Amos & Kolter , 2017 ) solve the optimization problem until the duality gap converges , and the fixed point models ( Bai et al. , 2019 ; Winston & Kolter , 2020 ) solve the network-generated fixed point iterations until some numerical solver satisfy the convergence condition . Implicit models use these ( intermediate ) solutions to conduct various end tasks ( e.g. , regressions , or classifications ) . In this way , implicit models can impose desired behavioral characteristics into layers as inductive bias and hence , often show superior parameter/memory efficiency and predictability . Fixed points of graph convolution . Methods that find the fixed points of graph convolutions have been suggested in various contexts of graph-related tasks . Several works have utilized GNN along with RNN-like connections to ( approximately ) find the fixed points of graph convolutions ( Liao et al. , 2018 ; Dai et al. , 2018 ; Li et al. , 2015 ; Scarselli et al. , 2008 ) . Some have suggested to constrain the parameter space of GNN so that the trained GNN becomes a non-expansive map , thus producing the fixed point ( Gu et al. , 2020 ; Tiezzi et al. , 2020 ) . Others have proposed to apply an additional GNN layer on the embedded graph and penalize the difference between the output of the additional GNN layer and the embedded graph to guide the GNN to find the fixed points ( Scarselli et al. , 2008 ; Yang et al. , 2021 ) . It has been shown that regularizing GNN to find its fixed points improves the predictive performance ( Tiezzi et al. , 2020 ; Yang et al. , 2021 ) . Comparison between CGS and existing approaches . Combining the ideas of ( 1 ) computing/using the fixed points of the graph convolution ( as a representation learning ) and ( 2 ) utilizing the implicit differentiation of the model ( as a training method ) has been proposed by numerous studies ( Scarselli et al. , 2008 ; Liao et al. , 2018 ; Johnson et al. , 2020 ; Dai et al. , 2018 ; Bai et al. , 2019 ; Gallicchio & Micheli , 2020 ; Bai et al. , 2019 ; Tiezzi et al. , 2020 ; Gu et al. , 2020 ) . Majority of those studies assume that the convolution operators ( iterative map ) induce a convergent sequence of the ( hidden ) representation ; however , this assumption typically does not hold unless certain conditions hold for the convolution operator ( iterative map ) . When neither the convergence nor uniqueness holds , it can possibly introduce biases in the gradient computed by implicit differentiation ( Liao et al. , 2018 ; Blondel et al. , 2021 ) . Hence , to impose convergence , some methods restrict the learned convolutions to be strictly contractive ( i.e. , the hidden solutions are convergent ) by projecting the learned parameters into a certain region and solving the constraint training problems , respectively ( Gu et al. , 2020 ; Tiezzi et al. , 2020 ) . Unlike these methods restricting the parameters of the graph convolutions directly , CGS guarantees the convergence and the uniqueness of the fixed point by imposing the structural inductive bias on the iterative map , i.e. , using the contractive linear map that has a unique fixed point , thus alleviating the need to solve the constrained parameter optimization ( Tiezzi et al. , 2020 ) . 3 PROBLEM DESCRIPTION . The objective of many network-analytic problems can be described as : Find a solution vector Y ∗ from a graph G that represents the target network system . In this section , we briefly explain a general iterative scheme to compute Y ∗ from G. The problem specification G = ( V , E ) is a directed graph that is composed of a set of nodes V and a set of edges E. We define the ith node as vi and the edge from vi to vj as eij . The general scheme of the iterative methods is given as follows : H [ 0 ] = f ( G ) , ( 1 ) H [ n ] = T ( H [ n−1 ] ; G ) , n = 1 , 2 , ... ( 2 ) Y [ n ] = g ( H [ n ] ; G ) , n = 0 , 1 , ... ( 3 ) where f ( · ) is the problem-specific initialization scheme that transforms G into the initial hidden embedding H [ 0 ] , T is the problem-specific iterative map that updates the hidden embedding H [ n ] from the previous embedding H [ n−1 ] , and g ( · ) is the problem-specific decoding function that predicts the intermediate solution Y [ n ] . T is designed such that the fixed point iteration ( equation 2 ) converges to the unique fixed point H∗ : lim n→∞ H [ n ] = H∗ s.t . H∗ = T ( H∗ , G ) ( 4 ) The solution Y ∗ is then obtained by decoding H∗ , i.e. , g ( H∗ ; G ) ≜ Y ∗ . In many real-world network-analytic problems , we can obtain G and its corresponding solution Y ∗ , but not T , H [ n ] and Y [ n ] . Therefore , we aim to learn a mapping T from G to Y ∗ without using T , H [ n ] and Y [ n ] .

Summary: This paper proposes a graph neural network termed "Convergent Graph Solvers" that computes the stationary distribution of stochastic matrices. This is achieved by an implicit definition of a linear system and by computing the fixed point of that system. The paper shows on a range of problems that the proposed approach works well and that it is particularly fast for smaller graphs.