The Infinite Contextual Graph Markov Model

1 INTRODUCTION . It can be argued that one of the most daunting processes in machine learning is the selection of appropriate hyper-parameters for the task at hand . Indeed , due to the data-dependent nature of the learning problem , there usually exists no single model configuration that works well in all contexts . The most straightforward approach to mitigate this issue has typically been to rely on standard model selection techniques such as grid and random searches ( Bergstra & Bengio , 2012 ) , where the range of values to try are fixed a priori by the user . Nonetheless , there has always been an interest in alternative methods that automatically choliteratureose the “ right ” values for some hyper-parameters ( Gershman & Blei , 2012 ; He et al. , 2021 ) . In the Bayesian nonparametric ( BNP ) literature , which is of particular interest for this work , the complexity of Bayesian models automatically grows with the data ( Teh et al. , 2006 ) , e.g. , a BNP mixture model can adjust the number of its mixtures to better fit the empirical data distribution , thus freeing the user from the burden of choosing the most important ( if not all ) hyper-parameters . In recent years , much research effort has been devoted to the theoretical and practical development of Deep Graph Networks ( DGNs ) , which originated from Micheli ( 2009 ) ; Scarselli et al . ( 2009 ) . DGNs can deal with graphs of varying topology without the need for human intervention , and they rely on local and iterative processing of information commonly known as message passing ; for a thorough description of some of the most popular DGNs in the literature ( and of the more general graph representation learning field ) we refer the reader to recent surveys on the topic ( Bronstein et al. , 2017 ; Battaglia et al. , 2018 ; Bacciu et al. , 2020b ; Wu et al. , 2020 ) . Despite most of these methods belonging to the neural world , the Contextual Graph Markov Model ( CGMM ) stands out as a deep , unsupervised , constructive and fully probabilistic model that has shown competitive performances on downstream graph classification tasks ( Bacciu et al. , 2018 ; 2020a ) . CGMM trains a stack of Bayesian networks , where each layer is conditioned on the frozen posteriors of the nodes of the graph computed at previous layers . Each layer optimizes the likelihood of the data using the Expectation Maximization ( EM ) algorithm ( Moon , 1996 ) with closed-form solutions . Like its neural counterparts , for which the number of hidden units in each layer has typically been selected as a hyper-parameter , CGMM relies on model selection to choose the “ reasonable ” number of hidden states associated with the categorical latent variables . Differently from the neural methods though , CGMM is amenable to a BNP extension , as each layer is essentially a conditional mixture model . The main challenge we tackle in this work is the adaptation of CGMM to the elegant theoretical framework of BNP methods , in order to automatize the choice of its hyper-parameters , e.g. , the number of states . The principal difficulty lies in how to handle the variable-size number of neighbors of each node inside this framework , which in CGMM is solved by ( possibly weighted ) convex combinations of the neighbors ’ posteriors . The resulting model , called Infinite Contextual Graph Markov Model ( ICGMM ) , can generate as many latent states as needed to solve the unsupervised density estimation task at each layer . To the extent of our knowledge , this is the first Bayesian nonparametric model for adaptive graph processing . As a second contribution , we provide a faster implementation of our method that scales to the social datasets considered in this work while still providing state of the art results . We compare ICGMM against CGMM as well as end-to-end supervised methods on eight different graph classification tasks , following a fair , robust and reproducible experimental procedure ( Errica et al. , 2020 ) . Results show that ICGMM performs on par or better than the related models . We complement the analysis with studies on the effects of depth and generation of our model ’ s latent states . All in all , we believe that ICGMM is an important ( if not the first ) step towards a theoretically grounded and automatic construction of Deep Bayesian Graph Networks . 2 RELATED WORKS . The fundamental Bayesian nonparametric literature that is relevant to our work relates to the families of Dirichlet Processes ( DPs ) ( Gershman & Blei , 2012 ) and Hierarchical Dirichlet Processes ( HDPs ) ( Teh et al. , 2006 ) . In its most essential definition , a DP is a stochastic process that defines a probability distribution over other probability distributions . A DP is parametrized by a base distributionG0 , i.e. , the expected value of the process , and a scaling parameter α0 that controls the concentration of DP realizations aroundG0 ( Teh , 2010 ) . In particular , the Chinese Restaurant Process ( Aldous , 1985 ) , the Stick-breaking Construction ( Sethuraman , 1994 ) and the Pòlya urn scheme ( Hoppe , 1984 ) are all alternative ways to formalize a DP . Moving to HDPs is conceptually straightforward , in that it considers the base distributionG0 as a draw from another DP parametrized by a base distribution H and a scaling parameter γ . For a detailed treatment of learning with DP and HDPs , the reader can check a number of tutorials and surveys ( Teh et al. , 2006 ; Orbanz & Teh , 2010 ; Gershman & Blei , 2012 ) . Our work shares similarities with the Infinite Hidden Markov Model for temporal series ( Beal et al. , 2002 ) , with the fundamental differences that causality assumptions have to be relaxed to deal with graphs and that the hidden variables ’ distributions are conditioned on a varying number of observations . Most of the recent advances of the graph representation learning field are based on the so called feedforward DGNs ( Bacciu et al. , 2020b ) . These models rely on “ spatial ” graph convolutional layers , i.e. , the state of each node in the graph is determined by applying a permutation invariant function to its neighboring states computed at the previous layers . Combined with the depth of the architecture , these models propagate contextual information across the graph , a process also known as “ message passing ” ( Gilmer et al. , 2017 ) . However , to the best of our knowledge , the only neural method for graphs that automatically constructs part of its architecture in a principled way is the pioonering work of Micheli ( 2009 ) . In fact , the Neural Network for Graphs ( NN4G ) , known to be the first spatial DGN , relies on the Cascade Correlation learning algorithm ( Fahlman & Lebiere , 1990 ) to determine the number of layers to use for the task under investigation . Instead , despite being loosely related to our work , AutoML methods for graphs are yet another way to automatize the selection of all hyper-parameters of a DGN ( He et al. , 2021 ) . In particular , the Auto-GNN technique relies on Neural Architecture Search to discover , based on performance trends , an adequate configuration for the supervised task ( Zhou et al. , 2019 ) . We differ from these approaches in two fundamental respects : first , we build upon theoretical grounds rooted in the BNP literature ; secondly , we determine the right number of states in a completely unsupervised fashion . In what follows , we provide a formalization of the Infinite Contextual Graph Markov Model . Apart from the technical details , our hope is to show how the cross-fertilization of ideas from different research fields can help us advance the state of the art , both in the theoretical and empirical sense . 3 METHOD . This Section introduces the details of our method . Since we borrow ideas from two relatively distant fields , we define a unified mathematical notation and jargon as well as a high-level overview of the CGMM and HDP models to ease the subsequent exposition . We define a graph as a tuple g = ( Vg , Eg , Xg ) where Vg is the set of entities ( also referred to as nodes or vertices ) , Eg is the set of oriented edges ( u , v ) connecting node u to v , and the symbol Xg stands for the set of node attributes associated with the graph g. Also , the neighborhood of a node u is the set of nodes connected to u , i.e. , Nu = { v ∈ Vg| ( v , u ) ∈ Eg } . For the purpose of this work , we will define the ( categorical or continuous ) node feature of a node u with the term xu ∈ Xg . 3.1 BASICS OF CGMM . To best understand how and why this work extends CGMM , we now give a brief but essential description of its main characteristics . CGMM is , first and foremost , a deep architecture for the adaptive processing of graphs . Like other DGNs , it maps the entities of a graph , if not the graph itself , into latent representations . More specifically , we can get one of such representations for each layer of the architecture and then concatenate all of them to obtain richer node and graph embeddings . The latter is usually obtained as a global aggregation of the former . The second peculiarity of CGMM is that it is constructive , i.e. , trained in an incremental fashion : after one layer is trained , another one can be stacked atop of it and trained using the frozen outputs of the previous layer . This idea is borrowed from NN4G ( Micheli , 2009 ) , and it allows CGMM to relax the mutual dependencies between latent variables in a cyclic graph . However , because the local and iterative message passing mechanism used by spatial methods ( Micheli , 2009 ; Kipf & Welling , 2017 ) is responsible for information propagation across the graph , this relaxation is not restrictive . Thirdly , the node/graph embedding construction is fully probabilistic and unsupervised , since layer ` is represented as the Bayesian network on the left hand-side of Figure 1 . A latent variable q ` u is attached to each node u , and it is responsible for the the generation of the node feature xu . To take into account structural information , the hidden state q ` u is conditioned on the neighboring hidden states computed at the previous layer , i.e. , the set { q ` −1v | v ∈ Nu } . Importantly , the constructive approach allows us to treat the hidden ( frozen ) states of the previous layer as observable variables . Each layer is trained to fit the data distribution of node features using the EM algorithm , thus guaranteeing the convergence to a local minima . Once inference is performed , the state of each node is frozen and we can move to the subsequent layer . Lastly , the embedding of each node at layer ` is encoded as the posterior of its hidden state . 3.2 BASICS OF HDP . The HDP is a Bayesian nonparametric prior for the generation of grouped data using different infinite mixture models with shared mixture components . Let { x1 , x2 , . . . } be a set of observations that are grouped into J groups , i.e. , each observation xu belongs to the group ju ∈ { 1 , . . . , J } . Using the stick-breaking representation ( Sethuraman , 1994 ) , the HDP mixture model that generates the observations can be defined as ( Teh et al. , 2006 ) : β | γ ∼ Stick ( γ ) qu | ju , ( πj ) Jj=1 ∼ πju πj | β , α0 ∼ DP ( α0 , β ) xu | qu , ( θc ) ∞c=1 ∼ F ( θqu ) θ |H ∼H , ( 1 ) where F ( θqu ) denotes the emission distribution , parametrized by θqu , that generates the observation xu . The latent state qu indicates which mixture component should be used to generate xu . The value of qu is sampled from the distribution πju , which stands for the mixture weights of group ju . All ( πj ) J j=1 are obtained from a DP with concentration parameter α0 and base distribution β . Notably , all groups ’ mixture weights are defined on the same set of mixture components , meaning there is a form of parameter sharing across different groups . Finally , we sample the distribution β via the stick-breaking process Stick ( γ ) of Sethuraman ( 1994 ) . To generate a possibly infinite number of emission distributions , we exploit a prior distributionH that allows us to create new mixture components on demand . Thanks to the stick-breaking construction , even though an infinite number of mixture components can be used , only a finite number of them is istantiated during the inference phase . Hereinafter , we indicate with the symbol C the number of mixture components that are chosen by the HDP at inference time .

Summary: This paper is an extension of the Contextual Graph Markov Model, a deep unsupervised probabilistic approach for modeling graph data. The key idea is to leverage Hierarchical Dirichlet Processes, which enables the proposed approach to automatically choose the size of each layer’s latent representation. The authors conduct experiment on graph classification tasks, and the results are quite promising.