Convergence of Generalized Belief Propagation Algorithm on Graphs with Motifs

1 INTRODUCTION . Undirected graphical models , also known as Markov Random Fields ( MRF ) , provide a framework for modeling high dimensional distributions with dependent variables . Ising models are a special class of discrete pairwise graphical models originated from statistical physics . Ising models have numerous applications in computer vision Ravikumar et al . ( 2010 ) , bio-informatics Marbach et al . ( 2012 ) , and social networks Eagle et al . ( 2009 ) . Explicitly , the joint distribution of Ising model is given by P ( X ) = 1 Z exp ( β ( ∑ i hiXi + ∑ ( i , j ) JijXiXj ) ) , ( 1 ) where { Xi } i ∈ { ±1 } n are random variables valued in a binary alphabet ( also known as `` spins '' ) , Jij represents the pairwise interactions between spin i and spin j , hi represents the external field for spin i , β = 1/T is the reciprocal temperature , and Z is a normalization constant called partition function . Historically , Ising models are proposed to study ferromagnetism . However , researchers find the computational complexity is the main challenge of performing sampling and inference on Ising models . In the literature , there are multiple ways to tackle the computational complexity . One of the ways are Markov-Chain Monte Carlo ( MCMC ) algorithms . A well-known example is Gibbs sampling , which is a special case of the Metropolis–Hastings algorithm . Basically Gibbs sampling samples a random variable conditioned on the distribution based on the previous samples . It can be shown that Gibbs sampling generates a reversible Markov chain of samples . Thus , the stationary distribution of the Markov chain is the desired joint distribution over the random variables , and it can be reached after the burn-in period . However , it is also well-known that Gibbs sampling will become trapped on multi-modal distribution . For example , Smith and Roberts ( 1993 ) and Mengersen ( 1996 ) show that when the joint distribution is bi-modal , the Gibbs sampling iterations may be trapped in one of the modes , reducing the probability of convergence . Another popular way to go around the computational complexity is variational methods , which makes some approximation to the joint distribution . These methods usually turn the inference problem with respect to the approximate joint distribution into some non-convex optimization problem , and solve it either by the standard optimization methods , e.g , gradient descent , or by specialized algorithms like belief propagation . However , due to the non-convexity , those methods usually do not have theoretical guarantees that the solution converges to the global optimum . Belief propagation ( BP ) is an effective numerical method for solving inference problems on graphical models . It was originally proposed by Pearl ( 2014 ) for tree-like graphs . Ever since it plays a fundamental role in numerous applications including coding theory Frey et al . ( 1998 ) ; Richardson and Urbanke ( 2001 ) , constraint satisfaction problems Achlioptas and Moore ( 2006 ) , and community detection in the stochastic block model Decelle et al . ( 2011 ) . It is well-known that belief propagation is only exact for a model on a graph with locally tree-like structures . The long haunting question is , theoretically how does belief propagation perform on loopy graphs . We now describe the related work and our contributions . Related work and our contribution In a classic work , Yedidia et al . ( 2003 ) establishes the connection between belief propagation and the Bethe free energy . He shows that there is one-to-one correspondence between the fixed points of belief propagation and stationary points of the Bethe free energy . Following his work , it is known that the Bethe free energy at the critical points can be represented in terms of fixed point messages of belief propagation Montanari ( 2013 ) . In a recent work , Koehler ( 2019 ) further studies the properties of Bethe free energy at the critical points , and shows for ferromagnetic Ising models , initialized with all one messages , belief propagation converges to the fixed point corresponds to the global maximum of the Bethe free energy . However , those theories consider either asymptotic locally tree-like graphs , or loopy graphs with simple edges . Real technological , social and biological networks have numerous short and large loops and other complex motifs , which lead to non-tree-like structures and essentially loopy graphs with hyper edges . Newman Newman ( 2009 ) ; Karrer and Newman ( 2010 ) and Miller ( 2009 ) independently propose a model of random graphs with arbitrary distributions of motifs . And Yoon et al . ( 2011 ) generalizes the Belief Propagation to graphs with motifs . Our work builds on generalized belief propagation on graphs with motifs Yoon et al . ( 2011 ) and the convergence of belief propagation on ferromagnetic Ising models on loopy graphs with simple edges Koehler ( 2019 ) . In this paper , we show for ferromagnetic Ising models on graphs with motifs , with all messages initialized to one , generalized belief propagation converges to the fixed point corresponds to the global maximum of the Bethe free energy . 2 ISING MODELS ON GRAPHS WITH MOTIFS . Let us introduce the concept of graphs with motifs . In graphs with motifs , each vertex belongs to a given set of motifs . As shown in Fig.1a , different motifs can be attached to vertex i : a simple edge ( i , j ) , a triangle , a square , a pentagon , and other non-clique motifs . Graphs with motifs can be viewed as hyper-graphs where motifs play a role of hyper-edges . And the number of specific motifs attached to a vertex is equal to hyper-degree with respect to the specific motifs . In this paper , for simplicity , we only consider simple motifs such as simple edges , and cliques . Consider the Ising model with arbitrary order of interactions among vertices in each motif on a hyper-graph . Let Ml ( i ) denote a cluster of size l attached to vertex i , where vertices j1 , j2 , . . . , jl−1 together with i form the motif . And let X denote the random variable of spin configurations , the Hamiltonian of the model is E ( X ) = − ∑ i hiXi − ∑ ( i , j ) JijXiXj − ∑ ( i , j , k ) JijkXiXjXk − ∑ ( i , j , k , l ) JijklXiXjXkXl − · · · ( 2 ) where the first sum corresponds to the external fields at each vertex , the second sum corresponds to the pairwise interactions on simple edges , the third sum corresponds to the higher order interactions among spins in triangles , the fourth sum corresponds to the higher order interactions among spins in squares , and so on . As discussed in the previous section , most previous works focus on Ising models with pairwise interactions . In this paper , we are interested in Ising models with higher order interactions . For simplicity , we consider Ising models with only external fields and higher order interactions in triangles . Our derivation can be extended to more general cases . Consider Ising models with only external fields and higher order interactions in triangles , the Hamiltonian of the model is E ( X ) = − ∑ i hiXi − ∑ ( i , j , k ) JijkXiXjXk , ( 3 ) where ( i , j , k ) is a triangle , which can also be denoted as M3 ( i ) , M3 ( j ) , or M3 ( k ) . By Boltzmann ’ s law , the joint distribution is defined by P ( X ) = 1 Z e−βE ( X ) , ( 4 ) where Z is the partition function . Throughout this paper , we focus on ferromagnetic Ising models , which is defined below Definition 1 . An Ising model is ferromagnetic if Jijk ≥ 0 for all triangle motifs ( i , j , k ) and hi ≥ 0 for all i . We introduce a intermediate message µM3 ( i ) from a motif M3 ( i ) to spin i. µM3 ( i ) ( Xi ) = eβλM3 ( i ) Xi 2 coshβλM3 ( i ) . ( 5 ) In the literature , different works have different definitions of messages . µM3 ( i ) is not the message definition we eventually work with in this paper , but it helps to understand the connections between different works . So , abusing the terminology a little bit , we call it ‘ intermediate message ’ . By the definition of generalized Belief Propagation , the probability that spin i is in a state Xi is determined by the normalized product of incoming intermediate messages from motifs attached to spin i and the external field factor eβhiXi , Pi ( Xi ) = 1 A eβhiXi ∏ { M3 ( i ) } µM3 ( i ) ( Xi ) , ( 6 ) where A is a normalization constant . And the belief update rule is given by : µM3 ( i ) ( Xi ) = B ∑ { Xj=±1 } e−βE ( M3 ( i ) ) ∏ j ∏ { M3 ( j ) 6=M3 ( i ) } µM3 ( j ) ( Xj ) , ( 7 ) where E ( M3 ( i ) ) is an energy of the interaction among spins in the triangle M3 ( i ) , and B is a normalization constant . Multiplying Equation ( 7 ) by Xi and summing over all spin configurations , we obtain an equation for the effective field λM3 ( i ) , tanh ( βλM3 ( i ) ) = 1 Z ( M3 ( i ) ) ∑ { Xi , Xj1 , ... =±1 } Xie −βẼ ( M3 ( i ) ) , ( 8 ) where Ẽ ( M3 ( i ) ) = − 2∑ n=1 Λt ( jn ) Xjn − Jij1j2XiXj1Xj2 , ( 9 ) Λt ( j ) = hj + ∑ { M3 ( j ) 6=M3 ( i ) } λM3 ( j ) , ( 10 ) Z ( M3 ( i ) ) = ∑ { Xi , Xj1 , ... =±1 } e−βẼ ( M3 ( i ) ) . ( 11 ) For more detailed explanations of Equations ( 7 ) to ( 11 ) , please refer to Yoon et al . ( 2011 ) . Now , define a message from a spin i to motif M3 ( i ) as νi→M3 ( i ) = tanh ( λM3 ( i ) ) . More specifically , if the motif M3 ( i ) is a triangle ( i , j , k ) , the message can be alternatively represented as νi→M3 ( i ) = νi→jk = tanh ( λM3 ( i ) ) . From now on , let the reciprocal temperature β = 1 , we can further simplify Equation ( 8 ) as νi→jk = tanh ( hi + ∑ { m , n } ∈∂i\ { j , k } tanh−1 ( tanh ( Jimn ) νm→inνn→im ) ) , ( 12 ) where ∂i denotes the motifs attached to spin i . Equation ( 12 ) is the consistency equation for fixed point hyper-edge messages ν∗i→jk of the generalized belief propagation . Alternatively , we denote Equation ( 12 ) as νi→jk = φ ( ν ) i→jk .

Summary: The author considers a special class of attractive models and extends the convergence results of [Koehler]. In general, the convergence behavior of belief propagation algorithm is hard to be understood, however, attractive models are special class of models known to have simple optimization loss landscape. Therefore, the result of the paper is unlikely to be generalized to non-attractive models, limiting the application of the paper.