A Chain Graph Interpretation of Real-World Neural Networks

1 INTRODUCTION . During the last decade , deep learning ( Goodfellow et al. , 2016 ) , the study of neural networks ( NNs ) , has achieved ground-breaking results in diverse areas such as computer vision ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Long et al. , 2015 ; Chen et al. , 2018 ) , natural language processing ( Hinton et al. , 2012 ; Vaswani et al. , 2017 ; Devlin et al. , 2019 ) , generative modeling ( Kingma & Welling , 2014 ; Goodfellow et al. , 2014 ) and reinforcement learning ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , and various network designs have been proposed . However , neural networks have been treated largely as “ black-box ” function approximators , and their designs have chiefly been found via trialand-error , with little or no theoretical justification . A major cause that hinders the theoretical analysis is the current overly generic modeling of neural networks as function approximators : simply interpreting a neural network as a composition of parametrized functions provides little insight to decipher the nature of its components or its behavior during the learning process . In this paper , we show that a neural network can actually be interpreted as a probabilistic graphical model ( PGM ) called chain graph ( CG ) ( Koller & Friedman , 2009 ) , and feed-forward as an efficient approximate probabilistic inference on it . This offers specific interpretations for various neural network components , allowing for in-depth theoretical analysis and derivation of new approaches . 1.1 RELATED WORK . In terms of theoretical understanding of neural networks , a well known result based on the function approximator view is the universal approximation theorem ( Goodfellow et al. , 2016 ) , however it only establishes the representational power of NNs . Also , there have been many efforts on alternative NN interpretations . One prominent approach identifies infinite width NNs as Gaussian processes ( Neal , 1996 ; Lee et al. , 2018 ) , enabling kernel method analysis ( Jacot et al. , 2018 ) . Other works also employ theories such as optimal transport ( Genevay et al. , 2017 ; Chizat & Bach , 2018 ) or mean field ( Mei et al. , 2019 ) . These approaches lead to interesting findings , however they tend to only hold under limited or unrealistic settings and have difficulties interpreting practical real-world NNs . Alternatively , some existing works study the post-hoc interpretability ( Lipton , 2018 ) , proposing methods to analyze the empirical behavior of trained neural networks : activation maximization ( Erhan et al. , 2009 ) , typical input synthesis ( Nguyen et al. , 2016 ) , deconvolution ( Zeiler & Fergus , 2014 ) , layer-wise relevance propagation ( Bach et al. , 2015 ) , etc . These methods can offer valuable insights to the practical behavior of neural networks , however they represent distinct approaches and focuses , and are all limited within the function approximator view . Our work links neural networks to probabilistic graphical models ( Koller & Friedman , 2009 ) , a rich theoretical framework that models and visualizes probabilistic systems composed of random variables ( RVs ) and their interdependencies . There are several types of graphical models . The chain graph model ( also referred to as the LWF chain graph model ) ( Koller & Friedman , 2009 ; Lauritzen & Wermuth , 1989 ; Frydenberg , 1990 ) used in our work is a general form that unites directed and undirected variants , visualized as a partially directed acyclic graph ( PDAG ) . Interestingly , there exists a series of works on constructing hierarchical graphical models for data-driven learning problems , such as sigmoid belief network ( Neal , 1992 ) , deep belief network ( Hinton et al. , 2006 ) , deep Boltzmann machine ( Salakhutdinov & Hinton , 2012 ) and sum product network ( Poon & Domingos , 2011 ) . As alternatives to neural networks , these models have shown promising potentials for generative modeling and unsupervised learning . Nevertheless , they are yet to demonstrate competitive performances over neural network for discriminative learning . Neural networks and graphical models have so far been treated as two distinct approaches in general . Existing works that combine them ( Zheng et al. , 2015 ; Chen et al. , 2018 ; Lample et al. , 2016 ) mainly treat either neural networks as function approximators for amortized inference , or graphical models as post-processing steps . Tang & Salakhutdinov ( 2013 ) create a hybrid model , the stochastic feedforward neural network ( SFNN ) , by concatenating deterministic neurons with stochastic Bernoulli random variables , in order to represent multimodal distributions . Some also consider neural networks as graphical models with deterministic hidden nodes ( Buntine , 1994 ) . However this is an atypical degenerate regime . To the best of our knowledge , our work provides the first rigorous and comprehensive formulation of a ( non-degenerate ) graphical model interpretation for neural networks in practical use . 1.2 OUR CONTRIBUTIONS . The main contributions of our work are summarized as follows : • We propose a layered chain graph representation of neural networks , interpret feed-forward as an approximate probabilistic inference procedure , and show that this interpretation provides an extensive coverage of practical NN components ( Section 2 ) ; • To illustrate its advantages , we show with concrete examples ( residual block , RNN , dropout ) that the chain graph interpretation enables coherent and in-depth theoretical support , and provides additional insights to various empirically established network structures ( Section 3 ) ; • Furthermore , we demonstrate the potential of the chain graph interpretation for discovering new approaches by using it to derive a novel stochastic inference method named partially collapsed feed-forward , and establish experimentally its empirical effectiveness ( Section 4 ) . 2 CHAIN GRAPH INTERPRETATION OF NEURAL NETWORKS . Without further delay , we derive the chain graph interpretation of neural networks in this section . We will state and discuss the main results here and leave the proofs in the appendix . 2.1 THE LAYERED CHAIN GRAPH REPRESENTATION . We start by formulating the so called layered chain graph that corresponds to neural networks we use in practice : Consider a system represented by L layers of random variables ( X1 , . . . , XL ) , where X li is the i-th variable node in the l-th layer , and denote N l the number of nodes in layer l. We assume that nodes X li in the same layer l have the same distribution type characterized by a feature function Tl that can be multidimensional . Also , we assume that the layers are ordered topologically and denote Pa ( Xl ) the parent layers of Xl . To ease our discussion , we assume that X1 is the input layer and XL the output layer ( our formulation can easily extend to multi-input/output cases ) . A layered chain graph is then defined as follows : Definition 1 . A layered chain graph that involves L layers of random variables ( X1 , . . . , XL ) is a chain graph that encodes the overall distribution P ( X2 , . . . , XL|X1 ) such that : 1 . It can be factored into layerwise chain components P ( Xl|Pa ( Xl ) ) following the topological or- der , and nodes X li within each chain component P ( X l|Pa ( Xl ) ) are conditionally independent given their parents ( this results in bipartite chain components ) , thus allowing for further decomposition into nodewise conditional distributions P ( X li |Pa ( Xl ) ) . This means we have P ( X2 , . . . , XL|X1 ) = L∏ l=2 P ( Xl|Pa ( Xl ) ) = L∏ l=2 N l∏ i=1 P ( X li |Pa ( Xl ) ) ; ( 1 ) 2 . For each layer l with parent layers Pa ( Xl ) = { Xp1 , . . .Xpn } , p1 , . . . , pn ∈ { 1 , . . . , l − 1 } , its nodewise conditional distributions P ( X li |Pa ( Xl ) ) are modeled by pairwise conditional random fields ( CRFs ) with with unary ( bli ) and pairwise ( W p , l j , i ) weights ( as we will see , they actually correspond to biases and weights in NN layers ) : P ( X li |Pa ( Xl ) ) = f l ( Tl ( X li ) , e l i ( Tp1 ( Xp1 ) , . . . , Tpn ( Xpn ) ) ) ( 2 ) with eli ( Tp1 ( Xp1 ) , . . . , Tpn ( Xpn ) ) = bli + pn∑ p=p1 Np∑ j=1 Wp , lj , iT p ( Xpj ) . ( 3 ) Figure 1 Left illustrates an example three-layer network as layered chain graph and its chain component factorization . In Eq . ( 2 ) , f l is an arbitrary function that represents a probability distribution . For exponential family distributions ( Koller & Friedman , 2009 ) , Eq . ( 2 ) simply becomes P ( X li |Pa ( Xl ) ) ∝ exp ( Tl ( X li ) · eli ( Tp1 ( Xp1 ) , . . . , Tpn ( Xpn ) ) ) . Note that layered chain graph has a globally directed graph structure and has an equivalent modeling based on directed graphical model ( Bayesian network ) ( Koller & Friedman , 2009 ) , we elaborate on this point for interested readers in Appendix A . 2.2 FEED-FORWARD AS APPROXIMATE PROBABILISTIC INFERENCE . To identify layered chain graphs with real-world neural networks , we need to show that they can behave the same way during inference and learning . For this , we establish the fact that feed-forward can actually be seen as performing an approximate probabilistic inference on a layered chain graph : Given an input sample x̃1 , we consider the problem of inferring the marginal distribution Qli of a node X li and its expected features q l i , defined as Qli ( x l i|x̃1 ) = P ( X li = xli|X1 = x̃1 ) ; qli = EQli [ T l ( X li ) ] ( q 1 = x̃1 ) . ( 4 ) Consider a non-input layer l with parent layers p1 , . . . , pn , the independence assumptions encoded by the layered chain graph lead to the following recursive expression for marginal distributions Q : Qli ( x l i|x̃1 ) = EQp1 , ... , Qpn [ P ( xli|Pa ( Xl ) ) ] . ( 5 ) However , the above expression is in general intractable , as it integrates over the entire admissible states of all parents nodes in Pa ( Xl ) . To proceed further , simplifying approximations are needed . Interestingly , by using linear approximations , we can obtain the following results ( in case of discrete random variable the integration in Eq . 7 is replaced by summation ) : Proposition 1 . If we make the assumptions that the corresponding expressions are approximately linear w.r.t . parent features Tp1 ( Xp1 ) , . . . , Tpn ( Xpn ) , we obtain the following approximations : Qli ( x l i|x̃1 ) ≈ f l ( Tl ( xli ) , e l i ( q p1 , . . . , qpn ) ) ; ( 6 ) qli ≈ ∫ xli Tl ( xli ) f l ( Tl ( xli ) , e l i ( q p1 , . . . , qpn ) ) dxli : = g l ( eli ( q p1 , . . . , qpn ) ) . ( 7 ) Especially , Eq . ( 7 ) is a feed-forward expression for expected features qli with activation function g l determined by Tl and f l , i.e . the distribution type of random variable nodes in layer l. The proof is provided in Appendix B.1 . This allows us to identify feed-forward as an approximate probabilistic inference procedure for layered chain graphs . For learning , the loss function is typically a function of ( QL , qL ) obtainable via feed-forward , and we can follow the same classical neural network parameter update using stochastic gradient descent and backpropagation . Thus we are able to replicate the exact neural network training process with this layered chain graph framework . The following corollary provides concrete examples of some common activation functions g ( we emphasize their names in bold , detailed formulations and proofs are given in Appendix B.2 ) : Corollary 2 . We have the following node distribution - activation function correspondences : 1 . Binary nodes taking values { α , β } results in sigmoidal activations , especially , we obtain sigmoid with α = 0 , β = 1 and tanh with α = −1 , β = 1 ( α , β are interchangeable ) ; 2 . Multilabel nodes characterized by label indicator features result in the softmax activation ; 3 . Variants of ( leaky ) rectified Gaussian distributions ( T li ( X l i ) = X l i = max ( Y l i , Y l i ) with Y l i ∼ N ( eli , ( s l i ( e l i ) ) 2 ) ) can approximate activations such as softplus ( = 0 , sli ≈ 1.7761 ) and -leaky rectified linear unit ( ReLU ) ( sli = tanh ( eli ) ) including ReLU ( = 0 ) and identity ( = 1 ) . Figure 1 Right illustrates activation functions approximated by various rectified Gaussian variants . We also plotted ( in orange ) an alternative approximation of ReLU with sigmoid-modulated standard deviation proposed by Nair & Hinton ( 2010 ) which is less accurate around the kink at the origin . The linear approximations , needed for feed-forward , is coarse and only accurate for small pairwise weights ( ‖W‖ 1 ) or already linear regions . This might justify weight decay beyond the general “ anti-overfit ” argument and the empirical superiority of piecewise linear activations like ReLU ( Nair & Hinton , 2010 ) . Conversely , as a source of error , it might explain some “ failure cases ” of neural networks such as their vulnerability against adversarial samples , see e.g. , Goodfellow et al . ( 2015 ) .

Summary: This paper tries to interpret neural networks with chain graphs that provides theoretical analysis on various neural network components. Furthermore, this chain graph interpretation has been used to propose a new approach (architecture), which is a partially collapsed feed-forward. A layered chain graph representation is adopted to formulate the neural networks with layered chain graphs. This further establishes to interpret feed-forward as an approximate probabilistic inference with using linear approximations. Some concrete examples are shown to be analyzed  based on the chain graph formulation.