Probabilistic modeling the hidden layers of deep neural networks

1 INTRODUCTION . Recently , interpreting the hidden layers of Deep Neural Networks ( DNNs ) as the Gaussian Process ( GP ) has attracted a great deal of attention because it provides a novel probabilistic perspective to clarify the working mechanism of deep learning . Neal ( 1994 ) initially demonstrates the equivalence between GP and neural networks with a single fully connected layer in the limit of infinite neurons . Following this seminal work , Lee et al . ( 2018 ) ; Matthews et al . ( 2018 ) further extend the equivalence to neural networks with multiple hidden layers and Garriga-Alonso et al . ( 2018 ) ; Novak et al . ( 2018 ) establish the correspondence between the convolutional layers and GP . It is important to note that the GP explanations rely on a fundamental probabilistic premise that the activations of a hidden layer are independent and identically distributed ( i.i.d. ) . More specifically , based on the classical Central Limit Theorem ( CLT ) , the prerequisite of an activation of a hidden layer approaching a Gaussian distribution is that all the activations of the previous layer are i.i.d .. Though the Lyapunov CLT could relax the probabilistic premise to be independence only , all the previous works do not thoroughly discuss the probabilistic premise except assuming an i.i.d . prior for all the parameters of DNNs . Due to its extreme importance but unclear status , it is necessary to thoroughly investigate the probabilistic premise for improving the interpretability of DNNs . In this work , we demonstrate that the parameters of DNNs are correlated and depend on the training dataset , i.e. , they do not satisfy the i.i.d . prior . Moreover , we demonstrate that activations being i.i.d . can not hold for all the hidden layers of DNNs . In the context of Bayesian probability , we theoretically derive the necessary conditions for the activations of a hidden layer being i.i.d . given the assumption that the activations of the previous layer are i.i.d .. Subsequently , we experimentally show that typical DNNs , such as the Multilayer Perceptron ( MLP ) and the Convolutional Neural Networks ( CNNs ) , can not satisfy the necessary conditions . As a result , activations being i.i.d . is not valid for all hidden layers . In other words , GP with the i.i.d . assumption for the parameters of DNNs can not correctly explain all the hidden layers of DNNs . In addition , some previous works show that GP is sensitive to the curse of dimensionality ( Bengio et al. , 2005 ; Hinton et al. , 2012 ) and can not clarify the hierarchical representation , an essential of deep learning ( Matthews et al. , 2018 ) . Alternatively , we propose a novel probabilistic representation for the hidden layers based on the Gibbs distribution ( LeCun et al. , 2006 ) and the Product of Experts ( PoE ) model ( Hinton , 2002 ) . The probabilistic representation explains a hidden layer in two aspects : ( i ) the distribution of a hidden layer can be formulated as a Gibbs distribution , of which the energy function is determined by the activations , and ( ii ) the connection between two adjacent layers can be modeled by a PoE model . Specifically , the distribution of a single neuron or hidden unit ( e.g. , convolutional channel ) can be expressed as a PoE , in which all the experts are defined by the neurons or hidden units in the previous hidden layer . Compared to the GP explanations , the proposed probabilistic representation provides a more specific explanation for the hidden layers of DNNs without any assumption . Since the output of a hidden layer is commonly the input of the next layer , DNNs form a Markov chain , thereby corresponding to a joint distribution . Moreover , we confirm the equivalence between the Stochastic Gradient Descent ( SGD ) and the variational inference ( Mandt et al. , 2017 ; Chaudhari & Soatto , 2018 ) and prove the former is also equivalent to the energy minimization optimization based on the proposed probabilistic representation . As a result , the entire architecture of DNNs can be explained as a Bayesian network ( Nielsen & Jensen , 2009 ; Koski & Noble , 2011 ) . In particular , we demonstrate that the hidden layers close to the input formulate prior distributions of the training dataset and propose a novel regularization approach to improve the generalization performance of DNNs through pre-training the hidden layers corresponding to prior distributions . 2 PRELIMINARIES . We assume X and Y are two random variables and Pθ ( X , Y ) = P ( Y |X ) P ( X ) is an unknown joint distribution between X and Y , where P ( X ) describes the prior knowledge of X , P ( Y |X ) describes the statistical connection betweenX and Y , and θ indicate the parameters of Pθ ( X , Y ) . A dataset D = { ( xj , yj ) |xj ∈ RM , yj ∈ RL } Jj=1 is composed of i.i.d . samples generated from Pθ ( X , Y ) . A neural network with I hidden layers is denoted as DNN = { x ; f1 ; ... ; fI ; fY } and trained by D , where ( x , y ) ∈ D are the input of the DNN and the corresponding training label , and fY is an estimation of the true distribution P ( Y |X ) . As a result , x ∼ P ( X ) and fY ≈ P ( Y |X ) . The meaning of fi is two-fold : ( i ) denoting all the neurons in the ith hidden layer , and ( ii ) representing the activations of the ith hidden layer . In addition , Fi and FY are the random variables corresponding to fi and fY , respectively . In this work , we use the MLP ( Figure 1 ) to derive most theoretical conclusions unless otherwise specified . 3 BACKGROUND . As a fundamental probabilistic model , Gibbs distribution ( a.k.a. , energy based model , Boltzmann distribution , or renormalization group ) formulates the dependence of random variables X through associating an energy to each dependence structure ( Geman & Geman , 1984 ) . P ( X ; θ ) = 1Z ( θ ) exp [ −E ( x ; θ ) ] ( 1 ) where E ( x ; θ ) is the energy function , θ denotes the parameters , and Z ( θ ) = ∫ x exp [ −E ( x ; θ ) ] dx is the partition function . Especially , a Gibbs distribution can be easily reformulated as various probabilistic models by redefining E ( x ; θ ) , e.g. , it would become Markov Random Fields ( MRFs ) model if the energy function is defined as the summation of multiple potential functions , i.e. , E ( x ; θ ) = − ∑ k fk ( x ; θk ) ( Wang et al. , 2013 ; Goodfellow et al. , 2016 ) . Moreover , assuming a single potential function defines an expert , i.e. , Fk = exp [ −fk ( x ; θk ) ] , we can regard MRFs as a PoE model , i.e. , P ( x ; θ ) = 1Z ( θ ) ∏ k Fk , where Z ( θ ) = ∏ k Z ( θk ) ( Hinton , 2002 ) . A Gibbs distribution has two appealing properties . First , E ( x ; θ ) is the sufficient statistics of P ( x ; θ ) because Z ( θ ) only relies on θ , i.e. , P ( x ; θ ) entirely depends on E ( x ; θ ) . The property establishes a connection between P ( X ; θ ) and the deterministic function E ( x ; θ ) , which allow us to explain a deterministic hidden layer in a probabilistic way . Second , the energy minimization is the commonly used optimization for learning θ , i.e. , θ∗ = arg minθ E ( x ; θ ) ( LeCun et al. , 2006 ) , which is helpful to explain the training procedure of DNNs , because the energy minimization can be implemented by the gradient descent algorithm as long as E ( x ; θ ) is differentiable . Numerous efforts have been devoted to explain DNNs from the viewpoint of Gibbs distribution . The classical work is the Restricted Boltzmann Machine ( RBM ) ( Salakhutdinov & Hinton , 2009 ) , which is a simplified Gibbs distribution . Specifically , RBM only considers the connections between visible units x and hidden units h but overlooks the internal connections within x and h , thus it can not explain complex DNNs , e.g. , CNNs . To the best of our knowledge , Mehta & Schwab ( 2014 ) first explain the distribution of hidden layers as a Gibbs distribution , and Yaida ( 2019 ) indirectly proves the distribution of fully connected layers is a Gibbs distribution . Lin et al . ( 2017 ) clarify some properties of DNNs based on the Gibbs distribution . However , all previous works still can not derive an explicitly Gibbs explanation for complex hidden layers , e.g. , convolutional layers . 4 MAIN RESULTS . 4.1 THE PARAMETERS OF DNNS DO NOT SATISFY THE I.I.D . ASSUMPTION . We examine the statistical properties of the parameters of DNNs and show they are correlated . First , we derive a necessary condition for the parameters of DNNs satisfying the i.i.d . prior , Second , we experimentally show that typical DNNs can not satisfy the necessary condition on the MNIST and CIFAR-10 datasets . Hence , the parameters of DNNs do not satisfy the i.i.d . assumption . If all the parameters of DNNs are i.i.d. , the parameters of each neuron are identically distributed , e.g. , { αmn } Mm=1 ∼ P ( αn ) in the MLP ( Figure 1 ) . Since all the parameters are i.i.d. , different neurons should be uncorrelated , i.e. , ∀n 6= n′ , Corr ( αn , αn′ ) = 0 . In particular , the i.i.d . assumption enables us to use the sample correlation r ( αn , αn′ ) to estimate Corr ( αn , αn′ ) . r ( αn , αn′ ) = ∑M m=1 ( αmn−αn ) ( αmn′−αn′ ) √∑M m=1 ( αmn−αn ) 2∑M m=1 ( αmn′−αn′ ) 2 ≈ Corr ( αn , αn′ ) ( 2 ) where αn = 1M ∑M m=1 αmn is the sample mean of { αmn } Mm=1 . Therefore , taking into account the estimation error , |r ( αn , αn′ ) | close to zero is the necessary condition for all parameters being i.i.d .. We examine the necessary condition for all parameters being i.i.d . in various DNNs . First , we use the MLP to classify the benchmark MNIST dataset . The information about the network architecture and training methods is included in Appendix A.1 . After training the MLP well , we calculate the absolute sample correlations , namely , |r ( αn , αn′ ) | , |r ( βk , βk′ ) | , and |r ( γl , γl′ ) | , and derive three absolute correlation matrices , i.e. , A′N×N , B ′ K×K , and C ′ L×L , shown in Figure 2 . We find that most |r ( αn , αn′ ) | are close to zero , but most |r ( βk , βk′ ) | and |r ( γl , γl′ ) | are non-zero . That indicates the necessary condition can not hold for all the parameters of the MLP . Second , we construct a shallow neural network with a single fully connected hidden layer , namely , NN = { x ; f ; fY } , for classifying the MNIST dataset based on the seminal work ( Neal , 1994 ) in Appendix A.2 , and construct a CNN for classifying the CIFAR-10 dataset in Appendix A.3 . Using the same method in the MLP , we obtain the absolute sample correlations between the weights of different nodes in the both neural networks . The results in Appendix A.2 and Appendix A.3 show that the weights of DNNs do not satisfy the necessary condition . Third , we visualize all the parameters of the NN = { x ; f ; fY } for classifying a synthetic dataset in Appendix A.4 . Since the synthetic dataset is much simpler than MNIST , we can directly show the correlation of different parameters and their dependences on the dataset by visualizing them . Overall , these simulations demonstrate that the parameters of DNNs are not i.i.d ..

Summary: The authors show that parameters of a DNN do not satisfy the i.i.d. prior assumption and that neural layer activations considered as i.i.d. are not valid assumptions for all hidden layers of the network. One can therefore not rightfully use GPs to describe the network’s hidden layers. The authors suggest formulating the neurons per layer as energy functions thereby rendering a hidden layer as a Gibbs distribution and the connection between adjacent hidden layers as a PoE model.