Probabilistic modeling the hidden layers of deep neural networks
Authors: Xinjie Lan
Kenneth E. Barner
Venue:  (2020)
Keywords: Neural Networks
Gaussian Process
Probabilistic Representation for Deep Learning

Abstract: In this paper, we demonstrate that the parameters of Deep Neural Networks (DNNs) cannot satisfy the i.i.d. prior assumption and activations being i.i.d. is not valid for all the hidden layers of DNNs. Hence, the Gaussian Process cannot correctly explain all the hidden layers of DNNs. Alternatively, we introduce a novel probabilistic representation for the hidden layers of DNNs in two aspects: (i) a hidden layer formulates a Gibbs distribution, in which neurons define the energy function, and (ii) the connection between two adjacent layers can be modeled by a product of experts model. Based on the probabilistic representation, we demonstrate that the entire architecture of DNNs can be explained as a Bayesian hierarchical model. Moreover, the proposed probabilistic representation indicates that DNNs have explicit regularizations defined by the hidden layers serving as prior distributions. Based on the Bayesian explanation for the regularization of DNNs, we propose a novel regularization approach to improve the generalization performance of DNNs. Simulation results validate the proposed theories. 

Under review as a conference paper at ICLR 2020
PROBABILISTIC MODELING THE HIDDEN LAYERS OF DEEP NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper, we demonstrate that the parameters of Deep Neural Networks (DNNs) do not satisfy the i.i.d. prior assumption and activations being i.i.d. is not valid for all the hidden layers of DNNs. thus the Gaussian process cannot correctly model the distribution of the hidden layers. Alternatively, we propose a novel probabilistic representation for the hidden layers of DNNS in two aspects: (i) a hidden layer formulates Gibbs distribution, of which the energy function is determined by the activations, and (ii) the statistical connection between two adjacent layers can be formulated as a product of experts model. Based on the proposed probabilistic representation, we further confirm the equivalence between the variational inference and the stochastic gradient descent, and prove the later is also equivalent the energy minimization optimization. As result, the entire architecture of DNNs can be viewed as Bayesian network, in which the hidden layers close to the input formulate the prior distributions of the training dataset. In addition, we propose new regularization approach to improve the generalization performance of DNNs by pre-training the hidden layers corresponding to the prior distributions. Simulation results validate the proposed theories.
INTRODUCTION
Recently, interpreting the hidden layers of Deep Neural Networks (DNNs) as the Gaussian Process (GP) has attracted great deal of attention because it provides a novel probabilistic perspective to clarify the working mechanism of deep learning. Neal 1994) initially demonstrates the equivalence between GP and neural networks with a single fully connected layer in the limit of infinite neurons. Following this seminal work, Lee et (2018) Matthews et al. (2018) further extend the equivalence to neural networks with multiple hidden layers and Garriga-Arai Alonso et al. (2018); Novak et al. (2018) establish the correspondence between the convolutional layers and GP.
is important to note that the GP explanations rely on a fundamental probabilistic premise that the activations of hidden layer are independent and identically distributed (i..d.). More specifically, based on the classical Central Limit Theorem (CLT), the prerequisite layer approaching Gaussian distribution is that all the activations of the previous layer are i.i.d.. Though the Lyapunov CLT could relax the probabilistic premise to be independence only, all the previous works do not thoroughly discuss the probabilistic premise except assuming an i.i.d. prior for all the parameters of DNNs. Due to its extreme importance but unclear status, it is necessary to thoroughly investigate the probabilistic premise for improving the interpretability of DNNs.
an activation of a hidden
In this work, we demonstrate that the parameters of DNNs are correlated and depend on the training dataset, i.c., they do not satisfy the i..dd prior. Moreover, we demonstrate that activations being i..d. cannot hold for all the hidden layers of DNNs. In the context of Bayesian probability, we theoretically derive the necessary conditions for the activations of a hidden layer being i.i.d. given the assumption that the activations of the previous layer are i.i.d.. Subsequently, we experimentally show that typical DNNs, such as the Multilayer Perceptron (MLP) and the Convolutional Neural Networks (CNNs), cannot satisfy the necessary conditions. As result, activations being i.i.d. is not valid for all hidden layers. In other words, GP with the i.i.d. assumption for the parameters of DNNs cannot correctly explain all the hidden layers of DNNs. In addition, some previous works show that GP is sensitive to the curse of dimensionality (Bengio et al.) 20055: Hinton et al., 2012) and cannot clarify the hierarchical representation, essential of deep learning (Matthews al. 2018).

---

Under review as a conference paper at ICLR 2020
Alternatively, we propose novel probabilistic representation for the hidden layers based the Gibbs distribution (LeCun et al. 2006) and the Product of Experts (PoE) model (Hinton 2002). The probabilistic representation explains a hidden layer in two aspects: (i) the distribution of hidden layer can be formulated as a Gibbs distribution, of which the energy function is determined by the activations, and (ii) the connection between two adjacent layers can be modeled by PoE model. Specifically, the distribution of a single neuron or hidden unit (c.g., convolutional channel) can be expressed as PoE, in which all the experts are defined by the neurons hidden units in the previous hidden layer. Compared to the GP explanations, the proposed probabilistic representation provides a more specific explanation for the hidden layers of DNNs without any assumption.
Since the output of a hidden layer is commonly the input of the next layer, DNNs form Markov chain, thereby corresponding to a joint distribution. Moreover, we confirm the equivalence between the Stochastic Gradient Descent (SGD) and the variational inference (Mandt et al., 2017 Chaudhari Soatto 2018) and prove the former is also equivalent to the energy minimization optimization based on the proposed probabilistic representation. As a result, the entire architecture of DNNs can be explained as a Bayesian network (Nielsen Jensen 2009; Koski & Noble 201T). In particular, demonstrate that the hidden layers close to the input formulate prior distributions of the training dataset and propose a novel regularization approach to improve the generalization performance of DNNs through pre-training the hidden layers corresponding prior distributions.
PRELIMINARIES
We assume X and Y are two random variables and = joint distribution between X and Y, where P(X) describes the prior knowledge of X, X describes the statistical connection between X and Y. and 0 indicate the parameters of (X Y).
is an unknown
dataset D ((x), y') w E RM y E 1= is composed of i.i.d. samples generated from P,(X, Y). neural network with I hidden layers is denoted as DNN {æ; fi;...; fr; fy} and trained by D, where (x, y) D the input of the DNN and the corresponding training label, and fy is an estimation of the true distribution XX.. As a result, x ~ P(X) and fY = X(Y The meaning of fi is two-fold: (i) denoting all the neurons in the ith hidden layer, and (ii) representing the activations of the ith hidden layer. In addition, F; and Fy are the random variables corresponding to Si and fY, respectively. In this work, derive most theoretical conclusions unless otherwise specified.
the MLP (Figure [ to
X1
f11
fas
d12
Pws
fx(1)
f12
Xyn
:
faz
fis
:
O
M2.))
:
fax
fax
f1
fz
fy
Figure 1: A MLP- (w; (w; f25 fr in which fi has N neurons, i.e., fi (hin = an (x)) where gin (x) = EM Qmn Tmn + bim is the nahh lincar filter, Omn is the weight of the edge between &m and fins bin is the bias, 01 is non-linear activation function. Similarly, f2 = fa2lges (fi)) where gos (fi) EN Bak fin + bag. In addition, since fy is the softmax, fr = xxeeepinn ( (fa)) where gor(2) = Σk= 7h sex + byt, and the partition function is Zy = Σt= (x2))
2

---

Under review as a conference paper at ICLR 2020
3 BACKGROUND
As a fundamental probabilistic model, Gibbs distribution (a.k.a., energy based model, Boltzmann distribution, renormalization group) formulates the dependence of random variables X through associating an energy to each dependence structure (Geman & Geman 1984).
P(X;0) z expl - 0)]
where F(:;; 0) is the energy function, O denotes the parameters, and Z = - exp|-F(ar; 0) is the partition function. Especially, a Gibbs distribution can be easily reformulated as various probabilistic models by redefining E(:; 0). e.g., it would become Markov Random Fields (MRFs) model if the energy function is defined as the summation of multiple potential functions, i.c., E(x; 0) = Σx f& (a:; 0x) (Wang al., 2013 Goodfellow et al., 2016). Moreover, assuming a single potential function defines an expert, I.e., F = expl [x;
[z; 0,)l. we can regard MRFs as PoE model, i.e., P(æ; 0) z II. Fx. where Z(0) - II 2(O,) (Hinton 2002).
A Gibbs distribution has two appealing properties. First, E' 0) is the sufficient statistics of P(x; 0) because Z(0) only relies on 0, i.e., P(x; 9) entirely depends on E(:; The property establishes a connection between P(X;0) and the deterministic function E(x; 0), which allow us to explain a deterministic hidden layer in a probabilistic way. Second, the energy minimization is the commonly used optimization for learning 0, i.e., 0* = arg ming Exx; 0) (LeCun et al. 2006), which helpful to explain the training procedure of DNNs, because the energy minimization can be implemented by the gradient descent algorithm as long as E(x; 0) is differentiable.
Numerous efforts have been devoted to explain DNNs from the viewpoint of Gibbs distribution. The classical work is the Restricted Boltzmann Machine (RBM) (Salakhutdinov & Hinton| 2009), which is a simplified Gibbs distribution. Specifically, RBM only considers the connections between visible units and hidden units h but overlooks the internal connections within æ: and h, thus it cannot explain complex DNNs, e.g., CNNs. To the best of our knowledge, Mehta & Schwab) (2014) first explain the distribution of hidden layers proves the distribution of fully connected layers is a Gibbs distribution. Lin et al. (2017) clarify some properties of DNNs based on the Gibbs distribution. However, all previous works still cannot derive an explicitly Gibbs explanation for complex hidden layers, c.g., convolutional layers.
Gibbs distribution, and Yaida (2019) indirectly
4
MAIN RESULTS
4.1 THE PARAMETERS OF DNNS DO NOT SATISFY THE L.D.D ASSUMPTION
We examine the statistical properties of the parameters of DNNs and show they are correlated. First, we derive necessary condition for the parameters of DNNs satisfying the i.i.d. prior, Second, we experimentally show that typical DNNs cannot satisfy the necessary condition on the MNIST and CIFAR-10 datasets. Hence, the parameters of DNNs do not satisfy the i.i.d. assumption.
all the parameters of DNNs are i.i.d., the parameters of each neuron are identically distributed, c.g., {amm) m=1 ~ P(a.) in the MLP (Figure Dp. Since all the parameters are i.i.d., different neurons should be uncorrelated, i.c., Vn a n', Corr(a (x,)) = O. In particular, the i.idd assumption enables to use the sample correlation r(an, Ott)) to estimate Corr (an, ).
EM (0mn - (0)m
r(Omn Onn) V (0mn o..) "> (Ommn' o
Corr Onr)
1 where C. = M EM=1 Omn is the sample mean of {amn m= Therefore, taking into account the estimation error, |r( (r(a,,, close to zero is the necessary condition for all parameters being i.i.d..
1M
We examine the necessary condition for all parameters being i.i.d. in various DNNs. First, we use the MLP to classify the benchmark MNIST dataset. The information about the network architecture and training methods is included in Appendix A. After training the MLP well, we calculate the
absolute sample correlations, namely, [r(an, Ox)),, ( |r(3k, (xr),, and [r(,, ((),>>>). and derive three absolute correlation matrices, i.e., A' No B' K. and C² shown in Figure 2 We find that most
Ir(a,, a..))l are close to zero, but most [r Brr) and |r(7, % are non-zero. That indicates the necessary condition cannot hold for all the parameters of the MLP.
3

---

Under review as a conference paper at ICLR 2020
(0.08)
f2 (0.26)
fy (0.31)
0.8
0.8
0.8
0.6
0.4
0.2
0.2
0.2
AN x
Bxx
CLXL XL
Figure 2: Three absolute correlation matrices for the weights in fi, f2, and fr. In addition, 0.08, 0.26, and 0.31 are average absolute correlation coefficients for all the weights fa, f2, and fr, respectively.
Second, construct shallow neural network with single fully connected hidden layer. namely, NN = <x: f: fr for classifying the MNIST dataset hased on the seminal work (Neal 1994) in Appendix A.2] and construct CNN for classifying the CIFAR-10 dataset in Appendix A.3 Using
the same method in the MLP, we obtain the absolute sample correlations between the weights of different nodes in the both neural networks. The results in Appendix A.2 and Appendix A.3 show that the weights of DNNs do not satisfy the necessary condition.
Third, we visualize all the parameters of the NN {æ; f: fy } for classifying synthetic dataset in Appendix AA.4 Since the synthetic dataset is much simpler than MNIST, we can directly show the correlation of different parameters and their dependences on the dataset by visualizing them. Overall, these simulations demonstrate that the parameters of DNNs are not i.i.d..
4.2 ACTIVATIONS BEING I.I.D IS NOT VALID FOR ALL THE HIDDEN LAYERS OF DNNS
Although we demonstrate that the parameters of DNNs do not satisfy the i.i.d. assumption, we are still unclear whether the activations of hidden layer are i...d. In this section, we demonstrate that activations being i.i.d. is not valid for all hidden layers DNNs.
In the context of Bayesian probability, all the weights and biases of the MIP (Figure T have prior distributions, i.e., Omn P(Am)), Buk ~ P(Wnk), bn, P(B1n), and b2x ~ ((Bza), thus we regard Gin ΣM AmnX Xm + Bin as the random variable of gi (x) = Σm -m==1 Omn Fm + bin- and G2 Σ> Wnx Fin + Bz as the random variable of (fx) D Ank fin + dau- Hence, the random variables of two arbitrary activations in fi and f2. e.g., fin and f2x. can be expressed as Fin- (Gi,) and P22 (G2x), respectively.
Given two random variables A and B, the necessary conditions for them being i.i.d. are that they are uncorrelated (i.e., Cov(A, B)= 0) and have the same expectation (i.c., E(A) E(B)). Similarly, assuming [Fin 1== are i.i.d. and [.fin]. J.n ~ P(Fi), we derive the following necessary conditions for the activations of the second hidden layer, i.e. .{ (22 K K= being i.i.d..
v(k,k') E S1 ((k EZ S K, 1 K' Cov(Gax G2w) Cou(WN, Ww [²3 Cov(Wk, 32x)E((1)
K). must have
(3)
B2x + Bo) + Cou Bav) =
((F))((W)) + E(B2z) ((F)) E(Brx)
(4)
(-(-)
) is strictly increasing, differentiable, and invertible.
(5)
where W. ΣN Wak- Equation and are the necessary conditions for (G being i.d... Equation 5 constrains the activation function o2(.) to guarantee that if (G2e (G2))Kn are i.i.d. then {F2x k are i.i.d. as well. The necessary conditions are valid for arbitrary fully connected lavers long properly changing the subscripts. The detailed derivations are included in Appendix and and
Appendix [ Moreover, we prove that the necessary conditions also hold for convolutional layers based the connection between fully connected layers and convolutional layers in Appendix

---

Under review as a conference paper at ICLR 2020
Table 1: The statistical measures for examining the necessary conditions
Layer
Activation expectation
Covariance
Weight expectation
f1 f2
E(X)
Cov( An, A.d)) Cov(W We)
E(An)
B(R)
E(W
fy
2
Cov(CC, Cv)
E(Ci)
(Im) M
(mm)m=)
P(X), (fin)=11 ~ P(Fi), (f26)
P(12)
DM 2m Amn, where Omn P(Amn)
W4 En Wak. where Boo P(Wn))
C 2k Cki where "hk P(Cu)
noteworthy that the i.i.d. assumption for the parameters of DNNs indeed satisfy the necessary conditions for activations being i.i.d.. Nevertheless, since the i.i.d. prior is not an appropriate prior for the parameters of DNNs, it cannot guarantee activations of a hidden layer being i.i.d..
The theoretical necessary conditions indicate an applicable approach to examine whether activations of hidden layer being i.i.d. given the assumption that the activations
previous layer are i.i.d.. More specifically, since {F J. are i.i.d., we can use the sample mean fi to estimate E [F ). Because the samples of the training dataset 'D are also assumed to be i.i.d., we can train the MLP multiple times to draw multiple independent observations of n. and B2s, thus we can use the sample covariance to estimate Cov(We, Ww,), Cou(W B2w ). and etc.
Based on the empirical approach, we estimate the statistical measures (Table to examine whether the activations of the three layers (i.e., f1, fz. and fy) being i.id. in the MLP for classifying the MNIST dataset. Without loss of generality, we restrict all the layers the MLP from using bias to decrease computation complexity, thus the necessary conditions for f2 be simplified
V(k, E S1 = {(k, k') E Z W, & & K, & M & K}, we must have Cov(G2k G2w) Cou(WA, WR) 0
(6)
((F)) = E(F))E(W>)
First, Figure shows the correlation matrices, i.e., ANXN, WK*K. and CLx to check if their cor- responding covariances, i.e, Cou(An ), Cou(WR, Wx and Cov Ck+), are close to zero. It shows that many correlations are far from zero, so their corresponding covariances are not close to zero either. Second, we use the sample mean, e.g., ær to estimate Taking into account the estimation eπrоr,
expectation, e.g., E(X). regard E(X) 0 based x 0.131. However, fi 0.436 and f2 0.498 imply E(Fi) * O and E(F) j 0 even considering the estimation error. Therefore, the activations of the last two hidden layer, i.e., f2 and fY, cannot be independent because they cannot satisfy the first necessary condition (Equation 6 In addition, necessary condition for activations being identically distributed also cannot be satisfied by the MLP in Appendix F.I Furthermore, i..d. also cannot be satisfied in the CNN for classifying the CIFAR-10 dataset in Appendix
show the
demonstrate that the necessary conditions for activations being
f
f2
fy
1.0
1.0
1.0
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
AN
Wk
CL
Figure 3: Three absolute correlation matrices for showing whether Cov(An, And) Coo(Ws, We), and Coe(Ca, Cer) ) are close to zero.
5

---

Under review as a conference paper at ICLR 2020
Overall, conclude that activations being i.idd is not valid for all the hidden layers of DNNSs based on the theoretical necessary conditions and the experimental simulations on typical DNNs. In other words, cannot use the CLT to establish the equivalence between GP and all the hidden layers of DNNs, thus GP can not correctly clarify all the hidden layers of DNNs. That necessitates a more general probabilistic representation for explaining the hidden layers of DNNs.
4.3 THE DISTRIBUTION HIDDEN LAYER GIBBS DISTRIBUTION
This section proves that the distribution of hidden layer can be formulated as Gibbs distribution and the statistical connection between two adjacent layers can be modeled as a PoE model.
Assuming the output layer in the MLP (Figure I is the softmax, its distribution can be expressed as
P(Fr) (P(w)) = Zy
-ep((xx))
where Z = Et exp(Srt) is the partition function. Since J -EES 7ke s2x + byt, we have
K nal f2x + bui
P(Fy) = (P(fa)
exp
(9)
Zy
k=1
Based the properties of the exponential function, i.e., exp(a + = exp(a)) exp(b) and exp(a b) = [exp(b)]", we can reformulate P(Fx) as
P(Fy) (P(Fa) 1 Z II (exp(f2x)
(10)
where Z Zy /exx(bbbr). Since (exp(f2x)1) are scalar, we can introduce a new partition function ZF, Ex exp(fex) such that {zF, exp f2)( become a probability measure. As result, can further reformulate P(Fx) (P(s)t f as Product of Expert (PoH) model
K
((FY)=(P(FF = Z
exp(fxx)
He
(11)
ZF.
k=1
where Z = Zy Ze/(ex((b,)) II FZ [7m1 and each expert is defined Zrz exp(
It is noteworthy that all the experts { ZF, exp( f2x) form probability measure and establish an exact one-to-one correspondence to all the neurons in the second hidden layer fz, i.e., {fzx 1f2 Therefore, the distribution of f2 can be expressed as
P(F2) (P(f2)) ZF. exp(f2x))
(12)
Based on the definition of Gibbs distribution, Equation g and 12 show that fy and f2 formulate two multivariate Gibbs distributions and their energy functions are equivalent to the negative of the output nodes 1fg and the neurons { / respectively. In addition, Equation 11 limplies that the connection between P(fyt) and P(f2x) can be expressed as the PoE model, namely that P(fx)
equal to the product of all the experts {P(fa))]K_ with their respective weights {
Moreover, prove that the probabilistic representation is valid for other hidden layers in the MLP
Since (/2x- 22 Bnk Sin + b2a b2x) P(Fz) be expressed
N
exp exp(a2( P(F2) = (P(22)) Zr,
Σ Buk fin + b2
(13)
n=]
Based on the equivalence between the gradient descent algorithm and the first order approximation Battiti 1992), we prove that oz2 (EN Bak fIn + b2x)] can be approximated as
N
Bax fin + b2n)) ) s C21
E n=
Bak sin + bax] + O22
(14)

---

Under review as a conference paper at ICLR 2020
where C21 and C22 only depend the activations {fin 1inn)n in the previous training iteration, thus they can be regarded as constants and absorbed by Bu and b2g. The proof for the approximation is included in Appendix G Therefore, P(Fz) also can be modeled PoE model.
P (P((2e)
II
1 Z
exp(f(n))
(15)
F.
where Z%2 = Zzz/ /exp/exp(ba)) II [2,, [ and ZF, = En exp(fin).
Finally, the distribution of the first hidden layer f1 formulated
P(R)) = (P(im)
=exp(fin))
(16)
ZF.
Overall, we introduce a novel probabilistic representation for fully connected layers in two aspects. First, a fully connected layer formulates multivariate Gibbs distribution (Equation g and T6p and the energy functions is equivalent the negative activation. Second, the connection between two adjacent fully connected layers can be formulated as a PoE model (Equation [1 and 15
The proposed probabilistic representation has two advantages over the GP explanations. First, the FoE explanation does not require assumption, e.g., we formulate P(J2x) as PoE model, in which all the experts izz,, exp(fin)) 3. are defined by all the neurons {fin Jin As a comparison,
Gaussian distribution in the limit of infinite neurons, i.e., the GP explanation derives P(f2x) [_in]== are i.i.d.. Second, the Gibbs distribution provides (Jin) under the assumption that 1N
more explicit explanation for the functionality of fully connected layer. Specifically, Gibbs distribution derives the probability of the multiple features defined by the neurons occurring in the input, but the covariance function of GP cannot clearly explain the functionality of each neuron.
Moreover, we prove that convolutional layer formulates a specific Gibbs distribution, namely the MRF model, in Appendix H More specifically, a convolutional layer fi with K convolutional f + by. .)] formulate MRF model as channels, i.e., fa = ( = o, 12 ((k,m)
K expl Σ
P(F.) ZF. II explp)((i
1
(ft-1)
(17)
Zp.
where fi-1 (f is the input, convolutional channel, ((k,,) is the kth convolutional filter applying to the nth input channel, 6 the bias, (-)-) is non-linear operator, and the partition function /r,=ʃ (f=-)))dff The corresponding energy function Ep, =- Σt_ p (fi-1)-
- S(K.m) f by: 1) is the kth linear
In summary, the distribution of hidden layer fü can be formulated Gibbs distribution P(F.), especially the energy function EF, establish a connection between the deterministic function of the hidden layer and the Gibbs distribution P( (FF)). The correspondence between Gibbs distributions and hidden layers can be summarized by Table
Table 2: The correspondence between hidden layers and Gibbs distributions
Architecture
neuron fin
fully connected layer f1 {fin J !
conv. channel ft
conv. layer fi thi fk=1
energy function
discrete Gibbs (
energy function Ep:=-EN o (fi-a)
MRF pp, K v((f-1))
Explanation
Ep, =m
We use Equation| 16 and Equation] as examples to illustrate the correspondence.
7

---

Under review as a conference paper at ICLR 2020
Input
1,00
Neuron1
0.6
Neuron2
Neuron3
0.75
0.4
0.4
0.50
0.25
0.2
0.2
0.00
0.0
0.0
-0.25
-0.2
-0.2
-0.5
-0.4
-0.4
-1.0
-0.75
0122009/1.000.00
-0.6
(113.70/1.0g) 70.00//0000.20)
-0.6
(-38.53/0.0/0.07
Neuron4
Neuron5
0.6
Neuron6
Neuron7
0.6
0.5
0.2
0.2
0.2
0.0
0.00
-0.2
-022
-0.25
-0.2
-0.50
(140.20/1.0/0.20)
-0.8
(28.87/1.0/0.20)
-0.6
+133.82/0.0/00.00
-0.5
(-133.36/) 1133.36/0.0/0.077
-0.6
Figure 4: The top-left figure shows 32 x2 synthetic image as the input, which is sampled from the Gaussian distribution (00, 1) and sorted in the primary diagonal direction by the descending order. The remaining figures visualize all the weights {amm) 1024 of the seven neurons. In addition, the subscript in each figure, ie., fu(),, P(.)). denotes the linear output gn (w). activation fa (a)), and Gibbs probability P(fx).
SIMULATIONS
This section uses synthetic datasets to demonstrate the proposed probabilistic representation: (i) the distribution of a fully connected layer is a multivariate discrete Gibbs distribution and (fi) the distribution convolutional layer is MRF model.
5.1 THE DISTRIBUTION A FULLY CONNECTED LAYER MULTIVARIATE DISCRETE GIBBS DISTRIBUTION
A fully connected layer with N neurons, i.e., {f ) formulates multivariate discrete Gibbs distribution to describe the probability of the N features defined by their respective neurons occurring in the input. The energy function is equal to the negative activation, i.c., Es, =nxx
1 (x)] ZF
(18)
where (x: olgn (z)),, sn (=)= Omn -n is a linear filter with Omn being the weights and by being the bias, (-) is the activation function, and Zr >>=>>= exp]fa(x)) is the partition function.
Since benchmark datasets consist of very complex features and typical DNNs have too complicated architecture, it is very hard to directly demonstrate the Gibbs distribution explanation. Alternatively. choose the neural network for classifying the synthetic dataset in Appendix A..4 because the neural network only has a single fully connected hidden layer with seven neurons, i.e., N 7, and the synthetic dataset only has four features. After the network trained well, we choose synthetic image the input and show all the information of the learned hidden layer in Figure 4
Since the input image is sorted in the primary diagonal direction, the input feature can be described as high values above the secondary diagonal (the red line) and low values below the red line. Figure shows that Neuron 1, Neuron 2 and Neuron correctly deseribe the input feature, thus they derive large positive outputs (i.e., 91 (=x = 122.09, (2(x=) = 113.70 10, and (x 140.20). However, Neuron and Neuron do not describe the input feature correctly, thus they derive large negative outputs (i.e., (x(x) - 134.08, g7(x)) ) 133.36). After applying the sigmoid function, we have fi = fz(x) = (z) = and fe = fr(x) = 0. Finally, we derive the probability of different neurons, i.e., P(f1) P(fe) P(fa) 0.20 and P(fa)
P(fr) 0.07, which indicates that the features defined by Neuron I, Neuron and Neuron 4 have high probability occurring in the input but the features defined by Neuron and Neuron 7 have low probability occurring in the input.
8

---

Under review as a conference paper at ICLR 2020
Digit
Digh
5
10
Figure 5: The first shows five synthetic images of handwritten digits, the second visualizes their respective histograms, and red curve indicates Gaussian distribution. N(0, 1024).
5.2 THE DISTRIBUTION CONVOLUTIONAL LAYER IS A MRF MODEL
Since the distributions of benchmark datasets are unknown, it is impossible to use them to verify the probabilistic explanation for convolutional layers. Alternatively, generate synthetic dataset obeying the Gaussian distribution (N00, 1024) based on the NIST dataset of handwritten digits. It consists of 20,000 32 32 grayscale images in 10 classes (digits from 9), and each class has 1,000 training images and 1 1,000 testing images. Figure 5 shows five synthetic images and their perspective histograms. The method for generating the synthetic dataset included in Appendix [ We design CNN { f1; f2; f3i fa; fy } for classifying the synthetic dataset. The CNN has two convolutional layers with ReLU (i.c., fi and fa), two max-pooling layers (i.c., f2 and fa) and softmax output layer fr. The detailed information about the CNN is included in Appendix
Based on the MRF explanation (Equation the distribution of f1 can formulated as
20
P(F1) ZF,
II exp (x)] ZF. k=1
exp
(x)]]
(19)
where - (x) S ox + the kth linear convolutional channel, S the linear convolutional filter and bf is the bias. The energy function of P(Fi) is equivalent to EF, = = A=1 fh.
In order to model high dimensional dataset, e.g., the 32 32 synthetic image, typically need a multivariate distribution with the same dimension, e.g., the dimension of the covariance function of a Gaussian process should be 1024 x 1024. However, most structured probabilistic graphical models, especially the MRF model, have two fundamental assumptions: stationary and Markov (Lyu & Simoncelli 2007). The former means that the distribution independent on the location, i.e. the distribution of pixels at different locations are identically distributed. The later indicates that the distribution of single pixel is independent on the other pixels given its neighbors. The two assumptions allow to use the histogram of samples to simulate the true distribution.
Digit
P(X) KLD:0.62
P(E) KLD:0.83
Tog P(X)
Loy P(F))
20
25
30
13.9
10-3
fi
The red curve indicates the truly prior distribution P(X) (00,, 1024). The blue curves are different histograms. (A) synthetic image æ is the input. the histogram of x, i.c. P(x), and KL(P(X)) = 0.62. (C) the histogram of Er, for estimating P(Fi) and (L)((())))(FF)) = 0.83.
Figure 6:

---

Under review as a conference paper at ICLR 2020
Energy
Channel1
Channel2
Channeld
KLD:0.94
KLD:0.49
KLD:10.16
KLD:16.15
KLD21.82
KLD:44.77
10
10
10
10
10
10
10
10
10
10
-000
100
-100
100
-100
Figure 7: The firs row shows synthetic image, the energy functions of P(F1) and five different experts, i.c., (P(FT) ZFr exp(ff")] The second row shows their respective distributions, and the KLDss denote
their distances to the distribution, ie.. KLD[P(X))IP(FF))
After the CNN well trained, we randomly choose testing image æ as the input to derive P(Fi ((F)) Since ~ P(X), we can use the histogram of the testing image, i.c., P(x). to estimate P(X) In addition, Er, is a sufficient statistics for P(Fi) such that we can use the histogram of Er, to estimate P(F1). As a result, we can verify the probabilistic representation by calculating the distance between P(Fi) and P(X), i.e,, KL P (P())))(F) We visualize their distributions in Figure which shows that ((F)) is very close to P(X), i... (L(P(X)]lq(F)) demonstrates the distribution of a convolutional layer can be modeled as a MRF model.
0.83, which
In addition, the experiment provides new viewpoint to demonstrate the activations of different convolutional channels are not i.i.d. Given another synthetic image, we visualize the output five convolutional channels in f1 in Figure 7 It shows that the activations of different channels at the same position are correlated, i.e., they are not independent. We also estimate the distribution each convolutional channel based on the histograms of their respective energy functions. It shows that the distributions of different channels are different, i.e., they are not identically distributed.
DISCUSSION
Based on the proposed probabilistic explanations for hidden layers, we discuss certain theoretical and practical topics of deep learning. First, further confirm the equivalence between SGD and the variational inference. Second, we demonstrate that the entire architecture of DNNs can be explained Bayesian network, in which the hidden layers close the input formulate prior distributions of the training dataset. Third, we propose a novel regularization method to improve the generalization performance of DNNs by pre-training the hidden layers corresponds the prior distributions.
6.1 SGD CAN EXPLAINED A VARIATIONAL INFERENCE
Recent works prove that SGD works as specific Bayesian posterior inference, i.e., the variational inference (Mandt et al. 2017 Chaudhari & Soatto 2018). Here we further confirm the Bayesian explanation for SGD based on the proposed probabilistic representation. Briefly speaking, given a DNN = {x; fi;...; fz: J. we prove that the conditional distribution of fx given æ, ie.. P(Fy IX), is also Gibbs distribution and its energy function is equivalent to the deterministic function formulated by the entire architecture of the DNN, i.e., the DNN describes a family of Gibbs distribution Q. Based the definition of variational inference. SGD aims to find Gibbs distribution Pa ( such that it can minimize the distance between P PXFr|X) and the true posterior distribution P X) determined by the training dataset D.
P" ( arg min (L) PEQ
(20)
Moreover, we prove that SGD is also equivalent to the energy minimization optimization, which in turn validates the Gibbs explanation for the hidden layer of DNNs. The proof is in Appendix [
10

---

Under review as a conference paper at ICLR 2020
We justify the theoretical explanation for SGD based on the CNN classifying the synthetic dataset and the MLP classifying the MNIST dataset in Appendix K.I In addition, we demonstrate that the learned parameters via SGD are not i..d. based on in Appendix K.2 which indirectly justifies that GP with the i.i.d. assumption for the parameters of DNNs cannot correctly elarify DNNs.
6.2 THE ENTIRE ARCHITECTURE OF DNNS CAN BE EXPLAINED AS A BAYESIAN NETWORK
Since the output of the hidden layer fi in the DNN {æ; fri is commonly the input of the next layer fi+1, 1, the DNN formulates Markov chain F1 + F, Fy. As result, the distribution of hidden layer the DNN should be regarded as a conditional distribution and the entire architecture of the DNN can be formulated joint distribution.
P(Fy; FI..; Xx = P(Py
P(P++1)F) P(F.())
(21)
Richard & Lippmann 1991 prove that P(Fy (X) an estimation of the true posterior distribution ) & P P(Y) and we demonstrate that P is a prior distribution of the training dataset and the equivalence between SGD and the variational inference, thus the entire architecture of DNNs can be explained as Bayesian network. That provides probabilistic explanation for the
hierarchy property of DNNs, which cannot be clarified by GP (Matthews el al. 2018
6.3 A NOVEL REGULARIZATION ALGORITHM FOR DNNS
Bayesian theory indicates that a better prior distribution corresponds to a better posterior distribution given the same likelihood distribution, thus pre-training the hidden layers corresponding to prior distributions should an effective approach to improve the generalization performance of DNNs. Erhan et al. (2010) first demonstrate that the unsupervised pre- pre-training is effective regularization for DNNs through capturing the prior knowledge of the input. However, since they do not know which hidden layer corresponds to prior distribution, they have pre-training all the hidden layers, which restricts its wide application due to the computation complexity. In addition, [e et al. (2018) prove that incorporating the prior knowledge of the input results in uniform stability and provides bound on generalization error for a specific DNN, i.e., the supervised auto-encoders.
Based on the proposed probabilistic representation, we first justify that a better prior distribution indeed has positive correlation to the generalization performance in Appendix[K. Subsequently, we propose a novel novel regularization algorithm improve the generalization performance of DNNs through pre-training the hidden layers only corresponding to prior distributions and show that outperforms classical regularizations, e.g., dropout (Srivastava al., 2014), for the image recognition task on the benchmark CUFAR-10 dataset in Appendix
7 CONCLUSION
In this paper, we demonstrate that the parameters of DNNs do not satisfy the i.i.d assumption and the activations being i.i.d. cannot be valid for all the hidden layers DNNs. As a result, GP cannot correctly explain all the hidden layer of DNNs. Alternatively, we propose novel probabilistic representation for DNNs in four aspects: (i) a hidden layer formulates a Gibbs distribution, which the energy function is determined by the activations, (ii) the connection between two adjacent layers can be modeled by a PoE model, (iii) the entire architecture of DNNs can be explained as a Bayesian network, and (iv) the SGD can be explained as the Bayesian variational inference algorithm.
There are two general directions for future research. The theoretical direction to examine the probabilistic representation in other popular DNNs, e.g., the residual networks, which further verify the proposed probabilistic representation. The practical direction is to validate the proposed regularization in more complex DNNs and datasets.
11

---

Under review as a conference paper at ICLR 2020
REFERENCES
Roberto Battiti. First and second order methods for learning; Between steepest descent and newton's 's method. Neural Computation, 1:141-666 1992.
Yoshua Bengio, Olivier Delalleau, and Nicolas Le Roux. The curse of dimensionality for local kernel machines. Technical Report 1258, 2005.
David Blei, Alp Kucukelbir, and Jon MaAuliffe. Variational inference: A review for statisticians. Journal of Machine Learning Research, 112:859-877 2017.
M. A. Carreira-Perpinee and Geoffrey Hinton. On contrastive divergence learning. Artificial Intelligence and Statistics, pp. 33-41, 2005.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. 2018 Information Theory and Applications Work- shop (ITA), pp. 1-10. IEEE, 2018.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 1 1((Fe)):255-66 660, 2010.
Adri Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison. Deep convolutional net- works shallow gaussian processes. In ICLR, 2018.
Geman and D. Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions. on Pattern Analysis and Machine Intelligence, pp. 721-741, June 1984.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
K Hc, X Zhang, S Ren, and J Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
G. Hinton, I.. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. Deep neural networks for acoustic modeling speech recognition. IEEE Signal Processing Magazine, 2012.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14:1771-1800, 2002.
Timo Koski and John Noble. Bayesian networks: an introduction, volume 924. John Wiley & Sons, 2011.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technique Report, 2009.
Lei Le, Andrew Patterson, and Martha White. Supervised autoencoders: Improving generaliza- tion performance with unsupervised regularizers. Advances in Neural Information Processing Systems, pp. 107-117, 2018.
Y. LeCun, Bottou, and P. Haffner. maddient-based learning applied document recognition. Proceedings of the IEEE, 11:2278-2324, 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, Mare' Aurelio Ranzato, and Fu Jie Huang. futorial on energy-based learning. MIT Press, 2006.
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In ICLR, 2018.
Henry W Lin, Max Tegmark, and David Rolnick. Why does deep and cheap learning work so well? Journal of Statistical Physics, 1223-1122211241 2017.
Siwei Lyu and Ecro. P. Simoncelli. Statistical modeling of images with fields of gaussian scale mixtures. In NeurlPS, 2007.
12

---

Under review as a conference paper at ICLR 2020
Stephan Mandt, Matthew D Hoffmann and David M Blei. Stochastic gradient descent approximate bayesian inference. The Journal of Machine Learning Research, 18(1):4873-4900 2017.
Alexander Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, and Zoubin Ghahramani. Gaus- sian process behaviour in wide deep neural networks. In ICLR, 2018.
Pankaj Mehta and David Schwab. An exact mapping between the variational renormalization group and deep learning. arXiv preprint arXiv:1410.3831, 2014.
Radford M. Neal. Priors for infinite networks. Technical report no. crg-tr-94-1, 1994.
Thomas Dyhre Nielsen and Finn Verner Jensen. Bayesian networks and decision graphs. Springer Science & Business Media, 2009.
Roman Novak, Lechao Xiao, Jachoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abo- lafia, Jeffrey Pennington, and Jaseha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. In ICLR, 2018.
M. D. Richard and R. Lippmann. Neural network classifiers estimate bayesian a posteriori proba- bilities. Neural Compution, 3:461-483, 1991.
Stefan Roth and Michael J. Black, Fields of experts: A framework for learning image priors. IEEE Conf. on CVPR, pp. 860-867, July 2005.
Stefan Roth and Michael Black, Fields of experts. International Journal of Computer Vision, 82: 205-229, 2008.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323:533-536, October 1986.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. AISTATS 2009, pp- 448 455, 2009.
U. Schmidt, K. Schelten, and Roth. Bayesian deblurring with integrated noise estimation. In IEEE Conf. on Computer Vison and Pattern Recognition (CVPR), pp. 2625-2632, 2011.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.
C. Wang, N. Komodakis, and N. Paragios. Markov random field modeling inference and learning in computer vision and image understanding: A survey. Computer Vision and Image Understanding, 117:1610-1627, 2013.
Sho Yaida. Non- gon-saassian processes and neural networks finite widths. arXiv:191000019, 2019.
arXiv preprint
13

---

Under review as a conference paper at ICLR 2020
Table 3: The architecture of the MLP for MNIST classification
Layer
Description
Dimension
Filter dimension (M x N)
Input
784
f1
FC(sigmoid)
128
128 x 784
f2
FCisigmoid)
64
64 x 128
fY
Output(softmax)
10
10 64
FC denotes the fully connected layer
Dimension: the number of nodes (i.e., neurons) in every layer.
Filter dimension (M x N): a layer has M filters with N x dimension
A THE PARAMETERS OF DNNS CANNOT SATISFY THE L.I.D PRIOR
A.I A MLP ON THE MNIST DATASET
We design MLP (Figure D to classify the benchmark MNIST dataset. The MLP has two hidden layers, each hidden layer fully connected layer with the sigmoid activation function, and the output layer is the softmax. Table 3 summarizes the architecture of the MLP.
After the MLP well trained (its training and testing accuracy is 97.7% and 97.1%, respectively), we quantify the correlation of different weights in the same layer based on Equation 22 and derive the correlation matrix for all the weights every hidden layer.
ΣM (amm
an (amm' ann)
r(ak,, On') =
2 Corr(an,
(22)
Σ Em= (0mn
(h.) DM (amn' (In))
2
where or,. = Em= Own is the sample mean of {Aim I.
To examine if different weights coefficients are close to zero, so we take the absolute value of the correlation matrix element-wisely, Figure visualizes the correlation matrices of each hidden layer, i.e., A'N B'x and Clxx
correlated,
only need check whether their correlation
We find that most weights in fi are uncorrelated because their correlation coefficients are very close zero, i.e., [r(Or,, Or,)) ] 0, but many weights in f2 and fy are correlated to others, i.e.. [r( Bw)) * 0 and [r(7. ( ) ş 0. In addition, we derive the average of the absolute correlation coefficients of all the weights in each layer and show them at the top of Figure g which further validates that the weights of some hidden layer are correlated.
Therefore, the parameters of the MLP cannot satisfy the i.i.d. prior.
f1 (0.08)
f2 (0.26)
fy (0.31)
1.0
1.0
1.0
0.8
0.8
0.4
0.4
0.4
0.2
0.2
0.2
A
B
CLxL
Figure 8: Three absolute correlation matrices weights fi, f2. and fr. addition. 0.08, 0.26, 0.31 are the average of the absolute correlation coefficients for all the weights in fI. f2. and fr, respectively.
14

---

Under review as a conference paper at ICLR 2020
Table 4: The architecture of the NN (mz f; fr for MNIST classification
Layer
Description
Dimension
Filter dimension (M x N)
æ f
Input FC(sigmoid)
784
128
128 x 784 10 x 128
fy
Output(softmax)
10
FC denotes the fully connected layer
Dimension: the number of nodes (i.e., neurons) in every layer.
Filter dimension (M N): a layer has M filters with x N dimension
A.2 A NEURAL NETWORK WITH A SINGLE HIDDEN LAYER ON THE MNIST DATASET
Based on the seminal work NNal 1994), we design a neural network with single fully connected hidden layer (Figure for classifying the benchmark MNIST dataset (LeCun al. 1998). The input layer dimension is 1 784, the hidden layer has 128 neurons with the sigmoid activation function, and the output layer is the softmax with 1 x 10 dimension. Table Jummariiss the architecture the network. In addition, {Ormn) 1M denote the weights of the nth linear filter in the fully conntected hidden layer and {B.u ] are the weights of the Ith linear filter in the output layer. Therefore, the hidden layer has 128 linear filters with 784 dimension, i.e., {amn 1784
Jm The output layer has totally 10 linear filters and their dimension is 1 x 128, i.e., [Bnt] 1121.
P
O (+(1)
Xm
INI
(F())
Figure 9: neural network with single hidden layer NN {x;; fy
After the network well trained (its training and testing accuracy is 97.4% and 96.9%, respectively), we calculate the absolute sample correlations, i.e., |r(a,, Onwr) and |r(B 3 ), and derive two absolute correlation matrices, i.e., A' and B'xL= based on Equation Figure 10 We find that most (r(a,,, ax')) are close to zero, but (r BBr) not. That indicates the parameters of the NN do not satisfy the necessary condition, thus they cannot satisfy the i.idd prior.
They are shown in
f(0.08)
fy (0.29)
1.0
1.0
0.8
0.8
0.4
0.4
022
0.2
ANx
B
Figure 10: Two absolute correlation matrices weights in f and fr. addition, and 0.29 are average of the absolute correlation coefficients for all the weights in f and fY, respectively.
15

---

Under review as a conference paper at ICLR 2020
Table 5: The architectures of the CNN for CIFAR-10 classification
Layer
Description
Output dimension
Filters dimension
Correlation matrix
Input
32 x
f
Conv (3 x 3) Maxpool
16 x 16 x 64
x x 3 x 64
A192x192
f2
Conv (3 x 3) + Maxpool
8 x 8 x 128
3
64 x 128
81192x8192
FC + Sigmoid
1024
8192 1024
C1024x1024
fa
FC Sigmoid
256
1024 256
D256x 256
fy
Output(softmax)
10
256 10
F10x10
A.3 A CNN ON THE CIFAR-I 10 DATASET
We design a CNN to classify the benchmark CIFAR-10 dataset (Krizhevsky 2009). The CNN has two convolutional layers without activation function, two max-pooling layers for dimension reduction, and two fully connected layers with the sigmoid activation function. In addition, the output layer is the softmax. Table summarizes the architecture of the CNN.
After we train the CNN well (its training and testing accuracy are 98.0% and 69.4%, respectively). We quantify the correlation of different weights in the same layer based Equation 2 thereby deriving the absolute correlation matrix for all the weights in every hidden layer.
Since has 3 channels and f1 has 64 channels, f1 has totally 3 x 64 192 filters. Hence, the dimension of the correlation matrix for f1 is 192 x 192. In particular, we flatten the output of the second convolutional layer, i.e., reshaping 8 8 128 to 8192, thus the filter dimension of fa is 8192 x 1024. Table 5 shows the dimension of the correlation matrices for all the layers in the CNN.
Figure 1 shows the absolute correlation matrices for every layer. We find that most weights correlated in fi and f2, i.c., the weights in the convolutional layers are correlated. In addition, the weighs the output layer are also shown be correlated. However, the weights in fa and fa are not correlated, because their correlation coefficients are close to zero. In other words, the weights in the fully connected layers are uncorrelated. Moreover, we derive the average of the absolute correlation coefficients of all the weights in each layer and show them at the top of Figure IT] which further validates the correlation of weights each layer. Overall, the parameters the CNN cannot satisfy the necessary condition. Therefore, they cannot satisfy the i.i.d. prior.
f1 (0.32)
f2 (0.31)
f3 (0.05)
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
A192 x 192
B8192 8192
C1024 1024
fa (0.17)
(0.26)
1.0
1.0
0.6
0.6
0.4
0.4
D256
E10x
Figure 11: absolute correlation matrices for weights layer of CNN. number the top of the figure is the average of the absolute correlation coefficients for all the weights of the layer
16

---

Under review as a conference paper at ICLR 2020
Image0
Imagel
Image2
Image3
1,00
0.6
0.75
0..4
0.4
0.50
0.2
0.0
0.2
0.0
0.00
0.00
0.25
-0.50
-0.50
0.8
-0.8
00.5
Figure 12: Four synthetic images with different features. All images sampled from (00 1). The only difference they are sorted in different diagonal directions by the ascending or descending orders. Image0 and Imagel are sorted in the primary diagonal direction by the ascending order and the descending order, respectively. Image2 and Image3 are sorted in the secondary diagonal direction by the ascending order and the descending order, respectively.
A.4 NEURAL NETWORKS WITH A SINGLE HIDDEN LAYER SYNTHETIC DATASET
In order to further demonstrate that the parameters of DNNs cannot satisfy the i.id. assumption, we construct a neural network, i.e., NN {æ; f: fy) shown Figure 9 to classify a synthetic dataset. The synthetic dataset consists of 4,800 32 x 32 grayscale images with 4 classes, and every synthetic image is sampled from the Gaussian distribution (o 1). To make the synthetic image have certain features that can be learned by neural networks, we sort the synthetic image in the primary or secondary diagonal directions by the ascending or descending orders. Thus, the synthetic dataset only has four features, i.e., four classes, and their corresponding images drawn above.
There are two important reasons why we choose the synthetic dataset. Above all, compared to benchmark datasets containing complex features, the synthetic dataset only has four simple features, thus we can easily show the correlation of different parameters by simply visualizing them. In addition, since single synthetic image only has single feature, we easily examine the connection between the parameters of networks and the synthetic image.
Neuron0
Neuron1
Neuron2
Neuron3
1.0
1.0
1.0
0.5
D.5
D.5
D.5
0.0
0.0
0.0
0.0
-0.5
-0.5
-0.5
-0.5
-1.
11.0
-1.0
-1.0
Neuron4
Neuron5
Neuron6
Neuron7
1.0
1.0
1.5
1.0
1.0
0.5
0.5
0.5
0.5
D.0
D.0
D.0
-0.5
-0.5
-0.5
-055
-1.0
-1.0
-1.0
-1.5
-1.0
Figure 13: Eight neurons the fully connected hidden layer. Their dimension 32 x
Since the synthetic dataset is very simple, we set the hidden layer f have only eights neurons and the output layer fy have four output nodes. After training the network well, we show all the eight neurons in Figure I3 In contrast to the i.i.d. assumption, their weights show strong correlation in three aspects. First, weights show strong internal correlation within each neuron, i.e., if weight has large magnitude, its neighbors have high probability to be large, and vice versa. Second, weights show strong external correlation between different neurons. For example, most weights of Neuronl and Neuron5 at the same position show strong positive correlation, and most weights of Neuron3 and 7euron7 at the same position show strong negative correlation. Third, the weights show strong dependence on the training dataset. More specifically, since all the features of the synthetic dataset concentrate the primary and secondary diagonal directions, most linear filters also demonstrate related features in the same directions.
17

---

Under review as a conference paper at ICLR 2020
matrix
1.00
Linear filters
0.50
Fiiter
00
00.25
Fiiter:
0.50
-0.75
Figure 14:
The left figure visualizes the four linear filters in the output layer. The right figure visualizes the correlation matrix of the four filters in the output layer.
Figure 14 visualizes the four linear filters in the output layer. Though they not show too strong internal correlation because their dimension is too small, their weights at the same position still show strong external correlation, e.g., the weights of FilterO and Filterl at the same position show strong negative correlation. Moreover, we calculate the correlation coefficients of all the four filters based on Equation 2 and derive the correlation matrix show in Figure shows that the correlation coefficient of Filter( and Filterl is close to 1,, which further validates that they have strong negative correlation, hence the parameters of the output layer do not satisfy the i.i.d. prior.
Compared to the simulations using benchmark datasets, the synthetic dataset enable to expose correlation different weights and their dependence on the training dataset in the more clearly way. Overall, we demonstrate that the parameters of neural networks are not i.i.d..
B PROOF: THE NECESSARY CONDITION FOR THE ACTIVATIONS OF HIDDEN LAYERS BEING INDEPENDENT
X1
f11
f21
0:2
Rus
fv(1)
f12
X
122
:
7er O
fv(L)
fix
:
fax
XM
fiN
f1
f2
fr
Figure 15: A MLP = {x;; fa; f2j fy }. in which f1 has N neurons, i.c., f1 = {fin = on ( where gin (w) = EM= amn Fm + bin is the rith linear filter, amn is the weight of the edge between am Sins Unn is bias, oi is non-linear activation function. Similarly. fa {aolgau (fr )) where (fi Dn= Brk fin buk. In addition, since fy is softmax, fy Zy explo (fz)) where (f2) Ek 7Is fas + byt, and the partition function Zy = Σt (x))
In the context of Bayesian probability, all the weights and the biases in the MLP (Figure 15 have prior distributions, i.e., a'm P(Amn), Buk P(Wn,), bin ~ P(BIn). and by: P (B2z)) thus we regard GIn = -MM Amn X.,. - Bin as the random variable of gıa (x)()) = _m==1 Omn Im thin and G2z = Σn Σx=1 Wot Fin + Bex the random variable of (f1) Σn Bak fin + bux- Therefore, the random variables arbitrary activations in fi and f2. e.g., fin and fax can be expressed Fi, a (G1n) and Fat 02 (G respectively.
18

---

Under review as a conference paper at ICLR 2020
Based the theorem (Appendix(D) that functions of independent random variables are independent, we can derive that if [F22)==> are independent and the activation function o (-) is invertible, then {G2 K are independent. In other words, in order to prove {F2x) IK f==1 are independent, we only need to prove {G2 1 being independent as long as (=(-) invertible.
The necessary condition for {G being independent is V(k, k*) E S {(k,k) E E?Ik + k. S ks K, < W' S Kj, the covariance Cov(G2k, G2R) = O, which can be formulated as
Cou(G2k G2x. = Coo Σ
W Fin + B2k,
Σ n'=1
Wirk Fine + B2x)
(23)
Based on Coo(X X+ + +Y, W + D) = W) Cou(X,
(X, 2) - Cov(Y, W) + Cou(Y,2), we can
derive that
Cou(G2R, G2x) Σ Σ
Cov(Wnk Fing Wirk Fin)) Σ
(enn Con B2x)
n=lw==
(24)
Σ
Cou(WWike Fine, B2k) + Cou( B2t: B2
12
Based on the definition of covariance, i.c., Cov(A, B) = E(AB) -
(((AB))-E(AA(B)))) E(A)E(B), we have Cou(W nk Fins Win Fin:) E(Wnk W W'K' Fin Fin)) ) - B(W F(W. Cou(Wai nk Fin, Brxr) F. B2 Pin) F(W nk Fin)) Fb( (Bzw)
(25)
Based on the law of total expectation, i.e, E(AB) = EA(AEB]AA we have
E( ni W.'k- FinFin)) = EW..W.rr (WnkWnw Www EFINFIn WWw (FinPin))) E(Wn Fin) = EWar (Wnk E..., Baw (Fin))
(26)
SWaneWnn. EF..,WW... (Fi.))
Since the statistical properties the input (Fim Fin are unrelated to the weights Buk and biases b2k- W. Fin: Warer wer (Fin FF.... (Fin E.r,..W...... (Fin) EF... (Fin),
have Fir,Fir
and E Wax (Fin) EF.. (Fin). As result, can derive that
B(WnkWace Fin Fint) )= = Ew,, W.. (Wnk W E (FinFin)) E(W.r Fin) Ew Bpy (W nk Baxs) Er,, (Fin) E(Wnk EW... (Wnk) ) Ep,.. (Fin)
(27)
Substituting Equation (27) for the corresponding expectations in Equation 25). we have FonnFFnFin W. Fink E(W., W,,, w.k) )(WFF...:))((......... E(FinFFr E(W, E we E(Fi..) E((i.)))
E(Fin) E(Wn.)E(Bar) E(F(.) Cou(Wnn B2 E(W B2 )
(28)
Assuming N are i..dd and Fin P(Fi), we have E(Finfind) E(FiN)E((W))) (F(F)). Hence, we can derive that
Coo Ww'N Fiw) Cov(W Ww(k) E E²(FF) Cou(Wnn Fin, Bar) = Coo(WWn Bz E(Fi)
(29)
Substituting Equation
29 for the corresponding covariances in Equation 24 we have
N
Cou(Gzz G2x)
Cov(Wnk, Wwr (F3))
Cou(Wnk nk. Bxx) E(F)
(30)
N
Cou(Wwrk B2x) + Cou(B2r, B2sr)
19

---

Under review as a conference paper at ICLR 2020
Based Cov(X X Y, W Z) we can derive that
Cor(X W) + Coo(X + 2)) Coo(Y + W) + Cov(Y + 2).
N
N
Cou(G2R, G2v) = Coc
W
Wies
Σ n==1
Wwx)) Cov
Σ
Wiki B2 E(Fi)
n=1
n=1
(31)
+ Cov(
Wiek, B2x) + Cou( B2k, B2x)
1. Therefore, given (2(-) is invertible and [Fin)nz1 are i.i.d., the necessary condition for { KK being independent can be expressed
k=1
Cov(G2k, G2k) = Co, WK W + Coo(W,, B2x) CCov(Ww, B2x) ((Fi) Cov(Bax, B2xr
(32)
where Wn - Wik and We = Wnk'.
Overall, in order to guarantee the activations of hidden layer are independent with each other in the context of Bayesian probability, the prior distributions of the weights and biases the hidden layer should satisfy the above necessary condition given the inputs of the hidden layer are i.i.d. and the activation functions are invertible.
C PROOF: THE NECESSARY CONDITION FOR THE ACTIVATIONS OF HIDDEN LAYERS BEING IDENTICALLY DISTRIBUTED
Given the same DNN drawn in Figure 15 we assume 02(+) being strictly increasing and differ- entiable, so (-(-) is invertible and its inverse 0.2 (-) is also strictly increasing. The cumulative distribution function of F22 can be expressed
1 Fax (f) = 6f2xx S f)
bla1(G22)) S f)
(33)
dGGr I o0 (f))
PG.. (oT
where 6F22 I f) is the probability of F2z takes on value less than or equal to f. Subsequently, we can obtain that
Przu (f) 0 Pau (
a4 (z))
0f
0f
(34)
PGsu (0 (()) (f)
Equation 34 means that if the activation function (2(+) is strictly increasing and differentiable,
(Fax J being identically distributed implies {G2x
1kk k= being identically distributed.
In the context of Bayesian probability, Get = EN= Win + Bax, we can derive that
F(G2))
Σ F(WnttFal) + F(B2a)
(35)
n=1
Based on the law of total expectation and Equation 27 we have
E(W nn Fin) Ew,. (Wnx) (Fin)
(36)
Assuming NN are i..dd and Fin Fi, we have E(Fin) E(Fi). Hence, we can derive that
N
E(G2x) = E(F)
Σ
E(W.)) + E(Bra)
(37)
n=
20

---

Under review as a conference paper at ICLR 2020
Based on the property of expectation, i..., E(A) E(B)) E(A+ B), we have
E(G2x) = B(FI)E _ Wrk) + E(B2a)
(38)
Therefore, given 02 is strictly increasing and differentiable and {Fin } are ii.d., the necessary condition for { F2 ==1 being identically distributed is that v(n, n') E S1, we have
((R))E(W)) + E(Bzz) ((F)) E(Bak)
(39)
where W₁ = ->= Wik and We = Σm= Wik'.
Overall, in order guarantee the activations of hidden layer identically distributed, the prior distributions of the weights and the biases of the hidden layer should satisfy the above necessary condition in the context of Bayesian probability given the inputs of the hidden layer i.i.d. and the activation functions are strictly increasing and differentiable.
D THEOREM: FUNCTIONS OF INDEPENDENT RANDOM VARIABLES ARE INDEPENDENT
Let X and Y be independent random variables on a probability space (9t, F.P. Let g and h be real-valued functions defined on the codomains of X and Y, respectively. Then g(X) and h(Y) are independent random variables.
Proof: Let A S R and B S R be the range of g and h, the joint distribution between g(X) and h(Y) can be formulate as (X) E A, E B). Let g (A) and h h-1(B) denote the preimages of and B, respectively, we have
( E AY) E B) = =
E E
(40)
Based on the definition of independence, we can derive that
P(x(X) E A,)(Y) E B) = P(X E (B)
(41)
((g(X) A)P()(Y) B)
Based on the definition of preimage, we can derive that
XX) E A. B) = P(((X) E A)P(M(Y) e B)
(42)
Therefore, g () and h h(Y) independent random variables.
E THE NECESSARY CONDITIONS FOR ACTIVATIONS BEING L.I.D. ARE VALID FOR CONVOLUTIONAL. LAYERS
Assuming DNN {æ; f1; Sr; fr) has two adjacent convolutional layers and f1+1. The first convolutional layer has N channels, i.e. f. ) and the second convolutional layer has K channels, i.e. ftt1= {f+t)]
Therefore, the dimension of the convolutional filters S in fi+1 are W x H x N x K, i.e., the dimension of single filter connecting the f channel and the f channel are D(S i+1
((s(k)))) = W x As result, the f channel can formulated as
f 0
s ((k.n)
f + 1
(43)
where 0:41 is the activation function and bet is the bias for the (i + 1)th channel.
21

---

Under review as a conference paper at ICLR 2020
Convolutional filters
Input
Output
X2
3/1
3/2
X4
X3
36
C
D
Y3
Y4
Xg
A convolution channel
Zg
Z4
2
91
92
Y1
A A1
B B-X2 32
8
2G
8g
8
y2
D
Ra
y3
Xx Xy
Dx
B NN
C
+
A fully connected layer
b
ò is the bias
W
H
B
G
X
Z2
+
C
E
F
Xg
Zz
B
C
E
Z
Blank elements in the matrix are zeros
&
Rg
Figure 16: The equivalence between convolution channel and fully connected layer.
Since Carriga-Alonse et al. (2018) demonstrate the equivalence between a convolution operation and regard the convolutional channel f as fully connected layer. Specifically, the input of the virtual fully connected layer is the same as the input of the convolutional channel, the output of the virtual fully connected layer is the vectorized output of the convolutional channel, and the weights of the virtual fully connected layer depend on the convolutional filter and the spatial location of the output of the convolutional channel.
the matrix multiplication,
Without considering the activation function, the equivalence between a convolutional channel and fully connected layer can be explained in Figure [6 which the first convolutional layer has two Igg and at = { ..), and the kth channel
channels, i.c., f: fæx; }}. where æ = {I11 of the second convolutional layer is f = (21, 22; 23, 24). addition, the dimension of the
convolutional filter Si+1 2 2 1, where ((h.1) [A, B; C, D) and 5(h.2) = [E, F; G, H). Figure 16 shows that the output of f can be expressed as a fully connected layer. The input of the fully connected layer the vector æ. the weights of each neuron is represented by each row of the matrix W, and the bias is b, thus f W x b. 1.
If consider the activation function and make precise statement, f can be formulated
f o, (W
(44)
where Wh 1 is the matrix corresponding to all the convolutional filters in the kth channel.
To derive the necessary conditions for arbitrary different channels are i.i.d., we need derive the necessary conditions for two arbitrary activations in two arbitrary different channels are i.i.d. For example, if the mth activation of the channel and the m' th activation of the k' channel in fi+1 are ii.d., the necessary conditions are Cov f (m), pk' f (m²)) = 0, E(f (m)) = E(F (m))).
Since we demonstrate the equivalence between convolutional channel and fully connected layer, the necessary conditions for activations being i.i.d. in a fully connected layer are also valid for convolutional layer. In other words, examine whether activations being i.id. is valid in convolutional layer under the assumption that all activations being i.i.d. in the previous layer.
also use the Equation 3 Equation 4 and Equation to
22

---

Under review as a conference paper at ICLR 2020
Table 6: The statistical measures for examining the necessary conditions
Layer
Activation expectation
Covariance
Weight expectation
f1
Covl An, A.d)) 2 Cov k, We)
E(X)
E(An)
fz
B(R)
E(W
fy
Cov(CC, Cv)
E(Ci)
{5m) M
P(X), (fin)==1 ~ P(Fi), (f2x)
P(F2)
DM 2m=l Amn, where Omn P(Amn)
W4 En Wak. where Boo P(Wn))
C 2km1 Cki where "hk P(Cw)
F ACTIVATIONS BEING I.I.D CANNOT BE VALID FOR ALL THE HIDDEN LAYERS OF DNNS
In this section, we demonstrate that activations being i.i.d. cannot be valid for all the hidden layers of DNNs through showing that the hidden layers of two typical DNNs, i.e, the MLP and the CNN, cannot satisfy the necessary conditions for activations being i.i.d..
F.1 THE MLP ON THE MNIST DATASET
The architecture of the MLP for classifying the MNIST dataset visualized in Figure 15 and its specific parameters summarized in Table Since the activation functions of all the hidden layers in the MLP are the sigmoid function, which is strictly increasing, differentiable, and invertible, we only need to examine the first two necessary conditions in the MLP.
Since we restrict all the hidden layers from using the bias, the first two necessary conditions for activations being i.i.d. can be simplified as
v(k Si = {(k, K²) E Z K K, S K}, we must have Coo(GZk G2v) = Co,(W We (F)) =
(45)
E(Fi)E(W)) E(F))E(())
(46)
Based on the architecture of the MLP, Table summarizes all the statistical measures for each layer of the MLP. Therefore, the key to examine if the layers of the MLP saisfy the necessary conditions to estimate the statistical measures.
To estimate the expectations E(X), E(Fi), and E(F2), we use the sample means H, Fr, and f2, respectively. The definition of the sample mean
M
N
K
T
Em
f
Σ
f()),
F2
fe(k)
(47)
M
N
K
m==1
n=1
k=1
After the MLP is well-trained, we derive the sample means as . = 00133, fi = 0.436, and f2 = 0.498 given the benchmark MNIST testing dataset.
Taking into account the estimation error, we can regard E(X) = 0. As a result, we can derive that v(n, Cov(Gi., G 0 and E(G E(G 0, thus fi satisfy the necessary conditions for activations being i.i.d..
However, Fi = 0.436 and fz = 0.498 imply that E(E) + 0 and E(Fb) + 0 even considering the estimation error. Therefore, we need to estimate Cor(Wk, Wer and E(W2) to examine whether the activations {f2x K being i.i.d..
Since the training dataset is commonly viewed i...d., train the MLP several times to derive multiple independent observation samples (Bnx(t))T to estimate the random variable W... thereby estimating (Wr, We) and B(Wx)).
the same training dataset to
23

---

Under review as a conference paper at ICLR 2020
f1 (6.75)
f2 (0.41)
fy (4.06)
1.0
0.5
E(A,)
E(W
0.0
E(C)
-0.5
-5
-2
-10
100
20
n
K
The expectation of the weights summation, i.e. E(A,), E(Ws), and E(C)), in each layer. The number at the top of the figure indicates the variance of the points drawn in the figure.
Figure 17:
More specifically, we use the sample correlation r(Bu. Bar) to check whether Coo,W,, We) are close to zero and use the sample mean 3r to estimate F(Ws).
Et (ww (t) 3g))
r(3x,BK)
=o CorrrWss,W,,
(48)
EI= 3x) Et= (3 (t) Br²)
where Ba - Et- (a(() is the sample mean of (Bx(1)}== and (L) = = (1)
In order to make a tradeoff between the estimation precision and computation complexity, we use the same training dataset train the MLP 20 times to derive the samples, i.e., (1)x (-1- For activation f24. has 128 weights because its previous layer has 128 neurons. Hence, single sample (t) = E (t). Based on the 20 samples, i.e., {3 [t) we can calculate r(3,, Bar) to examine whether Cow(Ws Wxx) are close zero and derive the sample mean to estimate E(Wa).
We use the same method to estimate Cov(An, And), E(An), Cov(Ci, Cr), and E(Ci).
Finally, we shows the absolute correlation matrices i.e., ANN WK*K, and CLXL, in Figure Since many elements in the three matrices are far from zero, we conclude that their corresponding covariances, i.e, Coc(An, Aw), Cov(W,, k, Wv). and Cov(Ck, Ckr), are not close to zero based on connection between correlation and covariance
Corr(W& We Cov(Wk; We)
(49)
OW.OWe
where ow2, > 0 is the standard derivation of Wk-
Figure visualizes the expectations, i.c., B(AA), E(W)), and E(C)), for each layer in the MLP. We find that some expectations show great variations, especially {E(A,)) and { In other words, the MLP cannot guarantee that all the expectations of the weights summation are equivalent though take into account the estimation error. Therefore, the MLP cannot satisfy the necessary condition for activations being identically distributed.
Overall, conclude that not all the hidden layers of the MILP satisfy the necessary conditions for activations being i..dd. under the assumption that the activations of the previous layer are i.i.d.
F.2 THE CNN ON THE CIFAR- 10 DATASET
The architecture of the CNN for classifying the CIFAR-10 dataset is in Table 5 In particular, the two convolutional layers do not have activations function, i.c., their activation functions can be viewed as y which is strictly increasing, differentiable, and invertible. The sigmoid activation function also strictly increasing, differentiable, and invertible, thus we only need to examine the first two necessary conditions in the CNN. To simplify derivation and decrease computation complexity, we assume the max- moo-loo layers are special activation functions and satisfy the third necessary condition for activations being i.i.d.
24

---

Under review as a conference paper at ICLR 2020
Table 7: The statistical measures for examining the necessary conditions of each layer in the CNN
fz
fa
fa
fr
Conv Maxpool
Conv Maxpool
FC
FC
Output Softmax
Description
Sigmoid
Sigmoid
Activation Dim.
16x 16x64
8x8x 128
1024x1
266x1
10x1
Filter Dim.
3xax3x64
3x3x64x128
8192x1024
1024x256
266x30
Weight Dim.
MXN
Nx
KxL
LXQ
QXR
(27x64)
(876x 128)
(0292x1024)
(1024x256)
(256x10)
Weight PDF
Amm-P((Amm)
dtgP((Pg)
J(rr)((Fr)
Sum. PDF
A, An-220 "mmm Amn
Bg G-Enn11 Buk
G-EA-1 Ca
Py- Pta
Re- Fge
Covariance
Cov(An))Ag))
Cov(By.Bpr)
CorlCr,Cp)
Cor(Dg.Dg)
Cou(Fe.FFr)
Sum. Exp.
B(An)
B(Ba)
B(C))
B(D,)
B(F,)
Activation Exp.
B(X)
B(F))
B(Ps)
E(Fs)
E(F4)
Dim. short dimension. Sum. short Weights Summation. Exp. short for Expectation.
Similar as the MLP, we also restrict all the layers of the CNN from using the bias, thus the first two necessary conditions for activations being i.i.d. can be simplified as
v(K,k') E S1 = {(k: k') E 22 H s K, S S K}, we must have Coo(G2t G2 Cov(W Wx (F) =
(50)
E(F))E(W)) = E(F))((W))
(51)
Table summarizes all the statistical measures for every layer of the CNN based on the architecture of the CNN Table The definitions of most random variables are similar as the MLP except the dimension of the weights. To check if two arbitrary convolutional channels are i.i.d., need to know all the weights of two convolutional channels. Based on the equivalence between convolu- tional channel and fully connected layer (Appendix E., all the weights of single convolutional channel consist of all the convolutional filters related to the convolutional channel. For example, the dimension of all the convolutional filters related convolutional channel in f1 are 3 x 3, thus single convolutional channel in fi has 27 weights, so the weights dimension of f1 is 27 x 64.
After training the CNN well (its training accuracy is 96.2% and testing accuracy 67.9%), we use the same methods as the MLP to estimate the above statistical measures in Table
f (0.29)
f2 (0.31)
f3 (0.29)
0.6
0.6
0.6
0.4
0.4
A64 x 64
B128 x 128
C1024 x 1024
(0.41)
fy (0.46)
0.8
0.8
0.6
0.6
0.2
0.2
D256 x 256
F10 10
Figure 18: Five absolute correlation matrices for the weight summations in layer. The number the top is the average of the absolute correlation coefficients for all the weights summations of the layer
25

---

Under review as a conference paper at ICLR 2020
f (0.00)
fz (1.88)
fg (550.28)
(13.91)
(1.13)
E(6+)
E(D
-10
-152
-45
22
40
80
50
102
1200
5.0
7.5
Figure 19: The expectation of weights summation each layer in the CNN. The number the top of the figure indicates the variance of the points drawn in the figure..
For the activation expectations of each layer in the CNN, we have = -0.023, f1 = 707.074,
f2 = - 6347.34877 836, fs 36.87 87. f4 = 15.131, thus all the expectations cannot be zero except the input layer, i... ( O. (FF)) # O, E(F2) + O, E(Fa) # 0. E(Fa) + O.
For the covariances, we show all the correlation matrices of each layer of the CNN in Figure 18 We find that many correlation coefficients are far from zero all the correlation matrices, thus their corresponding covariances are not close zero either. Based on the estimations for the activation expectation and the covariance between different weights summations in each layer, we can derive that activations being independent cannot be valid for all the layers of the CNN.
For the weights summation expectation, we visualize all the expectations for each layer in the CNN
in Figure 19 We find that some expectations have great variations, especially (E(C))= and
(E(Dg) q= which means that most expectations of weights summations of different neurons in fs and fa cannot be equivalent. As a result, the activations of the two fully connected layers in the CNN cannot be identically distributed.
Overall, we conclude that not all the hidden layers of the CNN can satisfy the necessary conditions for activations being i.i.d. under the assumption that the activations of the previous layer are i.i.d.
G THE EQUIVALENCE BETWEEN THE GRADIENT DESCENT ALGORITHM AND THE FIRST ORDER APPROXIMATION
In the context of deep learning, most learning algorithms belong to the gradient descent algorithm.
Given a DNN = {æ:3 fii fr; fr and cost function H [fr P(Y|X)), where P(Y|X) is the true distribution of the training label given the corresponding training dataset. Let 0 be the parameters of the DNN, the gradient descent aims to optimize 0 by minimizing H [ff P(Y [X] (Rumelhart et 1986). We typically update iteratively derive 0*, which can expressed
0 0 oV P(Y|X))
(52)
where Vo, H[f X)] is the Jacobian matrix of H[ff, X[Y]X)] with respect to O, at the ith iteration, and a > denotes a constant indicating the learning rate, Since P(Y|X) is constant, we denote H(fx) as Hifx. P(Y|X)) for simplifying the following derivation.
Since the functions of all hidden layers are differentiable and the output of a hidden layer is the input of its next layer, the Jacobian matrix of H(f)) with respect the parameters of the ith hidden layer, i.e,, Vecey (fv)). ), can be expressed as follows based on the chain rule.
I II j=it1
Vo() - Jifr
V Voryf
(53)
where 0(1) denote the parameters of the ith hidden layer.
Based on Equation 52 and Equation 53 (ii) can learned by the gradient descent method as
0 (, ((
(54)
Table & summarizes the backpropagation training procedure for the MLP in Figure 15
26

---

Under review as a conference paper at ICLR 2020
Table 8: One iteration of the backpropagation training procedure for the ML.P
Layer
Gradients update V Vorto
Parameters and activations update
fy
Vi Ssy Hify 12 Va t
f2
1e (2(51.0441(2) (0 (x.0014411)
f1
F Byy H(fy) 12
The uparrow and downarrow indicate the order of gradients and parameetenstacciiattion)) update, respectively.
If an arbitrary function differentiable point p" in RR and its differential is represented by the Jacobian matrix vnf, the first order approximation of f near the point can be formulated as
f()) f(p") = pv (p p') o(Ip p"ll)
(55)
where o(Ip p p²1] II) is quantity that approaches zero much faster than || p"ll approaches zero.
Based on the equivalence between the gradient descent method and the first order approximation (Battiti 1992), updating the activations of the hidden layers of the MLP in Figure [15 during the
backpropagation training procedure be approximated as
f2(fi, (2) R f2 (2)) (Ve.22)f2) (0.+1 (2())
(56)
fz 1 = fa (1)] (Vo,())f) (1)+ (1))
where falfi, (2)] denote the activations of the second hidden layer based on the parameters learned in the tth iteration, i.c., (,()), given the activations the first hidden layer, i.c. f1. The definitions of f2(f1, 0.+1 (2)) and (ax, (1)) the same 0 (2)).
Because f2 has neurons, i.e., f2 {f2x oz m== Bix fin + ber bex) 1 and each neuron has N 1 parameters, i.c. ( = {1S.e..... :NNA: b2x]]==1 the dimension of (2) equal K (N + 1) and (2. can be expressed
V2. (Vo,f2) (f
(57)
where Vo,J2- f.(2).00(22)
802
Substituting (Vo.f2) 1 for (2) in Equation 56 derive
= f2lf1. 0,() + f2) [fi: 1)TT 02+1 (Vo.f2) [f1:1] .0,(2)
(58)
If we only consider a single neuron, such as S2x. in S2. 02+1(2k) = [3 3NN b2x) and (2k) [i .. : b2g). thus [fi; 1Jz 0..1 (2k) En Pnk sin + bz.. As result, Equation 58 with respect to a single neuron can be expressed as
fze O 12+1(2))) (Vo.fe4)
12 Bak fin + b2k] (First order approximation)
(59)
N 12 3he fin + bbg] (Error) 1=
+ 0f (Vo,fex)
Equation 59 indicates that f2x [f1 0.+ (2k) can be expressed its first order approximation with an error component based on the activation in the previous iteration, i.e., f2x [f, 0, (2k)). Since 8f2 (f. is only related to fi and the parameters in the tth training iteration, i.e, d7,
Va, f2k (2). it can be regarded constant. Also note the error component not contains any parameters in the (t + 1)th training iteration. In summary, fex (f1. 0.+1 (2k)) can be reformulated as
N
(J1, (2k) RO C
| Bux Sia bau] + C2
(60)
n=l
1 D 87 fin byy). Similarly, the activations in the first hidden layer, i.e, fI also can be formulated as the first order approximation in the context of the gradient descent learning algorithm.
where C1 V f2. and C2 fek (fx.ff. , (2k) (Vo,fek)
27

---

Under review as a conference paper at ICLR 2020
ba
60
Conv-RRLUU-Poll.
Conv-RRlUU-Pool
Flattening
Output
Figure 20:
The above CNN has two convolutional layers, i.e., fi, f2, and one softmax output layer, i.c., fr. We flatten the output of f2 as the input of fv.
H THE PROPOSED PROBABILISTIC REPRESENTATION HOLDS FOR CONVOLUTIONAL LAYERS
This section proves that the proposed probabilistic representation is valid for convolutional layers. In the above CNN, the input has Q channels (x*) ) has N convolutional channels f1 = [ f2 has K convolutional channels f2 = {ff L 1k=1-
/==1 (e.g.. Q 3 are color images), fi
The output layer fY is the softmax with L nodes, thus its distribution can be formulated as
= ZFY
exp(Sni))
(61)
where ZF Et. exp(fyt) denotes the partition function. Since (st ES= we can derive
7a f2s + bbyl Mi
P[Fy)= = ZFy
expl aa f2s + b
(62)
1=1
where {f2s}/=1 the flattened output of f2. 7/t is the weight of the edge between fa, and ful and byt denotes the bias. Since f2 is convolutional layer with K convolutional channels, it can be reformulated {f2g)/m1 (fk) 1k=1- As result, P(Fy) be reformulated
=
exp Σ Thl
s +
bb))
(63)
ZFY
where Yhe is a subset of all the parameters {7/1 )/ such that ΣK_ The 83 = =EE Tkk the transpose of 7/k- Therefore, P(Fr) can reformulated
75t S2j. and
P(Fy) (P(f)))
Z II explr 1DH
1
(64)
where Z'y Z (exp((by)) Recall the element-wise matrix power, egg. exp(a)), where (a1; =aa;; 02; 0g) and = [b) by; by), we can derive that exp(a) =
exp(a)))x and exp(a) = exp(() ) exp((())))))) exp(ax) dx = [exp(arb) bi exp(azb2) exp(axba)), As result, exp(ab) = exp(a16) + azb2 - azb3) = 11 exp(a)*, where |a] is the element numbers of a.
Based on the element-wise matrix power, we can reformulate ((Fy) as
K II
1
P(Fx) = (P(fu) ~ D'
II
exp(>%i) f/h) =
[exp (uu
(65)
Fy A= 1
28

---

Under review as a conference paper at ICLR 2020
Moreover, introduce new partition function ZFz
Σ₁ exp(f2) to guarantee that
exp(f2) is probability measure. As a result, P(Fy) can be reformulated as rz
=
exp
Hf
(66)
ry k=1 [
1 Overall, P(Fy) can be reformulated as PoE model, in which each expert is defined by every convolutional channel in f2
where 7 = -/pp/TTk==
[k [2pz
P(f))
exp(f22
(67)
Zrb
Since f2 has K convolutional channels, i.c. f2 { { R= and each channel can be formulated as the summation of all the convolutional channels in f1, i.c.
f2 (12 02l
S (k)))
fr + b 1)
(68)
where S2 (((k,)) ff is the output of the kth convolutional filter applying into the 7th channel of f1, B is the bias, and (=(()) is the activation function. Therefore, we have
P(f))
expl02
S (k
ff + bb 1)]
(69)
Zpz
Based on the equivalence between the gradient descent learning and the first order approximation (Appendix G). Equation 69 can be approximated
P(
exp
Σ
S (R.
f + b 1)
(70)
F
n=l
Let us define linear function as (f(fi) = En
S (k))) fr +
bb 1, thus the distribution of the
kth convolutional channel in f2. i.e., P( ) can be expressed as
expl=X(f1)) P( ZFz
(71)
we regard all the linear filters (f)( k= as potential functions for modeling signal structures of fi. we can formulate P(F2) as a specific Gibbs distribution, i.e., the Markov Random Fields (MRFs), which can expressed as
K P
K
1
P(F2) ZFa k=1
exp
(t())
(72)
ZF.
where Zr: = exp E 2 (fi) is the partition function for P(Fz). summary, we prove that the convolutional layer fz formulates a MRF model to describe the statistical property of f1.
Here we prove that the convolutional layer fi also formulates a MRF model to describe the statistical property of the input æ. Based on Equation 70 have
N (k,)) 2
P( R
exp( Σ 7
f
(73)
a=1
where Z'rz" \ex((x((b))
29

---

Under review as a conference paper at ICLR 2020
Convolutional filter 5(km)
Input channel fi
Output channel f
X2
X3
3/
y/2
convolution operation
X4
25
X6
Yg
Y4
Xg
Xg
A matrix multiplication
X1
W
(k.)
B
D
X3
y1
A
B
C
D
X4
Y2
A
B
D
X5
y3
X6
Y4
Y1 = A- X1
+
X2
C X4 +
Y2 = A X2
+
8 Xx
D-xx
Blank elements
A-
Xy + D Xe
y4 A A X2
X9
Figure 21: The equivalence between a convolutional operation and a full connected layer.
Based on the equivalence between convolutional channel and a fully connected layer proven in Appendix E
we can regard convolutional operation matrix multiplication, which visualized As a result, we have
in Figure
N
P(
expl
(2) S2
f
exp(
(k, W ((().))
f
(74)
Z
Z'
==1
n=1
where (k,m) is matrix corresponding the convolution filter S (k),))
Based the element-wise matrix power, P(fE) be reformulated as
P(F))
1 Z
II FE n=1
exp (xp(W(kk (k,m) f
II
exp(FF) w((k.))
(75)
Z P
Similarly, we introduce new partition function ZFT En (fr) to guarantee that Zpp efP( is probability measure. As result, P(fh) can be reformulated as
P( P(Jn)
exp(Xx") ( w
(76)
7"
7FZ
Fb
n=1
where Zi 2/pg/ 1 12p 12p w! Therefore, P(f can be reformulated PoE model,
in which all the experts are defined P(ST) ZFr
eexp((r))
Since fi has N convolutional channels
fi = ( =
s(n
((,.2)
+ 4 1)
(77)
q=1
Based on the same derivation as Equations
70 71 72
P(FG) also can be expressed as MRF model
N
P(F)) = ZF.
II
P(f))
=
ZF.
exp
pf())
(78)
n=1
==
where (x) Eg s(m.9)) + -p-11 1) and the partition function ZF, - E. NN
(x))
Overall, the distribution of convolutional layer be formulated MRF model.
30

---

Under review as a conference paper at ICLR 2020
Algorithm The algorithm for generating the synthetic dataset
input:
NIST dataset handwritten digits by class
1:
repeat
2: binarizing an image of NIST to obtain 2
3:
extracting the central part of obtain with dimension 64 x 6d downsampling
4:
obtain Zed with dimension 32 32
extracting the edge Zed to obtain mask image Maa decomposing mid into four parts, ie., Momsiee Flguuiddshommnn Tinnsah-oonnsssrr and Minsidd. sampling (0, 1024) derive random vector &R 10124x1 sorting æ in the descending order derive & decomposing into four parts, i.c., & {Foussides Coutside- houndary, Cinside houndary Cinside } 10: placing each pixel of & = { tonusidd-tuubbry: until (20,000 synthetic images generated) output: The synthetic dataset
5:
6:
7:
8:
9:
Dimside) } into the above masks.
11:
I THE APPROACH TO GENERATE THE SYNTHETIC DATASET BASED ON NIST AND THE ARCHITECTURE THE CNN
In this section, we present novel approach to generate a synthetic dataset obeying the Gaussian distribution based on the NIST dataset of handwritten digits by class. The synthetic dataset has similar characteristics as the benchmark MNIST dataset. It consists of 20,000 32 x 32 grayscale images 10 classes (digits from 0 to 9), and each class has 1,000 training and 1 1,000 testing images. More specifically, the approach to generate the synthetic dataset is summarized in Algorithm 1 and Figure 22 visualize the spatial relation of a- outside & @outside-oundary @inside-boundary @nsside
Original image
Image edge
Synthetic image
O
Outside
Outside boundary
Inside boundary
Inside
Outside
Outside boundary
Inside boundary
Inside
The first row shows an original image, its edge, and the corresponding synthetic image based on the original one. The second row white pixels show the spatial position of the four mask parts, i.c., Monsidee Monside Thunsit-bouulary Miissd boundry, and Minside- The third shows the synthetic image corresponding each mask part, i.c. & (Faussidee Boutside boundrry Pimside-oummary Pinside)
Figure 22:
http:://www.nnt..g.ge////n/tttt 1
https:: /tt:://wwwww www. nist www..iist..w.///d/nnntt.
31

---

Under review as a conference paper at ICLR 2020
Table 9: The architectures of the CNN for synthetic image classification
R.V.
Layer
Description
CNN
X
Input
32 32 x
F1
f1
Conv
ReLU
30
f2
Maxpool
15
F2
f3
Conv (5 5) + ReLU
x 11 x 60
f4
Maxpool
x 5 X 60
Fy
fy
Output(softmax)
1 1 x 10
R. is the random variable of the hidden layen(s).
The architecture of the CNN for classifying the synthetic dataset is summarized in Table
THE EQUIVALENCE BETWEEN SGD AND VARIATIONAL INFERENCE
J.1 THE DISTRIBUTION OF THE OUTPUT LAYER GIVEN THE INPUT IS A GIBBS DISTRIBUTION
In Section 6.2 we demonstrate the entire architecture of DNN = {æ; fii...; fr) corresponds joint distribution P FYy Fy..; Fi x). As result, the conditional distribution of fy given a, i.e., P(Fy|X X), can be formulated as
P(Fr|X) = / P(Fy:Fr..
(79)
Since P(Fr; Fy.; = P(Fy|F1) .....(((FFE))) ... P(Fi and only F2 depends on F1, we can derive
P(Fr|X) |((P)E)) P(Fa)Fa)dF) P(BAEA dF2
P (80)
Since f ((EE) deterministic function defined by the corresponding hidden layer, P(Fy F X can be simplified as
P[F (X)), where fa is the
P(Fx /..BE..BBBBE)) P(F 1E dF2
P((Y)EE)
(81)
Similarly, since only F3 depends F2, we obtain
x( - P(FBI(() P(Fslfs(f(( .
(82)
we iteratively apply the integral with respect to F3 until F,, we can obtain
P(Fr|X) P P(Frlf((- (
(83)
Since the output layer fy is commonly a fully connected layer. we can formulate P(Fx|X) as multivariate discrete Gibbs distribution based on Equation g
P( ZDNN(x) exp expl fr (fr( (f()---f((((())))...
(84)
It is important to note that the energy function EDNN ( fy (x) - fx (f ie., EDNN(.) is determined by the entire architecture of DNNs. In addition. since P(Fy X) is a conditional distribution of Fy given X, the partition function ZDNN (x) should integrate all possible activations of the output layer fy to guarantee P Fy (X) being a valid conditional probability, i.e., 7DNx((:) f explfy (f (
(x))......
For clarity, here we only consider integral no matter a Gibbs distribution is discrete or continuous, is special of integral. All functions involving should include parameters ff, in the conditioning set, but we omit for the same reason.
because Σ
32

---

Under review as a conference paper at ICLR 2020
J.2 SGD CAN BE EXPLAINED AS VARIATIONAL INFERENCE ALGORITHM
As a dominant paradigm for Bayesian posterior inference P(H E) & P(E)H) the variational inference converts the inference problem into optimization problem, where the prior distribution P(H) is the probability of arbitrary hypothesis H with respect to the observation E, and P(E)H)
the likelihood distribution. Specifically, the variational inference posits family of approximate distributions Q and aims to find a distribution Q (H) that minimizes the Kellback--eibler (KL) divergence between ((HE)) and Q(H) (Blei al. 2017).
Q* arg min (EP(HEEE)))
(85)
QEQ
Notably, recent works demonstrate that SGD performs variational inference during training DNNs (Mandt et al. 2017: Chaudhari & Soatto 2018). Following these works, further confirm the Bayesian explanation for SGD based on the proposed probabilistic for DNNs.
Given DNN {æx; fiü...; ; fr; fr ), we choose the cross entropy H[P(Y|X) ))) XXF the loss function, where P) is the true posterior distribution determined by the training dataset D and P (X) is the conditional distribution of fy given æ derived from the DNN.
H(P(Y|X), P( P(yy [x) H XL(P(Y|X) P(Fy [X)
(86)
Σ (F())
where H[ is the entropy of X(Y and (()()) is the true posterior distribution given the training data æ. For example, if L 4 and P(y az) is [0, 1, Oj. so P[y(())a:) 1 and [3)[ = 0. P(fr (x) is the learned conditional distribution of fy given the same æ, e.g., P(fr 2) could be [0.2, 0.7, 0.1,
In the context of deep learning, SGD aims to find the P'(Fy|XX such that it can minimize the loss function H X), P(Fx [X]] given the training dataset D.
Pa (Fy = arg H (
(87)
XiEO
Given the training dataset D. X is constant, i.e., ( = 0, thus we can obtain
P" ("
arg min KL(P(YIX) P( P(Fr|X) XX]
(88)
P(Fy|X)EE
we can confirm the equivalence between SGD and variational inference based on the proposed probabilistic explanations for DNNs. More specifically, the DNN defines a family of conditional Gibbs distribution P(Fy (X) (Equation 84) and SGD aims to find P* [Fy|XX such that it can minimize the distance to the true posterior distribution XXY
Comparing the Equation
88 and 85
J.3 SGD IS EQUIVALENT TO THE ENERGY MINIMIZATION OPTIMIZATION
In most deep learning applications, P(ylar) a:) commonly an one-hot vector, i.e., P(yla: only has single non-zero element. If assume that (" is the index of the non-zero element of P(y]ææ), i.e., ([)( = 1, we can simplify (L X) P(Fr [X]] = -log
Based on the definition of XXF (Equation
84
we can obtain
KL(P(Y|X) |
|oog (()x("))z)))
fur ffr--fr(---- (2) (..)).....))) )) + ZNN (z)
(89)
F + ZoNN
where fyr is the 1'th output node of the output layer fr. In particular, since log ZoglZzNx (x)] is not related to the output fy, it can be viewed as a constant with respect to the gradient calculation indicated by T Finally, combining Equation (8) and [89] we can obtain
P" ('(Fx
arg min EDNN (fut-
(90)
The above equation indicates that SGD is equivalent to the energy minimization optimization, which in turn validates the Gibbs explanation for the hidden layer of DNNs.
33

---

Under review as a conference paper at ICLR 2020
CNN
Label 2
Label
Label 8
Testing error (%%
Eanrgy
Eaurgs
Eaurgy
8
8
16
50
13
MLP
Label 4
Label
Label 10
Teslng enror (%)
8 x 8 N
-100
Energy
2000
Elvergy
2003
.009
8
4000
1000
20
10
29
08
13
20
Figure 23: The first row shows the testing error of the CNN and the variation of FECx (fyt (z) given three testing datasets with different labels over 30 training epochs. The second row shows testing error of the MLP and the variation of EaLLP(fe' (a)) given three testing datasets with different labels over 30 training epochs.
K EXPERIMENTS VALIDATING THE THEORETICAL EXPLANATION FOR SGD
K.1 EXPERIMENT: SGD IS EQUIVALENT TO THE ENERGY MINIMIZATION OPTIMIZATION
To validate the proof that SGD learns P with the minimal energy from the family of Gibbs distribution determined by a DNN, we use SGD to train the CNN defined in Table 9 for classifying the synthetic dataset shown in Figure 5] Specifically, we the entire training dataset train the CNN over 30 epochs. During each training epoch, we calculate the energy function of P(Fr X)) given a testing dataset with the same label. Since the testing dataset has the same label, have the same index of the single non-zero element of P zz). As a result, we can easily examine if the corresponding energy function ECNN (fy (z) decreasing during the training procedure.
Based on the architecture of CNN in Table 9 ECNN (f
i (x) can be formulated as ECNx (fy (z) - fyr- (f
(91) In Figure 23 the first row visualizes the variation of ECCNN (fgt- ær) over 30 training epochs. Though there are certain fluctuations, ( indeed shows decreasing trend. Due to limited space, we only visualize the energy minimization procedure given three labels, but the procedure is valid for any label. addition, we demonstrate the equivalence based on the MLP (Table 3 for classifying the MNIST dataset. The energy function of P(Fx|X) corresponding the MLP is defined as - fyt- (f)(f(()))
(92) In Figure 23 the second row shows the same energy decreasing trend during training the MLP. Overall, we demonstrate the equivalence between SGD and the energy minimization optimization, which in turn validates the Gibbs explanation for DNNs.
EMLP(fuL-H
K.2 EXPERIMENT: THE PARAMETERS LEARNED BY SGD ARE NOT I.I.D.
To validate the parameters learned by SGD not i.i.d., we use SGD to train the MLP defined in Table 3 and the CNN defined in Table Following the commonly used initialization approach, we initialize all the weights of the two DNNs by Gaussian random numbers, i.e., all the parameters can be viewed as i...d. before training. We use the absolute value of the sample correlation defined by Equation 2 to quantify the dependence between different weights during the training procedure.
Figure 24 visualizes the the variation of the average correlation between different weights of the three layers in the MLP. Initially, the average correlation for f1 and f2 are very small, because parameters are Gaussian random values. As the training procedure continues, we find that all the correlation values are increasing. Notably, the increase for fz is huge, i..., the weights of fz cannot be i.i.d. anymore. However, the increase for fi is very small, i.c., all the weights of fi still could be i.i.d.. Therefore, not all the learned parameters in the MIP are i.i.d.
34

---

Under review as a conference paper at ICLR 2020
MLP
Average correlation
0.250
10
6.003
0.225
Testing error
0.200
0.32
0.055
0.150
0.125
0.045
0.300
0.26
10
15
10
15
10
15
(A)
Figure 24: (A) shows the testing error over 15 training epochs. (B), (C), and (D) show the average correlation between different weights of three layers in the MLP over 15 training epochs.
Figure visualizes the variation of the average correlation between different weights of the five layers in the CNN. Similar as the weights in the MLP, the average correlation for fi and fz initially are very small, because they are initialized as Gaussian random values. As the training procedure continues, they have a sharp increasing and quickly achieve stationary, thus the parameters of fi and f2 are not i.i.d. anymore. addition, the average correlation for fs and fa also have an increasing trend, but their increase very small, i.e., the parameters of fa and f4 still could be i.i.d. Overall, we conclude that not the learned parameters in the CNN are i.i.d..
CNN
Testing error (%)
Average ation
0.25
0.20
0.10
10
15
20
25
30
10
15
20
25
30
5
10
20
25
30
(B)
0.375
0.08
0.350
Average
0.300
0.02
0.06
0.275
30
(D)
(E)
(F)
Figure 25; (A) shows the testing 30 training epochs. (B). (C), (D). (E) and (F) show the average correlation between different weights five layers the CNN over training epochs.
K.3 EXPERIMENT: BETTER PRIOR DISTRIBUTION INDICATES A BETTER GENERALIZATION PERFORMANCE
We design three CNNs = {x; fi: f2: fa: f4; fs: fr) in Table 10 to classify the synthetic dataset shown in Figure 5) We use the entire training dataset to train the CNNs over 30 epochs and the entire testing dataset to examine the generalization performance of the CNNs.
Based the MRF explanation for convolutional layers, the first convolutional layer fi formulates a MRF model as the prior distribution P(Fi), which can be expressed as
P(F))
I exp (xpp)((x)) zi, exp 2t (x)
(93)
ZF,
where spf (x) Sh oz ox bb the kth linear convolutional channel, S the linear convolutional filter and by is the bias.
35

---

Under review as a conference paper at ICLR 2020
Table 10: The architectures of CNNs for classifying the synthetic dataset
R.V.
Layer
Description
CNN1
CNN2
CNN3
X
Input
32 32
32 32
32
32
1
F1
fi
Conv 3) +ReLU
x 30 4
30
x
30
36
f2
Maxpool
15 x 15 x
15 X
15 x 12
15 x
15
36
f3
Conv (5 x 5) ReLU
11 x 11 20
11 11
20
11 11
20
F2
Maxpool
5 x 5 x
5 20
5
x 20
F3
f6
Fully connected
1 1 20
1x 20
1 1 20
Fy
fY
Oulput(sofitmax)
x 1 x 10
1x 10
X x 10
R. V. is the random variable of the hidden layer(s).
Since the synthetic dataset obeys the Gaussian distribution P(X), ) we can measure the quality the prior distribution by averaging KLD[P(XX) (X)) P(Fi) over all the testing images for every training epoch. In addition, we use the testing error to measure the generalization performance. Also note that the only difference between the tree CNNs is they have different K numbers of convolutional channels in f1, ie., different CNNs have different prior distributions. Therefore, we can examine the effect of the prior distribution by checking the generalization performance of the three CNNs.
Figure 26 shows the learned prior distributions and the corresponding testing error over 30 training epochs. We can find that there is positive correlation between KLD KLD(P(X)) P((F) )] and the testing error. Specifically, the testing error decreases as the learned prior distribution P(F1) approaching to the truly prior distribution P(X), and the testing error becomes stable when P(F,) achieves stationary for all the three CNNs.
Moreover, find that the CNN with more convolutional channels achieves the better generalization performance. In terms of Bayesian theory, that because a convolutional layer with more convolu- tional channels can formulate a more powerful prior distribution, such that it can convey more prior knowledge of the dataset, thereby rendering better generalization performance. Overall, we justify that better prior distribution indicates a better generalization performance.
CNN1-04
CNN1-04
CNN2-12
CNN2-12 12
CNN3-36
CNN3-36
KLD (P())
1.8
Testing error (%)
1.4
1.2
20 (A) Training epoch
10
15
10
20
(B) Training epoch
overall all testing images. (B) shows the generalization performance the three CNNs represented by the testing error. The number behind the three CNNs legend denotes the number of convolutional channels of f2.
Figure 26: (A) shows the average
36

---

Under review as a conference paper at ICLR 2020
L A REGULARIZATION ALGORITHM BASED ON THE PROBABILISTIC REPRESENTATION
L.1 THE PROPOSED REGULARIZATION ALGORITHM
Based on the probabilistic representation, the first hidden layer formulates a MRF model prior distribution of the training dataset and other hidden layers recursively serve as prior distributions for their respective next layer in CNN. However, the backpropagation calculates the gradient in the backward direction, i.e., updating the gradient of a hidden layer corresponding to a prior distribution must go over all hidden layers behind it. In addition, the degradation problem could makes the learned prior distribution not precise enough to regularize DNNs and results in overfitting, especially in very deep neural networks (He al. 2016).
In this section, we propose to a new regularization learning algorithm for DNNs, which includes two steps: (i) we learn the prior distributions of DNNs directly from the training dataset { } as the blue arrow shown in Figure 27 and (ii) we use the backpropagation to train the initialized DNNs by the learned prior distributions as the red arrow shown in Figure 27
Since directly learning prior distribution from {> 1/== is less complicated than the backpropagation learning algorithm, we can relieve the effects of degradation problem and derive prior distributions more precisely, thereby achieving the better generalization performance of DNNs.
{>>)//=
important question need to answer is how many hidden layers of DNNs need to be pre-trained in the first step. Since most prior information is stored in the training dataset { the hidden layer closest to the input of DNNs, i.e., the first hidden layer f1, should learn the most prior information from {x 1 as long as f1 has powerful representation ability through including sufficient hidden units, e.g., convolutional filters. Therefore, the proposed regularization method only pre-train the first hidden layer fi in order decrease the computation complexity. Pre-training more hidden layers could obtain better generalization performance, which can be a direction for future research.
We choose the Field of Experts (FoE) model to directly learn a prior distribution of (æ> [=')/=1 because of two reasons. Above all, FoE achieves great performance on learning prior knowledge of given datasets various applications, e.g., image denoising (Roth & Black 2005), image restoration Roth & Black (2008), and image inpainting (Schmidt et al.) 2011). More importantly, FoE can be regarded as a Convolutional Neural Networks (CNN) with a single convolutional layer, hence we can easily use the learned FoE model to initialize the first hidden layer of CNNs as long as they have the same architecture. FoE model can formulated as
(ron(a:)
II ( [f(()))) k=1
(94)
Z
where f-(-)) a linear filter, f pespent is an expert function, and ZFE Su I (fx (eexpent [fa (x)] is the partition function. We the contrastive divergence learning method train PFoE (a: under the Kullback-Leibler divergence (KLD) criterion (Carreia--errinnan & Hinton] 2005). More detail of the training procedure is in (Roth & Black 2005] 2008). After deriving well-trained FoE model, we use the learned linear filters [f to initialize the first convolutional layer of CNNs. Finally, we use the backpropagation algorithm train the initialized CNN for a specific application. such as image recognition. We summarize the proposed regularization learning algorithm in Algorithm
(x/) P(F1) P. (F+1lFF ... P(.
Figure 27: The training procedure of a DNN = {a: f::...; fr). The red arrow indicates the traditional training algorithm, e.g. backpropagaton, which derives an output of the DNN given training dataset {x') } in the forward direction and calculate the gradient of parameters with respect to the training labels Y for updating the parameters in the backward direction.
37

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Regularization learning algorithm
input:
= { yv 1 DNN = { fii fr; fr and (F)
1: setup
2: initialize iteration O, and training epochs K
3: specify the architecture the DNN and (a))
4: regularization learning
5: train proE(ar) given (æv, Ji
6: initialize fi the well-trained (n))
7: repeat
8: train the DNN given D
9: until (k- = K)
output: the well-trained DNN for image recognition
L.2 SIMULATIONS
In this section, we validate the proposed regularization learning algorithm for image recognition application on the CIFAR-10 benchmark dataset, which includes 70,000 32 x 32 color images in 10 classes, and each class has 6,000 training images and 1,000 testing images. Specifically, we randomly choose 2,000 training images of each class as a new training dataset and use the same testing dataset. Smaller training dataset means overfitting being more likely occur, which can help us validate the proposed regularization learning method more convincingly. In addition, we covert color images to grayscale to simplify computation.
In order to directly learn regularization from {x' 1 we design a FoE model with 12 Gaussian Scale Mixture (GSM) experts, thus it can be expressed as
PFoE (x) Zne: II (. (GGMM [Ax
(95)
En Thi (Vff(()): of (.))) of is fixed base variance, = {8(1), ... )(11)) is range of scales, fa(.) is 3 x 3 convolutional filter, and Thi denotes the weight of each Gaussian distribution. The parameters of the FoE model is 0 {f,x), π)) where The architecture of the CNN for image recognition summarized in II It is noteworthy that the number of convolutional filters in fi and the filter dimension are the same the above FoE model, thus we can use the well-trained FoE to initialize fi as a regularization for the CNN classifying the CIFAR- 10 dataset based on Algorithm 2.
where fk FGSM (x=)
= [/1,..., S12] and π { ) 12,12,
In order to make comprehensive comparison, choose commonly used regularizations, i.e., the dropout (drop rate is 0.3) and the L2 norm, as references. We use the testing error to exam the generalization performances of different regularizations. Figure 28 visualizes that the proposed regularization method (abbr. Bayes Prior (BP) successfully decreases the testing error. In particular, it outperforms dropout (32.26%) and achieve similar performance T.2 norm (29.58%). Therefore, this experiment validates the effectiveness of the proposed regularization learning method.
Table 11: The architectures of CNN for CIFAR-10 classification
Layer
Description
Output
Input
32 x 32 x
f1
Cony (3 x 3)
30
fz
ReLU
30 x 30 x 12
f3
Conv (5 x 5)
26 x 26 x 32
f4
ReLU + Maxpool
13 13 32
Js
Conv (5 x 5)
9 128
fe
ReLU + Maxpool
4 x x 128
fr
Fully connected
1 512
fs
ReLU
1 512
Fully connected
I 1 X 32
f10
ReLU
1 1 1 x 32
fY
Outpul(softmax)
1 10
38

---

Under review as a conference paper at ICLR 2020
Testing error of CNN without any regularization (33.95%)
Testing error of CNN with droupout (32.26%)
Testing error of CNN with L2 norm (29.58%)
0.
Testing error of CNN with Bayes prior (29.90%)
Testing erior
04-
0.85
0.25
Training epoch
Figure 28:
The average testing error of the CNN on the last 5 training epochs based various regularizations. Bayes prior indicates the proposed regularization and its testing error is 29.90%.
Moreover, the proposed regularization method can be easily combined with other regularizations methods to improve the generalization performance further. We combine two arbitrary aforemen- tioned regularization methods together and examine their generalization performances under the same experimental conditions above. Figure 29 shows that the combination of two regulariza- tions together achieves better generalization performance than any single one. In particular, the combination of BP and 1.2 norm achieves the lowest testing error (28.60%). Also note that if combine three regularizations together, we achieve the best generalization performance (27.69%). This experiment further validate the proposed regularization learning algorithm.
Testing error of CNN with Bayes Prior (BP) (29.90%)
Testing error of CNN with droupout + L2 norm (29.29%)
0.65
Testing error CNN with BP + droupout (28.91
Testing error CNN with BP + L2 norm (28.60%)
0.5
Testing error CNN with BP + both (27.69%)
0.65
rate
0.5
0.45
0.5
0.2
10
20
25
30
Training epoch
Figure 29: The average testing error of the CNN on last training epochs hased on various combined regularizations. Bayes prior + hoth indicates that we combine all three regularizations together.
39


Reviews:
[{"review_id": "rylK6ur15r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors show that parameters of a DNN do not satisfy the i.i.d. prior assumption and that neural layer activations considered as i.i.d. are not valid assumptions for all hidden layers of the network. One can therefore not rightfully use GPs to describe the network’s hidden layers. The authors suggest formulating the neurons per layer as energy functions thereby rendering a hidden layer as a Gibbs distribution and the connection between adjacent hidden layers as a PoE model. \n\nThe paper is well written and well postulated. \n\n> Fig 4: What is the information presented by each neuron? How would this have looked with the i.i.d. prior in place.\n\n> There are places in the paper where one must refer to the supplementary, for example sections H and J with the simulations. Do consider moving these crucial sections to the main paper.\n\n> One recurring thought I had when the authors bring up Bayesian Hierarchical model, is that most of the BHMs rely on i.i.d assumptions both in the prior space and with the observations. How would you stand by your claim of explaining a DNN's layers to be modelled as a BHM? \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJx3GcA6KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nMain contribution of the paper\n- The paper argues that the base assumption, the i.i.d. of the activated elements (activations) in the hidden layers, the existing methods (lee.et.al 2018) hold is not convincible.\n- Instead, the author proposes a new way to probabilistically model the hidden layers, activations, and layer/layer connections.\n- Based on the probabilistic model, the paper proposes a new regularizer.\n\nMethods\n- The author argues that the activation is not iid by empirically showing that the trained MLP (in most cases) does not un-correlated.\n- The author proposes a new probabilistic model for MLP, and CNN assuming the Gibbs distribution to each activation and also assuming the product of expert (poE) model to explain the layer/layer relationship.\n- And according to their model, CNN will be explained by the MRF model.\n- The author proposes a regularization term regarding layer/layer connection.\n- They argue that the SGD training can be seen as a first-order approximation of the inference of the hidden activations in MLP.\n\nQuestions\n- See the Concerns\n\nStrongpoints\n- The probabilistic explanation of the MLP and the CNN seems novel and was interesting to the reviewer\n- The proposed explanation assumes a weaker condition compared to the existing methods.\n\nConcerns\n- The main concern is that the reviewer cannot fully convince that i.i.d. assumption is wrong. \nEven though the trained MLP does not support the i.i.d. condition, one can suppose that the reason would be the typical training method (SGD), just finding the local minima in a deterministic way.\nMaybe the proof in Appendix.G. supports the argument of the author, but the reviewer failed to clearly agree with the argument.\nA clear explanation regarding the issue would be required.\n- As far as the author understands, the paper proposes a probabilistic (Bayesian) model for explaining MLP, but it seems that they just used SGD for training the model. \nIn that case, the reviewer is little suspicious of the role of the proposed regularization in that the regularization comes from Bayesian formulation, but the model was trained in a deterministic way.\nThe reviewer wants to ask the author that \n(1) is it possible to infer the model in a Bayesian manner such as sampling?\n(2) Is there any justification for using SGD when conducting the experiments regarding the regularization? If it is related to Appendix.G, clearer explanation would be appreciated.\n- As far as the reviewer understands, the regularization deals with the practical part of the paper. It would be better to see the effect of the regularization of widely used networks such as small-layered ResNet or others.\nIf the proposed formulation has other practical strongpoints, it would be nice to clarify them.\n- The explanation using Gibbs distribution and PoE looks similar to RBM. The reviewer strongly wants a clear explanation of the difference and the strongpoints compared to RBM.\n\nConclusion\n- The author proposed a new probabilistic explanation of the neural network, which seems novel and worth reporting.\n- However, the reviewer failed to fully agree on some steps in the process of the paper.\nTherefore, the reviewer temporary rates the paper as weak-reject, but this can be adjusted after seeing the answers of the author.\n \nInquiries\n- See the concerns parts.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgMJnNvFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the Paper:\n  \nThe authors claim the that i.i.d hypothesis that is often used in the prior when looking for the equivalence between neural networks and GP is not valid. Then, they propose a new interpretation of neural networks as Gibbs distributions (in the case of fully connected layers) and a MRF in the case of convolutional layers. Some simulations are done to verify this.\n\n\nDetailed comments:\n\nOverall I believe that the writing of the paper is very sloppy and difficult to read and follow. It is not clear that the GP interpretation cannot be valid simply because the activations and weights are not i.i.d. I.I.D is a sufficient condition but not required in the central limit theorem. For example, the sum of correlated Gaussian variables also tends to a Gaussian distribution. The same for non-Gaussian variables. Therefore, I do not think that the GP interpretation of the NN is wrong simply for that. This simply shows that the i.i.d. prior may be suboptimal.\n\nSumming up, I think that this paper needs more work. Currently, I do not think I can extract anything useful from it.\n\nThe prior is subjective and can be chosen by the user. So if an i.i.d. prior is actually chosen, the corresponding Bayesian  neural network will converge to a GP.\n\nAn i.i.d. prior may be sub-optimal. However, it can be used to interpret neural networks as GP. There is no problem with that.\n\nIt is well known that the sum of random variables can also converge to a Gaussian distribution even though they are not independent. This questions the claims of the paper.\n\nThe witting of the paper needs to be improved. There are several expressions that do not sound well. E.g., \", GP with i.i.d.\"\n\nZ'y in Eq. (9) should depend on l.\n\nIt is not clear what is the distribution of a hidden layer (activations weights etc..).\n\n\"...and f Y is an estimation of the true distribution P (Y |X)\" what do you mean by that?\n\nEq. (8) seems to be a prob. distribution for the random variable Fy. However, the authors give an expression for f_yl, which does not make sense.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes a claim that the iid assumption for NN parameters does not hold. The paper then expresses the joint distribution as a Gibbs distribution and PoE. Finally, there are some results on SGD as VI. Reviewers have mixed opinion about the paper and it is clear that the starting point of the paper (regarding iid assumption) is unclear. I myself read through the paper and discussed this with the reviewer, and it is clear that there are many issues with this paper.

Here are my concerns:
- The parameters of DNN are not iid *after* training. They are not supposed to be. So the empirical results where the correlation matrix is shown does not make the point that the paper is trying to make.
- I agree with R2 that the prior is subjective and can be anything, and it is true that the "trained" NN may not correspond to a GP. This is actually well known which is why it is difficult to match the performance of a trained GP and trained NN.
- The whole contribution about connection to Gibbs distribution and PoE is not insightful. These things are already known, so I don't know why this is a contribution.
- Regarding connection between SGD and VI, they do *not* really prove anything. The derivation is *wrong*. In eq 85 in Appendix J2, the VI problem is written as KL(P||Q), but it should be KL(Q||P). Then this is argued to be the same as Eq. 88 obtained with SGD. This is not correct.

Given these issues and based on reviewers' reaction to the content, I recommend to reject this paper. 
Decision: Reject