論文 / 著書情報
Article / Book Information
題目(和文)
Title(English)

Generalization Analysis on dependency of Model Complexity inBayesian Deep Neural Network

著者(和文)

永安修也

Author(English)

Shuya Nagayasu

出典(和文)

学位:博士(理学),学位授与機関:東京工業大学,
報告番号:甲第12656号,
授与年月日:2024年3月26日,
学位の種別:課程博士,
審査員:渡邊 澄夫,金森 敬文,山下 真,中野 張,髙邉 賢史

Citation(English)

Degree:Doctor (Science),
Conferring organization: Tokyo Institute of Technology,
Report number:甲第12656号,
Conferred date:2024/3/26,
Degree Type:Course doctor,
Examiner:,,,,

学位種別(和文)

博士論文

Type(English)

Doctoral Thesis

Powered by T2R2 (Science Tokyo Research Repository)

Dissertation

Generalization Analysis on dependency ofModel Complexity in Bayesian Deep
Neural Network

Shuya Nagayasu
School of Computing
Department of Mathematical and Computing Science
Tokyo Institute of Technology2023

Abstract
Deep Neural Networks (DNNs) are widely used as machine learning models for computer
vision, natural language processing, robotics, and many other areas. The high performanceof DNNs comes from the fact that the rules in machine learning models such as bias-variance
tradeoﬀ do not hold. Recent year studies revealed the relationship between Stochastic gradientdescent and Stochastic gradient langevin dynamics which realize Bayesian learning. On theother hand, many studies try to realize the Bayesian Deep Neural Networks. From theseviewpoints, Bayesian learning in DNNs gets more important.
In this thesis we theoretically show that the performance of deep neural networks is different from the classical statistical models in perspective of Bayesian learning through thestatistical analysis of generalization error in two cases. One is the case that there exist multiple optimal probability distributions which are nearest to data generating process. In thiscase, the generalization error increases while the number of data increases. This case occurs
when the model has relatively smaller complexity to the data. The other is the case that themodel has larger number of parameters than necessary. In this case, the generalization error
does not increase even if the number of the model parameters increases.

3

Contents
1 Introduction
1.1 Deep Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Bayesian Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Neural Networks in Bayesian Learning . . . . . . . . . . . . . . . . . . . . . . .
1.4 Goals of studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7
7
7
8
8

2 Bayesian learning theory
2.1 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Bayesian learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Asymptotic analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.1 Kullback-Leibler divergence . . . . . . . . . . . . . . . . . . . . . . . . .
2.3.2 Free energy and Generalization Error . . . . . . . . . . . . . . . . . . .
2.3.3 Regular Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Singular learning theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9
9
9
10
10
10
11
12

3 Learning theory for non-identiﬁable optimal probability distributions
3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Theoretical conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Main theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Proof of main theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.1 Experiment1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5.2 Experiment2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Appendix 3.A Maximum value of 2-dimensional Gaussian . . . . . . . . . . . . . . .

15
15
16
18
19
24
24
26
27
28
28

4 Free Energy of Bayesian Convolutional Neural Networks
4.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Main Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Proof of main theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1 Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.2 Notations of parameters . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.3 No Skip Connection Case . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.4 Skip Connection Case . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31
31
32
34
35
35
36
37
39

5

4.5

Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.2 Result of experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6.1 Diﬀerence with or without Skip Connection . . . . . . . . . . . . . . . .
4.6.2 Comparison to Deep Neural Network . . . . . . . . . . . . . . . . . . . .
4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Appendix 4.A Fully connected case . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.A.1 Main theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.A.2 Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.A.3 Proof of Main Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Conclusion

40
40
41
41
41
41
42
42
42
43
48
51

Chapter 1

Introduction
1.1

Deep Neural Networks

Deep Neural Networks(DNNs) are the most used as machine learning models in the lastten years. Though the original idea of Neural Network was shown in 1940s, the basis ofmodern Deep Learning arose in 1989 LeNet [23]. The structure of LeNet was the combination of convolutional layers and fully-connected layer same as modern Convolutional Neural
Networks(CNNs) and the backpropagation was used. After twelve years of stagnation after
success of LeNet, AlexNet [21] triggered modern using of Neural Networks. AlexNet is a
development of LeNet and earlier architecture using GPU. Now many architectures of Neural
Networks are used such as Residual Network (ResNet)[15] for computer vision, Generative
Adversarial Network [11] for image processing, Transformer for natural language processing
and many others for many areas. However, the clear theoretical explanation to the high performance of Neural Networks are unknown yet. In particular, it is well known that bias-variance
trade-oﬀ shown in conventional learning models does not occur in deep neural networks. In
present, there exist some diﬀerent theories such as double descent [29] or Bayesian perspectiveof SGD [36].

1.2

Bayesian Learning

The name of Bayesian learning or Bayes inference comes from the Thomas Bayes who is an
English statistician in 18th century. He discovered the basic probabilistic law of conditional
probability distribution. For a long time, Bayesian inference is used only for analytically calculated cases such as exponential family and conjugate prior because the calculation of marginal
likelihood needs the integration in Bayesian inference. The situation changed with the development of Metropolis–Hastings algorithm which is one of Markov chain Monte Carlo methods(MCMC) in statistical physics [13]. Metropolis-Hastings algorithm has the problem of rejection rate, Hamiltonian Monte Carlo(HMC) which comes from the Hamiltonian dynamics inphase space of statistical mechanics solved this problem [7]. In modern Bayesian inference,
Hamiltonian Monte Carlo or No-U-Turn sampler [17] which is an improvement of HMC ismainly used for practical way, but the situation changed in Bayesian learning for DNNs. The
name of Bayesian learning or Bayes inference comes from the Thomas Bayes who is an English
statistician in 18th century. He discovered the basic probabilistic law of conditional probability
7distribution. For a long time, Bayesian inference is used only for analytically calculated casessuch as exponential family and conjugate prior because the calculation of marginal likelihood
needs the integration in Bayesian inference. The situation changed with the development ofMetropolis–Hastings algorithm which is one of Markov chain Monte Carlo methods (MCMC)
in statistical physics [13]. Metropolis-Hastings algorithm has the problem of rejection rate,
Hamiltonian Monte Carlo(HMC) which comes from the Hamiltonian dynamics in phase spaceof statistical mechanics solved this problem [7]. In modern Bayesian inference, Hamiltonian
Monte Carlo or No-U-Turn sampler [17] which is an improvement of HMC is mainly used forpractical way, but the situation changed in Bayesian learning for DNNs.

1.3

Neural Networks in Bayesian Learning

Earlier ideas of Neural Networks in Bayesian Learning were discussed by MacKay [24, 25].
In these papers, the gaussian approximation of posteriors which is the variational inference
was mainly used. Neal [30] showed the idea of applying MCMC for Bayesian Neural Networks. Variational Approximation and MCMC are the basic methods of realizing Bayesianposterior for neural networks. Variational Autoencoder [20] and Monte Carlo dropout [9] arerepresentative variational methods in practical use. On the other hand, Hamiltonian Monte
Carlo or Langevin Dynamics are mainly used among MCMC. In particular, the developmentof Stochastic Gradient Langevin Dynamics(SGLD) [49] makes it easier to conduct MCMC in
Neural Network because of the similarity to Stochastic Gradient Descent. In addition, recent
studies showed that the randomness of Stochastic Gradient Descent(SGD) makes them closer
to random walk and SGLD [36]. These studies connected the behavior of SGD with theﬂatness [16] through Bayesian learning and the explanation of generalization ability of Deep
Learning. From both viewpoints of direct use of Bayesian Neural Networks and the theoryof SGD in Neural Networks, the understanding of the behavior of Bayesian Neural Networks
has become important.

1.4

Goals of studies

In this thesis we show the two cases of theoretical analysis in Bayesian generalization
error about neural networks [28, 26]. In the ﬁrst case, we clarify the generalization error ofBayesian learning when optimal probability distributions are not unique. Optimal probability
distribution is the probability distribution nearest to the data generating process in learningmodel. In conventional learning theory, such distributions are assumed to be unique. However,in case the data generating process is more complicated than model approximation ability,
this assumption cannot always be applied. In particular, if there exists the symmetry indata generating process which is mismatched to the learning model, this case occurs. In thesecond case, we reveal the generalization error of Convolutional Neural Network with andwithout skip connection in Bayesian learning. Skip connection is used for many architecturesof CNNs. With skip connection case, we show that the variance term of generalization error
only depends on the complexity of data generating process. As far as currently known, themodels which have such property in Bayesian learning are only fully-connected neural networkwith ReLU activation function and CNNs with skip connection. Through analyzing these twocases we revealed the characteristics of DNNs in Bayesian learning.

Chapter 2

Bayesian learning theory
In this chapter, we explain the framework of Bayesian Learning theory.

2.1

Notations

Table2.1 show the deﬁnitions and notations used in this chapter.
Table 2.1: Notation
NotationE[· · · ]Ew [· · · ]EX [· · · ]
S
Fn
Gn

2.2

Deﬁnition
∫
∏n
n
i )dX
∫ · · · i=1 q(Xn∫ · · · p(w|X )dw· · · q(x)dx−Ex [log q(x)]− log ZnEx [log q(x)/p(x|X n )]Nameaverage of generating of samples
average of posterior
average of true distribution
entropy
the free energy
the generalization error

Bayesian learning

First, we explain the basic calculation of Bayesian learning. In supervised learning, Let
X n = (X1 , · · · Xn ) and Y n = (Y1 , · · · Yn ) be training data and labels or output data. The
natural number n is the number of the data. These data and labels are generated from atrue joint distribution q(x, y) = q(y|x)q(x). The prior distribution φ(w), the learning modelp(y|x, w) is given on the bounded parameter set W . Then the posterior distribution is deﬁned
by
∏
1
φ(w)
p(Yi |Xi , w)nn
Z(Y |X )np(w|X n , Y n ) =

(2.1)

i=1

where Zn

= Z(Y n |X n ) is normalizing constant denoted as marginal likelihood:
∫
Zn =

φ(w)

n
∏
i=1

9

p(Yi |Xi , w)dw.

(2.2)

The posterior distribution estimates the parameters probabilistically. The posterior predictive
distribution is deﬁned as the average of the model by posterior:∫p∗ (y|x) = p(y|x, X n , Y n ) = p(y|x, w)p(w|X n , Y n )dw.
(2.3)
The posterior predictive distribution p∗ (y|x) is an estimation of true conditional distribution
q(y|x). In case unsupervised learning, the training data is X n = (X1 , · · · Xn ) and the true
distribution is q(x). The learning model is p(x|w). In this case, the posterior distribution andthe posterior predictive distribution are deﬁned by
n
∏
1
φ(w)
p(Xi |w)p(w|X ) =Z(X n )
i=1∫∗np (x) = p(x|X ) = p(x|w)p(w|X n )dw.
n

(2.4)
(2.5)

Hereinafter in this chapter, we explain about the case unsupervised learning. The samethings hold also in case supervised learning.

2.3

Asymptotic analysis

2.3.1

Kullback-Leibler divergence

Kullback-Leibler(KL) divergence is a statistical distance measuring the probability distributions. For probability distributions p(x), q(x) which supports P, Q satisfy the Q ⊂ P , KLdivergence is deﬁned by∫q(x)DKL (q(x)|p(x)) = q(x) log
(2.6)
dx.
p(x)
KL divergence has the following properties.
• DKL (q(x)|p(x)) ≥ 0• DKL (q(x)|p(x)) = 0 if and only if p(x) = q(x)• DKL (q(x)|p(x)) ̸= DKL (p(x)|q(x)) in general.

2.3.2

Free energy and Generalization Error

The generalization error in Bayesian learning is KL divergence between true distribution
and estimated distribution.
Gn = DKL (q(x)|p∗ (x))(2.7)

The free energy Fn is a negative log marginal likelihood also as known as stochastic complexity
Fn = − log Zn .

(2.8)

Minimizing the average of free energy in data generating is equivalent to minimizing true joint
distribution of data and marginal distribution
E[Fn ] = −E[log Zn ][]n∏q(X n )
= −E[logq(Xi )] + E logp(X n )

(2.9)
(2.10)

i=1= nS + DKL (q(X n )|p(X n )).

(2.11)

Average of the generalization error is diﬀerence between the average of Free energy of n andn + 1:
Lemma 2.3.1.
E[Gn ] − S = E[Fn+1 ] − E[Fn ].
Proof. For p(x|X n ) = p(Xn+1 |X n ), the following equation holds
∫
n
∏
1
n
p(Xn+1 |w)φ(w)p(Xi |w)dw
p(x|X ) =Zn

(2.12)(2.13)

i=1

=

Zn+1
.
Zn

(2.14)

The average of a sample of negative log of equation(2.14) completes the lemma.

2.3.3

Regular Case

The set of optimal parameters W0 is deﬁned as the set of all parameters that minimize
the KL divergence of q(x) and p(x|w),∫q(x)W0 = {w ∈ W | q(x) log
dx is minimized.}(2.15)
p(x|w)The log density ratio function for w0 ∈ W0 and w ∈ W is deﬁned asp(x|w0 ).
p(x|w)(2.16)1∑
Kn (w) =f (Xj , w0 , w).
n(2.17)f (x, w0 , w) = log
The empirical error function Kn (w) is deﬁned asnj=1

The log loss function L(w) and the average error function K(w) are deﬁned by
L(w) = −EX [log p(X|w)],(2.18)

K(w) = EX [f (X, w0 , w)] = L(w) − L(w0 ).

(2.19)

K(w0 ) = 0, K(w) ≥ 0.

(2.20)

K(w) satisﬁes

If K(w)can be approximated by a quadratic form, in other words, the Laplace approximation can be applied to the posterior distribution, average of Free energy has the following
asymptotic expansion with the number of parameters of the learning model d [35, 32]
E[Fn ] = n(S + Bias) +dlog n + O(1)2

(2.21)

where S is entropy of true distribution and Bias is
DKL (q(x)|p(x|w0 )).

(2.22)The generalization error is calculated from Free energy by using equation(2.3.1) [2]:( )d1
E[Gn ] = Bias ++o.
(2.23)
2nn

2.4Singular learning theory

Laplace approximation cannot be applied to the average Kullback-Leibler divergence of
hierarchical model such as Gaussian Mixture or neural networks because of the degenerationof Fisher information matrix. In this section, we brieﬂy review the theory of asymptotic
behavior of Bayesian generalization error and free energy in such singular cases.[42]Theorem 2.4.1 (Resolution of singularities). Let W be a bounded closed set in Rd . Assume
that K(w) is nonnegative analytic function on W and that the set {w ∈ W : K(w) = 0} is
not empty. Then there exists ϵ > 0, {Wi : Wi ∈ W }, and {Ui : Ui ∈ Rd } which satisfy{w ∈ W : K(w) ≤ ϵ} =∪Wi

(2.24)iand, in each pair Wi and Ui , there exists an analytic map g : Ui → Wi which satisﬁes
K(g(u)) = u2k ,(2.25)′(2.26)|detg (u)| = b(u)|u |hwhere k, h(k > 0, h ≥ 0) are d-dimensional multi-indexes, and b(u) > 0 and g ′ (u) is Jacobianmatrix.
Deﬁnition 2.4.2 (Real Log Canonical Threshold). Let W be a bounded closed set in Rd .
Assume that K(w) is an analytic function of w ∈ Rd and φ(w) is C ∞ function with compact
support W ⊂ Rd . Then, the zeta function is deﬁned by following with a complex variable z∫
K(w)z φ(w)dw.
ζ(z) =W

This function is holomorphic in Re(z) > 0. A real log canonical threshold (RLCT) is deﬁned
by the negative maximum pole of ζ and its multiplicity is deﬁned by the order of the maximum
pole.

The normalized marginal likelihood is deﬁned by∫0
Zn =exp(−nKn (w))φ(w)dw.
w∈W
(0)

(1)

(2)

The normalized marginal likelihood Zn can be divided into Zn and Zn as
Zn(0) = Zn(1) + Zn(2) ,
∫
(1)
Zn =
exp(−nKni (w))φ(w)dw,
w∈Wi ,K(w)<ϵ
∫
(2)
Zn =
exp(−nKni (w))φ(w)dw.

(2.27)(2.28)(2.29)w∈Wi ,K(w)≥ϵ
(1)

(2)

For Zn , Zn ,Lemma 2.4.3. Let ϵ > 0 be a monotonically decreasing function of n which satisﬁes
lim ϵ = 0,√
lim nϵ = ∞.
n→∞

n→∞

(2.30)(2.31)

If ϵ satisﬁes these conditions, then√Zn(2) = Op (exp(− n)).
Lemma 2.4.4. If the log density ratio function has a relatively ﬁnite variance,
(
)
∫
∫
√
(log n)m−1(log n)m−1∗
λ−1Zn(1) =tξ(u))dt+odutexp(−t
+
n
p
i
nλ
nλ

(2.32)(2.33)

holds. In this equation, λ and m are respectively the real log canonical threshold and themultiplicity of zeta functions, and ξn (u) is an empirical process that converges in distribution
to a Gaussian process. du∗ is a measure represented by u, k, h of Ui on which there are thereal log canonical threshold and the multiplicity.
The realizable case of this lemma is proven in [41] and non-realizable case is in [42]. From
Lemma2.4.4 and 2.3.1 we can get the asymptotic form of free energy and the generalization
error in singular case that
E[Fn ] = n(S + Bias) + λ log n − (m − 1) log log n + O(1),
( )
λ
1
.
E[Gn ] = Bias + + onn(2.34)(2.35)

Chapter 3

Learning theory for non-identiﬁable
optimal probability distributions
3.1

Motivation

In statistical learning theory, a probability distribution which generates a sample is called atrue distribution and one with a parameter is called a statistical model or a learning machine.
A probability distribution is estimated by applying a training algorithm to a statistical model.
Then, the diﬀerence between the true distribution and the estimated one is deﬁned by somemeasure, for example, the Kullback-Leibler (KL) divergence. In practical applications, thetrue distribution is unknown, hence the free energy and the generalization loss, which givethe relative diﬀerence of KL divergence, are used to evaluate the estimated one.
The theoretical values of the free energy and the generalization loss strongly depend on
the geometrical situations of the true distribution and a statistical model. A statistical modelis called regular if the parameter which minimizes the KL divergence of a true distribution
and the statistical model is unique and Hessian matrix of the KL divergence at the minimum
point is regular. For the regular case, the asymptotic behavior of the generalization loss was
revealed by Akaike[1], while that of the free energy was revealed by Schwarz[35]. These results
have been applied to statistical model selection criteria, i.e. Akaike(AIC), Bayesian(BIC),
Deviance(DIC)[37], and Adjusted Bayesian(ABIC)[2].
If a statistical model is not regular, then it is called singular. Many practical probabilistic
models such that neural network, normal mixture model, Bayesian network are singular. On
some singular models, asymptotic behavior of generalization loss and free energy are revealed
when statistical model includes the true distribution in other words bias equals to 0[51, 33].
These studies based on the algebraic geometrical methods[41]. The model selection criterion
for singular cases:WAIC[44], WBIC[46], sBIC[6] are also suggested. In former studies insingular case, there exists an assumption which holds when bias equals to 0. However, it is
not clear whether the assumption holds or not with larger bias. In particular if probability
distribution on optimal parameter is not unique, this assumption does not always hold. Insuch cases asymptotic behavior of generalization loss and free energy are unknown. Whether
WAIC and WBIC select appropriate models or not is also unclear.
In this chapter, we show the asymptotic behavior of the generalization error and thefree energy when the optimal probability distributions are not unique. We show that the15

generalization error gets larger as the sample size increases in this case. This behavior isnot shown in conventional studies of singular model in Bayesian statistics. This behavior isnear to Deep Double Descent [29]. In addition, we show that with small sample size, thebias of generalization is smaller than apparent bias. This situation occurs when complex true
distribution is estimated by singular statistical model such as neural networks.

3.2

Theoretical conditions

In this section, we summarize some theoretical conditions in the previous researches andin this paper. Many studies required the following condition.
Deﬁnition 3.2.1 (Regular). Let J(w0 ) be Hessian matrix of KL divergence of q(x) andp(x|w) on optimal parameter w = w0 . If w0 is a single point and J(w0 ) is a regular matrix,
the statistical model is regular.
This is one of the conditions of asymptotic normality. It is known that if the statisticalmodel is regular, asymptotic behaviors of the free energy and the generalization loss haveasymptotic expansion[35][2],dlog n + O(1),
2
( )
d
1
E[Gn ] = L(w0 ) +
+o
.
2n
n
E[Fn ] = nL(w0 ) +

(3.1)
(3.2)

If Hessian matrix is not regular, the statistical model is singular. In this case, asymptotic
behaviors of the free energy and the generalization loss are also known[41][42] as,E[Fn ] = nL(w0 ) + λ log n − (m − 1) log log n + O(1),
(
)
λ
m−1
1
E[Gn ] = L(w0 ) + −+o,n n log nn log n

(3.3)
(3.4)

where λ > 0 is a rational number called the real log canonical threshold (RLCT) and m ≥ 1
is a natural number called the multiplicity. Deﬁnitions of these numbers are described in
Deﬁnition 2.4.2. These equations require an assumption(see Assumption 3.2.3).
The following condition is also set in many studies.
Deﬁnition 3.2.2 (Realizable). If a statistical model satisﬁes q(x) = p(x|w0 ), q(x) is said tobe realizable by p(x|w).
This condition means a statistical model includes true distribution. In other words, bias
equals to 0 in this condition.
To focus on model selection criteria, AIC[1] and DIC[37] require both of regular andrealizable for asymptotic equality to generalization loss. So, these criteria are useful whenbias is small and eigenvalues of Hessian matrix is large. On the other hand, BIC[35] andABIC[1] don’t require realizable condition for asymptotic equality to free energy. sBIC[6] don’t
require regular condition for asymptotic equality to free energy. WAIC[44] and WBIC[46]
don’t require both of regular and realizable conditions. Thus, these conditions are applicableconditions of Information criteria.

In singular and realizable case, asymptotic behaviors of generalization errors are revealed in some statistical models in Bayesian estimation :Normal mixture[51], reduced rank
regression[5], naive Bayesian network[33], Markov model[55], and latent Dirichlet allocation[14].
These researches reveal variance of each statistical model when bias equals to 0 by calculating
RLCT and the multiplicity. By studying not only RLCT and the multiplicity but also higher
order random variable terms, [19] constructs the Bayesian test of normal mixture. Asymptotic behavior of free energy of Bayesian network when the data are not simple independent
identical random variables is also studied[50]. This is one of the studies removing conventional
assumption in singular case.
Those singular case studies require realizable condition. On the other hand some regularcase studies don’t require realizable condition. To treat these conditions simultaneously thefollowing condition is suggested.
Assumption 3.2.3. If the following condition is satisﬁed, it is said that the log density ratio
function has a relatively ﬁnite variance.
∃c0 > 0, ∀w ∈ W, ∀w0 ∈ W0 , EX [f (X, w0 , w)] ≥ c0 EX [f (X, w0 , w)2 ],(3.5)

The following theorem holds for this assumption.
Theorem 3.2.4. If a statistical model is regular or realizable, the log density ration function
has a relatively ﬁnite variance.
The proof of this theorem is written in [47]. From this theorem it follows that the theorems
under Assumption3.2.3 also hold with regular case or realizable case. Asymptotic equivalenceof generalization loss and WAIC, free energy and WBIC, equation 3.3, 3.4 required thisassumption. Thus, these information criteria and equations hold in regular or realizable case.
For example, it is known that if a statistical model is regular, λ = d/2 and m = 1 therefore
equation3.3 is same as equation3.1 and therefore equation3.4 is same as equation3.2.
This assumption holds when bias is near to 0 or the statistical model is regular, but otherthan those conditions, it is not unclear whether this assumption holds or not. Thus it is notalso unclear that whether WAIC and WBIC are useful or not and equation3.3, equation3.4
hold. In particular, when a statistical model does not satisfy the following condition thatmodel does not always satisfy Assumption3.2.3.
Deﬁnition 3.2.5 (Essentially Unique). If p(x|w0 ) is unique probability distribution for all
w0 , the statistical model is essentially unique.
The following theorem holds for this condition.
Theorem 3.2.6. If the log density ratio function has a relatively ﬁnite variance, the statisticalmodel is essentially unique.
Proof If a statistical model satisﬁes Assumption3.2.3, there exists a positive real number
c0 > 0 such that for any w01 , w02 ∈ W0 ,0 = L(w01 ) − L(w02 ) = EX [f (X, w01 , w02 )] ≥ c0 EX [f (X, w01 , w02 )2 ],(3.6)

and it follows that f (x, w01 , w02 ) = 0 for all x, which means p(x|w01 ) = p(x|w02 ).
From contraposition of this theorem, if a statistical model is not essentially unique, thestatistical model does not satisfy Assumption3.2.3. In this paper, for constructing the theory
in non-essentially unique case, we discuss with following relaxed version of Assumption3.2.3.

Assumption 3.2.7.
∃c0 > 0, ∀w ∈ W, ∃w0 ∈ W0 , EX [f (X, w0 , w)] ≥ c0 EX [f (X, w0 , w)2 ],(3.7)

From deﬁnition, Assumption3.2.7 includes the assumption3.2.3. Assumption3.2.7 holds
if there exists even one w0 for all w satisfying the inequality even though Assumption3.2.3
required all w0 for all w.
Example. The following is an example for the case that the optimal probability distribution
is not unique. We suppose supervised learning of q(y|x) using a statistical model p(y|x, a, b).
The variables a, b are parameters. We suppose true distribution q(y|x), q(x) is)(1(y − f (x))2√,q(y|x) =
exp −
2
2π
{ 1
4 (−2 ≤ x ≤ 2) ,q(x) =
0 (otherwise) x + 2 (−2 ≤ x < −1)
1(−1 ≤ x < 1) .
f (x) =
−x + 2(1 ≤ x ≤ 2)
We also suppose a statistical model p(y|x, a, b),()(y − σ(ax + b))21p(y|x, a, b) = √ exp −22π1σ(x) =.
1 + exp(−x)

(3.8)
(3.9)

In this situation, the KL divergence between q(y|x) and p(y|x, a, b) can be calculated as∫∫q(y|x)
q(y|x)= q(y|x)q(x) log
dxdy,
KL(q(y, x)|p(y, x|a, b)) = q(y, x) log
p(y|x, a, b)p(y|x, a, b)(3.10)∫
1q(x)(f (x) − σ(ax + b))2 dx.
(3.11)=
2
Note that q(x) and f (x) are even functions. In addition, σ(ax + b) and σ(−ax + b) areline symmetric on x = 0. Therefore, KL(q(y|x)|p(y|x, a, b)) = KL(q(y|x)|p(y|x, −a, b)) holds.
Thus, we can ﬁnd two optimal parameters as (a0 , b0 ) and (−a0 , b0 ). These two points satisfyp(y, x|a0 , b0 ) ̸= p(y, x| − a0 , b0 ). A numerical calculation shows that the optimal parametersin this case are (5.13, 7.71), (−5.13, 7.71). Note that each neighbor of optimal parameter(5.13, 7.71), (−5.13, 7.71) is same as regular condition, thus in this case statistical modelsatisﬁes the Assumption3.2.7.

3.3

Main theorem

We estimate a probability distribution q(x) by Bayesian inference using a statistical modelp(x|w) and prior distribution φ(w). It is assumed that the set of parameters W is compact
and that p(x|w) is continuous for w.
Problem treated in this paper. We assume about the set W0 .

Assumption 3.3.1. We assume that the set W0 can be represented by a disjoint union of
W0i (i ∈ I) which satisﬁes∪i∈I W0i = W0 ,(3.12)

W0i ∩ W0j = ∅(i ̸= j),(3.13)

where in each subset W0i for w0i ∈ W0i optimal probability distribution p(x|w0i ) is unique on
the other hand if i ̸= j, w0i ∈ W0i , w0j ∈ W0j , p(x|w0i ) ̸= p(x|w0j )In particular, if I has only a single element, it is same as essentially unique case.
The main result of this paper is the following Theorem3.3.2.
Theorem 3.3.2. If a statistical model satisﬁes Assumption3.2.7 and Assumption3.3.1, index
set I are natural numbers the free energy and generalization loss has an asymptotic expansion
that√(3.14)
E[Fn ] = nL(w0 ) − nµ + λ̂ log n − (m̂ − 1) log log n + O(1),()µ
1E[Gn ] = L(w0 ) − √ + o √(3.15)
2 nnwhere µ, λ̂, m̂ are real numbers satisfying µ ≥ 0, λ̂ > 0, and m̂ ≥ 1.

3.4

Proof of main theorem

In this section, we discuss the proof of Theorem3.3.2 in case the number of optimal probability distributions is ﬁnite.
To show Theorem3.3.2, let us represent the set of parameters as
W = ∪mi=1 Wi ,where W0i ⊂ Wi . We deﬁne the empirical log function as
1∑
log p(Xj |w).
nn

Ln (w) = −(3.16)

j=1

We also deﬁne the index empirical log function Lni (w) and the index empirical error function
Kni (w) as
Lni = Ln (w0i ).
n1∑
Kni (w) =f (Xj , w0i , w).
n(3.17)(3.18)

j=1(i)

The marginal likelihood of each domain Zn is also deﬁned by∫Zn(i) =exp(−nLn (w))φ(w)dw.
w∈Wi(3.19)

By using Ln (w) = Kni (w) + Lni , we have∫(i)
Zn = exp(−nLni )exp(−nKni (w))φ(w)dw.

(3.20)w∈Wi

The index normalized marginal likelihood of each domain Zn0i is deﬁned by
∫
0i
Zn =
exp(−nKni (w))φ(w)dw.

(3.21)w∈Wi

In accordance with these deﬁnitions, the marginal likelihood is given by∫Zn =exp(−nLn (w))φ(w)dw =
w∈W

m
∑

exp(−nβLni )Zn0i .

(3.22)

i=1

The normalized marginal likelihood Zn0i can be divided into Zn1i and Zn2i asZn(0i) = Zn(1i) + Zn(2i) ,
∫
(1i)
Zn =
exp(−nKni (w))φ(w)dw,
w∈Wi ,K(w)<ϵ
∫
(2i)
Zn =
exp(−nKni (w))φ(w)dw.

(37)(3.23)(3.24)w∈Wi ,K(w)≥ϵ√(2i)
From Lemma2.4.3, with appropriate ϵ,Zn = Op (exp(− n)) holds. The optimal parameter
set in each Wi is W0i ; therefore, in this set, the optimal probability distribution is onlyp(x|w0i ). In Assumption3.2.7, w0 corresponding to w ∈ W0i is only w0i . Thus∃c0 ∀w ∈ Wi ∀w0i EX [f (x, w0i , wi )] ≥ c0 EX [f (x, w0i , wi )2 ].

(3.25)

holds. This condition is corresponding to Assumption3.2.3. Then, applying the Lemma2.4.4,
there exists a measure du∗i such that
)
(
∫
∫
√
(log n)mi −1(log n)mi −1
∗
λi −1
(1i)
dui
t
exp(−βt + β tξni (ui ))dt + op
Zn =. (3.26)nλ i
nλ iThus, the marginal likelihood is
Zn =

m
∑

exp(−nLni )Zn(0i)(3.27)

i=1

=

m
∑
i=1

(log n)mi −1
exp(−nLni )
nλ i

(∫

du∗i

∫
t

λi −1

exp(−t +√)tξni (ui ))dt + op (1) . (3.28)

From (4), the free energy is given by
Fn = − log Zn= f1 − f1 + g1 − g1 + g2 − g2 − log Zn ,= f1 + f2 + f3 + op (1),(3.29)

whereg1 = − log
g2 = − log
f1 = − log

(m
∑
i=1
(m
∑
i=1
(m
∑

)
e

(−nLni −λi log n+(mi −1) log log n),(3.30))e(−Θ(β,ξni )−nLni −λi log n+(mi −1) log log n),(3.31))
e(−nLni )k=1,(3.32)(∑

f2 = −f1 + g1 = − log(∑
f3 = −g1 + g2 = − logm(−nLni −λi log n+(mi −1) log log n)i=1 e∑m (−n L )
ni
k=1 e

)
(3.33),m(−Θ(ξni )−nLni −λi log n+(mi −1) log log n)i=1 e∑m (−nL −λ log n+(m −1) log log n)
ni
i
i
i=1 e

)
.

(3.34)In the above equations, we have used the notation,
Θ(ξni )(∫du∗i

= − log∫ ∞dt tλi −1 exp(−t +√)tξni (ui )) .

(3.35)(3.36)

0

In the following, we examine the asymptotic behaviors of the three terms eq3.29. First, tostudy f1 , we deﬁne imax and Y byimax = argmax (−Lni ),(3.37)i

Y = −nLnimax .

(3.38)

From the deﬁnition,

log

(m
∑



)
e(−nLni )− Y = log 1 +

i=1

m
∑


e(−nLni −Y )  .

(3.39)i̸=imax

Since Y + nLni ≥ 0,
0 < log(m∑)
e(−nLni )− Y ≤ m log 2.

(3.40)

i=1

Therefore,
f1 = Y + Op (1) = −nLnimax + Op (1).
Note that the average of Lni is L(w0i ), which does not depend on i. Let us deﬁne a randomvariable,Ln (w0i ) ≡√n(−Ln (w0i ) + L(w0i )).

(3.41)

By using the central limit theorem, Ln (w0i ) (i = 1, 2, ..., m) converges in distribution to an
m-dimensional Gaussian random variable L (w0i ) on w0i ∈ W0 whose average is zero andvariance-covariance matrix V = (Vij ) is
Vij = EX [(log p(X|w0i ) + L(w0 ))(log p(X|w0j ) + L(w0 ))].

(3.42)

Then we obtain
f1 = nL(w0 ) −√n max Ln (w0 ) + Op (1).
w0 ∈W0(3.43)

Using L (w0 ), the asymptotic behavior of its average is given by√
E[f1 ] = nL(w0 ) − E[ n max L (w0 )] + O(1).
w0 ∈W0(3.44)

Now, let us examine the second term f2 in eq3.29. If there exist multiple imax , we deﬁne imax
as the i whose RLCT is smallest, and if the RLCTs are the same, we deﬁne imax as the iwhose multiplicity is biggest. Using imax so deﬁned, the asymptotic behavior of the second
term f2 is given by)(∑m(−nLni −λi log n+(mi −1) log log n)ei=1∑m (−nL )−f2 = log
(3.45)ni )i=1 e= −λimax log n + (mkmax − 1) log log n)(∑(−nLnk −Y −(∆λk ) log n+(∆mk ) log log n)
1+ mk̸=imax e∑+ log(−nLni −Y ) )1+ m
k̸=imax e()
1 + op (e−na )= −λimax log n + (mimax − 1) log log n + log
1 + op (e−na )(3.46)= −λimax log n + (mimax − 1) log log n + op (e−na ),(3.49)(3.47)(3.48)

where ∆λi and ∆mi are λi − λimax and mi − mimax respectively, and a ≥ 0 is the diﬀerence
between Y and the second biggest −nLni .
E[f2 ] = λimax log n − (mimax − 1) log log n + O(1).

(3.50)

Next, let us study the third term f3 in eq3.29. We deﬁne a random variable ai as follows.
e−nLni −λi log n+(mi −1) log log nai = ∑m (−nL −λ log n+(m −1) log log n) .
ni
i
i
i=1 e

(3.51)The sum of ai over i is 1. Using ai is 1. We can describe the third term using ai , so we candescribe the free energy as follows.
√n max Ln (w0 ) + λimax log n − (mimax − 1) log log nw0 ∈W0(m)∑− log
ai e−Θ(ξni ) + Op (1).

Fn = nL(w0 ) −i=1(3.52)(3.53)

Lastly, in order to derive the asymptotic behavior of the average E[Fn ], we show that the
average of the random variable)(m∑
f4 ≡ log
ai e−Θ(ξni )i=1

is ﬁnite. By the Cauchy-Schwarz inequality,−t + supu |ξni (ui )|2
t + supu |ξni (ui )|2 √≤ tξni (ui ) ≤2
2(3.54)

holds. In the integral range of ui ,[0, 1]d , we have
∫
∫
1
1
∗
− log dui − log dt tλi −1 exp(− t) −sup |ξni (ui )|22
2 ui ∈[0,1]d(3.55)≤ Θ(ξni (ui ))∫∫1
3
sup |ξni (ui )|2 .
≤ − log du∗i − log dt tλi −1 exp(− t) +2
2 ui ∈[0,1]d(3.56)(3.57)

Using Jensen’s inequality,
f4

≥

m
∑

ai (−Θ(ξni (u)))

i=1

≥

m
∑

(
ai

∫
log

(3.58)

du∗i + log∫

dt tλi −1 exp

i=1

t
− 3β
2

1
−
sup |ξni (ui )|2
2 ui ∈[0,1]d)(3.59)

holds. From the nature of empirical processes, ξni (ui ) converge in low to Gaussian processes
ξi (ui ) andlim E[ sup |ξni (ui )|2 ] = lim E[ sup |ξi (ui )|2 ]

n→∞

n→∞

ui ∈[0,1]d(3.60)ui ∈[0,1]d)(∑ mai e−Θ(ξni (u)) , the only term that may
holds[40]. In regard to the lower bound of log
i=1
(∑ m
)
−Θ(ξni (u)) ] can be shown to be
diverge as a random variable is ξni (ui ), so E[logi=1 ai ebounded below. In addition, because sup |ξni (ui )|2 ≥ 0 holds, we haveui ∈[0,1]d

f4

(
)(
∫
∫
)
1
1 ∑m
2
≤ log max dt tλi −1 e− 2 t du∗ime 2 i=1 supu |ξni (ui )|
i

(
)
∫
∫
m
1
1∑
= log max dt tλi −1 e− 2 t du∗i + log m +supu |ξnk (ui )|2 .
i
2(3.61)(3.62)i=1(∑ m)−Θ(ξni (u)) ] isHence, as we did for the lower bound it can be shown that E[logi=1 ai ebounded from above. By summing up the above equations, the asymptotic behavior of E[Fn ]can be described as
E[Fn ] = nL(w0 ) −
+

m
∑

√

nE[ max L (w0 )]w0 ∈W0αi (λi log n − (mi − 1) log log n) + O(1),(3.63)

k=1

where αi is the probability that i = imax . By putting λ̂ =we obtain
E[Fn ] = nL(w0 ) −√∑m

k=1 αi λi and m̂ =nE[ max L (w0 )] + λ̂ log n − (m̂ − 1) log log n + O(1)w0 ∈W0∑m

k=1 αi mi ,(3.64)

holds.
Using Theorem2.3.1, and assuming that E[Gn (1)] has an asymptotic expansion, we ﬁnd that()
1
1
E[Gn ] = L(w0 ) − √ E[ max L (w0 )] + o √.
(3.65)
2 n w0 ∈W0nFrom this Theorem3.3.2 is proved.

3.5

Experiment

In this section, we show the results of an experiment for the case when the optimal
probability distribution is not unique.

3.5.1

Experiment1

We set the true distribution as)(
1(y − f (x))2.
q(y|x) = √
exp −
0.08
0.08π


(−2 ≤ x < −1)x + 2f (x) = 1(−1 ≤ x < 1)
−x + 2 (1 ≤ x ≤ 2){1(−2 ≤ x ≤ 2)
q(x) = 40 otherwise.
We use following statistical model and prior distributions.
)((y − σ(ax + b))2
1
.
exp −
p(y|x, a, b) = √
0.08
0.08
1
σ(x) =.
1 + exp(−x)
a ∼ Unifrom(0, 20)b ∼ Unifrom(−20, 20)

Figure 3.1: Experimental value and theoretical free energy depending on the sample size. The
error bar is the SE of the average of free energy. The theoretical value of log likelihood(nL(w0 ))is subtracted from each value.
This statistical model has two optimal parameters
w01 = (5.13, 7.71), w02 = (−5.13, 7.71).
At these points,p(x|w01 ) ̸= p(x|w02 )holds. In this case, eq3.64 gives the theoretical asymptotic behavior of the free energy versus
inverse temperature for β = 1. Note that the KL-divergence between q(y, x) and p(y, x|a, b)
in each neighborhood of the optimal parameter is regular, so λ = 1 and m = 1. The expectation of the maximum value of a 2-dimensional Gaussian distribution follows a 1-dimensional
Gaussian distribution (see the appendix). We will show that the theoretical behavior of freeenergy obeys√√
V[log(p(y|x, a0 , b0 )) − log(p(y|x, −a0 , b0 ))]E[Fn (1)] = nL(w0 ) − n+ log n + O(1).
2π(3.66)√
In eq3.66, L(w0 ) and the coeﬃcient of n can be calculated by numerical integration. We
used the average of Fn calculated from the true distribution q(y|x), q(x) as the experimental
value of E[Fn ]. The prior distribution p(a), p(b) does not have an eﬀect on the asymptotic
behavior. For this reason, we used equally spaced ﬁxed values for integration. We comparedthis experimental values and theoretical values, except for the O(1) term.
We calculated the experimental values of E[Fn ] whose sample size were n = 100 to 600
every 100. For making variance the same size, we repeat to calculate Fn from 10000 times
to 60000 times in steps of 10000 for each sample sizes. This means E[Fn ] of n = 100 isaverage of 10000 samples of F1 00. Figure1 compares the experimental and theoretical values.
The experimental behavior of the free energy depending on the sample size is similar to
the theoretical behavior. Figure2 shows the diﬀerence between the theoretical value andexperimental value. This diﬀerence corresponds to O(1) term. This diﬀerence is remains on
this order regardless of the sample size. The experimental results support the theoreticalformula, Theorem3.3.2.

Figure 3.2: Diﬀerence between experimental value and theoretical value in Figure1. The error
bar is the same as in Figure1.

3.5.2

Experiment2

We experimentally show that if optimal probability distribution is not unique, the generalization loss gets larger as the number of data becomes larger. We set the true distribution
as 2-dimensional 4-component Gaussian mixture whereq(x) =()
11· √ exp − ||x − b̄(i) ||2
4
2
2π
i=1

4
∑
1

b̄(1) , b̄(2) , b̄(3) , b̄(4) = (1, 1)T , (1, −1)T , (−1, 1)T , (−1, −1)T .

We use 2-dimensional 2-component Gaussian mixture model and prior distribution where
(
)
(
)
1
1−a
1
a
(1) 2
(2) 2
exp − ||x − b ||p(x|a, b , b ) = √ exp − ||x − b || + √
2
2
2π
2π
â ∼ Normal(0, 4)
1a=.
1 + exp(−â)
(1)

(2)

(i)

bj , ∼ Unifrom(−10, 10) (i = 1, 2 j = 1, 2).
In this statistical model, mixture weight a is reparameterized for Markov chain Monte Carlo
method(MCMC). The average of each Gaussian in true distribution b̄(1) , b̄(2) , b̄(3) , b̄(4) is rotationally symmetric in π/2 for coordinate system of x. We set the optimal parameter ofstatistical model as (â, b̂1 , b̂2 ). We also set the R(θ) as the rotation matrix where[]cos θ − sin θR(θ) =.
sin θ cos θ(3.67)

By using the symmetric property of q(x) and the property of R(π/2) the following equation
holds.
∫∫(1) (2)q(x) log p(x|â, b̂ , b̂ )dx = q(R(−π/2)R(π/2)x) log p(R(π/2)x|â, R(π/2)b̂(1) , R(π/2)b̂(2) )dx∫= q(R(−π/2)x′ ) log p(x′ |â, R(π/2)b̂(1) , R(π/2)b̂(2) )|R(−π/2)|dx′∫= q(x′ ) log p(x′ |â, R(π/2)b̂(1) , R(π/2)b̂(2) )dx′ (x′ = R(π/2)x)(3.68)
From eq3.68, (â, R(π/2)b̂(1) , R(π/2)b̂(2) ) is also optimal parameter. Moreover,p(x|â, b̂(1) , b̂(2) ) ̸= p(x|â, R(π/2)b̂(1) , R(π/2)b̂(2) )(3.69)

holds in general, therefore optimal probability distribution is not unique.
We experimentally calculate the generalization lossE[Gn ] in this case. From n-dataX n
generating from q(x), we calculate posterior distribution p(a, b(1) , b(2) |X n ) and predictive
distribution p(x|X n ). In this experiment we use the No-U-Turn Sampler(NUTS)[17] to realize
the posterior distribution. We calculate the experimental value of Gn by the average oflog p(x|X n ) of 10000 test sample. We estimate E[Gn ] by the average of 100 times of thiscalculation.
Table 3.1: Experimental value of E[Gn ]sample size
100
200
300
400
500

E[Gn ]
2.568642.58427
2.581112.58471
2.58640

Standard error
0.010910.009910.010130.01082
0.01105

Table1 shows the results of the experiment. E[Gn ] increases by the sample size increase.
There is a reversal of change n = 200, that may be caused by the eﬀect of lower order term√in o(1/ n). This result supports the main result.

3.6

Discussion

We found that if the optimal probability distribution is not unique, the apparent bias or
the variance gets smaller for a ﬁnite sample number n corresponding to a Gaussian process
determined by the log loss of the optimal parameter set, and the reduction converges to 0
asymptotically. This behavior can be explained qualitatively as follows: when there are two
or more optimal probability distributions, the posterior distribution can be selected to be thenearest optimal probability distribution by bias of data, and this makes the generalization loss
smaller than the average generation of data. As the number of samples and the bias increase,data generates averagely and generalization loss gets larger.

Both experiment1 and experiment2 are the case while the true distribution is symmetric,
the statistical model is smaller than suﬃciently realizing them. In real data analysis, if thegeneralization error gets larger by the sample size larger, it can be detected that the statisticalmodel is smaller than appropriate size or the model does not ﬁt to the unknown symmetricproperty of true distribution. In particular, such situations may occur depending on therelationship between dimensions in high dimensional data.
The behavior of generalization error shown in this paper is near to phenomena in Deep
Double Descent[29]. The section ”Sample-wise non-monotonicity” in [29] shows the behavior
generalization error gets larger with increasing sample size. The test error shown in theexperiment of [29] does not converge to 0 by increasing the sample size. Therefore, Deep
Double Descent of sample-wise case is also the case that bias does not equal to 0. Although therelation between Deep Double Descent and the term discovered in this paper is unclear in therange of this paper, to develop the study of multiple optimal probability distribution case inparticular the unﬁtness between true and statistical model in high dimensional distributions, it
is possible to contribute clarifying the irregular behavior of neural network. The speciﬁcationof unﬁtness between true and statistical model can help the eﬃcient expansion of statisticalmodel to estimate true distribution with lower bias and generalization error.
In this paper, we showed that the asymptotic behavior of the free energy and generalization
1
1
loss are determined by n 2 and n− 2 order. Previous research[45] provides concrete example
in which there is a unique optimal probability distribution but Assumption3.2.3 does nothold. In that paper, the asymptotic behavior of the free energy and the generalization loss
1
2
are determined by n 3 and n− 3 order, so we predict that the lowest order determining the1
1
asymptotic behavior of the free energy and generalization loss are n 2 and n− 2 .
We showed that the asymptotic behaviors of the free energy and generalization loss are
determined by the maximum value of a Gaussian process. The probability distribution of
the maximum value of a Gaussian process or multivariate normal distribution can not becalculated analytically, but an approximate calculation, called the “tube method”[22] exists.
There is also a method for calculating the upper and lower bounds of the expectation of the
maximum value of a Gaussian process, called “chaining”[39]. This maximum value is what
determines the free energy and generalization loss in this paper.

3.7

Conclusion

We examined the case of when an important assumption in singular learning theory about
the log density ration function is loosened. In this case there is a new term that is determinedby a Gaussian process, whereby the generalization loss asymptotically increases as the sizeof the dataset increases. In the future, we should examine the asymptotic behavior of thegeneralization loss as a random variable, in particular the asymptotic equivalence of WAIC [44]and WBIC [46] in this case, and in the case in which the assumption is completely removed.

Appendix 3.A

Maximum value of 2-dimensional Gaussian

We will derive the following equation.
√
V[log(p(x|w01 )) − log(p(x|w02 ))]E[maxL (w0 )] =
2π

(A1)

In this equation, L (w0 ) is a 2-dimensional Gaussian distribution which average is 0 andvariance-covariance matrix is
Vij = E[(log p(x|w0i ) + L(w0 ))(log p(x|w0j ) + L(w0 ))](A2)

We deﬁne two random variables as
z1 = L (w01 ) − L (w02 ).
z2 = L (w01 ) + L (w02 )The random variables (z1 , z2 ) are also from 2-dimensional Gaussian distribution whose averageis 0 and variance-covariance matrix is
(
)(
)(
)
1 −1
V11 V121 1
1 1
V21 V22−1 1()
V11 + V22 − V12 − V21 V11 − V22 + V12 − V21=.
V11 − V22 − V12 + V21 V11 + V22 + V12 + V21
The marginal distribution about z1 is a 1-dimensional Gaussian distribution whose average is
0 and the variance is
V11 + V22 − V12 − V21 .
According to eq.(16) and eq.(34), we have
V11 + V22 − V12 − V21= E[(log p(x|w01 )(log p(x|w01 )] − L(w0 )2 + E[(log p(x|w02 )(log p(x|w02 )]− L(w0 )2 − 2(E[(log p(x|w01 )(log p(x|w02 )] − L(w0 )2 )= E[(log p(x|w01 ) − log p(x|w02 ))2 ]= V[(log p(x|w01 ) − log p(x|w02 ))]We deﬁne a random variable z3

{
z3 =

z1
0

(A3)

z1 ≥ 0z1 < 0

By using z3 we can describe the maximum value of L (w0 ) in the following way,
maxL (w0 ) = L (w02 ) + z3 .

(A4)Considering the average of L (w02 ) is 0, we ﬁnd thatE[maxL (w0 )] = E[z3 ].

(A5)

E[z3 ] is the expectation of a positive value in a Gaussian distribution. This integration of a
Gaussian whose variance is σ 2 can be calculated as
) [
)]∞
(
(
∫ ∞
σ
x2
x2
x
√ exp − 2 = − √ exp − 2
2σ
2σ
2π
2π
0
0
σ
(A6)
=√ .
2π

From (A3), (A5), and(A6), we have√E[maxL (w0 )] = E[z3 ] =Therefore, (A1) holds.

V[log(p(x|w01 )) − log(p(x|w02 ))].
2π

Chapter 4

Free Energy of BayesianConvolutional Neural Networks
4.1

Motivation

Convolutional Neural Networks (CNNs) are a type of Neural Networks mainly used forcomputer vision. CNNs have been shown high performance with deep layers[38, 21]. Residual
Network(ResNet)[15] adopted the skip connection for addressing the problem that the loss
function of CNN with deep layers does not decrease well through optimization. After successof ResNet, the CNNs with more than 100 layers are realized. The high performance of ResNet
has been explained by similarity to the ensemble learning [18, 31, 10]. On the other hand,
there is a common issue in neural networks that the reason why the overparametrized deepneural network generalized has been unknown yet.
In conventional learning theory, if the Fisher information matrix of a learning machine ispositive deﬁnite, and the data size is suﬃcient large, the generalization error of the learning
machine is determined from the number of its parameter in maximum likelihood estimator[1].
The similar property is shown in free energy and generalization error in Bayesian learning[35,32, 2]. From these characteristics of generalization error and free energy some information
criteria such as AIC, BIC and MDL are proposed. However, most of the hierarchical models
such as neural networks have degenerated Fisher information matrix. In such models, theBayesian generalization error and free energy are determined by a rational number called RealLog Canonical Threshold(RLCT) and that is smaller than the number of parameters [42, 43].
In particular, RLCTs are revealed in some concrete models such as three layered neuralnetworks[41, 4], normal mixtures [12, 51], Poisson mixtures[34], Boltzmann machine[52, 3],
reduced rank regression[5], Latent Dirichlet allocation[14], matrix factorization, and Bayesian
Network[53]. While RLCTs of many hierarchical models are revealed, that of neural networks
with multiple layer of nonlinear transformation has not been clariﬁed. Yet the possibility ofthat is shown in [48], the RLCT of Deep Neural Network is revealed[27]. On the other hand
the RLCT of neural networks other than DNN was not explored.
In Bayesian learning for neural networks, how to realize the posterior is important. Thereexist approaches for generating posterior, Variational Approximation and Markov chain Monte
Carlo(MCMC) methods. For Variational Approximation methods for neural networks, Variational Autoencoder[20] and Monte Carlo dropout[9] are practically used. Also for CNNs,
31

variational approach for Bayesian inference was proposed [8]. MCMC for neural networks,
Hamiltonian Monte Carlo and Langevin Dynamics are useful for sampling from posterior.
Stochastic Gradient Langevin Dynamics(SGLD)[49] which is a method applying Stochastic
Gradient Descent instead of Gradient Descent to Langevin Dynamics is popular MCMC for
Bayesian Neural Networks. [54] used SGLD for generating posterior of CNNs.
In this chapter we clarify the free energy and generalization error of Bayesian CNNs
with and without skip connection. In both case the free energy and generalization error
don’t depend on the number of parameters in redundant ﬁlters. Then, in case without skip
connection, the redundant layers aﬀect the free energy and generalization error whereas they
don’t aﬀect in case with skip connection. This chapter consists of seven main sections and one
appendix. In section4.2, we describe the setting of Convolutional Neural Network analyzed inthis chapter. In section4.3, we show the main theorem of this paper. In section4.4, we provethe main theorem of this chapter. In section4.5, we conduct the experiment of synthetic
data. In section4.6 and section4.7, we discuss the theorem in this chapter and conclusion. In
appendix4.A we explain former study about fully connected deep neural network case.

4.2

Convolutional Neural Network

In this section we describe the function of Convolutional Neural Network. First, we explain
CNN without skip connection. The kernel size is 3 × 3 with zero padding and 1-stride.
The activation function is ReLU. The numbers of the layers of the CNN are K1 (≥ 3) for
Convolutional Layers and K2 (≥ 3) for Fully Connected Layers.
Let x ∈ RL1 ×L2 ×H1 be an input vector generated from q(x) with bounded support andy ∈ {0, 1}HK1 +K2 be an output vector with q(y|x). We deﬁne w(k) ∈ R3×3×Hk−1 ×Hk , b(k) ∈ RHk
as weight and bias parameters in each Convolutional Layer (2 ≤ k ≤ K1 ). f (k) ∈ RL1 ×L2 ×Hk
is output of each layer for 1 ≤ k ≤ K1 . Conv(f, w) is the convolution operation with zero
padding and 1-stride:

Conv(fk−1∑∑ p=3,q=3k, w )l1 ,l2 ,hk =fl1 +p−1,l2 +p−1,hk−1 wp,q,hk−1 ,hk .

(4.1)

hk−1 p=1,q=1

We deﬁne g(bk ) : RHk → RL1 ×L2 ×Hk asg(b(k) )l1 ,l2 = b(k)(4.2)

for 1 ≤ l1 ≤ L1 , 1 ≤ l2 ≤ L2 . By using w(k) , g(b(k) ), and f (k−1) , f (k) is described byf (k) (w, b, x) = σ(Conv(f (k−1) (w, b, x), w(k) ) + g(b(k) ))(4.3)

where w, b are the set of all weight and bias parameters. σ() is a function that applies the
ReLU to all the elements of the input tensor.
The output of k = K1 + 1 layer is result of Global Average Pooling on k = K1 layer:
1 1∑1 2∑2 (K1 )f(w, b, x)l1 ,l2 .
L1 L2
l =L l =L

f (K1 +1) (w, b, x) =l1 =1 l2 =1(4.4)

Let w(k) ∈ RHk ×RHk−1 , b(k) ∈ RHk be weight and bias parameters in each Fully Connected
Layer (K1 + 2 ≤ k ≤ K1 + K2 ). For K1 + 2 ≤ k ≤ K1 + K2 − 1, f (k) is deﬁned byf (k) (w, b, x) = σ(w(k) f (k−1) (w, b, x) + b(k) ),(4.5)

f (K1 +K2 ) (w, b, x) = softmax(w(k) f (k−1) (w, b, x) + b(k) ),(4.6)

and for k = K1 + K2 ,where softmax() is a softmax function
e zi
softmax(z)i = ∑J

j=1 e

zj

.

(4.7)

The output of the model is represented stochasticallyy ∼ Categorical(f (K1 +K2 ) (w, b, x))(4.8)

where Categorical() is a categorical distribution.
Then we describe CNN with skip connection. The number of layers within the skip
connection is Ks and the number of skip connection is M . The output of the layer withskipped connection is described byf (mKs +2) (w, b, x) = σ(Conv(f (mKs +1) (w, b, x), w(mKs +2) )+ B (mKs +2) + f ((m−1)Ks +2) (w, b, x)).

(4.9)

In this case, CNN satisﬁes the following conditions
K1 = M K s + 2
HmKs +2 = const(1 ≤ m ≤ M ).

(4.10)The other conditions are the same as the case without skip connection.

Without Skip Connection

With Skip Connection

Figure 4.1: The structure of Convolutional Neural Network with and without Skip Connection
Figure4.1 shows the conﬁguration of Convolutional Neural Network analyzed in this paper.

4.3

Main Theorem

In this subsection, the main result of this paper is introduced. We assume that the model
CNN has enough complexity to approximate the data generating process. In such situation,
the data generating process is described by a CNN which is smaller than model. We deﬁne thedata generating network. Both in case with and without skip connection, the data generating
network satisﬁes the following conditions about the number of layers and ﬁlters,∗
K1∗ ≤ K1 , K2∗ ≤ K2 , H1∗ = H1 , HK
= HK1 +K2
1

and

 ∗

HK1∗
∗
Hk ≥ HK+K ∗ ∗1 2Hk(4.11)(K1∗ + 1 ≤ k ≤ K1 )(K1 + K2∗ + 1 ≤ k ≤ K1 + K2 − 1)(others)

where K1∗ , K2∗ , Hk∗

are the corresponding sizes of K1 , K2 , Hk on data generating network.
These conditions indicate that the data generating network has smaller number of layers andwidth on each layer. Then, we show the main theorem.

Theorem 4.3.1. (No Skip connection) Assume that the learning machine and the data
generating distribution are given by p(y|x, w, b) and q(y|x) = p(y|x, w∗ , b∗ ) in case without skip connection which satisfy the conditions (4.11) and (4.12), and that a training data{(Xi , Yi ) i = 1, 2, ..., n} is independently taken from q(x)q(y|x). Then the average free energy
satisﬁes the inequality,
E[Fn ] ≤ nS + λCN N log n + C(4.12)

where

)
1( ∗
∗
∗
|w |0 + |b∗ |0 + (K1 − K1∗ )(9HK(4.13)∗ + 1)HK ∗112
where |w∗ |0 , |b∗ |0 are the numbers of parameters of weights and biases in data generating
network.
λCN N =

Theorem 4.3.2. (Skip connection) Assume that the learning machine and the data generating
distribution are given by p(y|x, w, b) and q(y|x) = p(y|x, w∗ , b∗ ) in case with skip connection
which satisfy the conditions (4.10), (4.11) and (4.12), and that a training data {(Xi , Yi ) i =1, 2, ..., n} is independently taken from q(x)q(y|x). Then1λCN N = (|w∗ |0 + |b∗ |0 ).
(4.14)
2
If there exists asymptotic expansion of the generalization error E[Gn ] in theorem4.3.1 andtheorem4.3.2, that satisﬁes the following inequality
( )
λCN N
1
E[Gn ] ≤,(4.15)
+o
n
n
where
∫
Gn =

HK1 +K2q(x)∑(K ∗ +K ∗ )fi 1 2 (w∗ , b∗ , x) log

i=1

which corresponds to categorical cross entropy.

(K1∗ +K2∗ )fi(w∗ , b∗ , x)

(K1 +K2 )Ew,b [fi(w, b, x)]dx(4.16)

4.4

Proof of main theorem

In this section, we show the proof of main theorem.

4.4.1

Inequalities

Note that we describe the Frobenius norm of any order of tensor as ∥ · · · ∥. We denote
the Kullback-Leibler divergence of a data-generating distribution q(y|x) = p(y|x, w∗ , b∗ ) anda model p(y|x) that∫q(y|x)
K(w, b) = q(x)q(y|x) log
dxdy.
(4.17)
p(y|x, w, b)
Lemma 4.4.1. [27] Assume that a set W is contained in the set determined by the priordistribution {(w, b); φ(w, b) > 0}. Then for an arbitrary positive integer n,∫(4.18)
E[Fn ] ≤ nS − log
exp(−nK(w, b))φ(w, b)dwdb.
W

Lemma 4.4.2. [27] For arbitrary vectors s, t,∥σ(s) − σ(t)∥ ≤ ∥s − t∥.

(4.19)

Lemma 4.4.3. [27] For arbitrary w,w′ , b, b′ , and K1 + 1 ≤ k ≤ K1 + K2 , the following
inequality holds,∥f (k) (w, b, x) − f (k) (w′ , b′ , x)∥≤ ∥w(k) − w′(k) ∥∥f (k−1) (w, b, x)∥ + ∥b(k) − b′(k) ∥+ ∥w(k) ∥∥f (k−1) (w, b, x) − f (k−1) (w′ , b′ , x)∥.

(4.20)

Corollary 4.4.4. For arbitrary w,w′ , b, b′ , and 1 ≤ k ≤ K1 , the following inequality holds,∥f (k) (w, b, x) − f (k) (w′ , b′ , x)∥≤ 9∥w(k) − w′(k) ∥∥f (k−1) (w, b, x)∥ + L1 L2 ∥b(k) − b′(k) ∥+ 9∥w(k) ∥∥f (k−1) (w, b, x) − f (k−1) (w′ , b′ , x)∥+ δ (k) ∥w(k) ∥∥f (k−K2 −1) (w, b, x) − f (k−K2 −1) (w′ , b′ , x)∥.

(4.21)

where δ (k) equals to 1 if the network has Skip connection and k = mK2 + 2, otherwise it equals
to 0
Proof.
f (k) (w, b, x) − f (k) (w′ , b′ , x)
= σ(Conv(f (k−1) (w, b, x), w(k) ) + g(b(k) )) − σ(Conv(f (k−1) (w, b, x), w′(k) )) + g(b′(k) ))+ σ(Conv(f (k−1) (w, b, x), w′(k) ) + g(b′(k) )) − σ(Conv(f (k−1) (w′ , b′ , x), w′(k) ) + g(b′(k) )).
(4.22)

From deﬁnition of Conv(), the following equation holds.
∥Conv(f(k−1)(w, b, x), w(k))j ∥ ≤k−1
H∑∥f (k−1) (w, b, x)i ∥|((k)

wpqij )|1

p=1 q=1

i=1

≤ 9∥f

3 ∑
3
∑

(k−1)

(w, b, x)∥∥w:,:,:,j ∥(4.23)

By using lemma4.A.2, (4.22) and (4.23), corollary4.4.4 is proved.
Lemma 4.4.5. For arbitrary w, b, x,∥f (k) (w, b, x)∥ ≤ Dk ∥w(k) ∥∥w(k−1) ∥ · · · ∥w(2) ∥∥x∥
+ D0 ∥b(k) ∥ +k−2∑(4.24)

Dj ∥w(k) ∥∥w(k−1) ∥ · · · ∥w(k−j) ∥∥b(k−j) ∥.

j=1

where Dj , 0 ≤ j ≤ k is constant.
Proof. By considering the case all the parameters of w′ and b′ are 0, in Lemma 4.A.3, it
follows that∥f (k) (w, b, x)∥ ≤ 9∥w(k) ∥∥f (k−1) (w, b, x)∥ + L1 L2 ∥b(k) ∥
+δ

(k)

∥w

(k)

∥∥f

(k−K2 −1)

(w, b, x) − f(k−K2 −1)

(4.25)′′(w , b , x)∥.

Then mathematical induction gives the Lemma.

4.4.2

Notations of parameters

In order to prove the main theorem, we need several notations. We divide the ﬁltersof learning model in each convolutional layer 1 ≤ hk ≤ Hk into the 1 ≤ hk ≤ Hk∗ andHk∗ + 1 ≤ hk ≤ Hk . The former is denoted as A and the latter is denoted as B. The(k)convergent tensor E (k) ∈ R3×3×Hk−1 ×Hk and vector E0 ∈ RHk where the absolute values of√all elements are smaller than 1/ n are denoted by
( (k)
)
(k)
E
E
(k)
pqAA
pqAB
Epq
=
(4.26), (1 ≤ p ≤ 3, 1 ≤ q ≤ 3),(k)(k)
EpqBA EpqBB
(
)
(k)
EA0
(k)
E0 =
(4.27)
.
(k)
EB0
(k)

The positive constant tensor M(k) and vector M0 are deﬁned by the condition that allelements are in the interval [A, B],
(
)
(k)
(k)
M
M
pqAA
pqAB
(4.28)M(k), (1 ≤ p ≤ 3, 1 ≤ q ≤ 3),
pq =
(k)
(k)
MpqBA MpqBB
(
)
(k)
MA0
(k)
M0 =
.
(4.29)(k)
MB0
To prove Theorem 4.3.1 4.3.2, we show an upper bound of E[Fn ] is given by choosing a set
WE which consists of essential weight and bias parameters in convolutional layers and fully
connected layers.

4.4.3

No Skip Connection Case

Deﬁnition. (Essential parameter set WE without Skip Connection). A parameter (w, b) is
said to be in an essential parameter set WE if it satisﬁes the following conditions (1),(2) for2 ≤ k ≤ K1 ,(1) For 2 ≤ k ≤ K1∗
(
(k)
wpq
=

(w∗ )(k) + EpqAA
(k)

(k)

(
b(k) =−MpqBA(b∗ )(k) + EA0
(k)
−MB0

(k)

)

(k)

MpqAB

,

(k)

−MpqBB)(4.30),(4.31)

for 1 ≤ p ≤ 3, 1 ≤ q ≤ 3(2) For K1∗ + 1 ≤ k ≤ K1
(
(k)
wpq
=

(k)

(k)

ZpqAA

MpqAB

(k)

)

(k)

−MpqBA −MpqBB()(k)(b∗ )(k) + EA0
(k)
b =
,
(k)
−MB0

where

{
(k)
ZpqAA =∗ (k)where I22AA ∈ R(H )∗ (k)× R(H ),

(4.32)(4.33)(k)

I22AA + E22AA (p = q = 2)
.
(k)
EpqAA
(others)(4.34)

is an identity matrix.

Lemma 4.4.6. Assume that the weight and bias parameters of convolutional layers are in
the essential set WE in case without Skip Connection. Then there exist constants c1 , c2 > 0such thatc1∗(K )∥f:,:,A1 (w, b, x) − f (K1 ) (w∗ , b∗ , x)∥ ≤ √ (∥x∥ + 1),n(4.35)(K )∥f:,:,A1 (w, b, x)∥ ≤ c2 (∥x∥ + 1).

(4.36)

Proof. Eq.(4.79) is derived from Lemma 4.A.4. By the deﬁnitions (4.71), (4.72), for 2 ≤ k ≤K∗
fA (w, b, x) = σ(Conv(f:,:,A (w, b, x), (w∗ )(2) + E:,:,AA ) + g((b∗ )(2) + EA0 )),
(2)

(1)

(2)

(2)

(4.37)fA (w, b, x) = σ(Conv(f:,:,A (w, b, x), (w∗ )(k) + E:,:,AA )
(k)

(k−1)

(k)

+ Conv(f:,:,B (w, b, x), M:,:,AB ) + g((b∗ )(k) + EA0 )).
(k−1)

(k)

(k)

(k)

(k)

(4.38)(2)

In k = 2, |x| is bounded and M:,:,AB is a constant tensor, MB0 is large suﬃciently, f:,:,B (w, b, x) =0 because all the elements of the output of ReLU function f (2) (w, b, x) are nonnegative. For

(k)

(k)

(k)

(k)

3 ≤ k ≤ K1 , f:,:,B (w, b, x) = 0, since all elements of w:,:,BA , w:,:,BB , and wB0 are negative.
Hence, by Lemma 4.A.3, for 2 ≤ k ≤ K1∗ ,∥f:,:,A (w, b, x) − f (k) (w∗ , b∗ , x)∥
(k)

(k)

(k)

≤ 9∥E:,:,AA ∥∥f (k−1) (w, b, x)∥ + L1 L2 ∥EA0 ∥+ 9∥(w∗ )(k) ∥∥f:,:,A (w, b, x) − f (k−1) (w∗ , b∗ , x)∥.
(k−1)

(4.39)∗

and for K1∗ + 1 ≤ k ≤ K1 , by using f (K1 ) (w∗ , b∗ , x) as f (k) (w∗ , b∗ , x),∗∥f:,:,A (w, b, x) − f (K1 ) (w∗ , b∗ , x)∥
(k)

(k)

(k)

≤ 9∥E:,:,AA ∥∥f (k−1) (w, b, x)∥ + L1 L2 ∥EA0 ∥∗+ 9∥(w∗ )(k) ∥∥f:,:,A (w, b, x) − f (K1 ) (w∗ , b∗ , x)∥.
(k−1)

(4.40)√(k−1)(k)
The elements of tensors and vectors in E:,:,AA and E:,:,A0 are bounded by 1/ n order
√
(k−1)
(k)
term, hence ∥EAA ∥ and ∥EA0 ∥ are bounded by 1/ n order term. Moreover, ∥(w∗ )(k) ∥ is(k−1)
a constant term. For k = 2, f:,:,A (w, b, x) − f (k−1) (w∗ , b∗ , x) = x − x = 0. Then, by using√
mathematical induction for (4.39) and (4.40) , the all terms can be bounded by 1/ n terms,
hence we obtained the Lemma.
From [27], because of the output in k = K1 + 1 is nonnegative there exist the essential
parameters for fully connected layers such that the number of the convergent parameters E
equals to that of data generating network. From these lemmas, the main theorem can beproved.
(Proof of Theorem 4.3.1). By Lemma 4.A.1, it is suﬃcient to prove that there exists a
constant C > 0 such that∫C
exp(−nK(w, b))φ(w, b)dwdb ≥ λ(4.41)n
WE
From the property of KL-divergence, there exists the positive constant c4∫
c4∗∗∥f (K1 +K2 ) (w, b, x) − f (K1 +K2 ) (w∗ , b∗ , x)∥2 q(x)dx.
K(w, b) ≤2(4.42)

By using Lemma 4.A.5, if (w, b) ∈ WE ,c4 c23K(w, b) ≤2n
It follows that∫(∥x∥ + 1)2 q(x)dx =

c5
< ∞.
n

(4.43)∫exp(−nK(w, b))φ(w, b)dwdb()≥ exp(−c5 )min φ(w, b) Vol(WE ).
WE(w,b)∈WE(4.44)

where c5 > 0, min(w,b)∈WE φ(w, b) > 0, and Vol(WE ) is the volume of the set WE by theLebesgue measure. The convergent scale of Vol(WE ) is determined from the number of convergent parameter E in WE . Then,
Vol(WE ) ≥

C1
,
nλ

(4.45)

where


k=K
k=K
1 +K2
∑
1  ∑1∗∗λ=(9Hk−1+ 1)Hk∗ +(9Hk−1+ 1)Hk∗ 2
k=2
k=K1 +1
)
1( ∗
∗
∗
=
|w |0 + |b∗ |0 + (K1 − K1∗ )(9HK.
∗ + 1)HK ∗1
12(4.46)

We obtained theorem4.3.1.

4.4.4

Skip Connection Case

Deﬁnition. (Essential parameter set WE with Skip Connection). An essential parameter set
WE with Skip Connection satisﬁes the following conditions (1),(2) for 2 ≤ k ≤ K1 ,(1) For 2 ≤ k ≤ K1∗ , the same conditions as (4.71) and (4.72).
(2) For K1∗ + 1 ≤ k ≤ K1
(
)
(k)
(k)
−M
−M
(k)
pqAA
pqAB
wpq
=
(4.47),(k)(k)−MpqBA −MpqBB
(
)
(k)
−M
A0
b(k) =(4.48),(k)−MB0
Lemma 4.4.7. Assume that the weight and bias parameters of convolutional layers are in
the essential set WE in case with Skip Connection. Then there exist constants c1 , c2 > 0 such
that
c1
∗
(K )
∥f:,:,A1 (w, b, x) − f (K1 ) (w∗ , b∗ , x)∥ ≤ √ (∥x∥ + 1),n(K )∥f:,:,A1 (w, b, x)∥ ≤ c2 (∥x∥ + 1).

(4.49)(4.50)

Proof. Because of similar reason to lemma4.A.5, holds. By Lemma 4.A.3, for k = mKs + 1,
∥f:,:,A (w, b, x) − f (k) (w∗ , b∗ , x)∥
(k)

(k)

(k)

≤ 9∥E:,:,AA ∥∥f (k−1) (w, b, x)∥ + L1 L2 ∥EA0 ∥+ 9∥(w∗ )(k) ∥∥f:,:,A (w, b, x) − f (k−1) (w∗ , b∗ , x)∥(k−1)+ ∥w(k) ∥∥f (k−K2 −1) (w, b, x) − f (k−K2 −1) (w′ , b′ , x)∥.

(4.51)

If k ̸= mKs + 1 and 2 ≤ k ≤ K1∗ , inequality (4.39) holds. Same as the lemma4.A.5, from√∗(K ∗ )
mathematical induction, ∥f:,:,A1 (w, b, x) − f (K1 ) (w∗ , b∗ , x)∥ is bounded by 1/ n terms. For2 ≤ k ≤ K1 , f:,:,B (w, b, x) = 0 same reason as lemma4.A.5. For K1∗ + 1 ≤ k ≤ K1 , since all
elements of w(k) and b(k) are negative, the following equations are given.
{ (K ∗ )f 1 (w, b, x) (k = nKs + 1)(k)f:,:,A (w, b, x) =.
(4.52)
0(others)(k)

Hence, we obtained the Lemma.
Same as without Skip connection case, by using the result of [27] for fully connected layer
and inequality(4.44),(4.45), we obtained theorem4.3.2.

4.5

Experiment

In this section, we show the result of experiment of synthetic data.

4.5.1

Methods

We prepared 2-class labeled simple data shown in ﬁg4.2. The data is x ∈ R4×4 and thevalues of each element are in (−1, 1). The average of each element is 0.5 or −0.5 and added
the truncated normal distribution noise within the interval(−0.5, 0.5). The probability of eachlabel of data is 0.5. We trained CNN whose number of convolutional layer K1 = 2 and fully
connected layers K2 = 2 with SGD. The number of ﬁlter is H2 = 2 and the parameters are L2
regularized. We use the trained CNN named ”true model” as a data generating distribution.
Note that the label of original data ﬁg4.2 is deterministic, but the label of true model isprobabilistic. We prepare three learning CNN models. Each number of convolutional layers
is K1 = 2, 3, 4. Each model has skip connection every one layers or does not have skip
connection. The number of ﬁlters in each layer is Hk = 4. They have K2 = 2 fully connected
layers. The prior distribution is the Gaussian distribution which covariance matrix is 10 4 I
for weight parameter and 102 I for bias parameter. We trained the learning CNN models byusing the Langevin dynamics. The learning rate is 10−2 and the interval of sampling is 100.
We use the average of 1000 samples of learning CNN models as the average of posterior. We
estimated the generalization error by the test error of 10000 test data from true model andtrained each learning model 10 times and estimated the E[Gn ] from the average of test error.

Figure 4.2: The average of input x of each label

4.5.2

Result of experiments

Table 4.1: Experimental value and theoretical upper bound of the generalization error
model
K1 = 2
K1 = 3 no skip
K1 = 4 no skip
K1 = 3 with skip
K1 = 4 with skip

n× Test Error
16.0(1.9)
10.0(0.9)
58.4(2.3)
11.4(2.3)
15.6(1.2)

λCNN
13
32
51
13
13

dmodel /2
25
99
173
99
173

Table4.1 shows the result of the experiment. Test Error shows n times of the average of
10 test error in each model and the standard error of them. dmodel is a number of parametersof each model. All the CNN models include the true model, hence the bias is 0. Then fromequation(4.15), theoretical upper bound of the generalization error is λCNN /n. In table4.1,
the experimental values of all models are smaller than dmodel /2. Moreover, in case with skip
connection, the experimental value did not so increase as the increase of the number of layer.
Then, in case K1 = 4 without skip connection, the experimental value increased from the case
K1 = 2. In case K1 = 3 without skip connection, the experimental value is smaller than thatof K1 = 2. Behavior of MCMC is considered to be the cause of this result. Since MCMC inhigh dimensional model needs the long series for convergence in general, the result is deviatedfrom theoretical predict.

4.6

Discussion

4.6.1

Diﬀerence with or without Skip Connection

In this paper for analyzing the overparametrized CNN, the data generating network issmaller than learning network both case of Skip Connection. Nevertheless, two cases of thedata generating network is diﬀerent, if the learning model network has double ﬁlter H (k) to
the data generating network in each convolutional layer, the model network can represent thegenerating network in diﬀerent case. The output of each layer is nonnegative hence the model
can represent the skip connection or the negative of that. If the model network doesn’t havelarger layer to the data generating network, the free energy of CNN with skip connection canbe both larger or smaller than that without skip connection by the data generating network.
Then, the layer of model network gets larger, the free energy of CNN with skip connection
does not change but that without skip connection gets larger and the free energy of CNN
with skip connection comes to have smaller free energy for all data generating network.

4.6.2

Comparison to Deep Neural Network

Firstly we compare the result of this paper to that of DNN in [27]. In case of DNN,
the free energy depends on the layers of the model and only on that of the data generating
network. This stands to the reason that mapping of the linear transformation in lower layer
can be represented in higher layer. On the other hand, convolution operation doesn’t havesuch property, hence the free energy of CNN without skip connection depends on the layer oflearning model network. However, with skip connection, there exists the essential parameter
which doesn’t depend on overparametrized layers and the free energy does not also dependon the layer of learning model network.

4.7

Conclusion

In this chapter, we studied Free energy of Bayesian Convolutional Neural Network with
Skip Connection and compared to the case without Skip Connection. Free energy of Bayesian
CNN with Skip Connection doesn’t depend on the layer of the model unlike the case without
Skip Connection. In Bayesian learning, the increase of Free energy is equivalent to generalization error, hence the generalization error has same property about the Skip Connection.
In particular, Free energy of CNN with skip connection does not depend on the number ofparameters in learning network but depends only on that in data generating network. Thisfeature shows the generalization ability of CNN with skip connection does not decrease withrespect to any overparameterization in Bayesian learning.

Appendix 4.A

Fully connected caseIn this appendix, we introduce the main theorem and proof of a paper[27] which is about
the free energy of full connected Bayesian Deep Neural Network.

4.A.1

Main theorem

Assume that the learning machine and the data generating distribution are given byp(y|x, w, b) and q(y|x) = p(y|x, w∗ , b∗ ) which satisfy the conditions eq.(4.11), eq.(4.12), andeq.(4.10), and that a sample {(Xi , Yi ) i = 1, 2, ..., n} is independently subject to q(x)q(y|x).
Then the average free energy satisﬁes the inequality,
E[Fn ] ≤ nS + λReLU log n + C.
For general cases,
λReLU =

1 ∗(d + H3∗ (H2 − H2∗ ))) ,2(4.53)

where d∗ is sum of the number of parameters in w∗ and b∗ . If the support of the input
distribution is bounded or contained in nonnegative region,
λReLU =

d∗2

(4.54)

These results show that the average free energy is bounded, even if the number of layers arelarger than necessary to estimate the data-generation network. In particular eq.(4.53) is equal
to the half the number of parameters in the data-generating network.

4.A.2

LemmasIn this section, we prepare several lemmas which are necessary to prove the main theorem.
Let the Kullback-Leibler divergence of a data-generating network q(y|x) = p(y|x, w∗ , b∗ )and a learning machine p(y|x) be∫q(y|x)
K(w, b) = q(x)q(y|x) log
dxdy.
p(y|x, w, b)
It is well-known that K(w, b) ≥ 0 for an arbitrary (w, b) and K(w, b) = 0 if and only ifq(y|x) = p(y|x, w, b).
Lemma 4.A.1. Assume that a set W is contained in the set determined by the prior distribution {(w, b); φ(w, b) > 0}. Then for an arbitrary positive integer n,∫
E[Fn ] ≤ nS − log
exp(−nK(w, b))φ(w, b)dwdb.
W

Proof. An empirical Kullback-Leibler divergence is deﬁned byp(Yi |Xi , w∗ , b∗ )1∑log,
Kn (w, b) =np(Yi |Xi , w, b)ni=1

which satisﬁes E[Kn ](w, b)] = K(w, b).
∑q(y n |xn )q(Yi |Xi )
=
exp(−
log
)
p(y n |xn )p(Yi |Xi , w, b)(4.55)= exp(−nKn (w, b)).

(4.56)ni=1

From the deﬁnition of free energy,q(y n |xn )] + nSp(y n |xn )∫= −E[log φ(w, b) exp(−nKn (w, b))dwdb] + nS.

E[Fn ] = −E[log(4.57)(4.58)

By applying Lemma.1 in [42],∫E[Fn ] ≤ − log≤ − log∫φ(w, b) exp(−E[nKn (w, b)])dwdb + nS

(4.59)

φ(w, b) exp(−nK(w, b))dwdb + nS

(4.60)

φ(w, b) exp(−nK(w, b))dwdb + nS,(4.61)∫≤ − logWwhere the last inequality is derived the fact that the restriction of integrated region makes
the integration not larger.
Lemma 4.A.2. For arbitrary vectors s, t,∥σ(s) − σ(t)∥ ≤ ∥s − t∥.

Proof. If si , ti ≥ 0 or si , ti ≤ 0, then |σi (s) − σi (t)| = |si − ti |. If si ≥ 0, ti < 0, then|σi (s) − σi (t)| = |si | ≤ |si − ti |. If si < 0, ti ≥ 0, then |σi (s) − σi (t)| = |ti | ≤ |si − ti |. Hence,∥σ(s) − σ(t)∥2 =∑|σi (s) − σi (t)|2 ≤i∑|si − ti |2 = ∥s − t∥2 .

iLemma 4.A.3. For arbitrary w,w′ , b, b′ , the following inequality holds,∥f (k) (w, b, x) − f (k) (w′ , b′ , x)∥≤ ∥w(k) − w′(k) ∥∥f (k−1) (w, b, x)∥ + ∥b(k) − b′(k) ∥+ ∥w(k) ∥∥f (k−1) (w, b, x) − f (k−1) (w′ , b′ , x)∥,(4.62)

where ∥w(k) ∥ is the operator norm of a matrix w(k) .
Proof.
f (k) (w, b, x) − f (k) (w′ , b′ , x)
= σ(w(k) f (k−1) (w, b, x) + b(k) ) − σ(w′(k) f (k−1) (w, b, x) + b′(k) )+ σ(w′(k) f (k−1) (w, b, x) + b′(k) ) − σ(w′(k) f (k−1) (w′ , b′ , x) + b′(k) ).

(4.63)

Hence, by using Lemma 4.A.2,∥f (k) (w, b, x) − f (k) (w′ , b′ , x)∥≤ ∥σ(w(k) f (k−1) (w, b, x) + b(k) ) − σ(w′(k) f (k−1) (w, b, x) + b′(k) )∥+ ∥σ(w′(k) f (k−1) (w, b, x) + b′k) ) − σ(w′(k) f (k−1) (w′ , b′ , x) + b′(k) )∥≤ ∥w(k) − w′(k) ∥∥f (k−1) (w, b, x)∥ + ∥b(k) − b′(k) ∥+ ∥w′(k) ∥∥f (k−1) (w, b, x) − f (k−1) (w′ , b′ , x)∥.

(4.64)

Hence, lemma is proved.
Lemma 4.A.4. For arbitrary w, b, x,∥f (k) (w, b, x)∥ ≤ ∥w(k) ∥∥w(k−1) ∥ · · · ∥w(2) ∥∥x∥+ ∥b(k) ∥ +k−2∑∥w(k) ∥∥w(k−1) ∥ · · · ∥w(k−j) ∥∥b(k−j) ∥.

(4.65)(4.66)

j=1

Proof. By substituting w′ := 0 and b′ = 0, in Lemma 4.A.3, it follows that∥f (k) (w, b, x)∥ ≤ ∥w(k) ∥∥f (k−1) (w, b, x)∥ + ∥b(k) ∥.
Then mathematical induction gives the Lemma.

(4.67)

In order to prove the main theorem, we need several notations. The convergent matrix(k)
E (k) and vector E0 deﬁned by the condition that the absolute values of all entries are smaller√

than 1/ n, which is denoted by(
E (k) =(k)(k)

EAA EAB(k)(k)
EBA EBB

)

(
,

(k)
E0 =

(k)

EA0
(k)
EB0

)
.

(4.68)(k)

The positive-small-constant matrix D(k) and vector D0 are deﬁned by the condition that all
entries are positive and smaller than δ > 0 where δ does not depend on n, which is denoted
by
(
)
(
)
(k)
(k)
(k)
DAA DAB
DA0
(k)
(k)
D =
, D0 =.
(4.69)(k)(k)(k)
DBA DBBDB0(k)

The positive constant matrix M(k) and vector M0 are deﬁned by the condition that all
entries are in the interval [1, 2],
(
)
(
)
(k)
(k)
(k)
M
M
M
(k)
AA
AB
A0
M(k) =, M0 =.
(4.70)(k)(k)(k)
MBA MBB
MB0
To prove Theorem 4.3.1, we show an upper bound of E[Fn ] is given by choosing a set WE
which consists of essential weight and bias parameters.
Deﬁnition. (Essential parameter set WE ). A parameter (w, b) is said to be in an essential
parameter set WE if it satisﬁes the following conditions, (1), (2), and (3).
(1) For 2 ≤ k ≤ N ∗ − 1, there exist convergent matrices E (k) and positive constant matrices
M(k) such that()(k)
∗ )(k) + E (k)
(w
Z
AA
AB
w(k) =(4.71)
,
(k)
(k)
−MBA
−MBB
(
)
∗ )(k) + E (k)(bA0(4.72)
b(k) =
,
(k)
−MB0
where

{
(k)
ZAB =
(k)

(k)

(3)

EAB (k = 3).
(k)
MAB (k ≠ 3)(k)(4.73)

Note that, for k = 2, ZAB , MBB , and MB0 are the empty matrix.
(2) For N ∗ ≤ k ≤ N − 1, there exist positive-small-constant matrix D(k) and positive constantmatrix M(k)
)
(
(k)
(k)
∗
I
+
D
M
N
−1
AA
AB
,
w(k) =(4.74)
(k)
(k)
−MBA
−MBB
(
)
(k)
M
A0
b(k) =,(4.75)(k)−MB0

where IN ∗ −1 is the identity matrix of HN ∗ −1 × HN ∗ −1 .
(N )(3) For k = N , there exist convergent matrix E (N ) and vector E0 such that
(
)
∗
(N )
(N )
w(N ) = (w∗ )(N ) P −1 + EAA
, MAB
b

(N )

∗ (N ∗ )= (b )−

N−1∑(4.76)

w(N −1) w(N −2) · · · w(k) b(k−1) + EB0 ,(N )(4.77)

k=N ∗∗∗where P ∈ R(HN ∗ −1 )×(HN ∗ −1 ) is deﬁned by matrices in eq.(4.74)(N −1)

P = wAA(N −2)wAA(N ∗ )· · · wAA .

Note that a positive constant δ > 0 is taken suﬃciently small such that arbitrary wAA (N ∗ ≤k ≤ N − 1) is invertible.
(k)

Lemma 4.A.5. Assume that the weight and bias parameters are in the essential set WE .
Then there exist constants c1 , c2 > 0 such that(N ∗ −1)∥fA

c1∗(w, b, x) − f (N −1) (w∗ , b∗ , x)∥ ≤ √ (∥x∥ + 1),n(N ∗ −1)∥fB(4.78)(w, b, x)∥ ≤ c2 (∥x∥ + 1).

(4.79)

Proof. Eq (4.79) is derived from Lemma 4.A.4. By the deﬁnitions (4.71), (4.72), for 4 ≤ k ≤N∗ − 1fA (w, b, x) = σ(((w∗ )(2) + EAA )fA (w, b, x) + (b∗ )(2) + EA0 ),
(2)

(2)

(1)

(2)

(3)
(3)
(2)
fA (w, b, x) = σ(((w∗ )(3) + EAA )fA (w, b, x)(3) (2)(3)+ EAB fB (w, b, x) + (b∗ )(3) + EA0 ),(k)(k) (k−1)
fA (w, b, x) = σ(((w∗ )(k) + EAA )fA(w, b, x)(k) (k−1)(k)+ MAB fB(w, b, x) + (b∗ )(k) + EA0 ).

(4.80)(4.81)(4.82)

Here, for 4 ≤ k ≤ N ∗ − 1, fB(w, b, x) = 0, since all entries of wBA , wBB , and wB0(k−2)
are negative and the output of ReLU function fB(w, b, x) is nonnegative. On the other
hand,
(k−1)

(k−1)

(k−1)

f (k) (w∗ , b∗ , x) = σ((w∗ )(k) f (k−1) (w∗ , b∗ , x) + (b∗ )(k) ).

(k−1)(4.83)

Hence, by Lemma 4.A.3, 2 ≤ k ≤ N ∗ − 1,
∥f1 (w, b, x) − f (k) (w∗ , b∗ , x)∥(k)(k−1) (k−1)≤ ∥EAA fA+ ∥(w∗ )(k) (fA

(k)

(3)

(2)

(w, b, x) + E01 ∥ + δk,3 ∥EAB fB (w, b, x)∥(4.85)(w, b, x) − f (k−1) (w∗ , b∗ , x))∥(4.86)

(k−1)

(k−1)

(4.84)

(k−1)

(k)

(3)

(2)

≤ ∥EAA ∥∥fA(w, b, x)∥ + ∥EA0 ∥ + δk,3 ∥EAB ∥∥fB (w, b, x)∥(4.87)+ ∥(w∗ )(k) ∥∥fA(w, b, x) − f (k−1) (w∗ , b∗ , x)∥,(4.88)(k−1)

(k−1)(3)(k)

where δk,3 = 1 if k = 1 or 0 otherwise. The entries of matrices in EAA , EAB , and EA0√
are bounded by 1/ n order term and the operator norm is bounded by the Frobenius norm,
√
(k−1)
(3)
(k)
hence ∥EAA ∥, ∥EAB ∥, and ∥EA0 ∥ are bounded by 1/ n order term. Moreover, ∥(w∗ )(k) ∥ is(k−1)
a constant term. For k = 2, fA(w, b, x) − f (k−1) (w∗ , b∗ , x) = x − x = 0. Then by using
mathematical induction we obtain the Lemma.
Lemma 4.A.6. Assume that the weight and bias parameters are in the set WE . Then there
exists a constant c3 > 0 such thatc3∗∥f (N ) (w, b, x) − f (N ) (w∗ , b∗ , x)∥ ≤ √ (∥x∥ + 1).
n(4.89)∗∗ ) be input vectors into the output layers of
Proof. Let h ∈ RHN and h∗ ∈ RHN ∗ (HN = HN∗
the learning and data-generating machines respectively. In other words, h and h∗ is deﬁned∗
such that f (N ) (w, b, x) = σ(h) and f (N ) (w∗ , b∗ , x) = σ(h∗ ). By the deﬁnition of the essential
(k)
(k)
(k)
parameter set (2), for N ∗ − 1 ≤ k ≤ N − 1, all entries of wBA , wBB and bB0 are negative.
(k)
(k)
(k)
Hence, for N ∗ ≤ k ≤ N − 1, f2 (w, b, x) = 0. For N ∗ ≤ k ≤ N − 1, all entries of wAA , wAB(k)and bA0 are positive. Hence, by using σ(t) = t for t ≥ 0,
(N )(N −1)

(N )

N
−1
∑

h = wAA wAA+ bA0 +(N ∗ ) (N ∗ −1)· · · wAA fA(N −1)wAA(w, b, x)(k) (k−1)· · · wAA bA0(4.90).

(4.91)

k=N ∗On the other hand,∗∗h∗ = (w∗ )(N ) f (N −1) (w∗ , b∗ , x) + (b∗ )(N ) .

(4.92)

If w is in the essential set of parameters,(N ∗ )(N −1)(N )= (w∗ )(N ∗ )∗(N −1)· · · wAA = ((w∗ )(N ) B −1 + EAA )wAA

wAA wAA(N )(N ∗ )(N )(N −1)∗∗∗(N ∗ )(N ∗ −1)(N ∗ −1)+ EAA wAA

(N ∗ )· · · wAA

· · · wAA .

(4.93)(4.94)

It follows that∥w(N ) w(N −1) · · · w(N ) f (N −1) (w, b, x) − w(N ∗) f (N −1) (w∗ , b∗ , x)∥(N )(N −1)≤ ∥wAA wAA≤ ∥(w∗ )(N ∗ )(N )· · · wAA fA(N ∗ −1)(w, b, x) − w(N ∗) f(fA(w, b, x) − f(N −1)(N ∗ )(N ∗ −1)(N ∗ −1)+ ∥EAB ∥∥wAA ∥ · · · ∥wAA ∥∥fAc4
≤ √ (∥x∥ + 1),n(w∗ , b∗ , x)∥(4.95)(4.96)(w∗ , b∗ , x))∥(4.97)(w∗ , b∗ , x)∥(4.98)(4.99)

where the last inequality is derived by Lemma 4.A.5. Also by the deﬁnition,
∥b

(N )

+

N
−1
∑
k=N

c4
∗
w(N −1) · · · w(k) b(k−1) − (b∗ )(N ) ∥ ≤ √ ,n∗(4.100)it follows thatc5∥h − h∗ ∥ ≤ √ (∥x∥ + 1).
nThen applying Lemma 4.A.2 completes the lemma.

Lemma 4.A.7. (1) If the support of q(x) is contained in a positive region, the same conclusion(3)(3)
as Lemma 4.A.5 holds by replacing ZAB in (4.73) with MAB .
(2) If the support of q(x) is contained in a bounded region, the same conclusion as Lemma
(3)
(3)
(3)
4.A.5 holds by replacing ZAB in (4.73) with MAB and by replacing −MB0 in (4.72) with amatrix in a suﬃciently small region.
(2)

Proof. In both cases, fB (w, b, x) = 0 in eq.(4.81) holds. Hence, the same conclusion ofLemma 4.A.5 holds.

4.A.3

Proof of Main Theorem

In this section we prove the main theorem.
Proof. (Main theorem). By Lemma 4.A.1, it is suﬃcient to prove that there exists a constantC > 0 such that∫C
exp(−nK(w, b))φ(w, b)dwdb ≥ λ
n
WE
∫
1
∗
K(w, b) =∥f (N ) (w, b, x) − f (N ) (w∗ , b∗ , x)∥2 q(x)dx.
2
By using Lemma 4.A.6, if (w, b) ∈ WE ,∫c23c4(∥x∥ + 1)2 q(x)dx =< ∞.
K(w, b) ≤
2n
n

where

It follows that∫exp(−nK(w, b))φ(w, b)dwdb)(≥ exp(−c4 )min φ(w, b) Vol(WE ).

(4.101)WE(4.102)(w,b)∈WE

where c4 > 0, min(w,b)∈WE φ(w, b) > 0, and Vol(WE ) is the volume of the set WE by theLebesgue measure. By the deﬁnition of the essential parameter set WE , its volume is determined by the dimension of the convergent matrices and vectors. Let 2λ be the number ofparameters in convergent matrices and vectors. Then
Vol(WE ) ≥

C1
,
nλ

where in general cases,
(
)
∗ −1
N∑
1
∗
∗
∗
∗
∗
λ=
HN
Hk∗ (Hk−1+ 1)∗ (HN −1 + 1) + H3 (H2 − H2 ) +2(4.103)

k=2

=

1 ∗
(d + H3∗ (H2 − H2∗ )) .
2

(4.104)

If the support of the input distribution is contained in a positive region or a bounded region,
(
)
∗ −1
N∑
1
∗
∗
∗
HN
Hk∗ (Hk−1
λ=+ 1)(4.105)∗ (HN −1 + 1) +
2
=

d∗
2

k=2

,

which completes the main theorem.

(4.106)

Chapter 5

Conclusion
This thesis shows the free energy and the generalization error of Bayesian learning in casesthat model has multiple optimal probability distributions and CNNs are overparamtetrized.
The ﬁrst study revealed that if there exist multiple optimal probability distributions, thegeneralization error gets larger if the number of data increases. This situation occurs whencomplex learning models are used for complex data such as learning of deep neural networks.
The second study revealed that even if CNNs are overparamtetrized, the generalization error
only depends on the complexity of data generating process with skip connection. Among themodels which are revealed the asymptotic behavior in Bayesian learning, only fully connected
DNNs and CNNs with skip connection have such property. We clariﬁed the peculiarities of
DNNs on Bayesian learning in these two studies.

51

Acknowledgment
I would like to thank the following people for helping with this work. My supervisor Prof
Sumio Watanabe supported me in many ways from Master course to PhD. I have learned manyfrom his approach to mathematical science. Colleagues in Watanabe laboratory helps me withthe discussions and advices about my work and daily life. Dr Naoki Hayashi in particular,
advised me after completing his PhD. Dr Susan Wei, University Melbourne invited me to IMSAPRM. I had a good opportunity to discuss singular learning theory as statistics. Edmund
Lau, University Melbourne discussed with me and show me around Melbourne city. Prof
Makoto Yamashita, Prof Satoshi Takabe, Prof Takafumi Kanamori and Yumiharu Nakano
reviewed this work and providing many helpful comments. Finally, I would like to express mystrong gratitude to my father and relatives for their support to me for years after my mother’s
death.

53

Bibliography[1] Hirotugu Akaike. A new look at the statistical model identiﬁcation. IEEE transactions
on automatic control, 19(6):716–723, 1974.
[2] Hirotugu Akaike. Likelihood and the bayes procedure. In Springer Series in Statistics,
pages 309–332. Springer New York, 1998.
[3] Miki Aoyagi. A bayesian learning coeﬃcient of generalization error and vandermonde matrix-type singularities. Communications in Statistics - Theory and Methods,
39(15):2667–2687, jul 2010.
[4] Miki Aoyagi and Kenji Nagata. Learning coeﬃcient of generalization error in bayesianestimation and vandermonde matrix-type singularity. Neural Computation, 24(6):1569–
1610, jun 2012.
[5] Miki Aoyagi and Sumio Watanabe. Stochastic complexities of reduced rank regression
in bayesian estimation. Neural Networks, 18(7):924–933, sep 2005.
[6] Mathias Drton and Martyn Plummer. A bayesian information criterion for singularmodels. Journal of the Royal Statistical Society: Series B (Statistical Methodology),79(2):323–380, feb 2017.
[7] Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. Hybrid
monte carlo. Physics letters B, 195(2):216–222, 1987.
[8] Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks withbernoulli approximate variational inference. arXiv preprint arXiv:1506.02158, 2015.
[9] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing
model uncertainty in deep learning. In international conference on machine learning,
pages 1050–1059. PMLR, 2016.
[10] Mudasir A Ganaie, Minghui Hu, AK Malik, M Tanveer, and PN Suganthan. Ensemble
deep learning: A review. Engineering Applications of Artiﬁcial Intelligence, 115:105151,
2022.
[11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances inneural information processing systems, 27, 2014.
55

[12] J. A. Hartigan. A failure of likelihood asymptotics for normal mixtures. In Proceedingsof the Barkeley Conference in Honor of Jerzy Neyman and Jack Kiefer, 1985, volume 2,
pages 807–810, 1985.
[13] W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. 1970.
[14] Naoki Hayashi. The exact asymptotic form of bayesian generalization error in latent
dirichlet allocation. Neural Networks, 137:127–137, may 2021.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning forimage recognition. In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 770–778, 2016.
[16] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1–42,
1997.
[17] Matthew D Hoﬀman, Andrew Gelman, et al. The no-u-turn sampler: adaptively setting
path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593–1623, 2014.
[18] Furong Huang, Jordan Ash, John Langford, and Robert Schapire. Learning deep resnet
blocks sequentially using boosting theory. In International Conference on Machine Learning, pages 2058–2067. PMLR, 2018.
[19] Natsuki Kariya and Sumio Watanabe. Asymptotic analysis of singular likelihood ratio
of normal mixture by bayesian learning theory for testing homogeneity. Communicationsin Statistics - Theory and Methods, pages 1–18, nov 2020.
[20] Diederik P. Kingma and Max Welling.
abs/1312.6114, 2013.

Auto-encoding variational bayes.

CoRR,[21] Alex Krizhevsky, Ilya Sutskever, and Geoﬀrey E Hinton. Imagenet classiﬁcation withdeep convolutional neural networks. Communications of the ACM, 60(6):84–90, 2017.
[22] Satoshi Kuriki and Akimichi Takemura. The tube method for the moment index inprojection pursuit. Journal of Statistical Planning and Inference, 138(9):2749–2762, sep
2008.
[23] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard,
Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip
code recognition. Neural computation, 1(4):541–551, 1989.
[24] David JC MacKay. Information-based objective functions for active data selection. Neuralcomputation, 4(4):590–604, 1992.
[25] David JC MacKay. Information-based objective functions for active data selection. Neuralcomputation, 4(4):590–604, 1992.
[26] Shuya Nagayasu and Sumio Watanabe. Free energy of convolutional neural network withskip connection. In Proceedings of Asian Conference on Machine Learning.

[27] Shuya Nagayasu and Sumio Watanabe. Bayesian free energy of deep relu neural networkin overparametrized cases. arXiv preprint arXiv:2303.15739, 2023.
[28] Shuya Nagayasu and Sumio Watanbe. Asymptotic behavior of free energy when optimal
probability distribution is not unique. Neurocomputing, 500:528–536, aug 2022.
[29] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya
Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of
Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021.
[30] Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science& Business Media, 1995.
[31] Atsushi Nitanda and Taiji Suzuki. Functional gradient boosting for learning residual-likenetworks with statistical guarantees. In International Conference on Artiﬁcial Intelligence and Statistics, pages 2981–2991. PMLR, 2020.
[32] J. Rissanen. Modeling by shortest data description. Automatica, 14(5):465–471, sep 1978.
[33] Dmitry Rusakov and Dan Geiger. Asymptotic model selection for naive bayesian networks. Journal of Machine Learning Research, 6(Jan):1–35, 2005.
[34] Kenichiro Sato and Sumio Watanabe. Bayesian generalization error of poisson mixture
and simplex vandermonde matrix type singularity. arXiv preprint arXiv:1912.13289,
2019.
[35] Gideon Schwarz. Estimating the dimension of a model. The annals of statistics, 6(2):461–
464, 1978.
[36] Sam Smith and Quoc V. Le. A bayesian perspective on generalization and stochastic
gradient descent.
[37] David J Spiegelhalter, Nicola G Best, Bradley P Carlin, and Angelika Van Der Linde.
Bayesian measures of model complexity and ﬁt. Journal of the royal statistical society:
Series b (statistical methodology), 64(4):583–639, 2002.
[38] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir
Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper
with convolutions. In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 1–9, 2015.
[39] Michel Talagrand. Gaussian processes and the generic chaining. In Upper and Lower
Bounds for Stochastic Processes, pages 13–73. Springer Berlin Heidelberg, 2014.
[40] A. W. van der Vaart and J. A. Wellner. Weak convergence. In Weak Convergence andEmpirical Processes, pages 16–28. Springer New York, 1996.
[41] Sumio Watanabe. Algebraic analysis for nonidentiﬁable learning machines. Neural Computation, 13(4):899–933, apr 2001.
[42] Sumio Watanabe. Algebraic geometrical methods for hierarchical learning machines.
Neural Networks, 14(8):1049–1060, 2001.

[43] Sumio Watanabe. Almost all learning machines are singular. In 2007 IEEE Symposium
on Foundations of Computational Intelligence, pages 383–388. IEEE, 2007.
[44] Sumio Watanabe. Asymptotic equivalence of bayes cross validation and widely applicable
information criterion in singular learning theory. Journal of machine learning research,
11(12), 2010.
[45] Sumio Watanabe. Asymptotic learning curve and renormalizable condition in statisticallearning theory. Journal of Physics: Conference Series, 233:012014, jun 2010.
[46] Sumio Watanabe. A widely applicable bayesian information criterion. The Journal of
Machine Learning Research, 14(1):867–897, 2013.
[47] Sumio Watanabe. Mathematical theory of Bayesian statistics. CRC Press, 2018.
[48] Susan Wei, Daniel Murfet, Mingming Gong, Hui Li, Jesse Gell-Redman, and Thomas
Quella. Deep learning is singular, and that’s good. IEEE Transactions on Neural Networks and Learning Systems, pages 1–14, 2022.
[49] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11),
pages 681–688, 2011.
[50] K. Yamazaki and Y. Motomura. Hidden node detection between observable nodes basedon bayesian clustering. Entropy, 21(1):32, jan 2019.
[51] Keisuke Yamazaki and Sumio Watanabe. Singularities in mixture models and upperbounds of stochastic complexity. Neural Networks, 16(7):1029–1038, sep 2003.
[52] Keisuke Yamazaki and Sumio Watanabe. Singularities in complete bipartite graph-typeboltzmann machines and upper bounds of stochastic complexities. IEEE Transactions
on Neural Networks, 16(2):312–324, mar 2005.
[53] Keisuke Yamazaki and Sumio Watanbe. Stochastic complexity of bayesian networks.
arXiv preprint arXiv:1212.2511, 2012.
[54] Yingxue Zhang, Soumyasundar Pal, Mark Coates, and Deniz Ustebay. Bayesian graphconvolutional neural networks for semi-supervised classiﬁcation. In Proceedings of the
AAAI conference on artiﬁcial intelligence, volume 33, pages 5829–5836, 2019.
[55] Piotr Zwiernik. An asymptotic behaviour of the marginal likelihood for general markov
models. The Journal of Machine Learning Research, 12:3283–3310, 2011.

Publications
Peer-reviewed Journal Papers
1. Shuya Nagayasu, Sumio Watanbe. ”Asymptotic behavior of free energy when optimal
probability distribution is not unique.” Neurocomputing, Vol.500, pp. 528-536, August2022.

Peer-reviewed International Conference
1. Shuya Nagayasu, Sumio Watanabe ”Free Energy of Bayesian Convolutional Neural Network with Skip Connection.” The 15th Asian Conference on Machine Learning, November, 11-14, 2023, Istanbul, Turkey. to appear in Proceedings of Machine Learning
Research.

Not-Peer-reviewed International Conference
1. Shuya Nagayasu, ”Generalization Error of Bayesian Deep Neural Network with nonanalytic activation function” The Institute for Mathematical Statistics Asia-Paciﬁc Rim
Meeting January, 4-7, 2024.Melbourne, Australia

Domestic Conference
1. 永安修也, 渡辺澄夫. ”最適な確率分布が一意でないときのベイズ学習曲線”, 信学技報, vol.
119, no. 453, NC2019-94, pp.107-112, 2020 年 3 月. 電気通信大学
2. 永安修也, 渡辺澄夫. ”最適パラメータの学習モデルが一意でないときのベイズ学習の漸近解析”, 第 25 回情報論的学習理論ワークショップ, 20221 年 11 月 22 日（火）, つくば国際会議場 茨城Preprint
1. Shuya Nagayasu, Sumio Watanabe ”Bayesian Free Energy of Deep ReLU Neural Network in Overparametrized Cases” https://doi.org/10.48550/arXiv.2303.15739

59