Regularization Methods for
the Restricted Bayesian NetworkBESOMICONIP 20162016-10-17
Yuuji Ichisugi and Takashi Sano

Artificial Intelligence Research Center (AIRC),
National Institute of Advanced Industrial Scienceand Technology(AIST)

Outline• Our research goal:– Implement a cerebral cortex model• Our developing model: BESOM model• Local minimum problem of BESOM• Regularization methods• Network translation technique in order to use
EM algorithmOur research goals• Long term goal : Human-like intelligence
by WBA approach• Short term goals:– Implement a cerebral cortex model

http://www.irasutoya.com/2015/05/ai.html• Our working hypothesis :The cerebral cortex is a kind of Bayesian network– Implement visual area, language area, motor
area etc. using the cerebral cortex modelCerebral cortex• Realizes human's intelligence.
– Sensory, Motor, Language, ...

• It is important to reveal the informationprocessing principle of the cortex.

from http://en.wikipedia.org/wiki/Brodmann_area

Bayesian network models ofcerebral cortex• Pattern recognition[George and Hawkins 2005][Hasegawa and Hagiwara 2010]• Electrophysiological phenomena[Lee and Mumford 2003] [Rao 2005] [Chikkerur, Serre, Tan andPoggio 2010][Hosoya 2010][Hosoya 2012]• Psychophysical phenomena[Chikkerur, Serre, Tan and Poggio 2010]• Anatomical structures[George and Hawkins 2005] [Ichisugi 2007] [Rohrbein, Eggert andKorner 2008] [Ichisugi 2011]• Motor areas[Hosoya 2009]• The others [Litvak and Ullman 2009][Ichisugi 2011]

A cerebral cortex seems to be a huge Bayesian networkwith layered structure like Deep Learning.

What is Bayesian network?
– Very efficient and expressive datastructure for probabilistic knowledge.
• If a joint probability table can be factored into small
conditional probability tables (CPTs), time andspace complexity will decrease.
ex.： P ( S , W , R, C ) = P (W | S , R ) P (C | R ) P ( S ) P ( R )
CPTs

SW

RC

P(S=yes)
0.2
S
no
no
yes
yes

R
no
yes
no
yes

P(R=yes)
0.02
P(W=yes|S,R)
0.12
0.8
0.9
0.98

R P(C=yes|R)
no 0.3yes 0.995

Similarities between Cerebral
Cortex and Bayesian network– Asymmetric and bidirectional connections
between lower and higher areas.
– Local and asynchronous communications.
– Non negative values.
– Normalization of values.
– Hebb's learning rule.
– Context dependent recognition.
– Behavior based on Bayesian Statistics.

Deep Learning using a
Bayesian network is thought tobe promising• Because of its similarity to the humanbrain• Inference in Bayesian networks cansometime be executed with low
computational complexity• Top-down information flow
• It is easy to build in prior knowledge about
learning targets
...

...

...

...

。。。

...

...

...

...

。。。

。。。

...

BESOM (BidirEctional SOM) [Ichisug 2007]• A Bayesian network model of cerebral cortex• Combination of Bayesian Networks, Deep
Learning, Self-Organizing Maps andIndependent Component Analysis– Incomplete technology, however• Our goal:– Scalability of computation amount– Scalability of accuracy– Usefulness as a machine learning algorithm– Plausibility as a neuroscientific modelBESOM Ver.3.0 features• Restricted Conditional Probability Tables:
1 𝑚𝑚
𝑃𝑃 𝑥𝑥 𝑢𝑢1 , … , 𝑢𝑢𝑚𝑚 = Σ𝑘𝑘=1 𝑃𝑃 𝑥𝑥 𝑢𝑢𝑘𝑘𝑚𝑚
• Scalable recognition algorithm [Ichisugi, Takahashi 2015]• Regularization methods:– Win-rate penalty– Lateral-inhibition penalty– Neighborhood learning– Edge selection

Today's topics

4 layer BESOM
for supervised learning
Output Layer
L2
L1

Output node 10 units

...

...

...

...

。。。

...

...

...

3x3 nodes
x 30 units

...

。。。

。。。

...

5x5 nodes
x 20 units
28x28 nodes
x 2 units

Input Layer
Input image (MNIST handwritten digits)
28x28 pixcels

Ovals are nodes (random variables)
White circles inside are units (possible values for the random variables)

Objective of learning• Calculate MAP estimator of the parameter 𝜃𝜃 tθ * = arg max ∏ P(i (i ) | θ ) P(θ )θ i =1 t= arg max ∏∑ P(h, i (i ) | θ ) P(θ )θ i =1 hTo estimate parameter , the online EM (ExpectationMaximization) algorithm or its approximation is used.

Structure of BESOM network

Input

Node = random variable = cortical column
unit= value= minicolumn

(V2)

(V1)

(LGN)

No connections in each layer.
Fully connected between different layers.

Recognition

Find the values of hidden variables
with the highest posterior probability.
(MPE: most probable explanation)

Connection weights= CPT= synapase weights

Input (observed data) is given at the lowest layer.

Learning

Increase the connection weights between active units(mini-columns) and decrease the other weights.

Problem of utilization ratio of units
Seems to be very
bad local minimum.
- Wastes units.
- Low recognition rates.
63.6% MNIST
Learned withproposed priors.

Learned with no priors.
Most units never becomeactive.

Each image is the mean image of inputs which activate the unit.
(Selected 10 units of L2 nodes are shown.)
White image indicate the unit never become active.

Win-Rate penalty• All units should be used evenly.
• Penalties are imposed when thehistograms of win-rates are difference
from the uniform distributions.

Problem: When the parameter has a complex
prior distribution, it is not obvious how to
perform the EM algorithm efficiently.

Equivalent network• Fortunately, the network with win-ratepenalty can be expressed as an approx.
equivalent network without prior.
– Then, EM is straightforwardly applicable.
,X1

RX1

RX2

Nodes whichimposes penalties.

X2

X3

RX3

...

Lateral-Inhibition penalty• Nodes which shares the same child nodes
should be independent.
– Otherwise, redundant representationis acquired by learning.

• Penalties are imposed when designatedpairs of nodes are not independent.

Equivalent network• This penalty can also be represented by
an approximately equivalent network
without prior.
,

X1

RX1X2

RX2X3

Nodes which
imposes penalties.

X2

X3

RX3X1

...

Evaluation Result (MNIST)With
Win-Rate Penalty

Without
Win-Rate Penalty

With
Lateral-Inhibition

80.6%

81.8%

Without
Lateral-Inhibition

82.2%

63.6%

For both penalties, the recognition rate washigher than when no penalties were applied.
This result also shows that two prior distribution
can be applied simultaneously; however, it does
not show the best accuracy in this case.

Status of utilization of units
With
Win-Rate Penalty

With
Lateral-Inhibition
penalty

Without
Lateral-Inhibition
penalty

Without
Win-Rate Penalty

Conclusion• Two regularization methods for parameter
learning of layered Bayesian networks like deeplearning are proposed.
– Win-Rate penalty and Lateral Inhibition penalty– Standard EM can be used for learning by networktranslation technique• They may alleviate both local minima andoverfitting problems.