Deep Restricted Bayesian
Network BESOMNICE 20172017-03-07
Yuuji Ichisugi

...

...

...

...

Artificial Intelligence Research Center (AIRC),
National Institute of Advanced Industrial Scienceand Technology (AIST), Japan...

...

...

。。。

...

。。。

。。。

1...

BESOM (BidirEctional Self Organizing Maps) [Ichisug 2007]• A computational model of the cerebral cortex– A model of column network, not spiking neurons• Design goals:– Scalability of computation– Usefulness as a machine learning system– Plausibility as a neuroscientific model• As a long-term goal, we aim to reproducefunctions of such as the visual areas and thelanguage areas using this cerebral cortex
model.
2

Architecture of BESOM model
Recognition step: The entire network behaves like a Bayesian network.
Learning step: Each node behaves like a Self-organizing map.
Node = random variable = macro-column
Unit = value of random variable = minicolumn
(V2)

(V1)

(LGN)

Outline• Bayesian networks and the cerebral cortex• BESOM Ver.3 and robust pattern recognition• Toward BESOM Ver.4

Models of visual cortex
based on Bayesian networks• Various functions, illusions, neural responses and anatomicalstructure of the visual cortex were reproduced by Bayesiannetwork models.
– [Tai Sing Lee and Mumford 2003]– [George and Hawkins 2005]– [Rao 2005]– [Ichisugi 2007]– [Litvak and Ullman 2009]– [Chikkerur, Serre, Tan and Poggio 2010]– [Hosoya 2012]– ...

The visual cortex seems to be a huge Bayesian networkwith layered structure like Deep Neural Networks.

What is Bayesian networks?
– Efficient and expressive data structure ofprobabilistic knowledge [Perl 1988]• Various probabilistic inference can be executed
efficiently if a joint probability table can be factored
into small conditional probability tables (CPTs).

P( S , W , R, C )  P(W | S , R) P(C | R) P( S ) P ( R )CPTsSW

RC

P(S=yes)
0.2
S
nono
yes
yesR
noyes
no
yes

P(R=yes)
0.02
P(W=yes|S,R)
0.12
0.8
0.9
0.98

R P(C=yes|R)
no 0.3yes 0.995

6

Loopy Belief Propagation[Murphy, Weiss, Jordan 1999]• Efficient approximate inference algorithm– Iterative algorithm with local andasynchronous computation, like brain.
– Although there is no guarantee ofconvergence, it is empirically accurate.
BEL( x)   ( x) ( x)

U1

 ( x)   P( x |u1 , , um )  X (uk )u1 ,,u m...

πX(uk)k ( x)   Y ( x)λX(uk)Xll

 Y ( x)   ( x) Y ( x)lj l X (uk )    ( x)xj

 P( x |u ,, u )  (u )1u1 ,,u m / u km

Xiki

πYl(x)
Y1UmλYl(x)

...

Yn 7

Belief propagation and micro
circuit of cerebral cortex• The similarity between belief propagationand the six-layer structure of the cerebral
cortex has been pointed out many times.
[George and Hawkins 2005][Ichisugi 2007][Rohrbein, Eggert and Korner 2008][Litvak and Ullman 2009]

IIIIII
IVVVI

8Approximate Belief Propagation
1l tXY z Yt  WXY oYtot 1X[Ichisugi 2007]

Approximates Pearl's algorithm[Pearl 1988]
with some assumptions.

lt 1XYY children ( X )t 1Tk UX WUXbUtp tX1 t 1k
 UXUbUU  parents ( X )rXt 1  o tX1  p tX1Zt 1Xi

Z X , oXX  (r ) ( rt 1XiWUXt 1X 1ot 1Xt 1Xp )bX

WXYZ Y , οYYz tX1  ( Z Xt 1 , Z Xt 1 ,  , Z Xt 1 )Tb tX1  (1 / Z Xt 1 )rXt 1
where x  y  ( x1 y1 , x2 y2 , , xn yn )TYuuji ICHISUGI, "The cerebral cortex model that self-organizes
conditional probability tables and executes belief propagation", Inproc. of IJCNN2007, Aug 2007.

Similarity in information flowIIIIIIIVVVI[Gilbert 1983][Pandya and Yeterian 1985]
Gilbert, C.D., Microcircuitry of the visual-cortex, Annual review ofneuroscience, 6: 217-247, 1983.
Pandya, D.N. and Yeterian, E.H., Architecture and connections of corticalassociation areas. In: Peters A, Jones EG, eds. Cerebral Cortex (Vol. 4):
Association and Auditory Cortices. New York: Plenum Press, 3-61, 1985.

Lower Area

Higher Area

Anatomical structure

1l tXY z Yt  WXY oYtot 1Xlt 1XYY children( X )

kUXoX

t 1
Tk UX
 WUXbUt

p tX1 lUXbU k UXt 1

U  parents ( X )rXt 1  o tX1  p tX1Zt 1X  (r ) ( rt 1Xiit 1X 1

ot 1Xt 1Xp )z tX1  ( Z Xt 1 , Z Xt 1 , , Z Xt 1 )Tb tX1  (1 / Z Xt 1 )rXt 1
where x  y  ( x1 y1 , x2 y2 , , xn yn )TParent nodes

ZXk XYoYl XYbXZYChild nodes

Information flow of the approx. BPThe intermediate variables of this algorithm can be assigned
to each layer of the cerebral cortex without contradicting theknown anatomical structure.

Detailed circuit that calculates
the approximate BP(bU1 )1(bU1 ) 2(bU 2 ) 3(bU 2 )1+(bU 2 ) 2+

( kU1 X )1

( kU1 X ) 2+

(bU 2 ) 3++( p X )1(o X )1U1+( kU 2 X )1( kU 2 X ) 2( pX )2(o X ) 2( rX )1( l XY1 )1( l XY1 ) 2++X( rX ) 2Z Y1(oY1 )1(oY1 ) 2( l XY2 )1+( l XY2 ) 2(oY1 ) 3+Z Y2(oY2 )1/ZXZX(bX )1U2(oY2 ) 2/

(oY2 ) 3(b X ) 2+

(o X )1(bX )1(o X ) 2(b X ) 2Y1Y2

The left circuit
calculates values oftwo units, x1 and x2,
in node X in the above
network.

Correspondencewith local cortical circuit(bU1 )1(bU1 ) 2I

(bU 2 )3(bU 2 )1+(bU 2 ) 2

Mini-column likestructure+( kU1 X )1( kU1 X ) 2+

(bU 2 )3++

( p X )1(o X )1II+( kU 2 X )1( kU 2 X ) 2( pX )2(o X ) 2( rX )1( l XY1 )1( l XY1 ) 2++

Many horizontal
fibers in I, IV

III( rX ) 2Z Y1I
II(oY1 )1III(oY1 ) 2( l XY2 )1+( l XY2 ) 2+Z Y2(oY2 )1/ZXZX(bX )1IV(oY1 ) 3VIVVI(oY2 ) 2/(oY2 )3(b X ) 2+

(o X )1(bX )1(o X ) 2(b X ) 2VVI

Many cells
in II, IVOutline• Bayesian networks and the cerebral cortex• BESOM Ver.3 and robust patternrecognition• Toward BESOM Ver.4

Toward realizationof the brain function• If the cerebral cortex is a kind of Bayesian
network, we should be able to reproducefunction and performance of it using
Bayesian networks.
– As a first step, we aim to reproduce some partof the functions of the visual areas and thelanguage areas.
– Although there were some difficulties such as
computational cost and local minimum
problem, now they have been solved
considerably.

BESOM Ver.3.0 features• Restricted Conditional Probability Tables:• Scalable recognition algorithm OOBP [Ichisugi, Takahashi 2015]• Regularization methods to avoid local minima– Win-rate and Lateral-inhibition penalty [Ichisugi, Sano 2016]– Neighborhood learning
Computational amount of
one step of iteration ofOOBP is linear to thenumber of edges of thenetwork.
Recognition algorithm OOBP

Yuuji Ichisugi and Naoto Takahashi,
An Efficient Recognition Algorithm for Restricted
Bayesian Networks, In proc. of IJCNN 2015.
Yuuji Ichisugi and Takashi Sano,
Regularization Methods for the Restricted Bayesian
Network BESOM, In Proc. of ICONIP2016, Part I, LNCS
9947, pp.290--299, 2016.

15

The design of BESOM is motivatedby two neuroscientific facts.
１．Each macro-column seems to be
like a SOM.
２．A macro-column at a upper area
receives the output of the macrocolumns at the lower area.

minicolumns

V4...
macro-columns

V2...
V1...
16

If a SOM receives input from other SOMs,
they naturally become a Bayesian Network
Learning rule (without neighborhood learning)xiwij  wij   ( y j  wij )converges to the probability
thatfires when fires, thatis, the conditional probability.

y=(0, 0, 1, 0, 0,wij

0, 1, 0, 0, 0,yj

0, 0, 0, 0, 1)T17

InputNode = random variable = macro-column
unit= value= minicolumn

(V2)

(V1)

(LGN)

Connection weights= Conditional probabilities= synapse weights

Input (observed data) is given
at the lowest layer.

Recognition

Learning

increased

Find the values with the highest posterior
probability. (MAP)

decreased

Updates connection weights
with Hebb's rule.

18

Connections oflong distancelateral inhibition
Hidden layer

Input[Ichisugi, Sano 2016]

...

...

Input layer

Yuuji Ichisugi and Takashi Sano,
Regularization Methods for the Restricted Bayesian Network BESOM,In Proc. of ICONIP2016, Part I, LNCS 9947, pp.290--299, 2016.

Recognition...

Units that receive strong lateral inhibition
are less likely to become winners.

Input (observed data) is given at thelowest layer.

Learning...

Updates connection weights
with Hebb's rule.

BESOM can be used as if
it were a Deep neural network.
A…

BC

D

… RecognizedFeatures
…

…

Input

BESOM can be used as if
it were a bidirectional Deep neuralnetwork.
A

BPrediction= Prior…

CD… RecognizedFeatures
…

…

Input

Robust character recognition
utilizing context information[Nakada, Ichisugi 2017]

2GRAM
WORD

CHAR

WORD

WORD

CHAR

CHAR

Statistics about word bigram

CHAR

CHAR

Knowledge about words

CHAR

Char recognizers
Hidemoto NAKADA and Yuuji ICHISUGI,
Toward Context-Dependent Robust Character Recognition using
Large-scale Restricted Bayesian Network,
TECHNICAL REPORT OF IEICE, 2017. (In Japanese)

Tested networks are trained without noise.

0.00.025
0.05
0.075
0.1
0.125
0.15
0.175
0.2

Accuracy

CHAR

2GRAM

Noise RatioRobust character recognition
utilizing context information[Nakada, Ichisugi 2017]

2GRAM
WORD

CHAR

Prior

WORD

WORD

CHAR

CHAR

Statistics about word bigram

CHAR

CHAR

Ambiguous evidence
Knowledge aboutwords
complement
eachother.
CHAR

Char recognizers
Hidemoto NAKADA and Yuuji ICHISUGI,
Toward Context-Dependent Robust Character Recognition using
Large-scale Restricted Bayesian Network,
TECHNICAL REPORT OF IEICE, 2017. (In Japanese)

Likelihood

Tested networks are trained without noise.

0.00.025
0.05
0.075
0.1
0.125
0.15
0.175
0.2

Accuracy

CHAR2GRAM

Noise Ratio

Outline• Bayesian networks and the cerebral cortex• BESOM Ver.3 and robust patternrecognition• Toward BESOM Ver.4

Problem of BESOM Ver.3• Recognition and Learning is fast
but accuracy is not good enough, probably
because conditional probability tables are
too restricted.

• We are now investigating new conditional
probability table models (BESOM Ver.4):– Noisy-OR model [Pearl 1988] and Gate-nodes.
– More expressive and fast enough.

Gate Nodes
Control NodeOpenU

Close

If Open,U and Ｘ are connected.
If Close, disconnected.

ULike inhibitory connections
on dendrites.

Gate NodeXCPT of gate nodesX¬= 1− 1−1−¬= 1−
1−=0 ,=1 ,Using matrix of gates, Control node cancontrol connections between nodes.
CU1

U2

C

U1

U2

CU1

U2

OR

Gate Matrix
X1X2

X1

X2

OR ...
X1X226

Prototyping byQuasi Bayesian Networks• We are designing prototype models of the visual areas andthe language areas using Quasi Bayesian Networks, which aresimplified Bayesian networks that only makes a distinction
between zero and non-zero of probabilities.
N2

N1

N3• Parameter learning is not supported.
• Solutions are found by SAT solver.
N3

N1

N2P(N3|N1,N2)

N3

N1

N2

P(N3|N1,N2)

False

False

False

0.2

False

False

False

non-zero

False

True

False

0.3

False

True

False

non-zero

False

False

True

0

False

False

True

zero

False

True

True

0.9

False

True

True

non-zero

True

False

False

0.8

True

False

False

non-zero...

...

...

...

...

...

...

...

True CPT

Simplified CPT ofquasi Bayesian networkPrototype of chart parser
for context free grammar
Each gate opens if theconnection forms a partof the parse tree.

Φ1,2-4

S

Φ

2,3-4

Φ

1

NP

timeVPΦ

2

V

flies

PP

3

P

like4

NP

an-arrowPrototype ofvariable unification mechanism
Inference rule(e.g. modus ponens)

Values of unified variables

AA.

Premise1.

BATHEN

Premise2-B

Connected nodes
has the same value.

B.

.

Conclusion

The network can infer not only a conclusion from the premises, but also the premises from a conclusion.
The network will be able to learn inference rules from sample data of premises and conclusions.
29
This mechanism will become key technique to implement unification grammar parser such as CCG.

Conclusion• A BESOM can be used as a bidirectional Deep
Neural Network.
– Thanks to the restricted CPT model, recognition andlearning algorithms are scalable.

• Using Gate nodes, parser and unification
mechanism would be implemented. (ongoingproject)• Future work
– Sequence learning, short-term memory– Large scale implementation30