Deep Restricted Bayesian
Network BESOM...

2018-02-27
Yuuji Ichisugi

...

...

...

。。。

...

...

...

...

。。。

...

。。。

Artificial Intelligence Research Center (AIRC),
National Institute of Advanced Industrial Scienceand Technology (AIST), Japan
1

BESOM (BidirEctional Self Organizing Maps) [Ichisug 2007]• A computational model of the cerebral cortex– A model of column network, not spiking neurons• Design goals:– Scalability of computation– Usefulness as a machine learning system– Plausibility as a neuroscientific model• As a long-term goal, we aim to reproducefunctions of such as the visual areas and thelanguage areas using this cerebral cortex
model.
2

Architecture of BESOM model
Recognition step: The entire network behaves like a Bayesian network.
Learning step: Each node behaves like a Self-organizing map.
Node = random variable = macro-column
Unit = value of random variable = minicolumn
(V2)

(V1)

(LGN)

Outline• Bayesian networks and the cerebral cortex• BESOM Ver.3 and robust pattern recognition• Toward BESOM Ver.4

Models of visual cortex
based on Bayesian networks• Various functions, illusions, neural responses and anatomical
structure of the visual cortex were reproduced by Bayesian
network models.
– [Tai Sing Lee and Mumford 2003]– [George and Hawkins 2005]– [Rao 2005]– [Ichisugi 2007]– [Litvak and Ullman 2009]– [Chikkerur, Serre, Tan and Poggio 2010]– [Hosoya 2012]– ...

The visual cortex seems to be a huge Bayesian networkwith layered structure like Deep Neural Networks.

What is Bayesian networks?
– Efficient and expressive data structure ofprobabilistic knowledge [Perl 1988]• Various probabilistic inference can be executed
efficiently if a joint probability table can be factored
into small conditional probability tables (CPTs).

P( S , W , R, C ) = P(W | S , R) P(C | R) P( S ) P( R)
CPTs

S

W

R

C

P(S=yes)
0.2
S
no
no
yes
yes

R
no
yes
no
yes

P(R=yes)
0.02
P(W=yes|S,R)
0.12
0.8
0.9
0.98

R P(C=yes|R)
no 0.3yes 0.995

6

Loopy Belief Propagation[Murphy, Weiss, Jordan 1999]• Efficient approximate inference algorithm– Iterative algorithm with local andasynchronous computation, like brain.
– Although there is no guarantee of
convergence, it is empirically accurate.
BEL( x) = αλ ( x)π ( x)U1

π ( x) = ∑ P ( x |u1 ,L , um )∏ π X (uk )u1 ,L ,u m...

λX(uk)πX(uk)kλ ( x) = ∏ λY ( x)

X

l

l

π Y ( x) = π ( x)∏ λY ( x)lj ≠l

λ X (uk ) = ∑ λ ( x)xj

∑ P( x |u ,L , u )∏ π (u )1u1 ,L ,u m / u k

m

X

i≠k

i

πYl(x)Y1

Um

λYl(x)...

Yn 7

Belief propagation and micro
circuit of cerebral cortex• The similarity between belief propagationand the six-layer structure of the cerebral
cortex has been pointed out many times.
[George and Hawkins 2005][Ichisugi 2007][Rohrbein, Eggert and Korner 2008][Litvak and Ullman 2009]

I
II
III
IV
V
VI

8

Approximate Belief Propagation+1= z Yt + WXY oYt
l tXY

o

t +1
X

=

[Ichisugi 2007]⊗∏lApproximates Pearl's algorithm[Pearl 1988]
with some assumptions.
1 𝑚𝑚
𝑃𝑃 𝑥𝑥 𝑢𝑢1 , … , 𝑢𝑢𝑚𝑚 = Σ𝑘𝑘=1 𝑃𝑃 𝑥𝑥 𝑢𝑢𝑘𝑘
𝑚𝑚

t +1
XY
Y ∈children ( X )

t +1
T
= WUX
k UX
bUt

p tX+1 =

t +1
k
∑ UX

U

bU

U ∈ parents ( X )rXt +1 = o tX+1 ⊗ p tX+1
Z

t +1
X

i

Z X , oXX= ∑ (r ) (= r
t +1
X
i

WUX

t +1
X 1

=o

t +1
X

t +1
X

•p )

bX

WXY

Z Y , οYYz tX+1 = ( Z Xt +1 , Z Xt +1 , L , Z Xt +1 )T
b tX+1 = (1 / Z Xt +1 )rXt +1
where x ⊗ y = ( x1 y1 , x2 y2 , L , xn yn )TYuuji ICHISUGI, "The cerebral cortex model that self-organizes
conditional probability tables and executes belief propagation", Inproc. of IJCNN2007, Aug 2007.

Similarity in information flow
I
II
III
IV
V
VI

[Gilbert 1983][Pandya and Yeterian 1985]
Gilbert, C.D., Microcircuitry of the visual-cortex, Annual review ofneuroscience, 6: 217-247, 1983.
Pandya, D.N. and Yeterian, E.H., Architecture and connections of corticalassociation areas. In: Peters A, Jones EG, eds. Cerebral Cortex (Vol. 4):
Association and Auditory Cortices. New York: Plenum Press, 3-61, 1985.

Lower Area

Higher Area

Anatomical structure

+1
l tXY
= z Yt + WXY oYt

o

t +1
X

=

⊗

∏l

t +1
XY
Y ∈children ( X )

kUX
oX

t +1
T
k UX
= WUX
bUt

p tX+1 =lUXbU∑ k UXt +1

U ∈ parents ( X )rXt +1 = o tX+1 ⊗ p tX+1
Z

t +1
X

= ∑ (r ) (= r
t +1
X
i

i

t +1
X 1

=o

t +1
X

t +1
X

•p )

z tX+1 = ( Z Xt +1 , Z Xt +1 , L , Z Xt +1 )T
b tX+1 = (1 / Z Xt +1 )rXt +1
where x ⊗ y = ( x1 y1 , x2 y2 , L , xn yn )TParent nodes

k XY
oY

l XY
ZX

bX

ZY

Child nodes

Information flow of the approx. BP
The intermediate variables of this algorithm can be assigned
to each layer of the cerebral cortex without contradicting the
known anatomical structure.

Detailed circuit that calculates
the approximate BP(bU1 )1(bU1 ) 2(bU 2 ) 3
(bU 2 )1(bU 2 ) 2

(kU1 X )1(bU 2 ) 3

(kU 2 X ) 2(kU 2 X )1(l XY1 )1

U2( pX )2( p X )1(o X )1

U1(kU1 X ) 2+(o X ) 2(rX )1X(rX ) 2Z Y1(l XY1 ) 2(oY1 )1(oY1 ) 2(l XY2 )1(oY1 ) 3(l XY2 ) 2

Z Y2(oY2 )1(oY2 ) 2

ZX(bX )1(oY2 ) 3(b X ) 2

ZX(o X )1(bX )1(o X ) 2(b X ) 2

Y1

Y2

The left circuit
calculates values of
two units, x1 and x2,
in node X in the above
network.

Correspondencewith local cortical circuit
Mini-column likestructure(bU1 )1(bU1 ) 2(bU 2 ) 3
(bU 2 )1(bU 2 ) 2

(kU1 X )1(kU1 X ) 2+(bU 2 ) 3

(kU 2 X )1

( p X )1(o X )1(l XY1 )1

Many horizontal
fibers in I, IV(kU 2 X ) 2( pX )2(o X ) 2(rX )1(rX ) 2(l XY1 ) 2

Z Y1

I
II

(oY1 )1

III(oY1 ) 2(l XY2 )1IV(oY1 ) 3(l XY2 ) 2

V

Z Y2

VI

(oY2 )1(oY2 ) 2
ZX(bX )1(oY2 ) 3(b X ) 2

ZX(o X )1(bX )1(o X ) 2(b X ) 2

Many cells
in II, IV

Outline• Bayesian networks and the cerebral cortex• BESOM Ver.3 and robust patternrecognition• Toward BESOM Ver.4

Toward realizationof the brain function• If the cerebral cortex is a kind of Bayesian
network, we should be able to reproducefunction and performance of it using
Bayesian networks.
– As a first step, we aim to reproduce some partof the functions of the visual areas and thelanguage areas.
– Although there were some difficulties such as
computational cost and local minimum
problem, now they have been solved
considerably.

BESOM Ver.3.0 features• Restricted Conditional Probability Tables:
1 𝑚𝑚
𝑃𝑃 𝑥𝑥 𝑢𝑢1 , … , 𝑢𝑢𝑚𝑚 = Σ𝑘𝑘=1 𝑃𝑃 𝑥𝑥 𝑢𝑢𝑘𝑘𝑚𝑚
• Scalable recognition algorithm OOBP [Ichisugi, Takahashi 2015]• Regularization methods to avoid local minima– Win-rate and Lateral-inhibition penalty [Ichisugi, Sano 2016]– Neighborhood learning
Computational amount of
one step of iteration of
OOBP is linear to thenumber of edges of the
network.
Recognition algorithm OOBP

Yuuji Ichisugi and Naoto Takahashi,
An Efficient Recognition Algorithm for Restricted
Bayesian Networks, In proc. of IJCNN 2015.
Yuuji Ichisugi and Takashi Sano,
Regularization Methods for the Restricted Bayesian
Network BESOM, In Proc. of ICONIP2016, Part I, LNCS
9947, pp.290--299, 2016.

15

The design of BESOM is motivatedby two neuroscientific facts.
１．Each macro-column seems to be
like a SOM.
２．A macro-column at a upper area
receives the output of the macrocolumns at the lower area.

mini
columnsV4...
macro-columns

V2

...
V1

...
16

If a SOM receives input from other SOMs,
they naturally become a Bayesian Network
Learning rule (without neighborhood learning)wij ← wij + α ( y j − wij )𝑤𝑤𝑖𝑖𝑖𝑖 converges to the probability
that 𝑦𝑦𝑗𝑗 fires when 𝑥𝑥𝑖𝑖 fires, thatis, the conditional probability
𝑃𝑃(𝑦𝑦𝑗𝑗 |𝑥𝑥𝑖𝑖 ).
y=(0, 0, 1, 0, 0,

xi

wij

0, 1, 0, 0, 0,

yj

0, 0, 0, 0, 1)T17

InputNode = random variable = macro-column
unit= value= minicolumn

(V2)

(V1)

(LGN)

Connection weights= Conditional probabilities= synapse weights

Input (observed data) is given
at the lowest layer.

Recognition

Learning

increased

Find the values with the highest posterior
probability. (MAP)

decreased

Updates connection weights
with Hebb's rule.

18

Connections oflong distancelateral inhibition
Hidden layer

Input[Ichisugi, Sano 2016]

...

...

Input layer

Yuuji Ichisugi and Takashi Sano,
Regularization Methods for the Restricted Bayesian Network BESOM,In Proc. of ICONIP2016, Part I, LNCS 9947, pp.290--299, 2016.

Recognition...

Units that receive strong lateral inhibition
are less likely to become winners.

Input (observed data) is given at thelowest layer.

Learning...

Updates connection weights
with Hebb's rule.

BESOM can be used as if
it were a Deep neural network.
A

B

C

D

… RecognizedFeatures

…

…

…

Input

BESOM can be used as if
it were a bidirectional Deep neuralnetwork.
A

B

C

D

… RecognizedFeatures
Prediction= Prior
…

…

…

Input

Robust character recognition
utilizing context information[Nakada, Ichisugi 2018]

2GRAM

WORD

CHAR

WORD

WORD

CHAR

CHAR

Statistics about word bigram

CHAR

CHAR

Knowledge about words

CHAR

Char recognizers
Hidemoto Nakada and Yuuji Ichisugi,Context-Dependent Robust Text Recognition using Large-scale
Restricted Bayesian Network, In Proc. of BICA 2017, Procedia
Computer Science, Vol. 123, pp.314--320, 2018.

Tested networks are trained without noise.

0.0
0.025
0.05
0.075
0.1
0.125
0.15
0.175
0.2

Accuracy

CHAR

2GRAM

Noise RatioRobust character recognition
utilizing context information[Nakada, Ichisugi 2018]

2GRAM

WORD

CHAR

Prior

WORD

WORD

CHAR

CHAR

Statistics about word bigram

CHAR

CHAR

Ambiguous evidence
Knowledge about
words
complement
each
other.
CHAR

Char recognizers
Hidemoto Nakada and Yuuji Ichisugi,Context-Dependent Robust Text Recognition using Large-scale
Restricted Bayesian Network, In Proc. of BICA 2017, Procedia
Computer Science, Vol. 123, pp.314--320, 2018.

Likelihood

Tested networks are trained without noise.

0.0
0.025
0.05
0.075
0.1
0.125
0.15
0.175
0.2

Accuracy

CHAR

2GRAM

Noise Ratio

Outline• Bayesian networks and the cerebral cortex• BESOM Ver.3 and robust patternrecognition• Toward BESOM Ver.4

Problem of BESOM Ver.3• Recognition and Learning is fast
but accuracy is not good enough, probably
because conditional probability tables are
too restricted.
1 𝑚𝑚
𝑃𝑃 𝑥𝑥 𝑢𝑢1 , … , 𝑢𝑢𝑚𝑚 = Σ𝑘𝑘=1 𝑃𝑃 𝑥𝑥 𝑢𝑢𝑘𝑘𝑚𝑚
• We are now investigating new conditional
probability table models (BESOM Ver.4):– Noisy-OR model [Pearl 1988] and Gate-nodes.
– More expressive and fast enough.

Gate Nodes
Control Node
Open

U

Close

If Open,U and Ｘ are connected.
If Close, disconnected.

U

Like inhibitory connections
on dendrites.

Gate NodeCPT of gate nodes

X

𝑃𝑃 𝐺𝐺 = 0 𝑐𝑐, 𝑢𝑢 = 1 − 1 − 𝑤𝑤𝑢𝑢 ¬𝑢𝑢 1 − 𝑤𝑤𝑐𝑐 c
𝑃𝑃 𝐺𝐺 = 1 𝑐𝑐, 𝑢𝑢 = 1 − 𝑤𝑤𝑢𝑢 ¬𝑢𝑢 1 − 𝑤𝑤𝑐𝑐 𝑐𝑐

X

Using matrix of gates, Control node cancontrol connections between nodes.
C

U1

U2

C

U1

U2

C

U1

U2

OR

Gate Matrix
X1

X2

X1

X2

OR ...
X1

X2
26

Prototyping by
Quasi Bayesian Networks• We are designing prototype models of the visual areas andthe language areas using Quasi Bayesian Networks, which aresimplified Bayesian networks that only makes a distinction
between zero and non-zero of probabilities.
N2

N1

N3

• Parameter learning is not supported.
• Solutions are found by SAT solver.
N3

N1

N2

P(N3|N1,N2)

N3

N1

N2

P(N3|N1,N2)

False

False

False

0.2

False

False

False

non-zero

False

True

False

0.3

False

True

False

non-zero

False

False

True

0

False

False

True

zero

False

True

True

0.9

False

True

True

non-zero

True

False

False

0.8

True

False

False

non-zero

...

...

...

...

...

...

...

...

True CPT

Simplified CPT ofquasi Bayesian network

Chart parser
for context free grammar[Takahashi, Ichisugi 2017]

Each gate opens if theconnection forms a partof the parse tree.

Φ

1,2-4

S

Φ

2,3-4

Φ

1

NP

time

Naoto Takahashi and Yuuji Ichisugi,
Restricted Quasi Bayesian Networks as a Prototyping Tool for
Computational Models of Individual Cortical Areas, In Proc. of
AMBN 2017, Proceedings of Machine Learning Research,
Vol.73, pp.188--199, 2017.

VP

Φ

2

V

flies

PP

3

P

like

4

NP

an-arrowPrototype of
logical inference mechanism
Inference rule(e.g. modus ponens)
𝑋𝑋, 𝑋𝑋 → 𝑌𝑌 ⊢ 𝑌𝑌

A.

Premise1.

Values of unified variables

A

B

A

THEN

Premise2-

B

Connected nodes
has the same value.

B

.

.

Conclusion

The network can infer not only a conclusion from the premises, but also the premises from a conclusion.
The network will be able to learn inference rules from sample data of premises and conclusions.
29
This mechanism will become key technique to implement unification grammar parser such as CCG.

A formal model of language areas
BA44,45

Addresses,
Sem.Categ.

Parser

Sem.Repl.
BA22

Words

Gates

Dict.

black black black black
black black black cat
black black black fish
black black black eat
black black black isEatenBy
black black cat black
black black cat cat
black black cat fish...

・ Modified version of Combinatory
Categorial Grammar [Steedman 2000].
・ This model may be implemented
by BESOM Ver.4.
・ Destruction of parts of this model
reproduces behavior of aphasia.
white dog eat dog
white dog eat cat
white dog eat fish
white dog chase dog
white dog chase cat
white dog chase fish
white dog isEatenBy dog...

Sentences are syntactically incorrect.
Selected words are semantically correct.

Sentences are syntactically correct.
Selected words are semantically incorrect.

Resembles Broca's aphasia

Resembles Werniche's aphasia

Conclusion• A BESOM can be used as a bidirectional Deep
Neural Network.
– Thanks to the restricted CPT model, recognition andlearning algorithms are scalable.

• Using Gate nodes, parser and logical inference
mechanism would be implemented. (ongoingproject)• Future work
– Model of PFC and ACC– Sequence learning, short-term memory– Large scale implementation31