Computational Model of theCerebral Cortex that Performs
Sparse Coding Using a Bayesian
Network and Self-Organizing MapsICONIP 20102010-11-22
National Institute of AdvancedIndustrial Science andTechnology (AIST) , Japan

Department ofComputer Science
The University of Tokyo, Japan

Yuuji Ichisugi

Haruo Hosoya

Parameter Learning ofa Cerebral Cortex Model based ona Bayesian Network
AMBN2010
2010-11-18,19
National Institute of Advanced Industrial
Science and Technology(AIST)
Yuuji Ichisugi

Abstract• Some computational neuroscientists havebegun to understand that the Bayesian
network is the essential mechanism of thecerebral cortex.
• We propose a biologically plausible
computational model that unifies a
Bayesian network model and sparsecoding model.
• This model is an extension of our previous
BESOM model [Ichisugi 2007].

Cerebral cortex• Realizes human's intelligence.
• The principle of the cortex has not been
revealed yet.

from http://en.wikipedia.org/wiki/Brodmann_area

Areas in cerebral cortex• Each area has its own function.
– Visual area, Motor area, Language area, etc.

• Bidirectional connection between areas.
Daniel J. Felleman and David C. Van Essen
Distributed Hierarchical Processing in the Primate Cerebral Cortex
Cerebral Cortex 1991 1: 1-47

Visual areas

Motor areas

Column and 6-layer structure ofcerebral cortex
macro-column(hyper-column)
500 µmminicolumn
50 µm
IIII
IIIII
IVVVI

III
IVVVI

2mm thickness

6-layer structureStructure of cerebral cortex• 10^2 areas• 10^6 macrocolumns• 10^8 minicolumns• 10^10 neurons• 10^14 synapses

From Wikipedia

Daniel J. Felleman and David C. Van Essen
Distributed Hierarchical Processing in thePrimate Cerebral Cortex
Cerebral Cortex 1991 1: 1-47

Macrocolumn
Minicolumn

IIIIIIIVVVII
II

2mm thickness

IIIIVx1

Vx2 .

VIxn.
.

w2wnw1y = φ ( ∑ wi xi )Synapsei

Output of neuron

Cerebral cortex models
based on Bayesian networks• [Lee and Mumford 2003]• [George and Hawkins 2005]• [Rao 2005]• [Ichisugi 2007]• [Rohrbein, Eggert and Korner 2008]• [Hosoya 2009]• [Litvak and Ullman 2009]• [Chikkerur, Serre, Tan and Poggio 2010]– These models try to explain the essential
mechanism of cortex, as opposed toprevious phenomenological models.

Rao's model [Rao 2005]• Bayesian network model to explainelectrophysiological phenomena.
Location

FeatureLF

Intermediate
Representati
onC

Visualinput

I

R. Rao. Bayesian inference andattention in the visual cortex.
Neuroreport 16(16), 1843-1848,
2005.

Chikkerur 's model[Chikkerur et al. 2010]
Explains electrophysiological and psychophysical phenomena.

Location
L (LIP)

Object identity
O (PFC)
Features
Fi (IT)

Location of feature Fi
Fil (V4)

Visual input
I (V2)
Chikkerur, S., T. Serre, C. Tan and T. Poggio,
What and Where: A Bayesian Inference Theory of Attention,Vision Research, 2010.

Special value that means
the feature do not exist
anywhere.

What is Bayesian network?
– Very efficient and expressive data
structure for probabilistic knowledge.
• If a joint probability table can be factored into small
conditional probability tables (CPTs), time andspace complexity will decrease.
ex.： P ( S , W , R, C ) = P (W | S , R ) P (C | R ) P ( S ) P ( R )
CPTs

SW

RC

P(S=yes)
0.2

S
nono
yes
yes

Rnoyesnoyes

P(R=yes)
0.02

P(W=yes|S,R)
0.12
0.8
0.90.98

Size 4+2+1+1=8

R P(C=yes|R)
no 0.3yes 0.995

Similarities between Cerebral
Cortex and Bayesian network– Asymmetric and bidirectional connections
between lower and higher areas.
– Local and asynchronous communications.
– Non negative values.
– Normalization of values.
– Hebb's learning rule.
– Context dependent recognition.
– Behavior based on Bayesian Statistics.

Precise correspondence
between Bayesian networks
and anatomical characteristics[Ichisugi 2007]

Belief propagation algorithm [Pearl 1988]
BEL( x) = αλ ( x)π ( x)U1π ( x) = ∑ P( x |u1 ,L , um )∏ π X (uk )u1 ,L,u mπX(uk)kλ ( x) = ∏ λY ( x)π Y ( x) = β1π ( x)∏ λY ( x)j ≠l

λ X (uk ) = β 2 ∑ λ ( x)x

Y1j

∑ P( x |u ,L, u )∏ π (u )1u1 ,L,u m / u km

Xii≠kIt's hard to be implemented by neurons.

UmλX(uk)

X

λYl(x)

πYl(x)

lll...

...

YnIn order to approximate• Assumption 1: Linear sum CPT model:– Qualitatively similar to noisy-OR model [Pearl 1988]
P( X | U1 ,L, U m )1 m= ∑ P( X | U i )m i =1• Assumption 2: Nodes have many parentand child nodes.

Messages of BP exclude
information from their targetλX(Uk)
U1

...

Uk

...

Um

U1

...

xi

Y1

...

Uk

...

Um

xi

Yl...

Yn

Y1...

πYl(x)Yl...

Yn• If there are many parents and children, theseinformation may be included.

Example: πYl(x) approximation• An messageπYl(x) from node X to node Yl
may include information λYl(x) from Yl .
π Y ( x)l= π ( x)∏ λY j ( x)

U1...

Uk...

Umj ≠lxi≈ π ( x)∏ λY j ( x)j= λ ( x)π ( x)Y1...
πYl(x)Yl...

Yn

Approx. Belief Propagation[Ichisugi 2007]+1l tXY= z Yt + WXY oYt

ot +1
X

Approximates Pearl's algorithm[Pearl 1988]
with some appropriate assumptions.

⊗t +1l∏ XY=Y ∈children ( X )
t +1
Tk UX
= WUXbUt

pt +1
X

=UbU

∑kt +1UX
U ∈ parents ( X )bX=ot +1X= ∑ (r ) (= rZ⊗pt +1Xt +1Xii

Z X , oX

Xt +1X

rt +1X

WUXt +1X 1=ot +1Xt +1X•p )z tX+1 = ( Z Xt +1 , Z Xt +1 , L , Z Xt +1 )T

WXY

Z Y , οYY

- Easy to be implementedby neurons.
t +1t +1t +1b X = (1 / Z X )rX- Linear time complexity
where x ⊗ y = ( x y , x y , L, x y )T in sparse network.
1 122nn

Connections between cortical layers[Gilbert 1983]

IIIIIIIVVVI
Lower Area

Higher AreaAnatomical structure[Pandya and Yeterian 1985]
Gilbert, C.D., Microcircuitry of the visual-cortex, Annual review ofneuroscience, 6: 217-247, 1983.
Pandya, D.N. and Yeterian, E.H., Architecture and connections of corticalassociation areas. In: Peters A, Jones EG, eds. Cerebral Cortex (Vol. 4):
Association and Auditory Cortices. New York: Plenum Press, 3-61, 1985.

kUXoX
lUX
bU

k XYoYl XYZXbXThis model

ZYIIIIIIIVVVIDetailed structure in columns(bU1 )1(bU1 ) 2I

(bU 2 ) 3(bU 2 )1+(bU 2 ) 2+(kU1 X )1(kU1 X ) 2+(bU 2 ) 3++( p X )1(o X )1II+(kU 2 X )1(kU 2 X ) 2(o X ) 2(rX )1(l XY1 ) 2++(l XY2 ) 2Z Y1(oY1 )1ZXZX(bX )1Y1(oY1 ) 3+Z Y2(oY2 )1/

X

III(rX ) 2(oY1 ) 2+U2( pX )2(l XY1 )1(l XY2 )1U1IV(oY2 ) 2/(oY2 ) 3(b X ) 2+(o X )1(bX )1(o X ) 2(b X ) 2VVIY2The left circuit
calculates values of
two units, x1 and x2,in node X in the abovenetwork.

Column structure of cortex
Many cells in layer 2, 4
BEL(U 1 = u11 )BEL(U 1 = u12 )I

BEL(U 1 = u31 )BEL(U 2 = u )2
1BEL(U 2 = u 22 )++

κU ( X = x1 )

κ U ( X = x2 )+1

BEL(U 2 = u32 )+κ U ( X = x1 )
2+π ( X = x1 )λ ( X = x1 )+1IIκ U ( X = x2 )
2π ( X = x2 )

ρ ( X = x1 )λ ( X = x2 )λY ( X = x1 )λY ( X = x2 )++1IIIρ ( X = x2 )Z Y1

1λ (Y1 = y11 )λ (Y1 = y12 )λY ( X = x1 )2+λ (Y1 = y31 )λY ( X = x2 )+2

Z Y2λ (Y2 = y12 )ZX//

BEL( X = x1 )BEL( X = x2 )λ (Y2 = y )λ (Y2 = y32 )+ZX
λ ( X = x1 )λ ( X = x2 )IV22BEL( X = x1 )BEL( X = x2 )K. Brodmann, Vergleichende Lokalisation der Grosshirnrinde. in:ihren Prinzipien dargestellt auf Grund des Zellenbaues,. J.A.
Barth, Leipzig, 1909.
This figure is taken from the following Web page.
http://web.sc.itc.keio.ac.jp/anatomy/brodal/chapter12.html

VVIHorizontal
fibers inlayer 1, 4

BESOM model [Ichisugi 2007] unifies someprevious models and Bayesian network
Malsburg model[Malsburg 1973]

Neocognitron[Fukushima 1980]

SOM[Kohonen 1995]

Selective attention
model[Fukushima 1987]

Bayesian network[Pearl 1988]

BESOM[Ichisugi 2007]

BESOM (BidirEctional SOM)
• Each node is a competitive layer of a SOM.
• Each unit represents a value of therandom variable.
Unit

NodeU1...

Uk... U m

Node... Yn

NodeXX
Y1...

YlYlBrain and BESOM model [Ichisugi 2007]
The cerebral cortex isa Bayesian network of10^6 nodes with 10^2states.

Basis

NodeFrmo Wikipedia...

BrainBESOM...

Daniel J. Felleman and David C. Van Essen
Distributed Hierarchical Processing in thePrimate Cerebral Cortex
Cerebral Cortex 1991 1: 1-47

Cortical column

Cerebral cortex

BESOM network

Area hierarchy

Basis hierarchy

AreaBasis

Cortical column

NodeMinicolumn

UnitNeuronVariableSynapse

Weight ofconnection

...

...

...

Node

Unit

minicolumn

IIIIIIIVVVIIIIUnit(bU1 )1(bU1 ) 2I(bU 2 ) 3(bU 2 )12mm thickness+(bU 2 ) 2+

( kU1 X )1+(bU 2 ) 3+

III
x1

Vx2 .

VIxn.
.

w2
wn

II+

( kU 2 X )1+( p X )1(o X )1IV

( kU1 X ) 2( kU 2 X ) 2( pX )2(o X ) 2( rX )1( l XY1 )1( l XY1 ) 2++III( rX ) 2ZY1(oY1 )1(oY1 ) 2w1( l XY2 )1+( l XY2 ) 2(oY1 ) 3+Z Y2(oY2 )1

y = φ ( ∑ wi xi )Synapse/ZXi

Output from neuron

ZX(bX )1IV(oY2 ) 2/(oY2 ) 3(bX ) 2+(o X )1(b X )1(o X ) 2(b X ) 2VVI

BESOM model andSparse-coding model
Malsburg model[Malsburg 1973]

Neocognitron[Fukushima 1980]

SOM[Kohonen 1995]

Selective attention
model[Fukushima 1987]

Bayesian network[Pearl 1988]

BESOM[Ichisugi 2007]

Sparse-coding model[Olshausen andField 1996]

Sparse-coding[Olshausen and Field 1996]• Sparse-coding of natural images
reproduces orientation selectivity of V1
simple-cells.
y (t ) = Wx(t )mE =|| x(t ) − W y (t ) || + ∑ | yi (t ) |T2i =1“Emergence of simple-cell receptive field properties
by learning a sparse code for natural images”.
Bruno A. Olshausen and David J. Field
Nature, 381:607-609 (1996)
This figure is copied from:http://watanabe-www.pi.titech.ac.jp/~shiga/ppt/ICA.ppt

What is sparse-coding ?
• A kind of unsupervised learning whose
goal is to express inputs using small
number of basis vectors.
• Computational merits:– Data compression.
– Avoids "curse of dimension."– Blind source separation.

• Biological merits:– Saves energy, synapse maintenance cost.

Idea of sparse-coding byBESOM• Nodes may become "inactive" state.
• Only small number of nodes become
active.
Active node

Input 1

Inactive node thatacts as if it does not exist.

Input 2

Input 3

InputBESOM network for sparse codingSpecial unit xφthat indicate thenode is inactive

Fixed to P( yi | xφ ) = P ( yi )NodeHidden layer

Hidden layer

Unit
Inputlayer

Inputlayer

...

...

Input (observed data) is given at the lowest layer.

Recognition

Learning
Inactivenodes

Hidden layer

Inputlayer

Hidden layer

Inputlayer...
Calculate MPE with "inactive bias."...
Increase the connection weights for active units.

Result of learning natural images

Randomly selecta 12x12 pixel image.
Preprocessed natural image.

We used the image database provided by Olshausen
http://redwood.berkeley.edu/bruno/sparsenet/

Approximate the input
with linear sum of about
zero to two basis images

Winner units
learns theinput image(withneighborhoodlearning)Summary of BESOM sparse-coding• A special value is introduced to each nodethat means the node is "inactive."• Two cerebral cortex models, Bayesian
network model and Sparse-coding model,
can be unified to a single model.
– The learning algorithm does not break thetheoretical framework of Bayesian networks.

• Learned basis images show orientation
selectivity, as in the primary visual area.

Take home message• The cerebral cortex is a Bayesian network.
• However, most neuroscientist do not know what
Bayesian networks are.

• Many cerebral models are being integrated
into one universal model based on Bayesiannetworks.
– Such a model will become the core technology forreproducing human-like high intelligence.

• Computational neuroscience needs
Bayesian network experts!

Additional slides

Reproducing contrast responses[Reynolds and Heeger 2009] model clearly reproduces very complicatedelectrophysiological phenomena, however, it contains a mysterious constant σ .

A( x, θ ) E ( x, θ )R ( x, θ ) =S ( x, θ ) + σ
The normalization model of attention(Reynolds JH, Heeger DJ, Neuron. 2009 Jan 29;61(2):168-85)

On the other hand, [Chikkerur et al. 2010] model reproduces some of these
phenomena very naturally.

P ( I | Fl i ) P ( Fl i )P( Fl | I ) =ii
P(I|F)
P(F∑ll )iJust applies Bayes' rule.

Fli= LP ( I | Fl i ) P( Fl i )iiiiP(I|F=i)P(F=i)+P(I|F=0)P(F∑llll = 0)i =1This term may cause the same effect as σ .

BESOM may be used
for pattern recognition• Similar structure toNeocognitoron[Fukushima 1980]

A• Lowest nodes
represents observed
values.

BCD… Recognizedpattern…

…

Hyper column in V1

…

Input

Simple formalizationof BESOM model• Objective of learning:– Calculate MAP estimator of the parameter θassuming each input i (t ) at time t is generated
from i.i.d.
⎡ t⎤θ * = arg max ⎢∏ P (i (i ) | θ )⎥ P(θ )θ⎣ i =1⎦⎡ t⎤= arg max ⎢∏∑ P(h, i (i ) | θ )⎥ P(θ )θ⎣ i =1 h⎦

Recognition and Learning steps)
Recognition step: h(t ) = arg max P (h, i (t ) | θ (t ))h)⎤⎡ tLearning step: θ (t + 1) = arg max ⎢∏ P (h(i ), i (i ) | θ )⎥ P (θ )θ⎦⎣ i =1P(h, i | θ ) : Probabilistic model. (Bayesian network.)
P (θ ) ：Innate knowledge about the parameter,
such as sparseness.

)h(t ) ：States of cortical columns. 10^6 dim.
i (t ) ：Input vector. 10^4 dim.?
θ (t ) ：All weights of variable synapses. 10^16 dim.

Learning conditional probabilities
with Hebb's rule[Ichisugi 2007]
Learning rule for unit xi :xiwijl ← wijl + α i (v lj − wijl )The weight is ML estimatorof the conditional probability.

wijl = P ( y lj | xi )wijly lj

v1=(0, 0, 1, 0, 0) v2=(0, 1, 0, 0, 0) v3=(0, 0, 0, 0, 1)

Structure of BESOM network
Node = random variable = cortical column

Input

unit
= value= minicolumn

(V2)

(V1)

(LGN)

No connections in each layer.
Fully connected between different layers.

Recognition

Find the values of hidden variables
with the highest posterior probability.
(MPE: most probable explanation)

Connection weights= CPT= synapase weights

Input (observed data) is given at the lowest layer.

Learning

Increase the connection weights between active units(mini-columns) and decrease the other weights.

How the network structure ofBayesian net is learned ?
- My speculation -Generative model can beacquired by ICA• In other words, ICA may acquire twolayered Bayesian network structure.

Y1

Y2

Y3Source signal

x=f(y)X1

X2

X3

Input signal

Hierarchical generative model• Hierarchical ICA may acquire multi-layered
Bayesian network structure.

Y1

Y2

Y3

X1

X2

X3
Other retinal areas or modalities.

How about motor areas ?

Reinforcement learning
in motor areas• Nodes acquire state-action pairs. Statevalues are learned by synapses connect tobasal ganglia.
(s,a)

s

Basal gangliaa• Matches the anatomical structure: corticobasal ganglia loop.
– This interpretation is an extension of Doya's model:
K. Doya, Complementary roles of basal ganglia and cerebellum in learning and motorcontrol, Current Opinion in Neurobiology 10 (6): 732-739 Dec 2000.

Effect of SparsenessIf no sparseness,orientation selectivity ofbasis images become
weak because everybase image becomesclose to the mean imageof input images.

Number of used units
to approximate
input image.

0 - 2 units

4 units(No sparseness)Learning natural images• Input:– We extracted image patch with 7x7=49 pixels
from a random position.
– Then, we gave the pixel intensities in theimage patch to the 49 binary input nodes(Y l ∈ {0,1}, l = 1L 49) .
• For example, for intensity 0.2, the value was set to
1 with probability 0.2.

• Visualization of CPT:P(Y l = 1 | X = xi )– 7x7 CPT elements ofarevisualized as the brightness of 7x7 pixels.

Brain is now understandable• because of remarkable progress of computerscience and neuroscience in recent 20 years.
– Maturity of AI and machine learning technology– Bayesian network [Pearl 1988]– Reinforcement learning [Sutton 1998]– Independent Component Analysis [Hyvarinen 2001]– Important findings of neuroscience– Sparse-coding at primary visual area [Olshausen 1996]– Reinforcement learning at basal ganglia [Schultz 1997]– Bayesian network models of cerebral cortex [Rao 2005] etc.