A Cerebral Cortex Model that Self-Organizes Conditional
Probability Tables and Executes Belief Propagation
Yuuji ICHISUGI

Abstract—This paper describes a neural network model of
cerebral cortex, BESOM model, that acquires conditional
probability tables for a Bayesian network using self-organizing
maps and estimates states of random variables with anapproximate belief propagation algorithm. The approximate
algorithm is derived from some assumptions. A neural networkthat executes the derived algorithm is in good agreement with
six-layer and column structures that represent the anatomical
characteristics of a cerebral cortex in many respects. This model
has scalable time and space complexities and is therefore
qualified to be a model of the brain, a large-scale information
processor.

I. INTRODUCTION
Through intensive neuroscience studies in recent years, theroles of individual areas in the cerebral cortex and enormous
amounts of information on the anatomical structure of areaconnections have been accumulated and organized. However,understanding of the cortex is not enough to reproduce themain functions of the cortex on a computer.
Self-organizing map (SOM) [1] is a model that
reproduces some cortex functions on a computer. SOM is amachine learning algorithm that is characterized bycompetitive learning and neighborhood learning. An SOM
can achieve clustering high-dimensional input under
unsupervised conditions and online. This agrees with one of
the features of cerebral information processing. The column
structure seen in the primary visual cortex and other areas isevidence that suggests that the cortex is a sort of SOM.
A Bayesian network[2] has also been used as a model forthe cortex[6]. It is a graphical model that represents a directed
acyclic graph of causal relations between random variables.
Observations of some random variables within the network
allow for estimation of the values of the remaining randomvariables based on conditional probability tables. For thisestimation, an efficient calculation method, called a belief
propagation algorithm, is used. This network has features
such as noise-resistant pattern recognition, ambiguous
information-based plausible inference and real-time
operation that are also in agreement with features of cerebral
information processing. The cortex areas form abidirectionally linked network. This structure is similar to thatof the Bayesian network.
One visual cortex model that has a mechanism similar toboth SOM and a Bayesian network is the Selective Attention
Model (SAM)[3]. The SAM, a hierarchical combination of
Yuuji ICHISUGI is with National Institute of Advanced Industrial
Science and Technology(AIST), Tsukuba Central 2, Tsukuba, Ibaraki
305-8568, Japan. (e-mail: y-ichisugi@aist.go.jp)

competitive learning nodes, has bidirectional signal routes, abottom-up signal that sends the results of individual nodes’
recognition, and a top-down one that sends predictions basedon past experiences and contexts. The SAM reproduces someof the features of cerebral visual information processing suchas noise-resistant recognition and target segmentation, and isconsidered to be a plausible model for a cerebral cortex.
In addition to the SAM, there are models for the cortex inwhich a top-down signal represents predictions [4][5][6].
These models, however, have no self-organizing mechanism.
Traditional models also have the issue of scalability. The
cortex is a large-scale information processor that consists of
14 billion neurons and should use a computational algorithm
that runs at a realistic speed through parallel processing.
However, the order of time and space complexity is notdiscussed in traditional models.
In this paper, we propose a BESOM (BidirEctional SOM)
as a model for the cortex that has both the features of the
SOM and the Bayesian network. The BESOM is ahierarchical SOM with bidirectional connections. Each SOM
is used to self-organize and obtain a conditional probability
table for the network. The BESOM replaces the SAM’slearning and recognition calculating formula with one basedon a belief propagation algorithm, thereby providing afooting for further expansion of the model and for efficient
calculation.
This paper consists of five subsequent sections. Section II
provides an overview of the BESOM model’s architecture.
Sections III and IV detail the learning and recognition steps,
respectively. In Section IV, scalability is also discussed.
Section V describes the agreement between this model andthe anatomical features of a cerebral cortex. Section VI
provides conclusions.
II.

ARCHITECTURE

The BESOM consists of nodes that are connected in theform of a directed acyclic graph. Each node consists ofseveral units. If two nodes are linked by an edge, the units
included in each node are connected in a complete bipartite
graph (Fig. 1). Each unit connection has a weight that varies
with learning. (Section IV describes the structure of units
more accurately.) It was assumed that a network structure
does not vary with learning and is given as a prior knowledge.
The BESOM repeats the learning step and the recognition
step alternately. A learning step updates weights ofconnections based on the result of the previous recognition
step. A recognition step calculates outputs of nodes, which
represent the state of the world, based on the observed values

Node X

Node

U1

Unit

...

Uk

... U m

Node

Y1

X

Y2

Y3

v1=(0, 0, 1, 0, 0) v2= (0, 1, 0, 0, 0) v3=(0, 0, 0, 0, 1)

X
Y1

...

Yl

... YnNode

YlFig. 1. BESOM Architecture. See Fig. 6 for unit structure and detailed
connections among units.

and the current conditional probability tables.
In the learning step, each node works as a SOM’scompetitive layer. Each node clusters input vectors that aresent from child nodes. The result of the SOM’s learning canbe regarded as a conditional probability, as described in thenext section. The updated conditional probabilities are used at
the next recognition step. The network of SOMs form ahierarchical structure such that SOMs at the higher layers of a
network express more abstract information that compresses
more input information than those at the lower layers.
In the recognition step, a network of nodes works as a
Bayesian network. Each node represents a random variable.
Each unit included in one node corresponds to a possible
value of the random variable. External inputs from sensors(observed values) are given as outputs of the lowest nodes(nodes that have no child nodes). In the recognition step, in
accordance with external inputs and each node’s conditional
probability table, an approximate belief propagation
algorithm (described in Section IV) is used to compute
posterior probabilities of individual variables. The results areused in the next learning step.
In terms of a model for a cerebral cortex, the BESOM isinterpreted as follows. The BESOM’s nodes and units areequivalent to the hyper-columns and columns of the cortex. In
the primary visual cortex, individual units are equivalent to
the orientation columns.
III.

LEARNING STEP

In the learning step, each node works as an SOM
competitive layer and clusters inputs from its child nodes.
Assume that Node X has n child nodes Yl (l = 1,L,n) .
In this step, the SOM receives estimated values of theindividual child nodes as inputs. Given that a estimatedllposterior probability of Node Yl ’s unit y i is BEL(y i ) , anlelement of an input vector v from Yl is expressed asfollows.

Fig. 2. Input vectors to the SOM in the learning step. Each black unit
corresponds to the value with maximum posterior probability.

⎧1 (if BEL( y lj ) = max i BEL( yil ))v =⎨⎩0 (otherwise)
l
j

(1)

Namely, an element corresponding to a unit with amaximum posterior probability is 1, while other elements are
0 (Fig. 2).
The estimated value of Node X becomes a winner forcompetitive learning. In the winner unit, a reference vector (avector of weights) is brought close to an input vector.
Assuming that the weight of the connection between Node
X ’s winner unit x i and Node Yl ’s unit y lj is w ijl , thisweight is then updated with the following rule:wijl ← (1 − α ) wijl + α v lj

(2)

where α is a learning rate. In this step, neighborhood
learning should also be done using a proper neighborhood
function.
Here, given that the neighborhood radius is sufficiently
small and negligible, and a learning rate α is equal to 1/ nlfor n-th learning of each unit, a weight w ij is then equal to aconditional probability P (Yl = y j | X = x i ) . (For details,lsee the appendix.)
IV.

RECOGNITION STEP

In the recognition step, an approximate belief propagation
algorithm is executed to estimate values of the individual
random variables. This section indicates the problems inintroducing a belief propagation algorithm into a neuralnetwork model and provides solutions to these problems. The
characteristics of the derived approximate algorithm are thendiscussed.
A. Approximation of conditional probability tables
Bayesian networks have a problem in that the size of aconditional probability table P ( X | U1 , L , U m ) increases
exponentially against the number of parent nodes, m . At thesame time, the execution cost of the belief propagation
algorithm [2] (Fig. 3) also increases in the same order. (InFig.3, the variable BEL (x ) is an abbreviation of

BEL( X = xi ) , that is an estimated posterior probability of

BEL( x) = αλ ( x)π ( x)a value xi (i = 1,L, s ) of a random variable X , whereπ ( x) = ∑ P ( x | u1 , L, um )∏ π X (uk )s is a number of possible values of X . The variable
π X (uk ) is an abbreviation of π X (U k = uik ) , that is amessage to a node X from a parent node U k concerning akvalue ui . Other variables are abbreviated in the samemanner.)In the case of BESOM as a model of a cerebral cortex, it
may be assumed that the individual nodes included in thesame layer learn different sets of features, and that theirindividual recognition results are almost independent of eachother. In addition, the number of input features required by
the status of a certain node may be assumed to be small
because of the sparseness of neuron firings. In this situation,
if at least one of Node X ’s parent nodes requires a feature x ,it can be estimated that this feature will exist. Based on theseassumptions, a conditional probability table for Node X canbe approximated with the sum of conditional probabilities
learned by parent’s SOMs.
m

P ( X | U 1 ,L ,U m ) ≈ ∑ P ( X | U i )

(3)

i =1

This assumption is similar to that of the noisy-OR model [2],so it can be expected that this situation will result in a simple
belief propagation algorithm and reduce the time and spacecomplexity substantially.
It is necessary to verify the establishment of (3) with aphysiological experiment or a computer simulation. In thispaper, (3) is assumed to be given and an approximate belief
propagation algorithm is derived from this assumption.
B. Information from message receiver
Each formula for π Yl ( x ) and λ X (u k ) in the algorithm
in Figure 3 excludes information came from the node thatwill receive a messages calculated by the formula. This
causes complexity of the algorithm. If a network has a tree
structure, the algorithm becomes simpler, as used in [6]. In
cerebral cortex, however, generally one area hasconnections to several higher areas, so we cannot assume
that the network is structured in a tree form.
In this paper, conversely, we assume that one node has
considerably many parent and child nodes. By assuming
this, information that supports values of a certain node’s
random variable are normally obtained from several nodes,
so inclusion of information from a message receiver may
not greatly affect the estimation results.
Based on this assumption, we use an approximation of“inclusion of information from a message receiver.”C. Approximate belief propagation algorithm
In addition to the two kinds of approximations describedabove, it is assumed that messages from parent nodes are

u1 ,L,u mkλ ( x) = ∏ λY ( x)llπ Y ( x) = π ( x)∏ λY ( x)
l

j

j ≠l

λ X (uk ) = ∑ λ ( x)x

Fig. 3.

∑ P( x | u ,L, u )∏ π (u )
1

m

X

i

i≠k

u1 ,L,u m / u k

Original belief propagation algorithm.

normalized.

∑ π (u ) = 1
X

(4)

k

uk

Then, a belief propagation algorithm can be approximated,
as shown in Figure 4. (See the appendix for a detailed
derivation.)In the approximated belief propagation algorithm, eachnode receives the values of λ(y l ) and ZYl from its childnodes and the values of BEL(uk ) from its parent nodes anddetermines values of its own λ ( x ) , BEL(x) and Z X . The
algorithm repeats these calculations until the values of theindividual variables converge, like an original loopy belief
propagation algorithm.
λ(y l ) is the results of the child nodes’ recognition with theuse of mainly bottom-up information. κ uk ( x ) is a prediction

based on information from parent nodes. BEL(x) is aposterior probability. Z X is used for the normalization ofBEL(x) and also represents the degree of agreement

between the predictions and observations in Node X .
It is interesting that the term Z Yl appears in a formula for

λY (x) . Inputs from a node with a good agreement betweenlpredictions and observations have a smaller effect on therecognition of parent nodes. In other words, the features notagreed with predictions are emphasized on the recognition.
(This effect results from Assumption (3).)
The values of individual variables, which only arise from avery simple calculation, can be sufficiently implemented witha neural network. In particular, an inner product calculation
conducted using λYl ( x ) or κ uk ( x ) is suitable for neuron
execution. In addition, a conditional probability table can beobtained from Hebbian learning by synapses linking to theseneurons. (Note that one conditional probability P ( x | u k ) islearnedbytwoneurons,κ u ( x ) and λ X (uk ) ,ksimultaneously.)
D. Scalability
Because input vectors in the learning steps are sparse, theobtained conditional probability table may also be sparse.

λYt +1 ( x) = Z Yt + ∑ λt ( y l )P( y l | x)
l

l

yl

n

λt +1 ( x) = ∏ λYt ( x)l =1lκ Ut +1 ( x) = ∑ P( x | u k ) BELt (u k )
k

uk

m

π t +1 ( x) = ∑ κ Ut +1 ( x)
k =1

Z

t +1
X

t +1

= ∑ρ

t +1

BEL(u k )kλ ( yl )l

ZX

BEL( x)

Z Ylk

ρ ( x) = λ ( x)πt +1

λ (uk )λ X (u k )κU (x)λ (x)λY ( x)t +1

Node Uk

Node X

I
II
III
IV
V
VI

Node Yl

( x)

( x)

x

BELt +1 ( x) = ρ t +1 ( x) / Z Xt +1
Fig. 4. Derived approximate belief propagation algorithm.

The number of non-zero elements of an input vector is aconstant number (number of child nodes) and independent of
the number of units s within each child node. The resulting
conditional probability table may inevitably have the samefeatures. Taking advantage of this characteristic, we canlower the order of time and space complexities. In the case of
cortex, synapses with a weight of 0 can be eliminated. Thiscan limit the explosive increase of synapses required.
A space complexity that is required to express one unit’s
conditional probability table (reference vector) is O(s) ,which can be reduced to O(1) by taking advantage of table’s
sparseness.
A time factor that requires an inner product calculation
made by λYl ( x ) and κ uk ( x ) will reach O(1) by the same
reasoning. Therefore, a time for all inner product calculations
in parallel computation within a node is O(1) . The timecomplexities in parallel computation for other variables areO(logs) or less.
As shown above, the approximate algorithm derived in thispaper is scalable. Hence, it can be said to be qualified as amodel for the cerebral cortex’s information processing
algorithm in terms of time and space complexities.
V. CORRESPONDING TO NEUROSCIENCE FINDINGS
A. Anatomical characteristics of a cerebral cortex
The cerebral cortex has a six-layer structure. Areas of thecortex are bidirectionally connected. These connections are
known to have the following regularities[7]. Bottom-up
connections are directed from layer III to IV. Some areas also
have connections from layer V to IV. Top-down connections
are directed mainly from layers V and VI to layer I. There arealso a few connections from layer III to layer I.
Furthermore, based on a column’s anatomical structure,
information input to layer IV is considered to be output froma-layer V via layers II and III within the column[8].

Fig. 5. Correspondence of approximate belief propagation algorithm
and the six-layer structure of cerebral cortex.

Considering these two findings, the six-layer cortex
structure forms a very strange construction in which theinformation in layer III, a provisional result of information
processing within the column, is sent to a higher area, and theinformation in layer V, a final result, is sent back to a lower
area. The functional meaning of this structure is unknown.
B. Corresponding to the approximate algorithm
Of the seven variables appearing in the approximate
belief propagation algorithm, five variables related to nodecommunication are applied to an area connection rule. The
result is shown in Figure 5. We selected a layer II, not I, forκ uk (x) because the layer I contains small number of cells.
Layers V and VI were selected for Z X based on a depth
related to BEL(x) , although a layer III can be considered.
As shown in the Figure 5, an approximate algorithm canbe directly corresponded to the connection rule, and cannot
be considered to be coincidentally correspondent to it.
(Top-down connections from layer III to I cannot be
explained by this model.)Figure 6 shows that the seven kinds of variables areplaced in a column so as to be consistent with anatomical
findings. A flow of information in the order of layers IV,
II/III and V, as described above, corresponds with that in anorder of variables of λYl (x) , λ (x) , ρ ( x ) and BEL(x) .
(Based on the findings, information further flows in theorder of layers V, VI and IV[8]. This information routecannot be explained by this model.)In addition, the following agreements can be observedwith the findings: 1) almost all information processing is
conducted vertically within the column, 2) many horizontal
fibers are seen in layers I and IV and 3) there are manysmall cells in layers II and IV.

BEL(U 1 = u11 )BEL(U 1 = u12 )I

BEL(U 1 = u31 )BEL(U 2 = u )2
1

BEL(U 2 = u 22 )++

κ U ( X = x1 )

κ U ( X = x2 )+1

BEL(U 2 = u32 )

+

2

+

π ( X = x1 )λ ( X = x1 )+1

κ U ( X = x1 )II

κ U ( X = x2 )2π ( X = x2 )λ ( X = x2 )

ρ ( X = x1 )λY ( X = x1 )λY ( X = x2 )

+

+

1

III

ρ ( X = x2 )Z Y1

1

λ (Y1 = y11 )λ (Y1 = y12 )λY ( X = x1 )2+

λ (Y1 = y31 )λY ( X = x2 )

+

2

Z Y2

λ (Y2 = y12 )ZX
ZXλ ( X = x1 )λ ( X = x2 )//BEL( X = x1 )BEL( X = x2 )IVλ (Y2 = y22 )λ (Y2 = y32 )

+

V
VI

BEL( X = x1 )BEL( X = x2 )Fig. 6. Neural network which executes the approximate belief propagation algorithm. This figure shows the flow of information between variables
within two units, x1 and x2, of Node X. Node X receives information from two parent nodes, U1 and U2, and two child nodes, Y1 and Y2. The network
structure is similar to the column structure of a cerebral cortex in many respects (see main text).

C. Same layer’s area connection problems
The cerebral cortex sometimes has horizontal connections
between two areas that cannot be explained in the current
BESOM model. We think these connections arecommunication route with a different purpose, such as
independent component analysis.
VI. CONCLUSION
An approximate belief propagation algorithm was derived
for bidirectionally connected SOM network, and it wasdemonstrated that the algorithm both corresponds well to the
six-layer and column structures of the cerebral cortex and isscalable. This model represents firing rate of neurons within
the cortex that might be observed in physiologicalexperiments.
In order to simulate this model and conduct large-scale tests,
the mechanism to make nodes within the same layer
independent each other must be elucidated. We are currently
tackling this challenge, by introducing a mechanism of
independent component analysis to this model. If this issue issolved, it will allow for simulations to be performed verifying
the adequacy of an approximate expression (3) in aconditional probability table.

In addition to the basic functions of the BESOM model
described in this paper, we think it is necessary to extend themodel, for instance, to include selective attention andshort-term memory mechanisms.
The BESOM network has a very high level of expressive
power. We have started to model a mechanism for acquiring
action sequences done by premotor and supplementary motor
areas, and also model a mechanism for acquiring a statetransition table of external world used for action planning by
prefrontal area. Both models are expressed by BESOM
network combined with a reinforcement learning mechanismthat is a model of the basal ganglia.
In the future, we aim to reproduce main brain functions on
computers.
APPENDIX
It can be shown that weights obtained in learning steps are
regarded as conditional probabilities as follows. Letv j (n) ∈ {0,1} be an j-th element of an input vector fromX’s child node Y and let w ij (n) be learning result, where nmeans n-th learning of a unit x i . (We do not consider

neighborhood learning here.) Let m (n) =n∑ v (i) be thecan be approximated as follows, using (3) and (8).

∑

j

i=1

u1 ,L,um / uk

number of times that a unit y j becomes the estimated valuei≠k∑ (∑ P( x | u ) +P( x | u ))∏ π (u )≈of Node Y. Let α = 1/n be a learning ratio. Givenw ij (1) = v j (1) = m(1) , the value of w ij (n) (n > 1)

P( x | u1 ,L, um )∏ π X (ui )u1 ,L,um / uk

u1 ,L,um / uk j ≠ kwij (n) = (1 − α ) wij (n − 1) + αv j (n)k+ P( x | uk )

j

X

i

i≠k

∑ ∏ π (u )Xi∑ ∑ P( x | u )∏ π (u ) + P( x | u )== ((n − 1)(m(n − 1) /(n − 1)) + v j (n)) / nju1 ,L,um / uk j ≠ kX= m( n ) / n∑ ∑ P( x | u )∏ π (u ) + P( x | u )This value is a ratio between “the number of times that aunit y j becomes the estimated value of Node Y” and “theof Node X.” Assuming that the estimation results are right,
this value is the conditional probability P ( y j | xi ) .
An approximate belief propagation algorithm used forlearning steps is derived as follows.
First, π (x) can be approximated as follows, using (3).

j

u1 ,L,um / uk j ≠ k

X

i

k

i≠k

≈ ∑ ∑ P( x | u j )∏ π X (ui ) + P( x | uk )number of times that a unit xi becomes the estimated value

u1 ,L,umji≈ π ( x) + P( x | uk )By substitution of the above equation, λX (uk ) can beapproximated as follows.

λ X (uk ) = ∑ λ ( x)x

∑ P( x | u ,L, u )∏ π (u )1u1 ,L,u m / u k

π ( x) = ∑ P( x | u1 ,L, um )∏ π X (ui )≈ ∑ λ ( x)(π ( x) + P( x | uk ))≈ ∑ (∑ P( x | uk ))∏ π X (ui )= ∑ λ ( x)π ( x) + ∑ λ ( x)P( x | uk )xi= ∑ ∑ P( x | uk )∏ π X (ui )u1 ,L,um k

i

x

(6)

Arranging the above results, we obtain the algorithm
shown in Figure 4.

uk∑ ∏ π (u )
X

i

(7)

REFERENCES

u1 ,L,um / uk i ≠ kHere, the following equation is established from (4),assumption for normalization.

∑∏ π X (ui ) = 1

[1]
[2]
[3]

(8)

u1 ,L,um / uk i ≠ k[4]Hence, (7) is converted to the following expression.

π ( x) = ∑∑ P( x | uk )π X (uk )[5]

π Y ( x) can be approximated as follows by “including

[6]

k

X

i≠k

i

= ∑∑ P ( x | uk )π X (uk )
k

m

x

i

k

k

i

i≠k

By “including information from a message receiver” and(6), the above formula is approximated as follows.

= (m(n − 1) + v j (n)) / n

u1 ,L,umiu1 ,L,um / uk i ≠ k= ((n − 1) wij (n − 1) + v j (n)) / n

u1 ,L,umXi≠k∑ ∑ P( x | u )∏ π (u )=equals the following.

j

j ≠k

uk

l

information from a message receiver”.

π Y ( x) = π ( x)∏ λY ( x)
l

j ≠l

[7]

j

≈ π ( x)∏ λY j ( x)j

= λ ( x)π ( x)The following equation included in a formula for λX (uk )[8]

T. Kohonen, Self-Organizing Maps. Springer-Verlag, 1995.
J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks ofPlausible Inference, Morgan Kaufmann, 1988.
K. Fukushima, Neural network model for selective attention invisual-pattern recognition and associative recall, APPLIED OPTICS 26(23): 4985-4992 Dec 1 1987.
M. Kawato, H. Hayakawa, T. Inui: A forward-inverse optics model ofreciprocal connections between visual areas. Network: Computation in
Neural Systems 4, 415-422, 1993.
R.P.N. Rao and D.H. Ballard, Predictive coding in the visual cortex: afunctional interpretation of some extra-classical receptive-field effects,
Nature Neuroscience, Vol.2 No.1, pp.79-87, Jan 1999.
George, D. Hawkins, J., A hierarchical Bayesian model of invariant
pattern recognition in the visual cortex, in proc. of IJCNN 2005, vol. 3,
pp.1812-1817, 2005.
Pandya, D.N. and Yeterian, E.H., Architecture and connections ofcortical association areas. In: Peters A, Jones EG, eds. Cerebral Cortex(Vol. 4): Association and Auditory Cortices. New York: Plenum Press,
3-61, 1985.
Gilbert, C.D., Microcircuitry of the visual-cortex, Annual review ofneuroscience, 6: 217-247, 1983.