© 2011 IEEE. Personal use of this material is permitted. Permission from IEEE must be
obtained

for

all

other

uses,

in

any

currentorfuturemedia,

includingreprinting/republishing this material for advertising or promotional purposes, creating
new collective works, for resale or redistribution to servers or lists, or reuse of any
copyrighted component of this work in other works.

Yuuji Ichisugi:
Recognition Model of Cerebral Cortex based on Approximate Belief Revision Algorithm,
In Proc. of The 2011 International Joint Conference on Neural Networks (IJCNN 2011),
pp.386--391, 2011.

The final version of this paper can be obtained from the following link.
「IEEE Xplore - Abstract Page」
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6033247
Digital Object Identifier: 10.1109/IJCNN.2011.6033247

Recognition Model of Cerebral Cortex
based on Approximate Belief Revision Algorithm
Yuuji Ichisugi

Abstract—We propose a computational model of recognitionof the cerebral cortex, based on an approximate belief revisionalgorithm. The algorithm calculates the MPE (most probableexplanation) of Bayesian networks with a linear-sum CPT(conditional probability table) model. Although the proposed
algorithm is simple enough to be implemented by a fixed circuit,results of the performance evaluation show that this algorithm
does not have bad approximation accuracy. The mean
convergence time is not sensitive to the number of nodes if thedepth the network is constant. The computation amount islinear to the number of nodes if the number of edges per node isconstant. The proposed algorithm can be used as a part of alearning algorithm for a kind of sparse-coding, which
reproduces orientation selectivity of the primary visual area.
The circuit that executes the algorithm shows better
correspondence to the anatomical structure of the cerebral
cortex, namely its six-layer and columnar features, than theapproximate belief propagation algorithm that has been
proposed before. These results suggest that the proposed
algorithm is a promising starting point for the model of therecognition mechanism of the cerebral cortex.

I. INTRODUCTION
The cerebral cortex is the part of the brain that is mostclosely related to humans’ high intelligence. It is strongly
desired to uncover how the cerebral cortex recognizes andlearns the outer world.
Some computational neuroscientists have begun tounderstand that the Bayesian network[1] is the essential
mechanism of the cerebral cortex[2-10]. Models based on
Bayesian networks can successfully explain the fundamental
mechanism of the cerebral cortex: namely, robust patternrecognition using prediction based on context[3].
Furthermore, previous studies have strongly suggested thatthe cerebral cortex is a Bayesian network, according tomodels that reproduce electrophysiological phenomena[4][9]and models that explain the roles of major anatomical
characteristics of the cerebral cortex[5][6]. Now, we can saythat we have acquired a very important clue for understanding
the brain.
Even if the cerebral cortex is a kind of Bayesian network,
an essential question still remains: what kind of algorithm isused for recognition? There are at least two candidates, thebelief propagation algorithm and the belief revisionalgorithm.
Most previous models[3-7,9] based on Bayesian networks
claim that the recognition of the cerebral cortex is achieved
Yuuji Ichisugi is with National Institute of Advanced Industrial Scienceand Technology(AIST) Tsukuba Central 2,Tsukuba,Ibaraki 305-8568, Japan(e-mail: y-ichisugi@aist.go.jp).

by belief propagation[1], which calculates the posterior
probabilities of all unobserved random variables; however,these models could not explain the theoretical relation
between the recognition algorithm and the learningalgorithm.
On the other hand, some models[8][10] assume that the
purpose of recognition is not to calculate posterior
probabilities but to find an MPE (most probable explanation),
which can be efficiently achieved by the belief revisionalgorithm[1]. The MPE is the set of values of randomvariables that most likely explains the given observed data.
One study[8] realized the approximate belief revisionalgorithm for the Markov Random Field (a model similar toBayesian networks) by a biologically plausible neural circuitof spiking neurons. Another study[10] showed that thecalculation of MPE with the hill-climbing method can be usedas part of an on-line learning algorithm, which is a kind ofsparse-coding algorithm[11] to reproduce the orientation
selectivity of the primary visual area.
At this point it has not been determined which algorithm,
belief propagation or belief revision (or some other algorithm
such as Markov chain Monte Carlo) is the brain’s actual
recognition algorithm.
In this paper, we show evidence that supports the belief
revision model. We propose an approximate belief revisionalgorithm that shows good performance despite its simplicity
and good correspondence to the anatomical structure of thecerebral cortex, namely its six-layer and columnar features.
II. BAYESIAN NETWORKS
Bayesian networks are the model of knowledge
representation that expresses causal relationships between
random variables using a directed acyclic graph. Randomvariables are expressed as nodes, and relationships between
random variables are expressed as edges. Each node has atable of conditional probability, which denotes the degree of
causal relation between the node and the set of its parent
nodes.
Belief propagation and belief revision are often used for
inferences in Bayesian networks. Both algorithms have somesimilarities with the cerebral cortex. For example, they
perform local and asynchronous communications and transfer
only non-negative values. Nevertheless, exact algorithms
would be too complex to be implemented by neurons naively.

λtY+1 ( x) = ∑ ( ρ t ( yl ) + λt ( y l )P( yl | x))lyl

λYt +1 ( x) = max( ρ t ( yl ) + λt ( yl ) P( yl | x))
l

yl

n

n

λt +1 ( x) = ∏ λtY ( x)λt +1 ( x) = ∏ λYt ( x)

κ Ut +1 ( x) = ∑ P( x | u k ) BELt (u k )κUt +1 ( x) = max P( x | uk ) BELt (uk )

l

l =1

k

l =1

k

uk

π t +1 ( x) = ∑ κUt +1 ( x)

m

k =1

k

k =1

ρ ( x) = λt +1 ( x)π t +1 ( x)Z Xt +1 = max ρ t +1 ( x)Z Xt +1 = ∑ ρ t +1 ( x)x

BEL ( x) = ρ t +1 ( x) / Z Xt +1
t +1

x

t +1

( x) / Zkρ t +1 ( x) = λt +1 ( x)π t +1 ( x)t +1

BEL ( x) = ρukm

π t +1 ( x) = ∑ κ Ut +1 ( x)

t +1

l

t +1
X

x *t +1 = arg max BELt +1 ( x)x

Fig. 1. The approximate belief propagation algorithm[5].

Fig. 2. Obtained approximate belief revision algorithm. x* is theestimated value of the node X.

III. APPROXIMATE BELIEF PROPAGATION AND BELIEFREVISION ALGORITHMS
The author has derived an approximate belief propagation
algorithm[5] based on the following two natural assumptions.
1. Each conditional probability table (CPT) satisfies thefollowing “linear-sum CPT model”, which isqualitatively similar to the noisy-OR model[1]:1 m∑ P( x | u k )m k =1P( x | u1 ,", u m ) =Actually, the normalization factor 1/m can be ignored
because it does not affect the normalized posterior
probabilities. (This point has not been mentioned in[5].)
2. Each node has a sufficient number of parent andchild nodes to make the following two
approximations valid ( π X (u i ) and λYl (x) aremessages from parents and children of a node
respectively):∑ ∑ P( x | u )∏ π (u )ju1 ,",u m / u k j ≠ k≈

Xprocess. The expression for λYl has been slightly changed
but the changes are not essential. These formulas calculate“messages” from a node X to its parent nodes

U k (k = 1,", m) and child nodes Yl (l = 1,", n) , using
old messages sent from its parents and children. After someiteration of message sending, the values of BEL are
expected to converge to the posterior probabilities.
Unfortunately we cannot apply the derivation processes
described in [5] to the belief revision algorithm because theexpressions in it consist of both max- and sigma-operators.
Therefore, we use an ad-hoc approach. As mentioned in[1], the belief revision algorithm can be obtained by merely
replacing sigma-operators with max-operators. So, wehypothesized that the approximate belief revision algorithm
may be obtained by replacing “sigma-operators whose origin
is not the CPT model” with max-operators. The obtained
algorithm is shown in Figure 2. The validity of this algorithm
is experimentally evaluated in the following sections.

ii≠k∑ ∑ P( x | u )∏ π (u )ju1 ,",u m

j

X

i

i

∏ λ ( x)N: the number of nodes at each layer

Yj

j ≠l

≈ ∏ λY j ( x)j

Although the validity of the approximations based on thesecond assumption has not been experimentally verified in[5], a similar approximation for Markov Random Fields has
been verified by simulation in [8].
The derived algorithm based on these two assumptions isshown in Figure 1. See [5] for more details on the derivation...
L: thenumberof layers...
...

hidden nodes

input nodes

Each node has S states.
Each node has at most E parent nodes.

Fig. 3. Structure of Bayesian networks used for evaluation.

B

0.9

0.9

0.8

0.8

0.7
0.6
0.5

ABR

0.4

OBR

0.3
0.2
0.1
0

0.7
0.6
0.5

ABR
OBR

0.4
0.3
0.2
0.1
0

6

7

8

9

10

5

Number of nodes at each layer

6

7

8

9

D
0.9

0.8
0.7
0.6
ABR
OBR

0.2

Convergence rates

0.9

0.3

0.1
0
4

ABR
4

OBR

3
2
1
0

5

Number of layers

6

7

8

9

10

Number of nodes at each layer

0.8
0.7
0.6
0.5

ABR
OBR

0.4
0.3
0.2

0
3

5

F

0.1

2

6

5

E
1

0.4

7

Number of nodes at each layer

1

0.5

8

10

Mean convergence times

5

Correct rates

C
9

Mean convergence times

1

1

Convergence rates

Correct rates

A

14
12
10
8
ABR
6

OBR

4
2
0

2

3

4

Number of layers

5

2

3

4

5

Number of layers

Fig. 4. Graphs of correct rates (A and D), convergence rates (B and E) and mean convergence times (C and F) for ABR (solid lines) and OBR (brokenlines). The upper graphs (A, B and C) plot results on shallow networks with various widths. (L=2, N=5, 6, ..., 10, S=2, E=5) erratum: L=2 to L=3
The lower graphs (D, E and F) plot results on narrow networks with various depths. (L=2, 3, ..., 5, N=5, S=2, E=full)IV. EVALUATION METHOD
A. Network Structure
We have executed algorithms on Bayesian networks thathave a layered structure similar to that of the brain (Figure 3).
The number of layers is denoted by “L” hereafter. There are
no edges between the nodes when they are in the same layer.
The observed data are input to the input layer, which is thelowest layer. The other layers are hidden layers, whichcontain unobserved (hidden) variables. All layers contain thesame number (denoted by “N”) of nodes. All randomvariables have the same number (denoted by “S”) of states.
Elements of the conditional probability tables are randomly
set, but values close to zero are avoided.
Two nodes at the neighboring layers are randomly
connected and each node has at most “E” parent nodes. Moreprecisely, each node except for nodes at the highest layer
randomly chooses “E” parent nodes allowing duplicationfrom the next higher layer.
Some simulations use fully connected neighboring layers(denoted by “E=full”).

B. Algorithms
In order to evaluate the proposed algorithm, we have
implemented three algorithms, a naive full search algorithm
to find strict MPE (STR), the original loopy belief revision(OBR) and the proposed approximate belief revision (ABR).
In OBR and ABR, at each iteration step, all messages are sent
to the neighboring nodes simultaneously. The initial values ofmessages are random values. To avoid overflow andunderflow, λ -messages are rescaled before using; therescaling makes the maximum value of the elements of a
λ -message be 1.
C. Measures of Performance
We measured three values: correct rates, convergence rates
and mean convergence times.
The correct rate is defined as the number of hidden
variables whose estimated values are correct MPE values,
divided by the number of all hidden nodes. The correct rate
can be calculated only for the small networks to which the
STR algorithm can be applied.
The convergence rate is defined as the number ofrecognition trials at which all hidden variables converge
within the predetermined number of iteration steps (50 steps,

25

-12

-13

ABR
OBR

-14

-15

Mean convergence times

-11

20

15

10

5

0
100

200

300

400

500

600

700

800

900

Number of nodes at each layer
-16
1

2

3

4

5

6

7

8

9

10

11

12

Iteration step
Fig. 5. An example of a transition of log posterior probabilities of
estimated values at the time after each iteration step. More accurately,
we use f(t)＝ log P(<estimated value at t > | <input>) – c as index,
where t is a number of iteration. In this case, OBR (broken line)
converges to the strict MPE and ABR (solid line) seems to converge toa sub-optimal solution. (L=5, N=5, S=2, E=full)in all simulations in this paper), divided by the number of alltrials.
The mean convergence time is defined as the total numberof iteration steps before convergence divided by the numberof converged recognition trials. Trials in which some hidden
variables did not converge are excluded for calculation of themean convergence time; therefore, this measure ismeaningful only for network configurations that achieve highconvergence rates.
All graphs in this paper plot the mean measured values of
100 recognition trials with differently initialized networks.

Fig. 6. Mean convergence times for shallow and wide networks.
(L=3, N=100, 200, …, 900, S=4, E=20)35

30

Mean convergence times

log posterior probabilities

-10

25

20

15

10

5

0
2

3

4

5

6

7

8

9

10

Number of layers
Fig. 7. Mean convergence times for deep networks.
(L=2, 3, ..., 6, N=10, S=2, E=5)

V. RESULTS
A. Accuracy for Small Networks
Firstly, we show the result of evaluation of ABR and OBR
on small networks.
Graphs in Figure 4 plot the correct rates, the convergence
rates and the mean convergence times.
The upper graphs plot results on shallow networks withvarious widths. The lower graphs show results on narrow
networks with various depths.
Although ABR is inferior to OBR, its accuracy is not sobad, despite its simplicity (Figure 4 A and D). It seems that
ABR is less likely to vibrate compared to OBR, which
vibrates easily when the network becomes deeper (Figure 4 B
and E). When the network converges, the mean convergencetime of ABR is slightly shorter than that of OBR (Figure 4 C
and F).
Although the correct rate of ABR is not 100%, theposterior probabilities of estimated values tend to increase as
iteration steps progress. And usually, the difference of log
posterior probabilities between the estimated values and thestrict MPE is not so large. Therefore, it is thought that theestimated values of ABR may be sub-optimal solutions.
Figure 5 shows an example of the transition of log posterior
probabilities in a recognition trial. In this case, OBR
converges to the strict MPE.

Fig.8. Learning result of natural images. Visualized conditional
probability tables have orientation selectivity. 4 rows of 20 images
are corresponding to 4 nodes with 20 states. Large values ofconditional probabilities of each state are shown by bright pixels.

B. Scalability
We have also evaluated the scalability of the ABR
algorithm.
Figure 6 and Figure 7 plot the mean convergence times of
ABR on shallow networks with various widths and narrow
networks with various depths, respectively. (We have chosen
these parameters carefully so that convergence rates becomealways nearly 1.)
Figure 6 shows that the mean convergence time is not sosensitive to N if the network depth is constant.
On the other hand, Figure 7 shows that the mean
convergence time is almost linear to the depth of the network.

BEL(u11 )BEL(u12 )max

BEL(u12 )maxκU ( x1 )2
2

BEL(u )

κ U ( x2 )

max

1

+

2

+

π ( x1 )U1λ ( x1 )U2λY ( x1 )Y1κ U ( x2 )2π ( x2 )λ ( x2 )ρ ( x1 )ρ ( x2 )ρ ( y11 )λY ( x2 )
1

1

max

X

max

1

κU ( x1 )λ ( y11 )max

ρ ( y12 )λY ( x1 )

Y2

2

max

λY ( x2 )2λ ( y12 )max

ρ ( y12 )λ ( y12 )ρ ( x1 )λ ( x1 )ρ ( x2 )λ ( x2 )ZX//

BEL ( x1 )BEL ( x2 )

ρ ( y 22 )λ ( y22 )max
BEL( x1 )BEL( x2 )Fig. 9. A biologically plausible circuit that calculates messages from node X to the two parent nodes, U1 and U2, and to the two child nodes, Y1 and Y2.
All nodes are binary, just for making the figure simple. See also Figure 10 which describes the details of the calculation of κU ( xi ) and λY ( xi ) . Theklcircuit structure has congruencies with the six-layer and the columnar features of the cerebral cortex structure (see main text).

According to current anatomical knowledge, the depth of
the hierarchy of cerebral areas of the macaque monkey isabout ten. This means the depth of the Bayesian network
represented by the cerebral cortex is not so deep. On the other
hand, the number of nodes may be more than one hundred
thousand if each macro-column (or hyper-column) of thecerebral cortex is assumed to be a node of a Bayesian network.
The scalability of ABR shown in this section leads us to
expect that ABR will work well on such large-scale networks.
In addition, the characteristics of ABR’s scalability might
explain why the evolution of the brain makes the number ofcolumns larger but does not makes the network depth deeper.
C. Computation AmountThe computation amount required for each iteration step of
ABR is linear to the number of nodes, if the number of edges
per node is constant. Indeed, we have verified thischaracteristic by measuring the computation time during theexperiments of Figure 6 and Figure 7.
Because ABR can be easily executed in parallel, massively
parallel computers will execute each iteration step of ABR in
a constant time. (OBR may also have same characteristic.)
This efficiency might explain a surprising fact: althoughmammals have very different-size brains, all of them work
well in real time.

VI. LEARNING NATURAL IMAGESIn this section, we show that ABR can be used as a part of alearning model of the cerebral cortex.
The used learning algorithm is almost the same as that
described in [10], which uses a hill-climbing method insteadof ABR, however. The neighborhood learning and sparseness
control described in [10] is omitted here. Input images aregray-scale natural images preprocessed by a 3×3 Laplacian
filter in order to emphasize the edges. We used a two-layered
network that consists of 4 hidden nodes with 20 values and12×12 input nodes with two values.
Each cycle of the learning process consists of a recognition
step and a learning step. When an input image (randomly
selected 12x12 pixel image) is given as observed data in the
input layer, MPE is calculated (recognition step). Then, theconditional probability tables are updated based on theestimated MPE values (learning step).
Figure 8 is the result of learning, which shows acquired
basis images (learned conditional probabilities) that haveorientation selectivity like receptive fields of simple cells in
the primary visual area[11].
This result indicates that ABR models a recognition part of
the actual brain’s learning of natural images with sufficientaccuracy.

mechanism of the cerebral cortex.
On the other hand, previous models based on belief
propagation
have
reproduced
someimportantelectrophysiological phenomena[4][9]. We think the belief
revision with some extension might also reproduce the same
phenomena because the meaning of the variable BEL is very
similar in both algorithms. We think this is important topursue in future work.
From the point of view of engineering, because theproposed algorithm efficiently works in large-scale Bayesian
networks with a linear-sum CPT model, its improved version
may be expected to be used as a part of efficientpattern-recognition mechanism.

BEL (u11 )BEL(u12 )Variable synapse P ( x | u1 )1i

P( x1 | u1i ) BEL(ui1 )maxκU ( x1 )1

λY ( x1 )1max

ρ ( y1j ) + λ ( y1j ) P( y1j | x1 )

ρ ( y11 )++Variable synapse P( y1j | x1 )λ ( y11 )ρ ( y12 )λ ( y12 )REFERENCES

Fig. 10. Detailed structure of input to the calculation units of κU ( xi )k(upper) and λY ( xi ) (lower).
lVII. CORRESPONDENCES OF THE ALGORITHM WITH THE
ANATOMICAL STRUCTURE OF THE CEREBRAL CORTEX
The circuit that executes the ABR shows better
correspondence to the anatomical structure of the brain than
the approximate belief propagation algorithm (ABP, Figure1) proposed before.
Figure 9 is a circuit that consists of simple arithmetic units.
The circuit calculates the values of messages to theneighboring nodes. Places of the plastic synapses that
represent conditional probabilities are shown in Figure 10.
We have matched arithmetic units that calculate thevariables κ , ρ , λYl and BEL with layer 2, 3, 4 and 5 ofthe cerebral cortex respectively. According to this circuit,
ABR has, like ABP[5], the following correspondences withthe anatomical structure: A) Bottom-up (feed-forward)
connections are mainly from layer 3 to layer 4 and top-down(feed-back) connections are mainly from layers 5/6 to layer
1[12]. B) Information is mainly processed within long andslender structures called cortical columns, C) Many
horizontal fibers are seen in layers 1 and 3 and D) Many cells
are seen in layers 2 and 4.
ABR has one more correspondence than ABP does. It has
been shown that max-operation can be implemented by abiologically plausible recurrent network of excitatory andinhibitory neurons[8]. The circuit of Figure 9 predicts theexistence of max-operation units at layers 2, 4 and 5. Indeed,
there exist recurrent circuits at layer 2/3, 4, 5 and 6 in theactual cerebral cortex[13]. In addition to this, as mentioned in[8], some electrophysiological experiments found max-like
responses in the cerebral cortex. These facts support thevalidity of belief revision models rather than belief
propagation models.
VIII. CONCLUSION
The evidence described in this paper suggests that the
proposed approximate belief revision algorithm is apromising starting point for the model of the recognition[1]

J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks ofPlausible Inference, Morgan Kaufmann, 1988.
[2] T.S. Lee, D. Mumford, Hierarchical Bayesian inference in the visualcortex. Journal of Optical Society of America, A. . 20(7): 1434-1448,
2003.
[3] D. George, J. Hawkins, A hierarchical Bayesian model of invariant
pattern recognition in the visual cortex, in proc. of IJCNN 2005, vol. 3,
pp.1812-1817, 2005.
[4] R. Rao., Bayesian inference and attention in the visual cortex.
Neuroreport 16(16), 1843-1848, 2005.
[5] Y. ICHISUGI, The cerebral cortex model that self-organizes
conditional probability tables and executes belief propagation, In Proc.
of International Joint Conference on Neural Networks (IJCNN2007),
pp.1065--1070, Aug 2007.
[6] F. Roehrbein, J. Eggert, and E. Koerner, Bayesian Columnar Networks
for Grounded Cognitive Systems, In Proc. of the 30th Annual
Conference of the Cognitive Science Society, pp.1423--1428, 2008.
[7] H. Hosoya: A motor learning neural model based on Bayesian networkand reinforcement learning, In Proceedings of International Joint
Conference on Neural Networks, 2009.
[8] S. Litvak, S. Ullman: Cortical Circuitry Implementing GraphicalModels, Neural Computation 21, 3010.3056, 2009.
[9] S. Chikkerur, T. Serre, C. Tan and T. Poggio, What and Where: A
Bayesian Inference Theory of Attention, Vision Research, 55(22), pp.
2233–2247, Oct 2010.
[10] Y. Ichisugi, H. Hosoya, Computational Model of the Cerebral Cortex
that Performs Sparse Coding Using a Bayesian Network andSelf-Organizing Maps, In Proc. of 17th International Conference on
Neural Information Processing (ICONIP 2010), Part I, LNCS 6443,
pp.33--40, Nov 2010.
[11] B.A. Olshausen, D.J. Field, Emergence of simple-cell receptive field
properties by learning a sparse code for natural images, NATURE 381(6583): 607-609 JUN 13 1996.
[12] D.N. Pandya, and E.H. Yeterian, Architecture and connections ofcortical association areas. In: Peters A, Jones EG, eds. Cerebral Cortex(Vol. 4): Association and Auditory Cortices. New York: Plenum Press,
3-61, 1985.
[13] C.D. Gilbert, Microcircuitry of the visual-cortex, Annual review ofneuroscience, 6: 217-247, 1983.