Computational Model of the Cerebral Cortex
that Performs Sparse Coding Using a Bayesian
Network and Self-Organizing Maps
Yuuji Ichisugi1 and Haruo Hosoya2
1National Institute of Advanced Industrial Science and Technology(AIST)
Tsukuba Central 2,Tsukuba,Ibaraki 305-8568, Japany-ichisugi@aist.go.jp
2
7th Bld. of Faculty of Science, The University of Tokyo,
Hongo 7-3-1, Bunkyo-ku, Tokyo 113-0033, Japan
hahosoya@is.s.u-tokyo.ac.jp

Abstract. The authors have proposed a computational model of thecerebral cortex, called the BESOM model, that combines a Bayesian
network and Self-Organizing Maps. In this paper, we add another modelof the cerebral cortex, called sparse coding, into our model in a biologically plausible way. In the BESOM model, hyper-columns in the cerebralcortex are interpreted as random variables in a Bayesian network. We
extend our model so that random variables can become “inactive.” In
addition, we apply bias at the time of recognition so that almost all of therandom variables may become inactive. This mechanism realizes sparse
coding without breaking the theoretical framework of the model basedon the Bayesian networks.
Keywords: Sparse coding, Bayesian network, Self-organizing maps, cerebral cortex, BESOM

1BackgroundSome computational neuroscientists have begun to understand that the Bayesian
network [2] is the essential mechanism of the cerebral cortex [6][8][9][11][12][13][14][15]. Bayesian networks are a technology for knowledge representation that caneﬃciently express the causal relationships among many random variables. Models based on Bayesian networks can successfully explain the fundamental mechanism of the cerebral cortex, namely, robust pattern recognition using prediction
based on context[8]. Furthermore, previous studies strongly suggest that thecerebral cortex is a Bayesian network, according to models that reproduce electrophysiological phenomena[9][15] and models that explain the roles of major
anatomical characteristics of the cerebral cortex[11][12]. Moreover, the mechanism of the motor area is explained by using a combination of a Bayesian network
and a reinforcement learning mechanism[13]. Another study realizes probabilistic
reasoning at the Markov Random Field (a model similar to Bayesian networks)
with a biologically plausible neural circuit of spiking neurons[14].
Yuuji Ichisugi, Haruo Hosoya:"Computational Model of the Cerebral Cortex that Performs Sparse Coding Using aBayesian Network and Self-Organizing Maps",
In Proc. of 17th International Conference on Neural Information Processing(ICONIP 2010), Part I, LNCS 6443, pp.33--40, Nov 2010.
The original publication is available at www.springer.com .
http://www.springer.com/computer/theoretical+computer+science/book/978-3-642-17536-7

Although previous models explain only some parts of the mechanism of thecerebral cortex functions, we believe that, in the not too distant future, thesemodels (maybe including [5]) will become integrated into one universal model
based on Bayesian networks.
The authors have also proposed a model of the cerebral cortex using a
Bayesian network, called the BESOM model [11][13]. This model inherits thebasic structure of Neocognitron and its successors[1], which are macroscopic
models of the visual area. Moreover, the mechanism of the Self-Organizing Maps(SOM)[3], whose origin is the model of the orientation columns of the primaryvisual area, is adopted as a learning algorithm. BESOM can also be regarded asa novel machine-learning algorithm that uses multiple SOMs, like [10] and [17].
On the other hand, there is another model, called the sparse coding model[4], of an aspect of the cerebral cortex. Sparse coding is a kind of unsupervised
learning whose goal is to acquire a basis on which to express an input vector by alinear summation of a smaller number of basis vectors. It has been suggested thatsparse coding is performed in the primary visual area[4] and primary auditory
area[16] of the cerebral cortex. Sparse coding eﬃciently compresses informationand is supposed to conserve both the energy spent by neurons and the cost of
maintaining synapses.
In this paper, we propose a biologically plausible computational model that
uniﬁes three mechanisms, a Bayesian network, SOM, and sparse coding. The
model is an extension of the previous BESOM model.

2

Overview of the Model

2.1

Bayesian Network and MPE

A Bayesian network[2] is a model of knowledge representation that expresses
causal relationships between random variables using a directed acyclic graph.
Random variables are expressed as nodes, and relationships between randomvariables are expressed as edges. Each node has a table of conditional probability,
which denotes the degree to which nodes are related to the set of its parent nodes.
In a Bayesian network, an MPE (most probable explanation) is the set ofvalues of nodes that most likely explains given observed data. Let i be a set ofvalues of observed random variables and h be a set of values of hidden variables(unobserved random variables). MPE ĥ is deﬁned by the following equations.
ĥ = argmax P (h|i) = argmax P (h, i)h(1)hwhere P (h, i) is the joint probability of h and i, which can be calculated by thefollowing formula if a Bayesian network is given:∏P (h, i) =P (x|parents(x))
(2)
x∈h∪i

where parents(x) denotes the set of values of parent nodes of node X.

2.2

The Structure of Two-Layered BESOM

Figure 1 shows the neural circuit of the two-layered BESOM network used in thispaper. The ellipses are nodes (random variables), the small circles are units (values that random variables can take), and the straight lines are the connections(conditional probabilities) between units.
Although every pair of units contained in the two layers (the hidden layerand the input layer) has a connection, most connections are omitted in Fig.1.
There is no connection between units in the same layer.
At the time of learning, each node in the hidden layer plays the role of acompetitive layer of SOM, learns the weights of the connections (conditionalprobabilities) between units, and compresses the input from its child nodes.
At the time of recognition, all nodes play the role of random variables ina Bayesian network. At this time, all edges of the Bayesian network are fromall nodes of the hidden layer to all nodes of the input layer. There is no edge
between nodes in the same layer.
The number of nodes, the number of units, and the network structure ofnodes are given ﬁrst, and are not changed by learning.
When BESOM is used as a cerebral cortex model, each node is a hypercolumn, each unit is a column (minicolumn), and each connection weight between
a pair of units is the weight of a synapse.
The correspondence of components in BESOM, SOM, a Bayesian network
and a cerebral cortex is summarized in Table 1.
Hidden nodes

Node...

UnitInput
nodes

...

Fig. 1. The structure of two-layered BESOM network.

2.3

Recognition Steps and Learning Steps

When a set of values of input nodes (observed data) is given, BESOM executesa recognition step and a learning step. By repeating this cycle, BESOM acquires
an approximated generative model of the outer world.
Let i(t) be the set of values of input nodes at time t. We assume that each
input is generated from i.i.d. (independent identical distribution). The objectiveof the learning is to maximize the likelihood of the parameter θ (the vector ofTable 1. The correspondence of components in BESOM, SOM, a Bayesian network
and a cerebral cortex.
BESOM
nodeunit

SOM
Bayesian network Cerebral cortex
competitive layer
random variable
hyper-column
element of input vector, value ofcolumn
unit of competitive layer random variables
parent node competitive layer
parent node (cause) upper area
child node
input layer
child node (result) lower area
output of unit the winner of competition MPE
response of column
weight ofelement ofconditional
weight of synapse
connectionreference vector
probability

all the elements of all conditional probability tables).
θ∗ = argmax(θ= argmax(θt∏P (i(i)|θ))P (θ)

i=1t ∑∏

P (h, i(i)|θ))P (θ)

(3)

(4)

i=1 h

Although the algorithm described in this paper is somewhat complex, its
essence can be simply expressed by the following two equations.
At the recognition steps, the estimated values of hidden variables ĥ(t) (MPE)
based on the current parameter θ(t) are calculated by the following equation.
ĥ(t) = argmax P (h, i(t)|θ(t))(5)

hAt the following learning step, the parameter θ(t + 1) is estimated by Eq. (4),with an approximation that replaces the marginalization of h with the estimatedvalue ĥ(i).
t∏θ(t + 1) = argmax( P (ĥ(i), i(i)|θ))P (θ)
(6)
θi=1

The exact calculation of Eq. (5) and Eq. (6) requires enormous amounts ofcomputation. On the other hand, the actual brain should execute both recognition and learning very eﬃciently with some clever approximation. In addition,
the brain should perform online learning; that is, θ(t + 1) should be calculated
only by using i(t), ĥ(t) and θ(t). We think the algorithm described in this paper
satisﬁes these restrictions and is thus a plausible brain model.
2.4

Basic Idea of Sparse Coding for BESOM

We add the mechanism of sparse coding to BESOM by introducing an “inactivestate” into each random variable (i.e., a hyper-column). If the inactive state isintroduced to each random variable and large numbers of nodes become inactiveat each recognition step, sparse coding will be realized in BESOM. (See Fig. 2.)
Actually, the way in which inactive states are introduced into a Bayesian network
is not obvious. Moreover, in order for the mechanism to become an appropriate
model of the cerebral cortex, it should be implemented in a biologically plausibleway. Detailed recognition and learning algorithms that realize the idea of inactive
states are described in the next section.
Active nodes Inactive nodesInput 1

Input 2

Input 3Fig. 2. Basic idea of sparse coding using BESOM. Diﬀerent sets of hidden nodes become
inactive depending on inputs.

3

Algorithm

3.1

Characteristics of the Acquired Bayesian Network

We restrict the Bayesian network acquired by the proposed algorithm to satisfy
the following conditions.
1. Let X be a node (a random variable). X’s value should be one of the followings + 1 values.
X ∈ {xϕ , x1 , x2 , · · · , xs−1 , xs }(7)
We call the value xϕ the “ϕ-value” and the values other than xϕ “non-ϕvalues.” The value xϕ means the node X is inactive.
2. A conditional probability table P (xi |u1 , · · · , um ) (i = ϕ, 1, · · · , s) should satisfy the following equation:
1 ∑P (xi |u1 , · · · , um ) =P (xi |uk )
mm

(8)

k=1

∑
The constant 1/m normalizes so that i P (xi |u1 , · · · , um ) becomes 1.
3. Let U be a node and X be a child node of U . The ϕ-value of U , uϕ , should
not have a causal relation to X. That is, the following equation should hold.
P (xi |uϕ ) = P (xi ) (i = ϕ, 1, · · · , s)(9)This condition will be satisﬁed when the learning converges because of thelearning rules described in Section 3.3 .

3.2

Recognition Step

In the original sparse coding algorithm[4], sparseness is realized by adding thepenalty term for activeness to the objective function. We apply this idea toBESOM.
Eq. (2), which calculates the joint probability of an MPE candidate h andan input i, is modiﬁed as follows, so that a penalty is given depending on thenumber of activity nodes:∏P (x|parents(x))(10)
P (h, i) = e−βA(h)x∈h∪iwhere β is the parameter that controls the sparseness, A(h) is the number ofactive nodes in h.
The formula for this joint probability is used at the recognition steps thatcalculate MPE. We do not calculate strict MPE in the present simulation. The
approximate MPE is calculated by the hill-climbing method. In the actual brain,
a variation of the belief revision algorithm[2], which seems biologically moreplausible[14], may be used.
If MPE is used as a learning step as it is, the learning will be likely to fall
into a local minimum. Then, to avoid the local minimums, we add a moderateamount of noise to the recognition results in the early stages of the learning.
3.3

Learning Step

At each a learning step, the weights of connections between units will be updated
according to the MPE calculated at the preceding recognition step. In the nodes
at the hidden layer, the units that correspond to the values of MPE are regardedas winners for competitive learning.
In this section, we clarify our explanation by explaining a simpliﬁed versionof the learning rule, which does not include neighborhood-learning rules.
Let us call the units corresponding to the value of MPE winner units. In theinput layer, winner units represent observed data. In the hidden layer, winnerunits represent estimated values of hidden variables. Let X be a node at thehidden layer and Yl (l = 1, · · · , n) be a child node of X at the input layer.
lThe connection weight wij
between the winner unit xi and unit yjl is updated
by the following formulas:
1. When i = ϕ:{lwϕj=

ΦYl(j = ϕ)
(1 − ΦYl )/s (j ̸= ϕ)(11)where ΦYl is the frequency that Yl becomes a ϕ-value in MPE; these ΦYl arelearned separately.
2. When i ̸= ϕ, j ̸= ϕ:
lllwij← wij+ α(vjl − wij)
(12)
where α is the learning rate, vjl is an input from the child node Yl , whosevalue is 1 if unit yjl is the winner unit, and is 0 otherwise.3. When i ̸= ϕ, j = ϕ :
lwiϕ
=1−s∑

lwij

(13)

j=1l
The connection weight wij
is learned by the above algorithm. When theestimated values of the hidden variables are regarded as true observed valuesland the learning rate α is appropriately scheduled, the connection weight wij
becomes a maximum likelihood estimator of the conditional probability P (Yl =yjl |X = xi )[11].
The obtained conditional probabilities are used at the next recognition stepto calculate joint probability, deﬁned as Eq. (10), assuming the constraints ofEq. (8).

4Experiment: Sparse Coding of Natural Images

We used images provided by Olshausen (the images ﬁltered with whitening/lowpass as described in [7]) and clipped to the range [0, 1]. The images are used as
input to a two-layered BESOM, with 4 nodes in the hidden layer and 49 in theinput layer. At each step, we extracted a image patch with 7x7=49 pixels froma random position. Then, we gave the pixel intensities in the image patch to thebinary input nodes. The value of each input node is set to 1 according to thepixel intensity taken as a probability. For example, for intensity 0.2, the value
was set to 1 with probability 0.2. The parameter that determines the sparsenessof node activity is set to β = 8.
Figure 3(a) shows a learning result. It shows the values of the conditional
probability tables of the ϕ-value units P (y1l |xϕ ) and non-ϕ-value unit P (y1l |xi ) (i =1, · · · , 9, l = 1, · · · , 49) as the brightness of 7x7 pixels.
It is shown that the non-ϕ-value unit of each node obtained the conditional
probability table with orientation selectivity like V1 simple cells[4]. The ϕ-value
unit of each node learned the mean of input images. In this experiment, we foundthat 0-3 nodes are activated according to the inputs.
Figure 3(b) shows a learning result with the sparseness parameter β = 0 . In
this case, every base image is close to the mean image of input because all nodes
tend to be active. The result shows weak orientation selectivity.

References
1. K. Fukushima, Neural network model for selective attention in visual-pattern recognition and associative recall, APPLIED OPTICS 26 (23): 4985-4992 Dec 1 1987.
2. J. Pearl , Probabilistic Reasoning in Intelligent Systems: Networks of Plausible
Inference, Morgan Kaufmann, 1988.
3. T. Kohonen, Self-Organizing Maps. Springer-Verlag, 1995.
4. Olshausen BA, Field DJ, Emergence of simple-cell receptive ﬁeld properties bylearning a sparse code for natural images, NATURE 381 (6583): 607-609 JUN 131996.

X1

X2

X3

X1

X4x9

X3

X4xφx1

xφx1

(a)

X2(b)

x9

Fig. 3. Learning result of sparse coding of natural images. The sparseness parametervalues are β = 8 (a) and β = 0 (b) .

5. Coward, L.A., The Recommendation Architecture: lessons from the design of large
scale electronic systems for cognitive science. Journal of Cognitive Systems Research2(2), 111-156, 2001.
6. Lee, T.S., Mumford, D. , Hierarchical Bayesian inference in the visual cortex. Journalof Optical Society of America, A. . 20(7): 1434-1448, 2003.
7. Bruno A. Olshausen and David J. Field, Sparse coding with an overcomplete basis
set: A strategy employed by V1? Vision Research, 37(23):3311.3325, 2003.
8. George, D. Hawkins, J., A hierarchical Bayesian model of invariant pattern recognition in the visual cortex, In proc. of IJCNN 2005, vol. 3, pp.1812-1817, 2005.
9. R. Rao., Bayesian inference and attention in the visual cortex. Neuroreport 16(16),1843-1848, 2005.
10. Oshiro N, Kurata K, Separating visual information into position and direction by
two inhibitory-connected SOMs. Artif Life and Robotics 9(2):86.89, 2005.
11. Yuuji ICHISUGI, The cerebral cortex model that self-organizes conditional probability tables and executes belief propagation, In Proc. of International Joint Conference on Neural Networks (IJCNN2007), pp.1065–1070, Aug 2007.
12. Florian Roehrbein, Julian Eggert, and Edgar Koerner, Bayesian Columnar Networks for Grounded Cognitive Systems, In Proc. of the 30th Annual Conference ofthe Cognitive Science Society, pp.1423–1428, 2008.
13. Haruo Hosoya: A motor learning neural model based on Bayesian network andreinforcement learning, In Proceedings of International Joint Conference on NeuralNetworks, 2009.
14. Shai Litvak, Shimon Ullman: Cortical Circuitry Implementing Graphical Models,Neural Computation 21, 3010.3056, 2009.
15. Chikkerur, S., T. Serre, C. Tan and T. Poggio, What and Where: A BayesianInference Theory of Attention, Vision Research, 2010.
16. Hiroki Terashima and Haruo Hosoya, Sparse codes of harmonic natural sounds andtheir modulatory interactions. Network: Computation in Neural Systems, 20(4):253267, 2009.
17. Tetsuo Furukawa, SOM of SOMs, Neural Networks, Vol.22, Issue 4, pp.463478,May 2009.