ORIGINAL RESEARCH ARTICLE
published: 21 November 2013
doi: 10.3389/fncom.2013.00165

COMPUTATIONAL NEUROSCIENCE

Information maximization principle explains the
emergence of complex cell-like neurons
Takuma Tanaka* and Kiyohiko Nakamura
Department of Computational Intelligence and Systems Science, Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology,
Yokohama, Japan

Edited by:
Tomoki Fukai, RIKEN Brain Science
Institute, Japan
Reviewed by:
Ko Sakai, University of Tsukuba,
Japan
James McFarland, University of
Maryland, USA*Correspondence:
Takuma Tanaka, Department ofComputational Intelligence andSystems Science, Interdisciplinary
Graduate School of Science andEngineering, Tokyo Institute ofTechnology, 4259-G3-46,
Nagatsuta-cho, Midori-ku,
Yokohama, 224-8502, Japane-mail: tanaka.takuma@gmail.com

We propose models and a method to qualitatively explain the receptive field propertiesof complex cells in the primary visual cortex. We apply a learning method based on theinformation maximization principle in a feedforward network, which comprises an input
layer of image patches, simple cell-like first-output-layer neurons, and second-output-layer
neurons (Model 1). The information maximization results in the emergence of the complex
cell-like receptive field properties in the second-output-layer neurons. After learning,
second-output-layer neurons receive connection weights having the same size fromtwo first-output-layer neurons with sign-inverted receptive fields. The second-output-layer
neurons replicate the phase invariance and iso-orientation suppression. Furthermore, on
the basis of these results, we examine a simplified model showing the emergence ofcomplex cell-like receptive fields (Model 2). We show that after learning, the output
neurons of this model exhibit iso-orientation suppression, cross-orientation facilitation,and end stopping, which are similar to those found in complex cells. These properties ofmodel neurons suggest that complex cells in the primary visual cortex become selective
to features composed of edges to increase the variability of the output.
Keywords: information maximization principle, complex cell, primary visual cortex, extraclassical receptive field,
computational model

1. INTRODUCTION
A fundamental question that is often raised in neuroscience is
how to determine the principle that underlies neural information
coding in the brain. In terms of sensory information processing,
this question can be answered by explaining how sensory neurons
acquire their selectivity to inputs. The primary visual cortex (V1)
is an ideal subject for this type of investigation because experimental results regarding the receptive field properties, single-cellelectrophysiology, and topographic selectivity map are accumulated in V1. These experimental results allow us to screen the
proposed principles of neural information coding by comparing
the behavior of the model on the basis of each principle withthe receptive field properties of neurons in V1. This screening
provides us with a way to deal with the general principle of neural information coding in the cerebral cortex. Several principles
having similar mathematical structures, such as the information
maximization principle (Linsker, 1988; Bell and Sejnowski, 1997)and sparse coding hypothesis (Barlow, 1959; Olshausen and Field,
1996), were proposed to explain the receptive field properties ofsimple cells in V1. The statistical independence of the output neurons is the most essential assumption of these models. In theframework of independent component analysis (ICA), the output neurons acquire selectivity to stimuli such that the outputsof these neurons are uncorrelated and as statistically independent as possible. The statistical independence is closely related tothe information maximization principle and sparse coding. If theoutput neurons are statistically dependent, the amount of information conveyed by these neurons reduces because many of them

Frontiers in Computational Neuroscienceshare the same information. Thus, maximizing the amount of
information conveyed by the output neurons gives a result similar to that obtained when increasing the statistical independenceof the output neurons. If the activity of neurons is independent and uncorrelated, the number of neurons firing simultaneously decreases, and therefore, the neuronal activity becomessparse.
Since publication of the groundbreaking work by Hubel andWiesel (1959), it is widely accepted that simple cells have waveletlike receptive fields, which respond when a wavelet or an edge
is positioned appropriately. The ICA models have revealed that
ICA of natural images generates output units with simple celllike receptive field properties. These results suggest that theassumption of the statistical independence of output neurons is
a promising principle of neural information coding. However,
the ability of this principle to explain the receptive field properties of complex cells has not been addressed. In contrast to
simple cells, the response of complex cells is predominantly determined by the orientation of gratings and edges, and these cells areless sensitive to the positions of edges (Hubel and Wiesel, 1962).
Thus, it is assumed that complex cells respond to abstract orientation and that the shift-invariant representation of the stimuli
in the visual field is accomplished by complex cells. However,recent studies reveal that complex cells are not simple orientation
detectors or shift-invariance detectors, and they exhibit surroundsuppression and facilitation by the gratings outside their classicalreceptive fields (Jones et al., 2001, 2002). Superimposing gratings
perpendicular to the preferred orientation in the receptive field of

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 1

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

complex cells suppresses the firing of these cells (Bonds, 1989).
These results suggest that the outputs of complex cells are nota simple pooling of simple cell inputs with similar orientationpreferences. Using learning principles that can explain the emergence of these complex properties, we will be able to approach
the general principle of neural information coding. Conversely,a general principle may shed light on what characterizes the features detected by complex cells. A number of theoretical studies
were reported to explain the properties of complex cells (Földiák,
1991; Hyvärinen and Hoyer, 2001; Berkes and Wiskott, 2005;
Karklin and Lewicki, 2005, 2009; Shan et al., 2007). Extraclassicalreceptive fields and complicated receptive field properties werereplicated by these studies. However, these models assume thatthe complex cell-like units receive inputs whose magnitude does
not depend on the polarity of the input image pixels (black
and white). This type of input facilitates the emergence of theshift-invariant complex cell-like units; however, this assumption
should be justified by a general principle.
In this paper, we show that this assumption is justified by using
the information maximization principle. In addition, we showthat the information maximization principle explains the receptive field properties of complex cells. The differential entropy ofthe output was used as the measure of information transmitted
from the input to the output in the previous ICA models (Belland Sejnowski, 1995; Shriki et al., 2001). In the Methods section, we introduce mutual information, entropy, and the models
we propose in this paper. In the first part of the Results section,
we use a three-layer feedforward network comprising an input
layer of natural image patches, simple cell-like first-output-layer
neurons, and second-output-layer neurons that receive inputs
from the first-output-layer neurons (Model 1). Our simulation
results obtained using a learning rule that maximizes information transmission show that second-output-layer neurons thatreceive inputs from first-output-layer neurons with rectifying
nonlinearity become shift invariant after the learning process.
Some second-output-layer neurons exhibit surround suppression similar to that reported for complex cells. A theoretical
calculation based on the fact that the edges detected by simple
cells are almost statistically independent proves that the phaseinsensitivity of complex cells maximizes the output entropy of
V1. Model 1 contains realistic first-output-layer neurons with
non-negative-definite firing rates. However, Model 1 is computationally expensive so we develop a simplified model of complex
cells and more precisely examine the properties of model neurons. In Model 2, the output neurons receive the absolute valueof the simple cell-like neurons as inputs. This simplification
reduces the computational load and accelerates the convergence.
In the Discussion section, we compare our proposed models withprevious models and discuss the biological implications of ourmodels.

2. METHODS
2.1. INFORMATION MAXIMIZATION PRINCIPLE

Information transmitted from input x to output y is measured by
the mutual information




 
 
I x; y = H (x) + H y − H x, y ,Frontiers in Computational Neuroscience

where H(x), H(y), and H(x, y) are the entropies of the input,
the output, and combination of the input and the output. Belland Sejnowski (1995) showed that the amount of informationtransmitted from the input layer to the output layer equals theoutput entropy plus a constant dependent only on the probability distribution of the input. The mutual information can begiven by
 
 
 
I x; y = H y − H y|x ,where H(y|x) is the entropy of the noise. In the noiseless system, the maximization of the mutual information of input
and output can be achieved by the maximization of output
entropy. Similar to Bell and Sejnowski (1995), we ignore theintrinsic noise of neurons although it is an important factor contributing to the variability of neuronal firings. We canignore the effect of the intrinsic noise because qualitatively similar results are obtained by ICA models when we add a small
noise to the input and the output of first-output-layer neurons.
Thus, by maximizing the output entropy, we expect to have an
information-efficient representation of the input in the output
layer.
Entropy is a measure of the variability of a random variable,
and is defined by
H(x) = −p(x) log p(x)dx,(1)

where p(x) is the probability density function of the randomvariable x. Taking the logarithm to base 2, we have the entropy
measured in bits. However, in the following sections, we takelogarithms to base e because this simplifies the analytical treatment. The entropy ofthe univariate normal distribution p(x) =− μ)− (x 2σ, where μ is the mean and σ is the standard
2√
deviation, is log 2πeσ2 ; the entropy is a monotonically increasing function of the standard deviation. The uniform distribution
√ 1
exp
2π σ2

2


p(x) =

1
b−a

0

a≤x≤b
otherwise(2)

has the largest entropy, log(b − a), among the probability distributions whose domain is between a and b (Cover and Thomas,
2006). In other words, the uniform distribution is the probability
distribution with the highest variability. This means that, under
the assumption of rate coding, the entropy of the output of a neuron is large if the maximal firing rate is large and if the firing rateover time is uniformly distributed. The entropy of the n-variate
normal distribution1
1
exp − (x − μ)T  −1 (x − μ)p(x) = 2(2π)n det (3)√
is log (2πe)n det , where μ is the mean vector and  isthe covariance matrix. The normal distributions with the mean

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 2

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

vectors
μ1 = μ 2 =

0
0

(4)

sensitivity. If the input vector x is transformed to the output vector y having the same number of elements, the entropy of theoutput is given byH(y) =and the covariance matrices
1 =

10
01

(5)

and
√
1√ 1/ 22 =1/ 2 1(6)have the same marginal distributions, i.e., x1 and x2 obey the standard normal distribution in both cases. Thus, the entropies of themarginalized x1 and x2 assume the same value for both distributions. However, the joint entropy of the joint distribution of x1
and x2 with  1 , log2 (2πe) ≈ 4.09
 bits, is greater than that of thejoint distribution with  2 , log2 (2πe)2 /2 ≈ 3.59 bits, because
the variables x1 and x2 are independent in the former and correlated in the latter. Entropy is maximized if the random variables
are independent of each other. We can see that the joint entropyof the output of the neurons takes a large value if these neurons
fire vigorously and independently. Entropy can therefore be used
as a measure of the output variability of a population of neurons.
The entropy of the output of neurons in the sensory areas is large
if their population firing pattern varies with the input, and it issmall if the firing pattern is less affected by the change of input.
The output of sensory neurons with large output entropy can beeasily utilized by higher sensory areas because different inputs arewell separated in the space of the output firing patterns. For thisreason, entropy is used as an objective function to train the models of sensory information processing. Another reason for using
entropy as an objective function is that entropy is a measure of

FIGURE 1 | Schematic representation of the models. (A) Joint entropyof the first-output-layer y+ and y− and second-output-layer z is maximizedin Model 1. V is the connection weight matrix between N = 4 input
neurons and 2N = 8 first-output-layer neurons, and W is the connection
weight matrix between the first-output-layer neurons and N = 4
second-output-layer neurons. The first-output-layer neurons yi+ take on
values greater than zero if ui > 0, and zero otherwise, whereas yi− take on
values greater than zero if ui < 0, and zero otherwise. Black and gray

Frontiers in Computational Neurosciencep(x) log det Jdx + H(x),

(7)

∂yi
where Jij = ∂xis the Jacobian matrix of the transform. Becausejthe (i, j) entry of the Jacobian matrix is the sensitivity of output i to the change of input j, the maximization of the output
entropy H(y) can be regarded as the maximization of the sensitivity of output neurons. It is desirable that sensory neurons are
maximally sensitive to the change of the stimuli. Note that det J
cannot always take an arbitrary large value when we use bounded
functions, such as in Equation 13, as the activation function of
output units.

2.2. MODEL 1

We assume that the system has an N-dimensional input vector x and output comprising a 2N-dimensional first-output-layer
neuron vector y = y+ , y− and an N-dimensional secondoutput-layer neuron vector z (Figure 1A). The output of thefirst-output-layer neurons is a deterministic function of input x
and is described byyi+ = R (ui )(8)yi− = R (−ui ) ,(9)ui = f (ai )

(10)

and

where

arrows indicate modifiable and fixed connections, respectively. Gray filled
circles and open circles are units with and without a nonlinear activation
function f (x), respectively. (B1) Entropy of the outputs u and z is
maximized in Model 2. Output zi is a function of the linear superpositionof inputs yj+ and yj− by the connection weight matrices W+ and W− ,respectively. (B2) Model 2 is equivalent to the maximization of the entropyof the output z, which is a function of the linear superposition of input |ui |by the connection weight matrix W̃.

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 3

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

and


ai =

Vij xj .

(11)

1≤j≤N

Here,

2.3. ENTROPY OF OUTPUT AND DERIVATION OF THE LEARNING RULE
OF MODEL 1x x≥0R(x) =
0 x<0(12)

is the ramp function, andf (x) = 2 arctan tanhwhere Wij± (t) and hi (t) are defined by Equations 29 and 32,
respectively, and  is the learning rate. The update was performed
1.63 × 106 times with  = 10−4 and then 3 × 104 times with  =10−5 .

x2(13)

is the activation function used in this paper. This activation function gives the same algorithm as the ICA algorithm of “tanh nonlinearity” (Hyvärinen et al., 2001) because
f  (x)/f  (x) = − tanh x. This activation function corresponds
to the assumption that the independent components follow
a sparse distribution p(x) = 1/(π cosh x) in terms of maximal likelihood. Different activation functions corresponding
to dense distributions such as that satisfying f  (x)/f  (x) =−x3 cannot predict the receptive field properties of simple
cells. Any decomposition matrix of natural images obtainedusing ICA algorithms can be used as matrix V = (Vij ). In all
simulations of this paper, we use the decomposition matrix
obtained by the method described in Section 2.4. The secondoutput-layer output z is a function of the first-output-layer
output y:

Here we describe the objective function, the entropy of the output
to be maximized, and the derivation of the learning algorithm.
First, we define the joint entropy of the first- and second-outputlayer neurons, and then we calculate the derivative of the joint
entropy with respect to the connection weights. In our model, theN-dimensional input vector gives rise to a 2N-dimensional firstoutput-layer vector and an N-dimensional second-output-layer
vector. The representation is overcomplete because there are more
output components than input components. Shriki et al. (2001)
considered the maximization of the entropy of the overcomplete
representation of the input. Although their model does not contain a multilayer structure and rectifying nonlinearity, the sameidea can be applied to Model 1. The probability density of y and z
is related to the input distribution and is given by the relation
P(x)P(y, z) ∝ ,det(χT χ)
where the susceptibility matrix χ ∈ R3N×N is defined as⎛ +⎞Y⎝
χ = Y− ⎠ .
Z

⎛

⎞






zi = f ⎝−hi +Wij+ yj+ − ȳj+ +Wij− yj− − ȳj− ⎠,(14)
1≤j≤N

1≤j≤N

Yij± =

accelerate the convergence. These terms do not affect the valuesof the parameters after convergence because the increase of ȳj± toȳj± + ȳj± is offset by the change of hi to hi − 1 ≤ j ≤ N Wij± ȳj± .
To enable readers to replicate our results without the need tofollow the details of the derivation of the algorithm, here we summarize the simulation process and describe how to evaluate theentropy of the output and derivation of the learning rule in thenext section. We use a batch learning process to accelerate thesimulation. The weights Wij+ , Wij− , and thresholds hi are updated
once every 100 steps using

Zij =

±

W ←W +±W (t)∂yi±∂xj= ±s (±ui ) f  (ai ) Vij ,
∂zi
∂xj

1≤k≤N

bi = −hi +h←h+h(t),

1≤ t ≤ 100

Frontiers in Computational Neuroscience

(16)

(20)








Wij+ yj+ − ȳj+ +Wij− yj− − ȳj− ,(21)

1≤j≤N

1≤j≤N

and

s(x) =

1≤ t ≤ 100



(19)

 

Wik+ s(uk ) − Wik− s(−uk ) f  (ak )Vkj ,= f  (bi )

(15)

and

(18)

Here we define

where Wij+ and Wij− are connection weights, hi is the offset of thesecond-output-layer neuron i, ȳj+ is the average of yj+ , and ȳj− isthe average of yj− . The terms −ȳj+ and −ȳj− are introduced to

±

(17)

⎧
⎪
⎨1

√1
⎪ 2

⎩

0

x>0
x=0,

(22)

x<0

d
R(f (x)) = s(x)f  (x) and f  (−x) = f  (x). We define
because dx√
s(0) = 1/ 2 to simplify the equation. Here the dependence ofY and Z on the time step t is not explicitly shown. Thus, we

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 4

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

input. Differentiating the first term with respect to Wij± , we obtainmaximize the entropy of the output
 H(y, z) = −  P y, z log P y, z dydzWij± (t) =



P(x)

=

dx
P (x) log det(χT χ)1 = E log det(χT χ) + H(x),
2
=−



∂ 1
T
± 2 log det I + C C
∂Wij
1
2


−1
I + CT C ∂Cklml

1 ≤ l, m ≤ N

where E[·] indicates averaging over the input distribution and weused P(y, z)dydz = P(x)dx (Shriki et al., 2001).
The first term of Equation 23 is given by the average of−1
I + CT C



=

∂Wij±

k



Ckm

±δjl f  (bi )s(±ul )ml

1 ≤ l, m ≤ N

f  (bi ) ±(y − ȳj± )+ Cil f (bi ) j

 1



1
T
T
log det χT χ = log det Y+ Y+ + Y− Y− + ZT Z . (24)22

∂  T ± C C lmml ∂Wij

1 ≤ l, m ≤ N



=

(23)

−1

I + CT C
−1= ± C I + CT C

Cimf  (bi )s(±uj )ijHere note thatTT
Y+ Y+ + Y− Y− =ijVki s(uk )2 f  (ak )2

1≤k≤N

−1

CT
+ C I + CT Cf  (bi )  ±yj − ȳj± , (29)
ii f (bi )1cosh x

(30)

where


2

+ s(−uk )2 f  (ak ) Vkj= VT diag(f  (ak )2 )V ,f  (x) =
(25)

ij

and
f  (x)= − tanh x.
f  (x)where diag(dk ) is a diagonal matrix whose diagonal entries are
d1 , d2 , . . . , dN . We also note that

ZT Z = VT diag(f  (ak ))CT C diag(f  (ak ))V ,
ij

ij

(26)

Differentiating the first term of Equation 28 with respect to hi , weobtainhi (t) =−1
I + CT C=−

Cij = f  (bi ) Wij+ s(uj ) − Wij− s(−uj ) .



=


1
log det VT diag(f  (ai )2 )V + VT diag(f  (ai ))2
× CT C diag(f  (ai ))V + H(x)1
log f  (ak ) + log det V
log det I + CT C +21≤k≤N+ H(x).

(28)

The second and third terms are the objective function used in the
ICA model to generate the connectivity of the first-output-layer
neurons (see Section 2.4). The fourth term is the entropy of the

Frontiers in Computational Neuroscience

∂Ckl
Ckm
∂hi


ml 1 ≤ k ≤ N−1= − C I + CT C
CT

From these relations we obtain
H(y, z) =−1
I + CT C

1 ≤ l, m ≤ N(27)

ml 1 ≤ k ≤ N

1 ≤ l, m ≤ N

where


(31)

f  (bi ).
ii f (bi )δik Cklf  (bk )Ckmf  (bk )(32)

Maximization of the entropy of the second output layer, H(z), can−1
be achieved by replacing (I + CT C) in the above learning rules−1with (CT C) . Results similar to those presented in this paper are
obtained by this modified learning rule, which is equivalent toModel 2 without the constraint Wij+ = Wij− .
2.4. NEWTON METHOD

We used the Newton method for the ICA model proposed in
Amari et al. (1997) and Palmer et al. (2008) to obtain the connection matrix from the input units to first-output-layer neurons
and perform the simulation of Model 2. In general, the entropyof real data in ICA models is not a convex function and has multiple local optima. Thus, the global optimum cannot be foundwww.frontiersin.org

November 2013 | Volume 7 | Article 165 | 5

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

using the Newton method in most cases. However, it accelerates
the convergence to one of the local optima. The following is thesummary of the learning algorithm by Palmer et al. (2008). The
entropy of the output is given by the expectation oflog det V +log f  (yi ),(33)

Vij xj .

(34)

1≤i≤N

where


yi =

1≤j≤N

According to their results, the optimal direction Vij (t) at step t
is given by
V(t) = B(t)V,(35)

where  is the learning rate.
The decomposition matrix V was obtained using this method.
We used randomly selected 20 × 20 pixels image patchesfrom the images and converted the pixels in these image
patches to 400-dimensional real-valued inputs, x. The meanof the pixels of the image was subtracted from each image.
The input images were not prewhitened (Olshausen and Field,
1997) because the input images without the prewhitening process yielded clearer results than the prewhitened ones. The
receptive field properties of the second-output-layer neurons
obtained using prewhitened images were qualitatively similar to those obtained using non-prewhitened images. The
update of the matrix V using Equation 44 was performed
105 times with  = 10−5 , 2.9 × 106 times with  = 10−4 , and9.7 × 106 times with  = 10−5 . The iteration of the learning process of V must be sufficiently long as insufficient
optimization disrupts the receptive field properties of output
neurons.

wheref  (y )Bij =1 + f  (yii) yi
1 + ηif  (y )



(yi )
yj − yi f  (yjj) κj σi2 ff  (y

i)
δij +
1 − δij , (36)κi κj σi2 σj2 − 1

f  (yi )2 − f  (yi )f  (yi ),f  (yi )2
σi2 = E yi2 ,κi = E

(37)
(38)

and
ηi = Ef  (yi )2 − f  (yi )f  (yi ) 2yi ,f  (yi )2(39)

2.5. MODEL 2

In Model 1, we decomposed N pixels into 2N first-output-layer
values and N second-output-layer values. In Model 2, we decomposed 2N input values from the rectified linear simple cell-likeelements into N sign-dependent simple cell-like neurons u and
N complex cell-like neurons z (Figure 1B1). In other words, we
fixed the output of the first half of the neurons to ui , which are
the nonlinear transformations of independent components of theimages, and vary the connection weights to the second half ofthe neurons to obtain complex cell-like properties. 2N inputs are
given by yi+ and yi− and are defined by Equations 8 and 9,
 respectively. The outputs u and z are functions of y = y+ , y− , and aredefined aswheref  (x)2 − f  (x)f  (x)f  (x)2=

1
cosh2 x.

(40)

Here the dependence on t is not explicitly shown. For the online
algorithm, we updated κi , σi2 , and ηi using1
κi (t) = κi (t − 1) +τ1cosh yi (t)2− κi (t − 1) , (41)1σi2 (t) = σi2 (t − 1) +yi (t)2 − σi2 (t − 1) ,τ1
τyi (t)2
cosh2 yi (t)− ηi (t − 1)

(42)



V(t),

1≤ t ≤ 100

Frontiers in Computational Neurosciencezi = f (ci ) ,

(46)

and

where
ci =








Wij+ yj+ − ȳj+ +Wij− yj− − ȳj− . (47)

(44)

1≤j≤N

We do not introduce the offset parameter hi in this model because
hi takes on values close to zero in the simulation of Model 1. Here
we define(43)

at each time step. The integration time constant τ was set to be
10, 000 steps in all simulations. The weights Vij were updated
once every 100 steps using
V←V+

(45)

1≤j≤N

and
ηi (t) = ηi (t − 1) +ui = yi+ − yi−W=

IN −IN
,
W+ W−

(48)

where IN is an N-dimensional identity matrix. We fix the connection weights of the first half of the neurons, u, to simple
cell-like receptive fields, and update W+ and W− using an ICA
algorithm. Because the covariance matrix of independent components is a diagonal matrix (Hyvärinen et al., 2001), W+ and W−

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 6

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

must be appropriately set to decorrelate the outputs. Below,weset W+ = W− = W̃, because assuming ȳi+ = ȳi− , E yi+ yj+ =






E yi− yj− , and E yi+ yj− = E yi− yj+ , all of which approximately hold for independent
components
of natural scenes, the
covariances E ui cj − E [ui ] E cj vanish only if W+ = W− .
Then, the output zi is given by⎛zi = f ⎝







⎞

W̃ij |uj | − |ui | ⎠ ,

(49)

1≤j≤N

where |ui | is the average of |ui |, which equals ȳj+ + ȳj− . If input

yi± is greater than 0, yi∓ is equal to 0; the probability density
p yi+ , yi− is not a function with finite values, and therefore, ICA
algorithms cannot be applied. However, assuming that p yi+ , yi−
is a continuous function, the entropy of the output of this system
is given by the expectation oflog det W +

log f  (ci ) + H(y)1≤i≤N= log det W̃ +

log f  (ci ) + N log 2 + H(y),

(50)

1≤i≤N

where the last two terms do not depend on W̃. Hence, the maximization of the first two terms is sufficient for the maximizationof the output. This is equivalent to the simulation of the ICA
model with the N-dimensional input |ui | − |ui | (Figure 1B2).
Note that ICA can be applied to it because the probability density of p(|ui |) takes finite values, and that the first two terms of−1
Equation 50 equal the first term of Equation 28 if (I + CT C)−1+−
T
is replaced by (C C) and Wij = Wij = W̃ij . We perform the
Newton method for N-dimensional inputs and outputs, which ismuch faster than the gradient descent of Model 1. The update of
the matrix W̃ obtained using Equation 44 was performed 1 × 105
times with  = 10−6 and then 2.99 × 107 times with  = 10−5 .
2.6. CHARACTERIZATION OF MODEL NEURONS

We fit the connection weights to the first-output-layer neurons
from the pixel at (i, j) with the Gabor function
y2
x2
A exp − 2 − 2
2σx
2σy


cos (kx − φ) + B,

(51)

FIGURE 2 | Representation of the connection weights. The connection
weights from input image patches to the simple cell-like first-output-layer
neurons are fitted by a Gabor function. The connection weights from thefirst-output layer neurons to second-output layer neurons are represented
by bars in the second-output-layer neurons. The color and opacity of eachbar indicate the sign (red indicates positive, i.e., the excitatory connections,and blue indicates negative, i.e., the inhibitory connections) and magnitudeof the connection weight from the first-output-layer neuron, respectively.
(A). In Model 1, the left box represents the connection weights from thefirst-output-layer neurons y1+ through yN+ , and the right box represents theconnection weights from y1− through yN− . (B). In Model 2, connection
weights W̃ij are represented by bars in a box.

The phase-dependent (F1) to phase-invariant (F0) component ratio (F1/F0 ratio) has been used to characterize simple
and complex cells (Shapley and Lennie, 1985; Skottun et al.,1991). Simple cells are identified by the F1/F0 ratio greaterthan 1, and complex cells are identified by an F1/F0 ratio
less than 1. We calculate the F1/F0 ratio of model neuron iby using



where


x = (i − x0 ) cos θ + j − y0 sin θ,y = − (i − x0 ) sin θ + j − y0 cos θ

8πwith parameters A, B, x0 , y0 , σx , σy , θ, φ, and k by using the gradient descent method (Figure 2). The sum of the square of thedifference between the fitted function and the connection weights
is less than 10% of the sum of the square of the connection
weights for 398 out of 400 first-output-layer neurons.

Frontiers in Computational Neuroscienceφ R (zi (φ)) sin φ


2

+



φ R (zi (φ)) cos φφ R (zi (φ))

2
, (52)

where φ is the phase of the grating and zi (φ) is the outputof neuron i in response to the grating of the phase φ. Under
this definition, the F1/F0 ratio equals 2 for the simple cell-like
case, zi (φ) = R(sin φ), and equals 0 for the perfectly phaseinvariant case, zi (φ) = 1. We first choose the optimal grating
for each neuron and obtain zi (φ) by varying the phase ofwww.frontiersin.org

November 2013 | Volume 7 | Article 165 | 7

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

the optimal grating. The optimal grating is chosen from thegratings with various radii (2, 3, 4, 5, and 6 pixels), center positions (x, y = 1, 2, . . . , 20), orientations (0◦ , 20◦ , . . . , 340◦ ),spatial frequencies (60◦ /pixel, 75◦ /pixel, . . . , 120◦ /pixel),
and phases (0◦ , 20◦ , . . . , 340◦ ). The center of the optimal grating is used as the center of the gratings when
examining the receptive field properties of model neurons(Figures 4, 6).
Neurons tend to adapt to static stimuli and decrease their firing rates. Moving gratings are frequently used to evoke a large
response in experiments. Because the response of this model
does not depend on the previous stimuli, we use single stationary stimuli as inputs. This does not necessarily mean thatthe results in this paper correspond to the experimental results
obtained using stationary stimuli. We compare the responses ofmodel neurons with experimental results obtained using moving
gratings.

3. RESULTS
3.1. MODEL 1: INFORMATION MAXIMIZATION IN A THREE-LAYERFEEDFORWARD NETWORK

Model 1 is a multilayer feedforward network (Figure 1A). This
network contains N input units (xi ), 2N first-output-layer neurons (yi+ and yi− ), and N second-output-layer neurons (zi ). We
used randomly selected 20 pixels × 20 pixels image patches fromnatural photographs distributed by Prof. Bruno Olshausen on
his homepage (Olshausen and Field, 1997) and converted the
pixels in these image patches to N = 400 real-valued inputs.
The input units correspond to the relay neurons in the lateralgeniculate nucleus, and first- and second-output-layer neurons
correspond to simple and complex cells in V1, respectively. The
intensity of a pixel in the input images is represented by a realvalue in the present paper. In the preprocessing of images, we
subtracted the mean pixel intensity from each image; the mean
pixel intensity of an image patch is not necessarily zero. First,
we performed the learning of the first-output-layer neurons, followed by the learning of the second-output-layer neurons. For
first-output-layer neurons to acquire simple cell-like receptive
field properties, we set the connection weights from the input
units to first-output-layer neurons to the decomposition matrix
V = (Vij ) obtained from the linear ICA of natural image patches.
A standard linear ICA algorithm decomposes the N-dimensional
input vector x into an N-dimensional independent component
vector whose elements can take both signs (Amari et al., 1996).
Because the firing rate of a simple cell is not less than zero, we
made two first-output-layer neurons yi+ and yi− from a nonlinear transformation of independent component i; if ui ≥ 0,
we set yi+ = ui and yi− = 0, and if ui < 0, we set yi+ = 0 and
yi− = −ui , where⎛ui = f ⎝⎞Vij xj ⎠

(53)

1≤j≤N

is the nonlinear transformation of independent component iby the sigmoidal activation function f (x). The first-outputlayer neuron yi+ is selective to the sign-inversion of the image

Frontiers in Computational Neuroscience

to which the first-output-layer neuron yi− is selective, andvice versa. The output of the second-output-layer neuron i isdefined by⎛⎞
Wij+ yj+ +Wij− yj− ⎠ ,(54)zi = f ⎝−hi +
1≤j≤N

1≤j≤N

where Wij+ and Wij− are the connection weights from the first-

output-layer neurons yj+ and yj− , respectively, and hi is thethreshold. We updated the connection weight matrices W+ and
W− , and maximized the entropy of the outputs y+ , y− , and z,
H(y+ , y− , z).
First, we examine the connection weights of the model neurons. Note that we do not impose the constraint Wij+ = Wij− ,which makes second-output-layer neurons invariant to the sign
inversion of input images. However, after the learning process,the connection weights Wij+ and Wij− result in similar values(Figure 3A, p < 0.01, permutation test of Spearman’s rank correlation coefficient). To visualize connection weights from firstoutput-layer neurons to second-output-layer neurons, we haveto represent each first-output-layer neuron compactly. We used aGabor function to fit the connection weights to the first-outputlayer neurons, as shown in Figure 2A, and represent the fitted
Gabor function with a bar. Thus, the first-output-layer neurons
are indicated by bars that represent the optimal orientation andthe spatial location of the fitted Gabor function. In the boxeson the right of Figure 2A, we plotted the bars corresponding to
the first-output-layer neurons. The left one shows the weights
Wij+ of a second-output-layer neuron, and the right one shows
the weights Wij− of the same neuron. The approximate relation

Wij+ ≈ Wij− holds in this neuron. Other examples are shown in
Figure 3B. The strongest excitatory and inhibitory connectionsof each neuron are represented by red (RGB[100%, 0%, 0%])and blue (RGB[0%, 0%, 100%]), respectively. The other connections are represented by a paler red (RGB[100%, 100(1 −r)%, 100(1 − r)%]) and a paler blue (RGB[100(1 − r)%, 100(1 −r)%, 100%]), where r is the ratio of the strength of the connection to the strongest excitatory or inhibitory connection.
These figures indicate that the second-output-layer neurons
tend to receive an input having the same sign from pairsof first-output-layer neurons with sign-inverted receptive fields
and neurons with similar orientation preferences in a small
region.
Figure 4 shows the response properties of second-outputlayer neurons. The image patches presented to the network as
the input are shown on the top of the panels. The output of
the neuron in Figure 2A in response to the phase-shifted gratings is shown in Figure 4A. This figure shows that the output
is positive for all phases, i.e., this neuron is less sensitive tothe phase of the gratings. Insensitivity to the phase of the gratings is a feature of complex cells. Previous experiments havecharacterized simple and complex cells by measuring the relative modulation or phase-dependent (F1) to phase-invariant (F0)component ratio (F1/F0 ratio) in their responses to the optimal gratings (Shapley and Lennie, 1985; Skottun et al., 1991).

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 8

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

FIGURE 3 | Connection weights from the first-output-layer neurons tothe second-output-layer neurons in Model 1. (A) Each dot corresponds
to the connection weights from two first-output-layer neurons yj+ and yj−
with sign-inverted receptive fields to the second-output-layer neuron zi .
(B) Connection weights from the first-output-layer neurons to 15
second-output-layer neurons.

Neurons with the F1/F0 ratio greater than 1 are identified as
simple cells, whereas those with the F1/F0 ratio less than 1
are identified as complex cells. The F1/F0 ratio of the neuron
in Figure 2A is 0.02, which suggests that this cell should beclassified as a complex cell. This phase insensitivity is a resultof the convergence of the connections having the same signfrom two first-output-layer neurons with sign-inverted receptive fields. If a second-output-layer neuron receives connection
weights having the same size from each pair of first-output-layer
neurons with sign-inverted receptive fields, the sign inversionof the input does not change the output of this second-outputlayer neuron. In addition, the convergence of connections having
the same sign from the first-output-layer neurons with similar
orientation preferences facilitates the phase insensitivity of thesecond-output-layer neuron. The black bars in Figure 4B represent the histogram of the F1/F0 ratio of second-output-layer
neurons. This histogram shows that almost all of them are classified as complex cells. In contrast, most of the model neurons
with randomly shuffled connection weights exhibit F1/F0 ratios

Frontiers in Computational Neuroscience

FIGURE 4 | Properties of second-output-layer neurons in Model 1.
(A) Modulation of the output of the second-output-layer neuron whose
connection weights are shown in Figure 2A by the phase of the gratings
presented on the top. (B) Histogram of the F1/F0 ratio of the modelneurons after the learning process (black) and the model neurons whose
connection weights are randomly shuffled (gray). (C) Change in the outputof the same neuron by the radius of the gratings. (D) Distribution of theoutput of 400 second-output-layer neurons in response to the optimal
gratings with the preferred radius and to the same gratings whose radius isenlarged by 6 pixels. (E) Modulation of the output of the same neuron by
the relative orientation of the gratings of the outer annulus to the inner
patch. The dashed line indicates the output of the neuron in response to
the stimulus with the grating of the inner patch only.

greater than 1 (Figure 4B, gray bars). Thus, the F1/F0 ratio close
to zero is not a result of the multilayer structure because thesecond-output-layer neurons in the network with random firstto second-output-layer connections do not exhibit the F1/F0 ratio

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 9

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

close to zero. On the contrary, this is a result of the information maximization in the multilayer network and the resultant
approximate relation Wij+ ≈ Wij− . These results suggest that thephase insensitivity of complex cells originates from an efficientencoding of the visual input. However, F1/F0 ratios of most modelneurons are much smaller than experimentally obtained values.
It is reported that a substantial proportion of complex cells have
F1/F0 ratios greater than 0.5 (Skottun et al., 1991). This discrepancy may suggest that the simple and complex cells in V1 are notas clearly segregated as the first- and second-output-layer neurons
in Model 1.
Stimuli presented in the silent receptive field surrounds canmodulate the response of the cells in V1 to the stimuli presented in the classical receptive field (Jones et al., 2002). In
most of the cells, the suppression is greatest when the orientation of the gratings in the silent surrounds is the same as the
optimal orientation of the classical receptive field. The modelneuron shown in Figure 2A is suppressed when the radius of thegrating is increased (Figure 4C). Inhibitory connections play animportant role in this suppression. Strong inhibitory connections
originate from the first-output-layer neurons with similar orientation preferences as the first-output-layer neurons with strong
excitatory connections. In this neuron, excitatory and inhibitory
areas are separated from each other (Figure 2A). The optimal
grating (the leftmost grating of Figure 4C) does not cover the
inhibitory area. To systematically examine the suppression of
the second-output-layer neurons, we first choose the optimal
disk of gratings with the preferred radius for each neuron, andthen enlarge them by 6 pixels. Figure 4D shows that enlarging
the radius of the optimal gratings decreases the output of mostof the second-output-layer neurons. The activity of a secondoutput-layer neuron is large if the presented grating is restricted
to the area of the receptive fields of first-output-layer neurons
with excitatory connections, whereas the activity diminishes if
the presented grating covers the receptive fields of first-outputlayer neurons with inhibitory connections. Jones et al. (2002)
reported that iso-orientation suppression was found in 94% of
the V1 cells. Similarly, almost all of the second-output-layer neurons (396/400) in Model 1 exhibit the surround suppression.
Figure 4E shows the response of the same neuron as in Figure 4C
to the surrounding annulus for various orientations in the presence of the inner patch of grating at its preferred orientation.
When the orientation of the grating of the outer annulus deviates from the preferred orientation, the degree of suppression
diminishes. This type of response, which is classified as isoorientation suppression (Jones et al., 2002), is a result of theconvergence of the excitatory and inhibitory connections fromfirst-output-layer neurons with similar orientation preferences.
The first-output-layer neurons with inhibitory connections correspond to the silent surroundings outside the classical receptive
field of complex cells. Because the surrounding gratings perpendicular to the center grating suppress the response, this modelneuron is classified into the group called “mixed general suppression and orientation alignment suppression” in Jones et al.
(2002). The responses to the stimulus with the surrounding grating perpendicular to the center grating are greater than that withthe surrounding grating parallel to the center in 376 out of 400

Frontiers in Computational Neuroscience

second-output-layer neurons (p < 0.01, binomial test). However,
the responses to the stimulus with the surrounding grating perpendicular to the center grating are greater than that to thecenter stimulus alone in only 175 second-output-layer neurons.
Jones et al. (2002) reported that 63% of neurons in V1 exhibited cross-orientation facilitation. Model 1 fails to predict theresult.
These results suggest that the second-output-layer neurons
acquire phase insensitivity and complex cell-like receptive field
properties. However, the simulation of Model 1 is computationally expensive, and the learning is very slow to converge.
Therefore, we examine the receptive field properties of the neurons in a simplified model.
3.2. MODEL 2: INFORMATION MAXIMIZATION IN A TWO-LAYERNETWORK WITH SIGN-INVARIANT INPUT

Connection weights Wij+ and Wij− take on close values in thethree-layer network of Model 1 after learning. To examine
why this relation holds, we constructed another model network. In Model 2, we maximize the entropy of outputs u andz (Figure 1B1). Here ui is identical to the nonlinear transformation of independent component i and corresponds to afirst-output-layer neuron in Model 1. Output z corresponds tosecond-output-layer neurons in Model 1, and is connected fromy+ and y− with connection matrices W+ and W− , respectively. We assume that the probability density functions of theoutputs of first-output-layer neurons are even functions, andthat the signs of these outputs are not correlated. This assumption is supported by the observation that less than 1% of
the 2 × 2 contingency tables of the signs of two first-outputlayer neurons contain entries that are greater than 0.26 or lessthan 0.24. Using this assumption, a simple calculation provesthat the entropy of the output is maximized if Wij+ = Wij−
(see Methods). Assuming that Wij+ = Wij− = W̃ij , Equation 54
leads to⎛zi = f ⎝−hi +⎞W̃ij |uj |⎠

(55)

1≤j≤N

(Figure 1B2). In this maximization, we use an ICA algorithmwith N input units, |ui |, and N output units, zi . Model 2 reduces
the computational load and accelerates the convergence of thelearning.
Figure 5 shows the connection weights of 100 output neurons
after learning. Because we assume that Wij+ = Wij− in this model,
the connection weights to a second-output-layer neuron can be
represented by a single box (Figure 2B). The inputs to the second
output-layer neurons have sparse distributions, whose kurtosis
is greater than 3, with the exception of only one model neuron.
None of the outputs of first-output-layer neurons, ui , have kurtosis greater than 3. The receptive fields for more than one thirdof the output neurons are similar to that of the output neuron
shown in Figure 2A. Some of them have excitatory and inhibitory
connections from input neurons with similar orientation preferences, e.g., the third neuron from the right in the second row.

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 10

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

FIGURE 5 | Connection weights from the input neurons to 100 output neurons in Model 2.

The connection weights from the input neurons with the sameorientation selectivity in distant areas of the image patches tend
to have inverted signs. Figure 6A shows that this type of neuron issuppressed if a larger grating is presented, in a way similar to thatshown in Figure 4C. Figure 6B shows that enlarging the radius of
the optimal gratings decreases the output of most neurons. 393
out of 400 neurons of Model 2 also exhibit surround suppression. The neuron in Figure 6C is also suppressed by enlarging theradius of the gratings. However, the orientation preference andspatial alignment of input neurons sending strong inputs to theneuron in Figure 6C are different from those in Figure 6A. The
neuron in Figure 6A is suppressed by parallel bars; in contrast, theneuron in Figure 6C is suppressed by a long bar. Thus, the neuron in Figure 6C is an end-stopped neuron (Hubel and Wiesel,1968), whereas the neuron in Figure 6A is not. Some neurons
have much more complex receptive fields and different response
properties. Figure 6D shows that the activity of a second-outputlayer neuron is suppressed if a grating perpendicular to thepreferred orientation of this neuron is superimposed on the center of the receptive field of the neuron. This type of suppression
was reported in complex cells (Bonds, 1989). This neuron has
excitatory and inhibitory connections from input neurons withorientation selectivity perpendicular to the preferred orientation.
In this type of neuron, the connections from the input neurons with orientation selectivity perpendicular to each other inthe same area of the image patches tend to have inverted signs.
This type of suppression is found in 264 out of 400 secondoutput-layer neurons (p < 0.01, binomial test). Figure 6E shows
an example of the response classified as cross-orientation facilitation (Jones et al., 2002). This neuron is suppressed if the radiusof the optimal grating is enlarged in a manner similar to thatshown in Figures 4C, 6A. When the orientation of the gratingof the outer annulus deviates from that of the inner patch, theresponse exceeds the response to the optimal grating only (shownby the dashed line). In this type of neuron, the connections fromthe input neurons with orientation selectivity perpendicular toeach other in distant areas of the image patches tend to havethe same sign. This type of receptive field configuration facilitates the response of the neuron when the orientation of theouter annulus is perpendicular to the preferred orientation of theneuron. This type of facilitation is found in 369 out of 400 secondoutput-layer neurons (p < 0.01, binomial test). The responses to

Frontiers in Computational Neuroscience

the stimulus with the surrounding grating perpendicular to thecenter grating are greater than that to the center stimulus alone
in 149 second-output-layer neurons only. Model 2 also fails to
predict the experimental result of cross-orientation facilitation(Jones et al., 2002). In some other neurons, the orientation preference is unclear (Figure 5). However, these neurons receive strong
excitatory and inhibitory connections from input neurons whose
preferred positions are restricted to a small area. Thus, theseneurons respond selectively to edges in these areas.
Ringach et al. (2002) reported that the circular variancedefined byθ R(zi (θ)) exp(2iθ)

1−
,
(56)
θ R(zi (θ))
where θ is the orientation of the grating and zi (θ) is the output of neuron i in response to the grating, tends to be greaterthan 0.5 for complex cells. Figure 6F shows that the circular variance of output neurons of Model 2 tends to be greater than0.5, which is consistent with experimental results of complex
cells.

4. DISCUSSION
Our models differ from previous models in several ways. First, togenerate sign-insensitive complex cells, Model 1 does not require
inputs to be insensitive to the signs of pixels. The model inHyvärinen and Hoyer (2001) assumes that complex cells receive
the square of the output of simple cells. Complex cells in themodel by Berkes and Wiskott (2005) are insensitive to the signof image patches because their output is given by the degree
two polynomials of pixel intensities. In the model by Shan et al.
(2007), the output of simple cells is transformed to the absolutevalue and subjected to a nonlinear transformation. The models by
Karklin and Lewicki (2005, 2009) also ignore the sign of the output of simple cells because the output of complex cells in thesemodels depends only on the variance of the simple cells. In contrast, Model 1 can generate sign-insensitive complex cells without
assuming sign-insensitive inputs to complex cells. The information maximization principle gives a possible explanation as to why
sign-insensitive complex cells arise from sign-sensitive simple cell
inputs. The results of Model 1 justify the assumption of Model
2 that the output neurons receive the absolute values of the output of simple cell-like neurons as inputs. Second, our models can

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 11

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

FIGURE 6 | Continued
depending on the radius of the gratings. (D) Modification of the output of aneuron by the relative orientation of the gratings superimposed on the
optimal grating. The dashed line indicates the output of the neuron in
response to the optimal grating only. (E) Modulation of the output of aneuron by the orientation of the gratings of the outer annulus. The dashedline indicates the output of the neuron in response to the stimulus with thegrating of the inner patch only. (F) Histogram of circular variance oforientation tuning of output neurons.

FIGURE 6 | Properties of output neurons in Model 2. (A) Change in the
output of a neuron by the radius of the gratings. (B) Distribution of theoutput of 400 neurons in response to the optimal gratings with thepreferred radius and to the same gratings whose radius is enlarged by 6
pixels. (C) End stopping exhibited by a neuron changing its output(Continued)

Frontiers in Computational Neuroscience

predict the receptive fields exhibiting surround suppression andfacilitation. The complex cell-like receptive fields produced by themodels of Földiák (1991) and Hyvärinen and Hoyer (2001) aredevoid of these properties. The models of Földiák (1991) andBerkes and Wiskott (2005) also differ from our models in that
their models require time-varying sequences of image patches
as inputs. Although these models give results similar to ourmodels, none of these models are based on the information maximization principle, but they are based on other optimization
criteria such as the parameter fitting of the probability distributions and the minimization of the temporal change of the
response.
In Model 2, we transform the i-th independent component,ai , to |ui | = |f (ai )|. Shan et al. (2007) used a nonlinear functionthat transforms the absolute value of the independent components into a standard normal distribution. In Model 2, |ui | isalmost uniformly distributed rather than normally distributed
because the entropy of f (ai ) is maximized when f (ai ) is uniformly distributed in the range of the bounded function f (x).
Our results show that a uniformly distributed input can form
complex cell-like receptive fields. The model proposed by Karklin
and Lewicki (2005) also resembles Model 2. Each output unit of
their model detects a specific set of covariances among input variables. For example, an output becomes large when |x1 | and |x2 |are large, and another output becomes large when |x1 | is large
and |x2 | is small. The linear superposition of |ui | plays a similarrole in Model 2. A large |ui | indicates that the absolute value ofthe i-th simple cell-like input is large, and a small |ui | indicatesthat the absolute value of the i-th simple cell-like input is small.
Their model requires the sparseness of the output distribution,
which is also required in Model 2, because the linear ICA algorithms can be derived by assuming the sparseness of the source
distribution.
The nonlinearity of V1 complex cells was studied by using
the models with the pooling of simple cell-like units (Sakai andTanaka, 2000; Martinez and Alonso, 2001). We attempted to
examine the second-order nonlinearity of the second-outputlayer neurons. However, reverse correlation of these neurons
did not exhibit second-order Wiener-like kernels that are similar to those observed for complex cells (Szulborski and Palmer,1990). This is presumably because the widths of on- and offregions of most first-output-layer neurons are as narrow as
1 pixel.
From our study, we speculate that complex features can bedetected by combining the inputs from a large number of simpler elements. By increasing the number of output layers in our

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 12

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

models, we will have higher-order neurons that are selective tomore complex features. The information maximization of a multilayer network with higher-order neurons would be capable ofexplaining the complicated selectivity of neurons in higher visualareas.
However, there are some limitations to our models, such as
the fact that our models use feedforward networks. Although thisstructure simplifies the models and facilitates the derivation of
the learning rules, cortical networks have feedback and recurrent structures as well as feedforward structures. It is known thatthe feedback from higher to lower sensory areas plays an essential role in sensory cortices. Bardy et al. (2006) reported that theinactivation of the feedback from the higher visual areas affected
the selectivity of the neurons in V1. They found that this inactivation changed the responses of a substantial proportion ofneurons classified as complex cells to simple cell-like responses,
indicating that the feedback from higher visual areas modifies
the receptive fields of complex cells. The response of complex
cells appears to be formed by both a feedforward mechanism
and a feedback and recurrent mechanism. Simple cells, which areassumed to provide inputs to complex cells in this paper, receive
recurrent connections so that they exhibit surround suppression(Burr et al., 1981; Walker et al., 2000). Ringach et al. (2002)
showed that simple cells with odd-symmetric receptive fields in
V1 are greater in number than those predicted from ICA models. This may be the reason for which reverse correlation of thesecond-output-layer neurons did not exhibit Wiener-like kernels
that are similar to those observed for complex cells. Although thereceptive fields of some second-output-layer neurons in our models are irregular and different from model neurons with typicalcomplex cell-like receptive fields, the introduction of higherorder neurons and recurrent connections (Tanaka et al., 2009)
could cause the first-output-layer neurons to exhibit properties
that are more similar to simple cells; in turn, it may increase
the number of model neurons with complex cell-like receptive
fields.

ACKNOWLEDGMENTS
This work was supported by a Grant-in-Aid for Scientific
Research on Innovative Areas “Mesoscopic Neurocircuitry” (No.
23115512) of The Ministry of Education, Science, Sports andCulture of Japan.

REFERENCES
Amari, S., Chen, T.-P., and Cichocki, A. (1997). Stability analysis of learningalgorithms for blind source separation. Neural Networks 10, 1345–1351. doi:
10.1016/S0893-6080(97)00039-7
Amari, S., Cichocki, A., and Yang, H. H. (1996). A new learning algorithm for blind
signal separation. Adv. Neural Inf. Process. Syst. 8, 757–763.
Bardy, C., Huang, J., Wang, C., FitzGibbon, T., and Dreher, B. (2006).
‘Simplification’ of responses of complex cells in cat striate cortex: suppressive surrounds and ‘feedback’ inactivation. J. Physiol. 574, 731–750. doi:
10.1113/jphysiol.2006.110320
Barlow, H. B. (1959). “Sensory mechanisms, the reduction of redundancy, andintelligence,” in The Mechanisation of Thought Processes, London: Her Majesty’s
Stationery Office. 535–539.
Bell, A., and Sejnowski, T. (1995). An information-maximization approach toblind separation and blind deconvolution. Neural Comput. 7, 1129–1159. doi:
10.1162/neco.1995.7.6.1129

Frontiers in Computational Neuroscience

Bell, A., and Sejnowski, T. (1997). The “independent components” of naturalscenes are edge filters. Vision Res. 37, 3327–3338. doi: 10.1016/S0042-6989(97)00121-1
Berkes, P., and Wiskott, L. (2005). Slow feature analysis yields a rich repertoire ofcomplex cell properties. J. Vis. 5, 579–602. doi: 10.1167/5.6.9
Bonds, A. B. (1989). Role of inhibition in the specification of orientation
selectivity of cells in the cat striate cortex. Visual Neurosci. 2, 41–55. doi:
10.1017/S0952523800004314
Burr, D., Morrone, C., and Maffei, L. (1981). Intra-cortical inhibition prevents
simple cells from responding to textured visual patterns. Exp. Brain Res. 43,
455–458.
Cover, T. M., and Thomas, J. A. (2006). Elements of Information Theory, 2nd Edn.
Hoboken, NJ: John Wiley and Sons, Inc.
Földiák, P. (1991). Learning invariance from transformation sequences. NeuralComput. 3, 194–200. doi: 10.1162/neco.1991.3.2.194
Hubel, D. H., and Wiesel, T. N. (1959). Receptive fields of single neurones in the
cat’s striate cortex. J. Physiol. (Lond.) 148, 574–591.
Hubel, D. H., and Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. 160,
106–154.
Hubel, D. H., and Wiesel, T. N. (1968). Receptive fields and functional architectureof monkey striate cortex. J. Physiol. 195, 215–243.
Hyvärinen, A., and Hoyer, P. O. (2001). A two-layer sparse coding model learns
simple and complex cell receptive fields and topography from natural images.
Vision Res. 41, 2413–2423. doi: 10.1016/S0042-6989(01)00114-6
Hyvärinen, A., Karhunen, J., and Oja, E. (2001). Independent Component Analysis.
(New York, NY: John Wiley and Sons, Inc.) doi: 10.1002/0471221317
Jones, H., Grieve, K., Wang, W., and Sillito, A. (2001). Surround suppression inprimate v1. J. Neurophysiol. 86, 2011–2028. doi: 10.1152/jn.00403.2001
Jones, H. E., Wang, W., and Sillito, A. M. (2002). Spatial organization and magnitude of orientation contrast interactions in primate V1. J. Neurophysiol. 88,
2796–2808.
Karklin, Y., and Lewicki, M. (2005). A hierarchical Bayesian model for learning nonlinear statistical regularities in nonstationary natural signals. NeuralComput. 17, 397–423. doi: 10.1162/0899766053011474
Karklin, Y., and Lewicki, M. (2009). Emergence of complex cell properties by learning to generalize in natural scenes. Nature 457, 83–86. doi: 10.1038/nature07481
Linsker, R. (1988). Self-organization in a perceptual network. Computer 21,
105–117. doi: 10.1109/2.36
Martinez, L. M., and Alonso, J.-M. (2001). Construction of complex receptive
fields in cat primary visual cortex. Neuron 32, 515–525. doi: 10.1016/S08966273(01)00489-5
Olshausen, B. A., and Field, D. J. (1996). Emergence of simple-cell receptive field
properties by learning a sparse code for natural images. Nature 381, 607–609.
doi: 10.1038/381607a0
Olshausen, B. A., and Field, D. J. (1997). Sparse coding with an overcomplete basis
set: a strategy employed by V1? Vision Res. 37, 3311–3325. doi: 10.1016/S00426989(97)00169-7
Palmer, J., Makeig, S., Delgado, K., and Rao, B. (2008). “Newton method for the
ICA mixture model,” in Proceedings of the 33rd IEEE International Conferenceon Acoustics and Signal Processing (ICASSP 2008), (Las Vegas), 1805–1808. doi:
10.1109/ICASSP.2008.4517982
Ringach, D. L., Shapley, R. M., and Hawken, M. J. (2002). Orientation selectivity in macaque v1: diversity and laminar dependence. J. Neurosci. 22,
5639–5651.
Sakai, K., and Tanaka, S. (2000). Spatial pooling in the second-order spatial
structure of cortical complex cells. Vision Res. 40, 855–871. doi: 10.1016/S00426989(99)00230-8
Shan, H., Zhang, L., and Cottrell, G. (2007). Recursive ICA. Adv. Neural Inf. Process.
Syst. 19, 1273–1280.
Shapley, R., and Lennie, P. (1985). Spatial frequency analysis in the visual system.
Ann. Rev. Neurosci. 8, 547–581. doi: 10.1146/annurev.ne.08.030185.002555
Shriki, O., Sompolinsky, H., and Lee, D. (2001). An information maximization approach to overcomplete and recurrent representations. Adv. Neural Inf.
Process. Syst. 13, 612–618.
Skottun, B., De Valois, R., Grosof, D., Movshon, J., Albrecht, D., andBonds, A. (1991). Classifying simple and complex cells on the basis of
response modulation. Vision Res. 31, 1079–1086. doi: 10.1016/0042-6989(91)90033-2

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 13

Tanaka and Nakamura

Infomax explains complex cell-like selectivity

Szulborski, R. G., and Palmer, L. A. (1990). The two-dimensional spatial structureof nonlinear subunits in the receptive fields of complex cells. Vision Res. 30,
249–254. doi: 10.1016/0042-6989(90)90040-R
Tanaka, T., Kaneko, T., and Aoyagi, T. (2009). Recurrent infomax generates cell
assemblies, neuronal avalanches, and simple cell-like selectivity. Neural Comput.
21, 1038–1067. doi: 10.1162/neco.2008.03-08-727
Walker, G., Ohzawa, I., and Freeman, R. (2000). Suppression outside
the classical cortical receptive field. Visual Neurosci. 17, 369–379. doi:
10.1017/S0952523800173055
Conflict of Interest Statement: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could beconstrued as a potential conflict of interest.

Frontiers in Computational Neuroscience

Received: 02 July 2013; accepted: 26 October 2013; published online: 21 November
2013.
Citation: Tanaka T and Nakamura K (2013) Information maximization principle
explains the emergence of complex cell-like neurons. Front. Comput. Neurosci. 7:165.
doi: 10.3389/fncom.2013.00165
This article was submitted to the journal Frontiers in Computational Neuroscience.
Copyright © 2013 Tanaka and Nakamura. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY).
The use, distribution or reproduction in other forums is permitted, provided
the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice.
No use, distribution or reproduction is permitted which does not comply withthese terms.

www.frontiersin.org

November 2013 | Volume 7 | Article 165 | 14