# Dynamics of Moment Neuronal Networks with Intra- and Inter-Interactions

Authors: Xuyan Xiang; Jianguo Wu
Journal: Mathematical Problems in Engineering
 (2015)

---

## Abstract

A framework of moment neuronal networks with intra- and inter-interactions is presented. It is to show how the spontaneous activity is propagated across the homogeneous and heterogeneous network. The input-output firing relationship and the stability are first explored for a homogeneous network. For heterogeneous network without the constraint of the correlation coefficients between neurons, a more sophisticated dynamics is then explored. With random interactions, the network gets easily synchronized. However, desynchronization is produced by a lateral interaction such as Mexico hat function. It is the external intralayer input unit that offers a more sophisticated and unexpected dynamics over the predecessors. Hence, the work further opens up the possibility of carrying out a stochastic computation in neuronal networks.


---

## Body

## 1. Introduction

The theory of moment neuronal networks (MNN) was developed recently [1], which takes into account both the first and the second order statistics of spike trains, generalizing the case of Poisson synaptic inputs to more biologically plausible renewal processes. Therefore, the MNN framework can be considered as an attempt toward a general framework of computation with stochastic systems. Further a more biologically reasonable MNN with intra- and interlayer interactions was developed by introducing intralayer inputs [2], which is analogous to the “networks with context units” [3] in ANNs. It shows that even a single unit with such system, as analogous to [4–6], is able to perform various complex nonlinear tasks like the XOR problem and that it can reach the trade-off between the output bias and variance by determining the optimal penalty factor due to a specific learning task.In this letter we will explore how the spontaneous activity is propagated across the feedforward network with both homogeneous and heterogeneous connections, derived from the intralayers inputs. The input-output firing relationship and the stability are first explored for a homogeneous network. Synchronization or desynchronization is then presented for heterogeneous network by random or lateral interactions. It is the external intralayer input unit that offers a more sophisticated and unexpectedly dynamics over MNN.This paper is organized as follows. The framework of MNN with intra- and interlayer interactions is presented in Section2. The dynamics of homogeneous and heterogeneous network are explored in Section 3.


## 2. Moment Neuronal Networks with Intra- and Inter-Interactions

For the decay rateL, when the membrane potential V
i
k
+
1
(
t
) of the ith neuron in the (
k
+
1
)th layer is between its resting state V
r and its threshold V
t, it satisfies the following dynamics:
(1)
d
V
i
(
k
+
1
)
t
=
-
L
V
i
k
+
1
t
-
V
r
d
t
+
d
I
i
,
syn
k
+
1
t
,
where the synaptic input I
i
,
syn
(
k
+
1
)
(
t
) is given by
(2)
d
I
i
,
syn
k
+
1
t
=
∑
j
=
1
p
(
k
)
,
i
r
ω
i
j
E
,
(
k
)
,
i
r
d
N
j
E
,
(
k
)
,
i
r
(
t
)
-
∑
j
=
1
q
(
k
)
,
i
r
ω
i
j
I
,
(
k
)
,
i
r
d
N
j
I
,
k
,
i
r
(
t
)
+
∑
j
=
1
p
(
k
+
1
)
,
i
a
ω
i
j
E
,
(
k
+
1
)
,
i
a
d
N
j
E
,
(
k
+
1
)
,
i
a
(
t
)
-
∑
j
=
1
q
(
k
+
1
)
,
i
a
ω
i
j
I
,
(
k
+
1
)
,
i
a
d
N
j
I
,
(
k
+
1
)
,
i
a
(
t
)
.
Here ω
i
j
E
,
(
k
)
,
i
r and ω
i
j
I
,
(
k
)
,
i
r are the magnitudes of excitatory postsynaptic potentials (EPSPs) and inhibitory postsynaptic potentials (IPSPs), {
N
i
E
,
(
k
)
,
i
r
(
t
)
} and {
N
i
I
,
(
k
)
,
i
r
(
t
)
} are renewal processes arriving from the ith and jth synapse, and p
(
k
) and q
(
k
) are the total number of active excitatory and inhibitory synapses in the kth layer. All notations with the superscriptia imply that they receive the external stimulus from intralayer interactions rather than from the network itself, similar to the “networks with context unit” [3]. It would present a more biologically reasonable network architecture than the moment neuronal networks. When V
i
(
k
+
1
)
(
t
) crosses the membrane threshold V
t from below, a spike is generated and the membrane resets to its resting potential V
r.The renewal process{
N
i
E
,
(
k
)
,
i
x
(
t
)
} can be approximated as [1, 2]
(3)
d
N
i
E
,
(
k
)
,
i
x
(
t
)
~
μ
i
E
,
(
k
)
,
i
x
d
t
+
σ
i
E
,
(
k
)
,
i
x
d
B
i
E
,
(
k
)
,
i
x
(
t
)
,
with
(4)
μ
i
j
E
,
(
k
)
,
i
x
=
1
T
i
j
E
,
(
k
)
,
i
x
+
T
ref
,
σ
i
j
E
,
k
,
i
x
2
=
Var
⁡
T
i
E
,
k
,
i
x
T
i
j
E
,
k
,
i
x
3
,
where 〈
T
i
E
,
(
k
)
,
i
x
〉 and Var
⁡
(
T
i
E
,
(
k
)
,
i
x
) are the mean and variance of the interspike intervals (ISIs) of the renewal process {
N
i
E
,
(
k
)
,
i
x
(
t
)
}, respectively, the superscriptix could be eitherir oria, and T
ref is the refractory period.For the conciseness of notation, we take the intrainput from the(
k
+
1
)th layer as external context input paralleled to the kth layer, which is denoted by the superscriptia. Furthermore, we suppose that q
(
k
)
,
i
x
=
p
(
k
)
,
i
x,  μ
(
k
)
,
i
x
≡
μ
I
,
(
k
)
,
i
x
=
μ
E
,
(
k
)
,
i
x,  σ
(
k
)
,
i
x
≡
σ
I
,
(
k
)
,
i
x
=
σ
E
,
(
k
)
,
i
x,   and  ω
i
j
I
,
(
k
)
,
i
r
=
r
i
r
ω
i
j
E
,
(
k
)
,
i
r
≡
r
ω
i
j
(
k
)
,
i
r
≡
r
ω
i
j
(
k
),  ω
i
j
I
,
(
k
)
,
i
a
=
r
i
a
ω
i
j
E
,
(
k
)
,
i
a
≡
r
i
a
ω
i
j
(
k
)
,
i
a,  ρ
m
n
(
k
)
≡
ρ
m
n
(
k
)
,
i
r; then (2) can be approximated as (see [2])
(5)
d
I
i
,
syn
(
k
+
1
)
t
≡
μ
¯
i
(
k
)
d
t
+
σ
¯
i
(
k
)
d
B
t
,
where
(6)
μ
¯
i
(
k
)
=
∑
j
=
1
p
(
k
)
ω
i
j
(
k
)
μ
j
(
k
)
(
1
-
r
)
+
∑
j
=
1
p
(
k
)
,
i
a
ω
i
j
(
k
)
,
i
a
μ
j
(
k
)
,
i
a
(
1
-
r
i
a
)
,
(7)
σ
¯
i
(
k
)
2
=
∑
m
,
n
=
1
p
(
k
)
‍
ω
i
m
(
k
)
σ
m
(
k
)
ω
i
n
(
k
)
σ
n
(
k
)
ρ
m
n
(
k
)
(
1
+
r
2
)
+
∑
m
,
n
=
1
p
(
k
)
,
i
a
‍
ω
i
m
(
k
)
,
i
a
σ
m
(
k
)
,
i
a
ω
i
n
(
k
)
,
i
a
σ
n
(
k
)
,
i
a
ρ
m
n
(
k
)
,
i
a
[
1
+
r
i
a
2
]
.
Here r
i
a is the ratio between inhibitory inputs and excitatory inputs from the intralayer; ρ
j
s
(
k
)
,
i
a is correlation coefficient between the jth and sth input from intralayer.In terms of Siegert’s expression [7], we have the expression of the mean and variance of the output ISIs:
(8)
T
i
(
k
+
1
)
=
2
L
∫
A
i
(
k
)
(
V
r
)
A
i
(
k
)
(
V
t
)
‍
a
x
d
x
,
Var
⁡
T
i
(
k
+
1
)
=
4
L
2
∫
A
i
(
k
)
(
V
r
)
A
i
k
(
V
t
)
‍
h
x
d
x
,
where
(9)
A
i
k
y
=
y
L
-
μ
¯
i
k
σ
¯
i
k
L
,
a
x
=
exp
⁡
x
2
∫
-
∞
x
exp
⁡
-
u
2
d
u
,
h
(
x
)
=
exp
⁡
(
x
2
)
∫
-
∞
x
exp
⁡
-
u
2
a
2
u
d
u
.
As discussed in [2], for the correlation between the inputs and outputs, we can assume that the followingheuristic relationship holds:
(10)
ρ
i
j
(
k
+
1
)
=
∑
m
,
n
ω
i
m
(
k
)
σ
m
(
k
)
ω
j
n
(
k
)
σ
n
(
k
)
ρ
m
n
(
k
)
∑
m
,
n
ω
i
m
(
k
)
σ
m
(
k
)
ω
i
n
(
k
)
σ
n
(
k
)
ρ
m
n
(
k
)
∑
m
,
n
ω
j
m
(
k
)
σ
m
(
k
)
ω
j
n
(
k
)
σ
n
(
k
)
ρ
m
n
(
k
)
.
Note that the right-hand side of (10) is the correlation of the inputs to the ith and the jth neurons in the (
k
+
1
)th layer, which includes the intra- and interinput correlations.The above relationships between inputs and outputs lay the foundation of the moment neural networks with intra- and interinteractions (MNNIII).


## 3. Dynamics of Networks

The question we intend to address here is how a spontaneous activity can be maintained in a feedforward network. For all simulations, the values of the decay rate, threshold, and resting potential were set equal toL
=
1
/
20 ms−1, V
t
=
20 mV, and V
r
=
0 mV, respectively. The same parameters have been employed elsewhere [8] and are thought to be in the physiological range for visual cortex cells and in agreement with most published results [9, 10]. The coefficient of variation, CV
=
var
(
T
)
/
〈
T
〉, is used to quantify the irregularity of a spike train. If CV
=
0, the spike train is regular; otherwise the spike train is random. As analogous to [1], we stop our simulation when the firing rate is slower than 0.001 Hz. All simulations were carried out with MATLAB.


### 3.1. Dynamics of Homogeneous Network

In this section, we focus on a homogeneous network, where all weights, afferent means, and variances were set to be identical. Now the quantities in (6) reduce to μ
1
(
k
)
=
μ
j
(
k
),  σ
1
(
k
)
=
σ
j
(
k
),  μ
1
(
k
)
,
i
a
=
μ
j
(
k
)
,
i
a,  σ
1
(
k
)
,
i
a
=
σ
j
(
k
)
,
i
a, and ω
12
(
k
)
=
ω
i
j
(
k
)
=
ω
(
k
)
=
ω,  ω
12
(
k
)
,
i
a
=
ω
i
j
(
k
)
,
i
a
=
ω
(
k
)
,
i
a
=
ω
i
a,  i
≠
j. As we discussed before, the propagation of correlation becomes trivial in such case since all cells become fully correlated after the first layer. To avoid this, we clamped the correlation coefficient; that is, we set ρ
12
(
k
)
=
ρ
i
j
(
k
)
=
ρ
(
k
)
=
ρ,  ρ
12
(
k
)
,
i
a
=
ρ
i
j
(
k
)
,
i
a
=
ρ
(
k
)
,
i
a
=
ρ
i
a,   i
≠
j. In simulations we set ρ
i
a
=
ρ
=
0 or 0.1, in agreement with experimental data reported in the literature [1, 11, 12], and we assume that each intralayer input, including numbers of neurons, afferent means, and variances, is identical; that is, p
(
1
)
,
i
a
=
p
(
k
)
,
i
a
≡
p
i
a,  μ
1
(
1
)
,
i
a
=
μ
1
(
k
)
,
i
a, and σ
1
(
1
)
,
i
a
=
σ
1
(
k
)
,
i
a.First we chooser
=
0.1, 0.5, 1.5 to illustrate how the activity is propagated across the networks, varied from r
i
a and p
i
a, on assumption that C
1
(
k
)
=
C
1
(
k
)
,
i
a
=
1. Unless otherwise specified, the initial CV in all simulations is always 1, and the weight is 0.5. In Figure 1, we show the results obtained for various values of μ
(
1
) and σ
(
1
) (we reported the coefficient of variation C
(
k
)
=
σ
(
k
)
/
μ
(
k
)). As stated in [1], each data point (μ
(
k
)
,
C
(
k
)) is connected with (μ
(
k
+
1
)
,
C
(
k
+
1
)) to illustrate how the activity is propagated across the networks. After the first few layers, for r
=
0.1 (top), neurons are found to be either silent or firing at relatively high frequency for all r
i
a and p
i
a. For r
=
0.5 (middle), however, the situation has changed. On the one hand, the left panel (middle) shows that even if there is no inhibitory input (r
i
a
=
0) for a small number of intraneurons (p
i
a
=
10), to be silent is certainty on fewer intrainput neurons. On the other hand, the right panel (middle) shows that not all networks are certainty to be silent. As stated earlier, for r
=
0.5 with fewer intrainput neurons (p
i
a
=
10), the network is certainty to be silent (left). However, with increasing size of intrainput units, the network might be fixing or firing at relatively high frequency (right). Instead, they fire at low or relatively high frequency; for example, p
i
a
≥
30 for r
1
i
a
=
0.1. It actually shows that the networks can be stable at low frequency; see discussions later. In addition, for r
=
1.5 (bottom), there is another kind of properties that neurons fire at relatively low frequency after a few times oscillations. Unlike a homogeneous MNN [1], neurons are certainty to be silent with a larger ratio (e.g., r
≥
0.4) after the first few layers. It is obvious that the intrainputs cause a more sophisticated dynamics of network than MNN.Figure 1
Propagation of activity in a homogeneous feedforward MNNIII. For each layerk, the output firing rate (μ
1
(
k
)) and the coefficient of variation (C
1
(
k
)) are reported on the abscissa and the ordinate, respectively. Points labeled 1 correspond to the first layer; points corresponding to successive layers are connected by lines. Results were obtained for p
=
100,  ρ
i
j
=
ρ
i
j
i
a
=
0,  μ
1
(
1
)
=
50 Hz (solid lines), 100 Hz (dashed lines), and r
=
0.1 (top), r
=
0.5 (middle), and r
=
1.5 (bottom). In each panel, the ratios from intralayer (r
i
a) are evenly spaced.Figure1 implies that the stable solution existed in some networks. Thus we explore the stable solution of such network. Figure 2 shows how the input-output firing rate relationship is varied from p
i
a and r
i
a, for r
=
0.5 and r
=
1.5. As analogous to [1], roughly speaking, the stable solution appears at low firing rate for networks of increasing size (either p or p
i
a size) with strong inhibition (either r or r
i
a). However, the preferred appearance of stable solution at low frequency is not the increasing size of both inter- and intralayer; for r
=
1.5 (top right, dashed lines), for example, any ratio r
i
a from intralayer (from 0 to 1.5) appears as a fixed point at low firing rate when p
=
300,  p
i
a
=
100 (top right), instead of p
=
300,  p
i
a
=
300 (bottom right).Figure 2
The input-output firing rate relationship for a homogeneous MNNIII varied from numbers of intralayer neuronsp
i
a and the ratio of intrainputs r
i
a. Results were obtained for ρ
i
j
=
ρ
i
j
i
a
=
0.1 and r
=
0.5 (solid lines), r
=
1.5 (dashed lines).We further explore the stability of such network by showing that the stable outputs varied from(
p
i
a
,
r
i
a
). Figure 3 shows the stable outputs versus r
i
a and p
i
a, with a fixed ratio from interlayer inputs r
=
0.8 (top) and r
=
1.5 (bottom) and with the number of neurons from interlayer inputs p
=
100 (left) and p
=
300 (right). The initial inputs fix 5 Hz in the current and latter simulations. It is found that the intralayer input unit appears to have a sophisticated role in the stability of such network, depending on the ratio r
i
a and the size p
i
a of context unit, together with the ratio r and the size p of network itself, that is, interlayer. For example, for r
=
0.8 (top), the stable output appears to increase with the increasing size p
i
a and the decreasing ratio r
i
a of context unit, which plays a general role; for r
=
1.5 and p
=
300 (bottom right panel), however, the situation has changed to play a negative role; see p
i
a
⩾
300.Figure 3
Stable outputs for a homogeneous MNNIII varied from numbers of intralayer neuronsp
i
a and the ratio of intralayer r
i
a. Results were obtained for r
=
0.8 (top) and 1.5 (bottom) and p
=
100 (left) and 300 (right), with ρ
i
j
=
ρ
i
j
i
a
=
0.1.We also explore the stability of such network by showing that the stable outputs varied from numbers of intralayer neuronsp
i
a and of interlayer p. Figure 4 shows that the network with a small size of intra- and interlayer will be either silent or firing at low frequency. Top panel shows that the stable output of such network seems to be symmetry between numbers of intralayers and interlayers when r
=
1. However, there are obvious asymmetry for r
<
1 or r
>
1; see middle and bottom panels in Figure 4. Middle panel shows that the network with a relatively large size of interlayer, for r
<
1, will be firing at relatively high rate and that the size of intralayer has negligible effect on the stable outputs such as p
>
250. Bottom panel presents unexpected results of r
>
1 that relatively high frequency is firing at the large size of intralayer and small size interlayer, rather than the large size of intralayer and interlayer. In particular, for p
>
300 and p
i
a
>
300, the network is unexpectedly firing at low frequency when r
=
1.5 and r
i
a
=
0.5; see the left of bottom panel. All those imply that the current network can lead to more sophisticated dynamics than MNN and the intrainput unit plays a very sophisticated role in dynamics of such network.Figure 4
Stable outputs for a homogeneous MNNIII varied from numbers of intralayer neuronsp
i
a and of interlayer p. Results were obtained for r
=
1 (top), r
=
0.5 (middle) and 1.5 (bottom) and r
i
a
=
0.5 (left) and r
i
a
=
1.5 (right), with ρ
i
j
=
ρ
i
j
i
a
=
0.1.Finally, we explore the stability of such network by showing that the stable outputs varied from the ratio of both inter- and intralayers, that is,r and r
i
a. Figure 5(a) shows the stable outputs versus r and r
i
a with p
=
p
i
a
=
300. It can be seen that the network will be either silent or firing at low frequency when r
>
1 and will be firing at relatively high rate when r
<
1, which implies that the ratio (between inhibitory and excitatory input) from the network itself plays a more important role than that from intrainput unit with such parameters. However, the network will converge to silent or to low firing frequency with strong inhibition r
>
1 or to high frequency with weak inhibition r
<
1. Unlike the stable output of MNN in [1] (a high level of inhibition is necessary to maintain a spontaneous activity, that is, a low output firing rate), a weaker inhibition (r
<
1 approaching 1), even with weaker intrainhibition (r
i
a
<
1 approaching 1), can converge to low firing frequency; see, for example, r
∈
(
0.8,1
) and r
i
a
∈
(
0.5,1
). It is the intrainput unit to maintain the low frequency activity. Thus there is an important gain in computational power over MNN. Note the appearance of oscillation for a homogeneous MNNIII as singularity, for example, with r
=
2,  r
i
a
=
0.1,  p
=
p
i
a
=
300, and  ρ
i
j
=
ρ
i
j
i
a
=
0.1 (Figure 5(b)).(a) Stable outputs for a homogeneous MNNIII varied from the ratio of both inter- and intralayer (r and r
i
a). Results were obtained for p
=
p
i
a
=
300, with ρ
i
j
=
ρ
i
j
i
a
=
0.1. (b) The appearance of oscillation for a homogeneous MNNIII.
(a)
(b)


### 3.2. Dynamics of Heterogeneous Network

We now turn to explore a more sophisticated dynamics of a heterogeneous network. We shed light on, for instance, how synchronization or desynchronization is produced by random or lateral connections when we remove the constraint on the correlations between neurons.In accordance with the results reported in the previous publications, we fixed the total number of neurons in each interlayer top
(
k
)
=
p
=
324. To show the influences of intralayer inputs on networks, we also fixed the total number of neurons in each intralayer to p
(
k
)
,
i
a
=
p
i
a
=
324. We further assumed that each intralayer receives the same inputs as the first intralayer; that is, w
i
j
(
k
)
,
i
a
=
w
i
j
(
1
)
,
i
a,  μ
i
(
k
)
,
i
a
=
μ
i
(
1
)
,
i
a,  C
i
(
k
)
,
i
a
=
C
i
(
1
)
,
i
a, and ρ
i
j
(
k
)
,
i
a
=
ρ
i
j
(
1
)
,
i
a for i
,
j
=
1
,
…
,
324.First we generated random connectionsw
i
j
(
k
)
,
w
i
j
(
1
)
,
i
a
∈
[
0,0.5
] and inputs μ
i
(
1
)
,
μ
i
(
1
)
,
i
a
∈
[
0,100
] Hz. We also assumed that C
1
(
1
)
=
C
1
(
1
)
,
i
a
=
1 and that ρ
i
j
(
1
)
,
ρ
i
j
(
1
)
,
i
a
∈
[
0,0.1
] for i
,
j
=
1
,
…
,
324.Figure6 shows the output CV versus the mean firing rate in the first 10 layers of the network itself (left) and the correlation coefficients for one cell in the first, second, and fourth (or fifth) layer (right) for r
=
1.2,  r
i
a
=
1.5; r
=
0.5,  r
i
a
=
0.5; and r
=
1.5,  r
i
a
=
0.9. Simulations show that all neurons synchronize after the 5th layer, that is, ρ
i
j
(
k
)
=
1 for k
≥
5, and that the network is stable. Top panels show that the network will be stable at relatively low frequency (<100 Hz) after the 4th layer; middle panels show that the network will be stable at relatively high rate (>100 Hz) after the 5th layer; however, the last panels show that the network can be stable at a very low frequency (<1 Hz). For example, Figure 7 shows the mean firing rate for different layers, corresponding to Figure 6 (top). Note the different scale for each layer. It can be seen from Figures 6 and 10 (top) that the ratios r and r
i
a change the stable outputs but do not change the synchronization. This result, in general, is in agreement with numerical experiments of feedforward spiking neuronal networks, showing that neurons get synchronized quite easily [13]. In fact, desynchronization rather than synchronization seems to be the major problem for a spiking neuronal network.Figure 6
Propagation of activity in a heterogeneous MNNIII with random connections. The output CV versus the mean firing rate in the first10 layers (left). The correlation coefficients for one cell (right). Results were obtained for r
=
1.2,  r
i
a
=
1.5; r
=
0.5,  r
i
a
=
0.5, and r
=
1.5,  r
i
a
=
0.9.Figure 7
The mean firing rate for different layers, corresponding to top panel in Figure6.In order to avoid synchronization, we used the same Mexican hat weight distribution as in [1]. To this end, we first rearranged all neurons {
1,2
,
…
,
p
} from the interlayer on a two-dimensional square lattice by assigning to neuron i
∈
{
1,2
,
…
,
p
} coordinates as
(11)
x
i
=
i
-
1
n
,
y
(
i
)
=
(
i
-
1
)
-
n
x
(
i
)
≡
mod
⁡
(
i
-
1
,
n
)
,
with p
(
k
)
=
p
=
n
2. Then for i
,
j
=
1,2
,
…
,
p, we set
(12)
ω
i
j
(
k
)
=
ω
i
j
=
M
0.1
x
i
-
x
j
,
0.1
y
i
-
y
j
,
where M
(
x
,
y
) is the Mexico hat function, defined for x
,
y
∈
R as
(13)
M
x
,
y
=
m
2
π
γ
exp
⁡
⁡
-
x
2
+
y
2
2
γ
2
-
1
2
exp
⁡
⁡
-
x
2
+
y
2
8
γ
2
,
with m
>
0,  γ
>
0 modulation parameters. For simulations we used the same modulation m
=
3,  γ
=
0.5 as [1] and the same other parameters as in the previous subsection.Figure8 shows the propagation of activity in a heterogeneous MNNIII with Mexico hat type connections for r
=
1.2,  r
i
a
=
1.5. The results indicate that the activity in the network becomes stable after the 6th layer (top left). Also, as indicated by the values of the correlation coefficients in the top right panel, the neurons do not synchronize. Middle and bottom panels show the mean firing rate for different layers. Figure 9 presents the correlation coefficients between the central cell and the other cells in layer 3 and in layer 10, corresponding to Figure 8; on average the correlation coefficient between neurons was around zero.Figure 8
Propagation of activity in a heterogeneous MNNIII with Mexico hat type connections. The output CV versus the mean firing rate in the first10 layers (top left). The correlation coefficients for three cells in layer 10 (top right). The mean firing rate for different layers (middle and bottom). Results were obtained for r
=
1.2,  r
i
a
=
1.5.The correlation coefficients between the central cell and the other cells in layer3 (left) and in layer 10 (right), corresponding to Figure 8.
(a)
(b)Figure 10
Propagation of activity in a heterogeneous MNNIII with random connections (top) and Mexico hat type connections (bottom). The output CV versus the mean firing rate in the first10 layers (left). The stable mean firing rate in layer 10 (right). Results were obtained for r
=
1.5,  r
i
a
=
1.5.It is interesting to note that, from the first to the second layer, there is a general reduction in firing rate. Then, after a few transition layers, the neuronal activity becomes stable. In summary, the activity in a heterogeneous MNNIII becomes stationary after a few layers. With random interactions, the network gets easily synchronized. However, with lateral interactions by Mexico hat function, desynchronization is presented; that is, firing rates of individual neurons tend to spread out.To compare the results with others, Figure10 shows the propagation of activity in a heterogeneous MNNIII with random connections (top) and with Mexico hat type connections (bottom) for r
=
1.5,  r
i
a
=
1.5. It only shows the output CV versus the mean firing rate in the first 10 layers (left) and the stable mean firing rate in layer 10 (right), respectively. Figure 10 implies that the MNNIII with random connections gets synchronized after the 6th layer, and with Mexico hat type connections it becomes stationary and fires with the bell-like firing rates after the 7th layer.First, the weight connections are key to propagate the activity of such network. A homogeneous network with clamped weight connections, as stated in Section3.1, converges to silent or to low/high firing frequency. However, a heterogeneous network gets synchronized or desynchronized due to random or lateral weight connections. It is the lateral weight connections introduced by the Mexican hat that push cells to fire with more widely spread firing rates than in a network with random interactions; see the former discussions in this subsection. In other words, weight connections determine the type of dynamics of network (say, convergence, synchronization, and desynchronization) and subsequently change the size of stable outputs. For r
=
1.5,  r
i
a
=
1.5, for example, a homogeneous network with initial input 50 Hz converges to 16.09 Hz after the 5th layer (approximate to r
=
1.5,  r
i
a
=
1.5 in the left panel of Figure 5); Figure 10 shows that a network with random connections gets synchronized at 9.36 Hz after the 6th layer (top) and with Mexico hat type connections it gets desynchronized at the stable bell-like outputs with a mean value of 19.56 Hz after the 7th layer (bottom). It is the weight connections that change the stable outputs of network. Additionally, the network simulations reported in this subsection show that a highly irregular output firing is also the result of correlations between neuronal activities.Second, all simulations show that the ratio of inhibition from inter- or intralayer (r or r
i
a) does not determine the type of dynamics but influences the stable outputs of network. For homogeneous network, it was verified in Section 3.1. For a network with random connections, Figures 6 and 10 (top) show that the ratio r or r
i
a changes the stable outputs but does not change the synchronization. Figures 8 and 10 (bottom) show that the lateral weight connections introduced by the Mexican hat push cells to fire with more widely spread firing rates than in a network with random interactions and that the ratio r only changes the stable outputs; for example, the mean of stable bell-like outputs in Figure 8 (18.05 Hz) is bigger than that in bottom panel of Figure 10 (11.45 Hz). It is the inhibition from network itself or from external intrainput that can enhance or weaken the (stable) firing outputs of network.Finally, all parameters influence the speed of the stability of network. For homogeneous network, it was included in discussions in Section3.1. For heterogeneous network, the weight connections can influence it, when other parameters are identical; see, for example, Figure 8, which shows that the speed of synchronization by random connections is slightly faster than that of desynchronization by lateral connections. Further the ratio of inhibition influences it; see, for example, Figures 6 and 10 (top), which show its validity for random connections, and Figures 8 and 10 (bottom) do so for the Mexican hat type connections. In addition, the heterogeneous MNNIII simulations reported in this subsection show that synchronization/desynchronization seems to be slightly slower or faster than MNN [1] with other identical parameters, since the external intralayer input unit can enhance or weaken the firing outputs. Thus, the network reported here offers a more sophisticated and unexpected dynamics over the MNN.


## 3.1. Dynamics of Homogeneous Network

In this section, we focus on a homogeneous network, where all weights, afferent means, and variances were set to be identical. Now the quantities in (6) reduce to μ
1
(
k
)
=
μ
j
(
k
),  σ
1
(
k
)
=
σ
j
(
k
),  μ
1
(
k
)
,
i
a
=
μ
j
(
k
)
,
i
a,  σ
1
(
k
)
,
i
a
=
σ
j
(
k
)
,
i
a, and ω
12
(
k
)
=
ω
i
j
(
k
)
=
ω
(
k
)
=
ω,  ω
12
(
k
)
,
i
a
=
ω
i
j
(
k
)
,
i
a
=
ω
(
k
)
,
i
a
=
ω
i
a,  i
≠
j. As we discussed before, the propagation of correlation becomes trivial in such case since all cells become fully correlated after the first layer. To avoid this, we clamped the correlation coefficient; that is, we set ρ
12
(
k
)
=
ρ
i
j
(
k
)
=
ρ
(
k
)
=
ρ,  ρ
12
(
k
)
,
i
a
=
ρ
i
j
(
k
)
,
i
a
=
ρ
(
k
)
,
i
a
=
ρ
i
a,   i
≠
j. In simulations we set ρ
i
a
=
ρ
=
0 or 0.1, in agreement with experimental data reported in the literature [1, 11, 12], and we assume that each intralayer input, including numbers of neurons, afferent means, and variances, is identical; that is, p
(
1
)
,
i
a
=
p
(
k
)
,
i
a
≡
p
i
a,  μ
1
(
1
)
,
i
a
=
μ
1
(
k
)
,
i
a, and σ
1
(
1
)
,
i
a
=
σ
1
(
k
)
,
i
a.First we chooser
=
0.1, 0.5, 1.5 to illustrate how the activity is propagated across the networks, varied from r
i
a and p
i
a, on assumption that C
1
(
k
)
=
C
1
(
k
)
,
i
a
=
1. Unless otherwise specified, the initial CV in all simulations is always 1, and the weight is 0.5. In Figure 1, we show the results obtained for various values of μ
(
1
) and σ
(
1
) (we reported the coefficient of variation C
(
k
)
=
σ
(
k
)
/
μ
(
k
)). As stated in [1], each data point (μ
(
k
)
,
C
(
k
)) is connected with (μ
(
k
+
1
)
,
C
(
k
+
1
)) to illustrate how the activity is propagated across the networks. After the first few layers, for r
=
0.1 (top), neurons are found to be either silent or firing at relatively high frequency for all r
i
a and p
i
a. For r
=
0.5 (middle), however, the situation has changed. On the one hand, the left panel (middle) shows that even if there is no inhibitory input (r
i
a
=
0) for a small number of intraneurons (p
i
a
=
10), to be silent is certainty on fewer intrainput neurons. On the other hand, the right panel (middle) shows that not all networks are certainty to be silent. As stated earlier, for r
=
0.5 with fewer intrainput neurons (p
i
a
=
10), the network is certainty to be silent (left). However, with increasing size of intrainput units, the network might be fixing or firing at relatively high frequency (right). Instead, they fire at low or relatively high frequency; for example, p
i
a
≥
30 for r
1
i
a
=
0.1. It actually shows that the networks can be stable at low frequency; see discussions later. In addition, for r
=
1.5 (bottom), there is another kind of properties that neurons fire at relatively low frequency after a few times oscillations. Unlike a homogeneous MNN [1], neurons are certainty to be silent with a larger ratio (e.g., r
≥
0.4) after the first few layers. It is obvious that the intrainputs cause a more sophisticated dynamics of network than MNN.Figure 1
Propagation of activity in a homogeneous feedforward MNNIII. For each layerk, the output firing rate (μ
1
(
k
)) and the coefficient of variation (C
1
(
k
)) are reported on the abscissa and the ordinate, respectively. Points labeled 1 correspond to the first layer; points corresponding to successive layers are connected by lines. Results were obtained for p
=
100,  ρ
i
j
=
ρ
i
j
i
a
=
0,  μ
1
(
1
)
=
50 Hz (solid lines), 100 Hz (dashed lines), and r
=
0.1 (top), r
=
0.5 (middle), and r
=
1.5 (bottom). In each panel, the ratios from intralayer (r
i
a) are evenly spaced.Figure1 implies that the stable solution existed in some networks. Thus we explore the stable solution of such network. Figure 2 shows how the input-output firing rate relationship is varied from p
i
a and r
i
a, for r
=
0.5 and r
=
1.5. As analogous to [1], roughly speaking, the stable solution appears at low firing rate for networks of increasing size (either p or p
i
a size) with strong inhibition (either r or r
i
a). However, the preferred appearance of stable solution at low frequency is not the increasing size of both inter- and intralayer; for r
=
1.5 (top right, dashed lines), for example, any ratio r
i
a from intralayer (from 0 to 1.5) appears as a fixed point at low firing rate when p
=
300,  p
i
a
=
100 (top right), instead of p
=
300,  p
i
a
=
300 (bottom right).Figure 2
The input-output firing rate relationship for a homogeneous MNNIII varied from numbers of intralayer neuronsp
i
a and the ratio of intrainputs r
i
a. Results were obtained for ρ
i
j
=
ρ
i
j
i
a
=
0.1 and r
=
0.5 (solid lines), r
=
1.5 (dashed lines).We further explore the stability of such network by showing that the stable outputs varied from(
p
i
a
,
r
i
a
). Figure 3 shows the stable outputs versus r
i
a and p
i
a, with a fixed ratio from interlayer inputs r
=
0.8 (top) and r
=
1.5 (bottom) and with the number of neurons from interlayer inputs p
=
100 (left) and p
=
300 (right). The initial inputs fix 5 Hz in the current and latter simulations. It is found that the intralayer input unit appears to have a sophisticated role in the stability of such network, depending on the ratio r
i
a and the size p
i
a of context unit, together with the ratio r and the size p of network itself, that is, interlayer. For example, for r
=
0.8 (top), the stable output appears to increase with the increasing size p
i
a and the decreasing ratio r
i
a of context unit, which plays a general role; for r
=
1.5 and p
=
300 (bottom right panel), however, the situation has changed to play a negative role; see p
i
a
⩾
300.Figure 3
Stable outputs for a homogeneous MNNIII varied from numbers of intralayer neuronsp
i
a and the ratio of intralayer r
i
a. Results were obtained for r
=
0.8 (top) and 1.5 (bottom) and p
=
100 (left) and 300 (right), with ρ
i
j
=
ρ
i
j
i
a
=
0.1.We also explore the stability of such network by showing that the stable outputs varied from numbers of intralayer neuronsp
i
a and of interlayer p. Figure 4 shows that the network with a small size of intra- and interlayer will be either silent or firing at low frequency. Top panel shows that the stable output of such network seems to be symmetry between numbers of intralayers and interlayers when r
=
1. However, there are obvious asymmetry for r
<
1 or r
>
1; see middle and bottom panels in Figure 4. Middle panel shows that the network with a relatively large size of interlayer, for r
<
1, will be firing at relatively high rate and that the size of intralayer has negligible effect on the stable outputs such as p
>
250. Bottom panel presents unexpected results of r
>
1 that relatively high frequency is firing at the large size of intralayer and small size interlayer, rather than the large size of intralayer and interlayer. In particular, for p
>
300 and p
i
a
>
300, the network is unexpectedly firing at low frequency when r
=
1.5 and r
i
a
=
0.5; see the left of bottom panel. All those imply that the current network can lead to more sophisticated dynamics than MNN and the intrainput unit plays a very sophisticated role in dynamics of such network.Figure 4
Stable outputs for a homogeneous MNNIII varied from numbers of intralayer neuronsp
i
a and of interlayer p. Results were obtained for r
=
1 (top), r
=
0.5 (middle) and 1.5 (bottom) and r
i
a
=
0.5 (left) and r
i
a
=
1.5 (right), with ρ
i
j
=
ρ
i
j
i
a
=
0.1.Finally, we explore the stability of such network by showing that the stable outputs varied from the ratio of both inter- and intralayers, that is,r and r
i
a. Figure 5(a) shows the stable outputs versus r and r
i
a with p
=
p
i
a
=
300. It can be seen that the network will be either silent or firing at low frequency when r
>
1 and will be firing at relatively high rate when r
<
1, which implies that the ratio (between inhibitory and excitatory input) from the network itself plays a more important role than that from intrainput unit with such parameters. However, the network will converge to silent or to low firing frequency with strong inhibition r
>
1 or to high frequency with weak inhibition r
<
1. Unlike the stable output of MNN in [1] (a high level of inhibition is necessary to maintain a spontaneous activity, that is, a low output firing rate), a weaker inhibition (r
<
1 approaching 1), even with weaker intrainhibition (r
i
a
<
1 approaching 1), can converge to low firing frequency; see, for example, r
∈
(
0.8,1
) and r
i
a
∈
(
0.5,1
). It is the intrainput unit to maintain the low frequency activity. Thus there is an important gain in computational power over MNN. Note the appearance of oscillation for a homogeneous MNNIII as singularity, for example, with r
=
2,  r
i
a
=
0.1,  p
=
p
i
a
=
300, and  ρ
i
j
=
ρ
i
j
i
a
=
0.1 (Figure 5(b)).(a) Stable outputs for a homogeneous MNNIII varied from the ratio of both inter- and intralayer (r and r
i
a). Results were obtained for p
=
p
i
a
=
300, with ρ
i
j
=
ρ
i
j
i
a
=
0.1. (b) The appearance of oscillation for a homogeneous MNNIII.
(a)
(b)


## 3.2. Dynamics of Heterogeneous Network

We now turn to explore a more sophisticated dynamics of a heterogeneous network. We shed light on, for instance, how synchronization or desynchronization is produced by random or lateral connections when we remove the constraint on the correlations between neurons.In accordance with the results reported in the previous publications, we fixed the total number of neurons in each interlayer top
(
k
)
=
p
=
324. To show the influences of intralayer inputs on networks, we also fixed the total number of neurons in each intralayer to p
(
k
)
,
i
a
=
p
i
a
=
324. We further assumed that each intralayer receives the same inputs as the first intralayer; that is, w
i
j
(
k
)
,
i
a
=
w
i
j
(
1
)
,
i
a,  μ
i
(
k
)
,
i
a
=
μ
i
(
1
)
,
i
a,  C
i
(
k
)
,
i
a
=
C
i
(
1
)
,
i
a, and ρ
i
j
(
k
)
,
i
a
=
ρ
i
j
(
1
)
,
i
a for i
,
j
=
1
,
…
,
324.First we generated random connectionsw
i
j
(
k
)
,
w
i
j
(
1
)
,
i
a
∈
[
0,0.5
] and inputs μ
i
(
1
)
,
μ
i
(
1
)
,
i
a
∈
[
0,100
] Hz. We also assumed that C
1
(
1
)
=
C
1
(
1
)
,
i
a
=
1 and that ρ
i
j
(
1
)
,
ρ
i
j
(
1
)
,
i
a
∈
[
0,0.1
] for i
,
j
=
1
,
…
,
324.Figure6 shows the output CV versus the mean firing rate in the first 10 layers of the network itself (left) and the correlation coefficients for one cell in the first, second, and fourth (or fifth) layer (right) for r
=
1.2,  r
i
a
=
1.5; r
=
0.5,  r
i
a
=
0.5; and r
=
1.5,  r
i
a
=
0.9. Simulations show that all neurons synchronize after the 5th layer, that is, ρ
i
j
(
k
)
=
1 for k
≥
5, and that the network is stable. Top panels show that the network will be stable at relatively low frequency (<100 Hz) after the 4th layer; middle panels show that the network will be stable at relatively high rate (>100 Hz) after the 5th layer; however, the last panels show that the network can be stable at a very low frequency (<1 Hz). For example, Figure 7 shows the mean firing rate for different layers, corresponding to Figure 6 (top). Note the different scale for each layer. It can be seen from Figures 6 and 10 (top) that the ratios r and r
i
a change the stable outputs but do not change the synchronization. This result, in general, is in agreement with numerical experiments of feedforward spiking neuronal networks, showing that neurons get synchronized quite easily [13]. In fact, desynchronization rather than synchronization seems to be the major problem for a spiking neuronal network.Figure 6
Propagation of activity in a heterogeneous MNNIII with random connections. The output CV versus the mean firing rate in the first10 layers (left). The correlation coefficients for one cell (right). Results were obtained for r
=
1.2,  r
i
a
=
1.5; r
=
0.5,  r
i
a
=
0.5, and r
=
1.5,  r
i
a
=
0.9.Figure 7
The mean firing rate for different layers, corresponding to top panel in Figure6.In order to avoid synchronization, we used the same Mexican hat weight distribution as in [1]. To this end, we first rearranged all neurons {
1,2
,
…
,
p
} from the interlayer on a two-dimensional square lattice by assigning to neuron i
∈
{
1,2
,
…
,
p
} coordinates as
(11)
x
i
=
i
-
1
n
,
y
(
i
)
=
(
i
-
1
)
-
n
x
(
i
)
≡
mod
⁡
(
i
-
1
,
n
)
,
with p
(
k
)
=
p
=
n
2. Then for i
,
j
=
1,2
,
…
,
p, we set
(12)
ω
i
j
(
k
)
=
ω
i
j
=
M
0.1
x
i
-
x
j
,
0.1
y
i
-
y
j
,
where M
(
x
,
y
) is the Mexico hat function, defined for x
,
y
∈
R as
(13)
M
x
,
y
=
m
2
π
γ
exp
⁡
⁡
-
x
2
+
y
2
2
γ
2
-
1
2
exp
⁡
⁡
-
x
2
+
y
2
8
γ
2
,
with m
>
0,  γ
>
0 modulation parameters. For simulations we used the same modulation m
=
3,  γ
=
0.5 as [1] and the same other parameters as in the previous subsection.Figure8 shows the propagation of activity in a heterogeneous MNNIII with Mexico hat type connections for r
=
1.2,  r
i
a
=
1.5. The results indicate that the activity in the network becomes stable after the 6th layer (top left). Also, as indicated by the values of the correlation coefficients in the top right panel, the neurons do not synchronize. Middle and bottom panels show the mean firing rate for different layers. Figure 9 presents the correlation coefficients between the central cell and the other cells in layer 3 and in layer 10, corresponding to Figure 8; on average the correlation coefficient between neurons was around zero.Figure 8
Propagation of activity in a heterogeneous MNNIII with Mexico hat type connections. The output CV versus the mean firing rate in the first10 layers (top left). The correlation coefficients for three cells in layer 10 (top right). The mean firing rate for different layers (middle and bottom). Results were obtained for r
=
1.2,  r
i
a
=
1.5.The correlation coefficients between the central cell and the other cells in layer3 (left) and in layer 10 (right), corresponding to Figure 8.
(a)
(b)Figure 10
Propagation of activity in a heterogeneous MNNIII with random connections (top) and Mexico hat type connections (bottom). The output CV versus the mean firing rate in the first10 layers (left). The stable mean firing rate in layer 10 (right). Results were obtained for r
=
1.5,  r
i
a
=
1.5.It is interesting to note that, from the first to the second layer, there is a general reduction in firing rate. Then, after a few transition layers, the neuronal activity becomes stable. In summary, the activity in a heterogeneous MNNIII becomes stationary after a few layers. With random interactions, the network gets easily synchronized. However, with lateral interactions by Mexico hat function, desynchronization is presented; that is, firing rates of individual neurons tend to spread out.To compare the results with others, Figure10 shows the propagation of activity in a heterogeneous MNNIII with random connections (top) and with Mexico hat type connections (bottom) for r
=
1.5,  r
i
a
=
1.5. It only shows the output CV versus the mean firing rate in the first 10 layers (left) and the stable mean firing rate in layer 10 (right), respectively. Figure 10 implies that the MNNIII with random connections gets synchronized after the 6th layer, and with Mexico hat type connections it becomes stationary and fires with the bell-like firing rates after the 7th layer.First, the weight connections are key to propagate the activity of such network. A homogeneous network with clamped weight connections, as stated in Section3.1, converges to silent or to low/high firing frequency. However, a heterogeneous network gets synchronized or desynchronized due to random or lateral weight connections. It is the lateral weight connections introduced by the Mexican hat that push cells to fire with more widely spread firing rates than in a network with random interactions; see the former discussions in this subsection. In other words, weight connections determine the type of dynamics of network (say, convergence, synchronization, and desynchronization) and subsequently change the size of stable outputs. For r
=
1.5,  r
i
a
=
1.5, for example, a homogeneous network with initial input 50 Hz converges to 16.09 Hz after the 5th layer (approximate to r
=
1.5,  r
i
a
=
1.5 in the left panel of Figure 5); Figure 10 shows that a network with random connections gets synchronized at 9.36 Hz after the 6th layer (top) and with Mexico hat type connections it gets desynchronized at the stable bell-like outputs with a mean value of 19.56 Hz after the 7th layer (bottom). It is the weight connections that change the stable outputs of network. Additionally, the network simulations reported in this subsection show that a highly irregular output firing is also the result of correlations between neuronal activities.Second, all simulations show that the ratio of inhibition from inter- or intralayer (r or r
i
a) does not determine the type of dynamics but influences the stable outputs of network. For homogeneous network, it was verified in Section 3.1. For a network with random connections, Figures 6 and 10 (top) show that the ratio r or r
i
a changes the stable outputs but does not change the synchronization. Figures 8 and 10 (bottom) show that the lateral weight connections introduced by the Mexican hat push cells to fire with more widely spread firing rates than in a network with random interactions and that the ratio r only changes the stable outputs; for example, the mean of stable bell-like outputs in Figure 8 (18.05 Hz) is bigger than that in bottom panel of Figure 10 (11.45 Hz). It is the inhibition from network itself or from external intrainput that can enhance or weaken the (stable) firing outputs of network.Finally, all parameters influence the speed of the stability of network. For homogeneous network, it was included in discussions in Section3.1. For heterogeneous network, the weight connections can influence it, when other parameters are identical; see, for example, Figure 8, which shows that the speed of synchronization by random connections is slightly faster than that of desynchronization by lateral connections. Further the ratio of inhibition influences it; see, for example, Figures 6 and 10 (top), which show its validity for random connections, and Figures 8 and 10 (bottom) do so for the Mexican hat type connections. In addition, the heterogeneous MNNIII simulations reported in this subsection show that synchronization/desynchronization seems to be slightly slower or faster than MNN [1] with other identical parameters, since the external intralayer input unit can enhance or weaken the firing outputs. Thus, the network reported here offers a more sophisticated and unexpected dynamics over the MNN.


## 4. Discussion

In the current paper, we have focused on the dynamics of feedforward MNNIII, akin to the dynamics of MNN. Synchronization or desynchronization is presented by random or lateral interactions. Due to more biologically reasonable intralayer inputs, such network offers a more sophisticated and unexpected dynamics over the MNN. As stated earlier, the intrainput unit plays a very sophisticated role in dynamics of such network. It can enhance or weaken the firing outputs by varying on the ratior
i
a and/or the size p
i
a of intralayer, incorporating with those of interlayer, that is, r and/or p. In the future we might shed light on the application in engineering, the dynamics, and/or learning rule of MNNIII with other network architectures such as recurrent and RBF network [14].



---

*Source: 381271-2015-01-20.xml*