脳機能分化・機能分割を実現する情報論的機構仮説
A hypothesis on the information-theoretic mechanism
realizing the functional differentiation and parcellation
津田一郎
Ichiro Tsuda
札幌市立大学AITセンター
AIT center, Sapporo City University

CHAIN セミナー@北大学術交流会館第３会議室
２０２４年６月２８日

An exploration of the principle ofemerging interactions in spatiotemporaldiversity (2017－2022+2022－2023)
PI: Ichiro Tsuda
Co-PI: Yuji Kawai, Takashi Ikeda, Ikki Matsuda, and Tatsuya Kameda

Summary of Achievements via Collaborative Works
Application of ERC for a genesis of functional
differentiation

Research questions on functional differentiationa. How was functional differentiation organized in the brain?
What is an organization mechanism?
Can it be described by conventional self-organization theories?
⇒ To answer these questions, constraint is a key concept.
Brodmann areas (functional map)b. How were neurons and neural modules generated?

Gohara 2010

Szentagothai 1992

c. How is dynamic organization of function
temporarily generated? functional parcellation
4

Glasser et al., 2016

Thermal contact of two materials with different temperatures
This is the case of no constraint: entropy increases in time, finally an
isothermal state occurs

𝑇1 > 𝑇2
𝑇1𝑡→∞

𝑇1 = 𝑇2

𝑇2

In a closed system, entropy increases
according to the second law of thermodynamics.

Then, only prediction is possible for macroscopic states.

Carnot Cycle

Adiabatic contraction →Isothermal expansion →Adiabatic expansion →Isothermal contraction: Constraints

𝑇1

𝑄1

𝑄2
T

In this cycle,
entropy change
should be zero.

𝑇2
1

2

Obtained work
W

4

𝑆1

3

𝑆2

S

If there is no constraint,
What happens?
Entropy increases in time.

Event space

We can arbitrarily select initial conditionsat any time
because of the increase of entropy.

If a certain constraint is provided
at a certain future time,
What happens?
time
小さな誤差のある事象の連鎖は大きな違いを生み出す

At each bifurcation (multi-furcation) point, events with highprobabilities do not always happen, so that rare events
as a whole series of events with low probabilities can happen withprobability 1, which is proved by principle of large deviation:
各ステップで必ずしも最適なパスが選ばれるわけThis principle assures the appearance
ではない oflow entropy events.

Entropy decreases in time.
We cannot arbitrarily select initialconditions; we have to find good i.c. tosatisfy the constraint.

Life is more than a Carnot cycle engine, which obtains work by realizing zero
entropy change, namely a reversible process, in the environment where entropy
increases, whereas living organisms make far-from-equilibrium conditions andproduce order out of chaos, which is an irreversible process.
Phase
space

∆𝑆 > 0present∆𝑆 > 0In chaotic dynamical systems,“entropy” increases in both future and past
direction of time.
→
Both prediction and past estimate arepossible: 𝑩𝒂𝒚𝒆𝒔𝒊𝒂𝒏 𝒊𝒏𝒇𝒆𝒓𝒆𝒏𝒄𝒆 holds.

time⇒ Implying that life can have an ability ofprediction and past estimate only in chaotic
environments

For cause A and effect X,

𝑝 𝑋 𝐴 𝑝(𝐴)𝑝 𝐴𝑋 =σ𝐴 𝑝(𝑋|𝐴)

𝐴→𝑋

(a) What is the difference of self-organization with constraints(SOC)from conventional self-organization (SO)?
SO:
Macroscopic orders are generated
via interactions of atoms and molecules at
microscopic levels

SOC:
Systems elements and subsystems are generated
via constraints acting on the system

Nicolis, G., Prigogine, I. (1977);
Haken, H.(1977)
Fixed boundary conditions!

Macroscopic
order

（ex）
(From Kitahata
http://www.chem.scphys.kyotou.ac.jp/nonnonWWW/kitahata/bz_1.html)

Microscopic
interactions（From Honours Project:
MolDyn - Molecular Dynamics
Veselin Dikov, Niko Manopulo, Darya Popiv, Ilya
Saverchenko, Levi Valgaerts)

Open boundary conditions!

Constraints acting on
a whole system

Functional differentiation: selforganization of functional
elements(fromhttp://www.yuchan.net/yuchan/dictionary/
new_ikuji/brain004.html)
10Our view: functional differentiation in the brain should be formulated
within the framework of self-organization with constraints, where thefunctional elements (or components, or subsystems) are produced byconstraints that act on a whole system.
Tsuda, I. Prog. Theor. Phys. 1984;
Rosen, R. Life Itself , Columbia Univ. Press, 1991;
Freeman, W. J., How Brains Make Up Their Minds, A Phoenix Paperback, 1999;
Tsuda, I. Behav. Brain Sci. 2001;
Freeman, W. J., Biol. Cybern. 92, 350−359 (2005).
Tsuda, I. et al. Entropy 2015;
Shimizu, H.: http://www.banokenkyujo.org/?page_id=48
11

Self-organization theory with constraints in neural networks
Ex)・C. Von der Malsburg, Kybernetik 1973
A model for the primary visual cortex: a construction of orientation sensitive cells
Constraint : a sum of the connected weights to each neuron should be constant.
・S. Amari, Bull. Math. Biol. 1980
Topographic mapping via competitive neural networks by mutual inhibitions
Constraint : winner-take-all
・T. Kohonen, Biol. Cybern. 1982
SOM can be viewed as Expectation Maximization algorithm (EM)Constraint : maximization of expectation
・H. Haken and J. Portugali, Information Adaptation (Springer, 2015)
Pattern recognition based on pattern formation
12
Constraint : attention parameters

Assertion(Emerging Science Principle)（創発原理）：
In living systems, self-organization with constraints is a blueprint, thereby”functional differentiation=a genesis of functional elements”

Functional differentiation/percellation and cell differentiation
follows developmental dynamical systems
13

Direct differentiation model

Glial progenitor
cells

neuronNeuro stem
cell

astrocyte

Intervening progenitor cells model

Neuro stem
cell

astrocyte

oligodendrocyte

It should stop here

oligodendrocyte

neuronFrom http://www.okano-lab.com/okanolab/stemcell

It should stophere

The existence of neuro stem cells in the third ventricle⇒Neurogenesis in dentate gyrus, side subventricularzone, etc. ⇒・Treatment model of brain injury via acceleration of differentiation by stimulation in the growth factor ofprogenitor cells
・Treatment model of brain injury via nerve graft with neuro stem cells stemming from fetus brains, stem
cells, and iPS cells
14⇒Finding the mathematical principle of neuronal differentiation
is important.

Dynamical systems approach is useful for cell differentiation

Constraint: the sum of
the number of moleculesin each cell is fixed

C. Furusawa and K. Kaneko, Science, 2012cf) Chambers et al., Nature 2007; Hayashi et al, 200815

Constrained mechanics
１．Holonomic：
・The degree of freedom of infinitesimal change equals the one of global change:
Integrable

２．Nonholonomic：
・The former degree of freedom does not equal the latter: nonintegrable
・d’Alembert’s principle(principle of virtual work)・deleting Lagrange multipliers⇒feedback control (selecting a solution following constraints
among extrema of functional)
・causal
３．Vakonomic mechanics (by V.V.Kozlov)：・Lagrange multipliers are independent variables, depending on both initial and final states⇒ Final state sensitivity
・Optimal control theory (Andronov-Pontryagin)（finding an extremum, giving constraints on variational manifold)・noncausal

CF) The case of constrained Hamiltonian systems ⇒for example, Dirac method

I. Tsuda, Y. Yamaguti, H.Watanabe
Entropy 2016, 18, 74

𝐒𝐭𝐞𝐩𝟏. on Ω(n) × 𝑅𝑙𝑑𝑥
= 𝑓 𝑥; 𝜆𝑑𝑡

The description of neural dynamics

𝐒𝐭𝐞𝐩𝟐. on Ω(n) × 𝑅𝑙 ×W𝑑𝑥= 𝑓 𝑥, 𝜆 + 𝐺 𝑥, 𝑡
𝑑𝑡

The description of the interactions between
neural systems and environments

𝐒𝐭𝐞𝐩 𝟑.

The description of constrained dynamics
𝑇δ𝐿 = δ ‫׬‬0 {∁ + 𝜇(𝑑𝑥
− 𝑓 𝑥, 𝜆𝑑𝑡− 𝐺 𝑥, 𝑡 )}𝑑𝑡 = 0,

𝐶: intentional constraints↓
3.1 In the case that 𝐶 is quantified, on Ω(n) × 𝑅𝑙 ×W× 𝑅𝑛
𝑑𝑥= 𝑓 𝑥, 𝜆 + 𝐺 𝑥, 𝑡𝑑𝑡𝑑𝜇
= h(μ, 𝑥, 𝑥)ሶ
𝑑𝑡

𝜇: Lagrange multiplier

The framework of self-organization with constraints for yielding
functional elements𝑇𝑑𝑥δ ‫׬‬0 {∁(𝑦(𝑥), 𝑡) + 𝜇(𝑥, 𝑥,ሶ 𝑡)( 𝑑𝑡 − 𝑓 𝑥, 𝜆

− 𝐺 𝑦(𝑥), 𝑡 )}𝑑𝑡 = 0∁ 𝑦(𝑥), 𝑡 : Intentional and Informational Constraintscf) Pattee, J. Social Biol. Struct. 1978;
Tschacher and Haken, New Ideas in Psychol. 2007𝑑𝑥
− 𝑓 𝑥, 𝜆𝑑𝑡− 𝐺 𝑦(𝑥), 𝑡 = 0: Dynamical Constraints

Note: Lagrange multipliers are a function not
only of a state variable x, but also of its timederivatives 𝑥ሶ and time t, and their equations of
motion are derived⇒ Vakonomic dynamics (Kozlov, V.V. 1983)⇒
Theorem 1
The vakonomic dynamical systems, derived
from any differentiable dynamical systems arelinearly unstable on subspace of Lagrange
multiplies.
18

Optimization
with
Exponential
Discount

Particularly, applicable to the problems of
convex constraints such as
least energy consumption
𝜌෤+

Theorem 2
There exist positivemeasure initial conditions
to reach a given final state.

19

Example: Finding optimal perturbations of Bonhoeffer-van der Pol equations𝑥3
𝑥ሶ = 𝑐 𝑦 + 𝑥 − − 𝑟 + 𝐼
3Forger, D.B. et al, JTB 230(2004)521-32𝑦ሶ = −(𝑥 − ሶ 𝑎 + 𝑏𝑦)/𝑐𝑎 = 0.7, 𝑏 = 0.8, 𝑐 = 3.0, 𝑟 = 0.342, 𝐼 = 0: 𝑠𝑢𝑏𝑐𝑟𝑖𝑡𝑖𝑐𝑎𝑙 𝐻𝑜𝑝𝑓 𝑏𝑖𝑓𝑢𝑟𝑐𝑎𝑡𝑖𝑜𝑛

・Changing the state from a limit cycle to a stable fixed point by adding an external signal
・Choosing an optimal external signal (variational principle)
External signal: z(t)boundary conditions : 𝑥 𝑡𝑖 , 𝑦 𝑡𝑖
𝐿 = 𝑧(𝑡)2 + 𝜇𝑥 𝑡𝑥ሶ − 𝑐 𝑦 + 𝑥 −𝑜𝑛 𝑙𝑖𝑚𝑖𝑡 𝑐𝑦𝑐𝑙𝑒; 𝑥 𝑡𝑓 , 𝑦 𝑡𝑓
𝑥3
3

−𝑟 −𝐼−𝑧 𝑡where C = 𝑧(𝑡)2𝑡𝑓

𝛿 න 𝑑𝑡 𝐿 𝑥, 𝑦, 𝑧, 𝜇𝑥 , 𝜇𝑦 = 0𝑡𝑖

𝑜𝑛 𝑓𝑖𝑥𝑒𝑑 𝑝𝑜𝑖𝑛𝑡+ 𝜇𝑦 (𝑡) 𝑦ሶ + (𝑥 − 𝑎 + 𝑏𝑦)/𝑐 ,Euler-Lagrange equations:

𝑑𝐿
𝑑𝑞𝑖

−

𝑑 𝑑𝐿
=0
𝑑𝑡 𝑑𝑞ሶ 𝑖𝑞𝑖 = 𝑥, 𝑦, 𝑧, 𝜇𝑥 , 𝜇𝑦・𝑞𝑖 = 𝜇𝑥 , 𝜇𝑦 → 𝐵𝑣𝑃 𝑒𝑞
・𝑞𝑖 =z →𝑧 = 𝜇𝑥 /2
𝑑𝜇𝑥・𝑞𝑖 =x →= −𝑐(1 − 𝑥 2 ) 𝜇𝑥 + 𝜇𝑦 /c・𝑞𝑖 = y →As 𝜇𝑥 0 , 𝜇𝑦 0

𝑑𝑡
𝑑𝜇𝑦
𝑑𝑡

= −𝑐 𝜇𝑥 + b𝜇𝑦 /c

are not given, start from appropriate values

As a conservation quantity, Hamiltonian can be defined.

𝐸𝑖𝑔𝑒𝑛𝑣𝑎𝑙𝑢𝑒𝑠 𝑜𝑓 𝐽𝑎𝑐𝑜𝑏𝑖 𝑚𝑎𝑡𝑟𝑖𝑥 𝑎𝑡 (𝑥𝑓𝑝 , 𝑦𝑓𝑝 , 0, 0):
𝜆1,2 ~ − 0.01103 ± 0.96677𝑖
𝜆3,4~0.01103 ± 0.96677𝑖

Obtain the arrival time of
trajectories at the boundary ofcylinder⇒ sensitive dependence on
initial conditions on limit cycle

𝜇𝑥 , 𝜇𝑦

Optimization dynamics ends

While optimization dynamics works𝑦𝑥

Original dynamics works again

Initial conditions on the outer limit cycle𝑡𝑓𝜌𝑡⇒ Another solvable method: exponential discount: 𝛿 න 𝑑𝑡 𝑒 𝐿 𝑥, 𝑦, 𝑧, 𝜇𝑥 , 𝜇𝑦 = 0𝑡𝑖In order to obtain global stable states, we adopted a genetic algorithm.

Evolutionary and developing dynamics via genetic algorithm
Dynamical and otherconstraints𝑑𝑥= 𝑓 𝑥; 𝑏𝑑𝑡with C

Dynamical Systems：{state space、dyn. rule（transition rule of states}
Family of D.S.：{D.S., bifurcation parameters}
・Genes defined by bifurcation parameters 𝑏
・Cells’ states（via activated proteins and/or electric activity）
defined by dynamical variables 𝑥
・A principle of evolution-development yielding functional differentiation
defined by self-organization with constraints C

Changing dynamical systems 𝑓 𝑥; 𝑏 by changing parametersb to satisfy
constraintsC.
⇒Finding a set of initial conditions satisfying constraints on(subspaces) of function space.

(b), (c) Applications of the variational principle
1. Mathematical modeling of functional modules
1.1 Developed a mathematical model in terms of coupled neural oscillators• The dynamics of each oscillator is defined by
𝛼
(𝑖,𝑘)
(𝑖,𝑘)
(𝑖,𝑘)
𝜃𝑡+1 = 𝜔
+ 𝜃𝑡
+
𝑁𝑝𝑐

෍
(𝑗,𝑙)∈𝐺(𝑗,𝑙)sin 𝜃𝑡𝑖𝑗(𝑖,𝑘)

for k th oscillator in i th module
Constraint: maximum transfer entropy

I.Tsuda, Y.Yamaguti, H. Watanabe, Proc. of ICCN;
Y. Yamaguti, I Tsuda, Neural Networks;
Y. Yamaguti, I. Tsuda, Y. Takahashi, Cogn. Neurodyn.

(𝑖,𝑘)− 𝜃𝑡 𝑖,𝑘 − 𝜓𝑘𝑙 + 𝜎𝛽 𝛽𝑡24

1.2 Functional Differentiation by Minimization of
Mutual Information between Neural Groups
Maximization( Y. Yamaguti & I. Tsuda, in preparation 2024)• Developing functionally differentiatedstructures by minimizing the amount ofmutual information between the states ofneural groups in RNN by gradientlearning.

Statistical Network𝑇
𝑋𝑔1 , 𝑋𝑔2
Noise

RNN
Input

g1

Estimated
MI

Gradient of
MI Loss(for Minimization)

Outputg2
Task Loss

L2 Regularization Loss

Task : Multi-frequency sinusoidal signal prediction
Outputs of trained RNN• The minimization of mutual information led to the formation offunctionally differentiated modules.
• During the training, the development of “functional” (correlational)
differentiation preceded the development of structural differentiation.

Weight matrix of RNN

Group 1

Group 2

Correlations
between Neural units2
𝑄 = σ𝑁𝑖=1(𝑒𝑖𝑖 − 𝑎𝑖 ), whereσ𝑗 𝑒𝑖𝑗 = 𝑎𝑖 (for random connections)< 𝑒𝑖𝑖 > = 𝑎𝑖2

The development of modularity Q
during learning

2. Mathematical modeling for the production of spiking neurons and glial cells
・Watanabe, H., Ito, T., Tsuda, I.,
Neuroscience Research 156 (2020) 206-216.
・I. Tsuda, Y. Yanmaguti and H. Watanabe,
Entropy 2016, 18, 74:pp1-13.

x (t + 1)

x(t + 1) = x(t )Mutual information I

x (t )

3.5
3
2.5
2
1.5
1
0.5
0

site k

0
1
2
3
4
5
6
7
8
9

0

The rule of statechange represented
by mapSpatio-temporal change of
information of external signal

15

10

5

20

25

30

iteration t

xn i
1
0.5

Time series of elementary
dynamics

0
1
2
3
4

site k

5
6
7
8
9

0

20

40

60

80

100

iteration t

initialStage of evolution

finalx (t + 1)

x(t + 1) = x(t )x (t )
Mutual information I

site k

3.5
3
2.5
2
1.5
1
0.5
0

3.5
3
2.5
2
1.5
1
0.5
0

0
1
2
3
4
5
6
7
8
9

0
1
2
3
4
5
6
7
8
9

10

5

0

15

20

25

30

10

5

0

15

20

30

25

iteration t

xn i

2
1.5
1
0.5

1
0.5

0
1
2

0
1

3
4
5

2
3
4

site k

6
7

5
6

8

7
8
9

initial

0

20

40

60

80

100

9

0

20

40

60

80

100

iteration t

Stage of evolution

finalx (t + 1)

x(t + 1) = x(t )x (t )

Mutual information I

site k

3.5
3
2.5
2
1.5
1
0.5
0

3.5
3
2.5
2
1.5
1
0.5
0

3.5
3
2.5
2
1.5
1
0.5
0

0
1
2
3
4
5
6
7
8
9

0
1
2
3
4
5
6
7
8
9

0
1
2
3
4
5
6
7
8
9

0

15

10

5

20

25

30

10

5

0

15

20

30

25

5

0

10

15

30

25

20

iteration t

xn i

1.5
1
0.5

2
1.5
1
0.5

1
0.5

0

0

0
1
2
3
4

site k

3
4
5

3
4
5

5
6

6
7

6
7

7
8
9

initial

1
2

1
2

0

20

40

60

80

100

8
9

0

20

40

60

Stage of evolution

80

100

8
9

0

20

40

60

80

100

iteration t

finalThe excitable system emerges, which
possesses characteristics of neurons
Membrane potential

X(t+1)
X(t+1)=X(t)Thresholdtime

Threshold

x1

x2
X(t)

Equilibrium state

The resulting dynamics of the network
is chaotic:
a neuron works and processes informationin chaotic environment.

x1, x2: initial conditions
31

Three types of dynamical systems were selected：
Success of differentiation of spiking neurons and glial cells
Identity map(via strong
coupling)

Excitable map(via weakcoupling)
Oscillatory map(whole rangesof couplingstrength)

Passivedepolarization:
Glial cells-type

Neuronal
spiking:
Neuronal cells
type

Active variations:
Neuronal or Glial cell
type

32

3. Mathematical modeling for
Evolutionary Reservoir Computers (ERC):
Success of Learning of Separation of Temporal and Spatial Patterns

Y. Yamaguti and I. Tsuda, Chaos 2021

33

3.1 Investigating the Neural Specificity:
Differentiation of Neurons• Negative correlation appears between
𝐼𝑠𝑝 and 𝐼𝑡𝑒𝑚𝑝 .
• Emergence of ‘spatial neuron’ and‘temporal neuron’
Initial network

auditoryneurons

After evolution

visualneurons
nondifferentiatedneurons⇒ synesthesia ?
34

3.2 Changes of the Network Architecture
The feedforward connections are more strengthened than the
feedback connections: about 5:1-3:2, depending on the evolutioncf) Local Network in
Rats primary visual cortex ~10:1,
Human frontal and temporal cortices ~ 5:1-7:1(Seeman et al 2018)
decreasedecreaseincrease
Random networks evolved to feedforward
networks with weaker recurrent networks⇒ Consistent with small-world network
by Kawai et al

Input

Input
neurons

Outputneurons

SpatialOutputTemporal
Outputdecrease35

4. reBASICS: reservoir of basal dynamics[Kawai, IT et al., ICANN, 2022; Kawai, IT et al. Neur. Netw. 2022 ]• Several small random neural networks are used as
modules and connected in parallel.
• Each module spontaneously produces stable timeseries with diverse phases and frequencies.
• They are functionally differentiated to create an
orthogonal basis.

𝑟1 𝑡
𝑢(𝑡)

𝑟2 𝑡
𝑦 𝑡
Recursive
least squares𝑟𝑀 𝑡Very large timing capacity[Kawai et al., ICANN, 2022; Neural Networks 2022]• reBASICS could learn the timing even for very long intervals of one minute or more.
• The total performance (timing capacity) of reBASICS was more than twenty times
larger than that of the innate training (existing approach).
• reBASICS could also learn the long-term Lorenz time series.

Correction of PD control of a robot using reBASICS[Kawai & Asada, 2024]

A cerebellar-inspired system that
adapts motor commands to achanging environment immediately

reBASICS
Actual position

Onset
signalPosition
error−
Trajectory
generator+

Approx.
IK

Angle error
Angle

Torque−+Desiredtrajectory

Without
correction

Error = 0.094+

KpD
controllererrorcorrection

Robot
arm

Position of theend effector

Actual angle

Error = 0.036

Desired trajectory
Actual trajectory

Rapid adaptation to the body’s physical changes
Change of links’ length

Change of links’ weight

Error correction

Error correction

Error[Kawai & Asada, 2024]

Learning epochs

What kind of chaos is adequate for information processing in the brain？
⇒Establishment of chaotic information processing
K. Matsumoto and I. Tsuda, 1985, 1987, 1988; R. Shaw 1982;
J. S. Nicolis 1982, 1991; J.S.Nicolis and I. Tsuda 1985, 1989

𝐼 𝑡

Kullback-Leibler divergence ⇒ Lyapunov exponent⇒Mutual information⇒The definition of time-dependent mutual information

情報流 Information flow←
BZ map←→→→←
情報流の揺らぎ大Large fluctuation ofInformation flow
Logistic map
情報流の揺らぎ小
Small fluctuation ofInformationflow←= 𝐼 0 𝑒 −𝛼𝑡 →𝑑𝐼= −𝛼𝐼 →𝑑𝑡𝑑𝐼
= −𝛼𝑑𝑡𝐼
同じ割合の情報損失
The same ratio of
information loss

Without noise

With noise

Without noise

With noise

同じ量の情報損失
The same amount of→
information loss
← 𝐼 𝑡 = 𝐼 0 − 𝛽𝑡 → 𝑑𝐼 = −𝛽 → 𝑑𝐼 = −𝛽𝑑𝑡𝑑𝑡←The case of large fluctuation of information flow
Mutual information can be decomposed into
bit-wise information.
Decomposition into
bit-wise information
𝐼 = (𝑖1 , 𝑖2 , 𝑖3 , ⋯ )K. Matsumoto and I. T, J. Phys. A 1988

⋅ 𝑖 ,𝑖 ,𝑖1

Input 𝐼⋮⋅ 𝑖 ,𝑖 ,𝑖⋅3

𝑖1, 𝑖2, 𝑖3, ⋯𝑖1, 𝑖2, 𝑖3, ⋯𝑖1, 𝑖2, 𝑖3, ⋯𝑖1 , 𝑖2

𝑖4

𝑖2 , 𝑖3 , 𝑖5

⋮

⋮

⋮

2

3

𝑖1 , 𝑖2 , 𝑖3chaotic system 1

1

2

情報の重ね合わせが可能
Superposition of information contents⋯

chaotic system 2

chaotic system 3

入力情報はネットワークの中に保存される
Input information is
dynamically stored in the
network
生物学的カオスはほぼこの性質を満たす
Biological chaos satisfy thisproperty

Shannon’s channel シャノンの通信路

相互情報量をビットごとの情報量に展開できる
Mutual information can be decomposed into
bit-wise information quantities

BZ map結合カオス力学系（⇒カオスネットワーク）を
情報の通信路とみなす（カオスは内的ノイズを有する）
Coupled chaotic dynamical systems (chaotic networks)
are viewed as Information channel(chaos possesses internal noise)

Logistic map

Constraints

Philosophy is necessaryℚ = 𝐸𝑅𝐶, 𝑟𝑒𝐵𝐴𝑆𝐼𝐶𝑆

↬

𝐸𝑛𝑣.

↬

Rapid adaptation after
functional differentiation

We developed a new type of reservoir computers, which
rapidly adapt to given environments and solve tasks, byrealizing functional differentiation according to given constraints.
＿＿＿＿＿＿＿___________________________________________________________________ℋ1 (ℚ1 , ℚ2 , ⋯ , ℚ𝑛 )↬ℋ2 (ℚ1 , ℚ2 , ⋯ , ℚ𝑛 )What is emergent information in a society consisting of interactive agents ℚ𝐬 ?
I personally hope, it should be conscience.

参考資料

大偏差原理(Large Deviation Principle: Donsker-Varadhan)
𝑋𝑛 : 𝑖. 𝑖. 𝑑. , 𝐸 𝑋1 = 𝑚, 𝑉(𝑋1 ) = 𝜎 2 = 𝑣 > 0とする．
𝑆𝑛 = σ𝑛𝑘=1 𝑋𝑘 , 𝑎 > 𝑚に対して𝑃(𝑆𝑛 > 𝑎𝑛)を求めたい．
𝑆
大数の法則（ 𝑛 → 𝑚, 𝑎. 𝑠.）より、𝑃 𝑆𝑛 > 𝑎𝑛 → 0 𝑛 → ∞𝑛

中心極限定理より𝑃 𝑆𝑛 > 𝑎 𝑛 + 𝑚𝑛

𝑆 −𝑚𝑛
= 𝑃( 𝑛 𝑛 > 𝑎) →2

1
∞ −(𝑥 )‫ 𝑒 ׬‬2𝑣 𝑑𝑥
2𝜋𝑣 𝑎

2

𝑥
∞
−( 2𝑣 )𝑒𝑑𝑥
‫׬‬
2𝜋𝑣 𝑛(𝑎−𝑚)1

したがって、𝑃 𝑆𝑛 > 𝑎𝑛 →これが０に収束していくオーダーを与えるのが大偏差原理

大偏差原理（Cramér’s theorem）
𝑋𝑛 : 𝑖. 𝑖. 𝑑. , ∀𝑡 ∈ 𝑅, 𝐸[𝑒 𝑡 𝑋1 ] < ∞とする．
𝜑 𝑡 = 𝐸(𝑒 𝑡𝑋1 )とおく．∀𝑎 > 𝐸(𝑋1 ) = 𝑚に対して，
1
lim 𝑛 log 𝑃 𝑆𝑛 > 𝑎𝑛 = −𝐼 𝑎 , すなわち𝑃 𝑆𝑛 > 𝑎𝑛 ~𝑒 −𝑛𝐼 𝑎 .
𝑛→∞

ここで𝐼 𝑎 = sup𝑡 (𝑎𝑡 − log 𝜑(𝑡)) , lim 𝐼 𝑎 = ∞, 𝐼 𝑎 ≥ 0 = 𝐼(𝑚)であり、
𝑎→±∞𝐼(𝑎)は下半連続(lower semicontinuous:∀𝜀 > 0, ∃𝛿 > 0, ∀𝑦, 𝑦 − 𝑥 < 𝛿, 𝐼 𝑦 > 𝐼 𝑥 − 𝜀 （下から幅𝜀で抑えられる）で、凸関数
つまり、𝐥𝐨𝐠 𝝋(𝒕)の𝒕に関するルジャンドル変換である𝑰(𝒂)が存在する．

Donsker-Varadhan表現
定理 For stochastic variables 𝑋, if stochastic distribution functions 𝑝 𝑥 , and 𝑞 𝑥 are defined, the following
equality holds.
𝐷𝐾𝐿 (𝑞| 𝑝 = sup𝑇:𝑋→𝑅 (𝐸𝑞 𝑇 𝑥 − log 𝐸𝑝 𝑒 𝑇 𝑥 )証明）𝑞(𝑥)の近似分布ℎ(𝑥) を次のように考える（経験分布ℎ(𝑥) を母集団分布𝑝 𝑥 で測る：ℎ(𝑥) は𝑝 𝑥 が属する空間と同じ空間からとるとする）。

−𝑇(𝑥)をエネルギー関数と考えて、ギブス分布（が存在すると仮定）
を近似分布とする。 ℎ 𝑥 =𝑒𝑇 𝑥

𝑒 𝑇 𝑥 𝑝(𝑥)‫𝑥𝑑 𝑥 𝑇 𝑒 𝑥 𝑝 ׬‬=𝑒𝑇 𝑥
𝐸𝑝 𝑒 𝑇 𝑥

𝑝(𝑥)

𝑞 𝑥

𝑞 𝑥

ℎ 𝑥 = 𝐸 𝑒 𝑇 𝑥 𝑝(𝑥)より、 𝐷𝐾𝐿 (𝑞| 𝑝 = 𝐸𝑞 log 𝑝 𝑥 , 𝐷𝐾𝐿 (𝑞| ℎ = 𝐸𝑞 log ℎ 𝑥𝑝

𝐷𝐾𝐿 (𝑞| 𝑝 − 𝐷𝐾𝐿 (𝑞| ℎ = 𝐸𝑞 𝑇 𝑥− log 𝐸𝑝 𝑒 𝑇 𝑥

より
𝐷𝐾𝐿 (𝑞| ℎ = 𝐷𝐾𝐿 (𝑞| 𝑝 − (𝐸𝑞 𝑇 𝑥ゆえに定理が従う。

− log 𝐸𝑝 𝑒 𝑇 𝑥 ) ≥ 0

定理（Ichiro Tsuda 2023） When 𝑞 is defined properly for 𝑝, the equality
𝐼 𝑎 = 𝐷𝐾𝐿 (𝑞| 𝑝 holds.
証明）
𝐼 𝑎 = sup𝑡 𝑎𝑡 − log 𝜑 𝑡

𝜑 𝑡 = 𝐸(𝑒 𝑡𝑋1 )𝐼(𝑎)は𝜑 𝑡 の𝑡 に関するルジャンドル変換： 𝑡, 𝜑 𝑡
𝑋1 の分布を𝑝(𝑋1 )とする。𝑎 =𝐼 𝑎 = sup𝑡 𝑎𝑡 − log 𝜑 𝑡

𝜑′(𝑡)
𝜑(𝑡)

=

= sup𝑡𝐸(𝑋1 𝑒 𝑡𝑋1 )𝐸(𝑒 𝑡𝑋1 )𝐸 𝑋1 𝑒 𝑡𝑋1
𝐸 𝑒 𝑡𝑋1→ 𝑎, −𝐼 𝑎=(𝑑(log 𝜑(𝑡)), −𝐼(𝑎))𝑑𝑡

より、 𝑎 は分布𝑞(𝑋1 ) ≡ 𝑝(𝑋1 )𝑒 𝑡𝑋1 による𝑋1 の平均。
𝑡 − log𝐸 𝑒 𝑡𝑋1= sup𝑡 𝐸𝑞 𝑡𝑋1 − log𝐸𝑝 𝑒 𝑡𝑋1
Donsker-Varadhan表現より、 𝐼 𝑎 = 𝐷𝐾𝐿 (𝑞| 𝑝⇒ This leads to a reasonable interpretation that the estimator of MI with deep neural networks
guarantees a sufficient sampling over even largely deviated from an average.

Network size and chaos• In a network with a finite size 𝑁 as 𝑔 increases larger than 1,
asymptotically stable state → limit cycle state → chaotic state.
[Sompolinsky et al., Physical Review Letters, 1988; Doyon et al., Acta Biotheoretica, 1994]

Stable &Self-sustained𝑔 < 1.0
Stablebut decaying𝑔 > 1.0

𝑔 > 1.0
Self-sustained
but unstable

Network size 𝑁• However, small 𝑁 = low degrees of freedom & low orthogonality

Chaotic
Stable
Limit
cycle

1

𝑔

Motor timing learning[Kawai et al., ICANN, 2022; NN, 2023]• An example of the task with an interval of 1 s.
• Performance R2: the square of the correlation coefficient between
target and output

The number of modules M[Kawai et al., submitted]