<figure><img src="image_1.png" /></figure>

HAL
open science

# Modeling and Dynamic Analysis on Animals' Repeated Learning Process

Mu Lin, Jinqiao Yang, Bin Xu

► To cite this version:

Mu Lin, Jinqiao Yang, Bin Xu. Modeling and Dynamic Analysis on Animals' Repeated Learning Process. 12th Engineering Applications of Neural Networks (EANN 2011) and 7th Artificial Intelligence Applications and Innovations (AIAI), Sep 2011, Corfu, Greece. pp.116-121, 10.1007/978-3-642-23960-1_15. hal-01571451

HAL Id: hal-01571451

https://inria.hal.science/hal-01571451

Submitted on 2 Aug 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Modeling and dynamic analysis on animals'
repeated learning process

Mu Lin<sup>1,2,*</sup>, Jinqiao Yang<sup>3</sup>, Bin Xu<sup>2</sup>

<sup>1</sup>School of Applied Mathematics, Central University of Finance and Economics, Beijing, 100081, China.

<sup>2</sup>Department of Mathematical Sciences, Tsinghua University, Beijing, 10084, China.

<sup>2</sup>School of Humanities, Dongbei University of Finance and Economics, Dalian, 116025, China.

*Corresponding Author: mlin@math.tsinghua.edu.cn

**Abstract.** Dynamical modeling is used to describe the process of animals' repeated learning. Theoretical analysis is done to explore the dynamic property of this process, such as the limit sets and their stability. The scope of variables is provided for different practical purpose, cooperated with necessary numerical simulation.

**Keywords:** dynamical modeling; animals' learning; animal training; piecewise-smooth system.

# 1 Introduction

Learning is always a hot issue in pedagogy and psychology research. J. Dewey considered learning as the process of individual activities, from the perspective of Pragmatism [1]. He proposed "learning by doing", and emphasized the effect of specific situation on the individual learning behavior. However, from the perspective of Constructivism, J. Piaget considered learning as a kind of interaction process between individual and the environment surrounding him, which makes his cognitive structure develop constantly and gradually [2].

We can see different learning behaviors everywhere. One common adopted way is repeated learning, such like reciting foreign language words repeatedly to remember them, and doing exercises frequently to learn driving. [3]. This process can make some adaptable changes on the mental level. And one of the most important changes is, learners can abstract an "schema" from the operating process of the skill, and this is so called "transfer of learning" [4].

In modern psychology, learning is such a kind of behavior which is not only owned by human beings, but also by animals. Lots of researchers studied animals' learning behavior. In experiments, E. Thorndike observed the escaping behavior of a cat which was trapped in a box. He argued that the cat keep running aimlessly at first but gradually drop the ways that cannot open the box and finally acquire the right way to escape [5]. B. Skinner observed the bar-pressing behavior of a mouse in the box, which can serve itself food, and argued that the
animals reacted to the reinforcement in particular ways. And this "reinforcement - reaction" connection is learned by the animals themselves [6].

There are a lot of studies of learning behavior of animals, although the learning contents maybe different, but the way of learning is always repeated learning, for it is widely believed that animals' learning ability is poor. Furthermore, lots of studies seem to tell us that based on this level of learning ability, the animals can perform the transfer of learning like human being. For example, in the Pavlovian classical experiments, the dog acquired the conditioned reflex of "bell ringing - salivating" [7]. So, if we make an animal learn a kind of behavior repeatedly, it indeed can influence the generation of other related behaviors. But if this influence exists, how it works? And how strong it can be? Both of these questions are important because they can help us know the nature of learning deeper. The present study will answer these questions, and try to provide a mathematical model of relationship between these two behaviors.

Mathematical modeling is in psychology and society has rapid development these years [8]. Social mechanisms are analyzed by analytical and numerical approaches. For example, He et al. found a cellular automata model to describe the learning process [9]. Dynamic analysis is used for modeling the problems that have definite dynamic relation or feedback relationship. It has been widely used to solve ecology and demography problems. There are obviously two sub-processes in this repeated learning problem, i.e., the learning are forgetting states. It belongs to the piecewise-smooth problems [10].

The remainder of this paper is as follows. Section 2 is the mathematical modeling process, and three types of switching rules are enumerated in Section 3. More precise dynamic analysis including some numerical simulation is done in Section 4. Section 5 is the conclusion.

## 2 Modeling Framework

We establish a dynamic model in this section first. As mentioned above, for artificial animals' repeated learning process, there are obviously two discrete states, the training condition and forgetting condition, noted as $Q = 1$ and $Q = -1$ separately. Note $x$ as the main memory ability, and $y$ as the relevance memory ability, $0 \le x, y \le 1$.

In the training condition, following the logistic model, the dynamic function of the main memory can be described as

$$ \dot{x} = \alpha_1 x(1-x). \quad (1) $$

While in the forgetting condition, by Ebbinghaus forgetting curve, the dynamic function of main memory ability can be described as

$$ \dot{x} = -\beta_1 x. \quad (2) $$

The variation of the relevance memory ability is related with the main memory ability. In the training condition, it can be described as

$$ \dot{y} = \alpha_2(x - l_1)y(1 - y), \quad (3) $$
where $\alpha_2 > 0$. There is one extra term $(x - l_1)$ describing the influence of the main memory ability to the related memory ability. More concretely, there is a separation $x = l_1$, $0 < l_1 < 1$, such that the relevance memory ability increases when the main memory ability $x > l_1$ and reduces when $x < l_1$. Similarly, the equation is

$$ \dot{y} = -\beta_2(l_2 - x)y(1 - y) \quad (4) $$

in the forgetting condition, where $0 < l_1 < l_2 < 1$, $\beta_2 > 0$.

The explicit solution of the simultaneous equations (1)-(4) is

$$ \begin{cases} x(t) = 1 - \frac{1}{1+e^{\alpha_1 t+c_1}} \\ y(t) = 1 - \frac{1}{1+(1+e^{\alpha_1 t+c_1})^{\frac{\alpha_2}{\alpha_1}} e^{-\alpha_2 l_1 t+c_2}} \end{cases} \quad \text{when } Q = 1; $$

$$ \begin{cases} x(t) = c_3 e^{-\beta_1 t} \\ y(t) = 1 - \frac{1}{1+e^{-\beta_2 l_2 t-c_3 \frac{\beta_2}{\beta_1} e^{-\beta_1 t+c_4}}} \end{cases} \quad \text{when } Q = -1, \quad (5) $$

where $c_i, i = 1, 2, 3, 4$ are arbitrary constants.

In order to get a complete deterministic system, a switching rule between the training and forgetting conditions is needed. Actually, there are four following types, as listed in the following section.

# 3 Dynamic analysis under different switching rules

## 3.1 Quantitative control switching rule

The quantitative control switching rule contains two preestablished bounds $w_1$ and $w_2$. During the training condition ($Q = 1$), the system switches to the forgetting condition ($Q = -1$) when $x$ reaches $w_2$. Vice versa. So it makes a periodic learning cycle consist of one training condition and one forgetting condition, See equation (6).

$$ Q(t^+) = \begin{cases} -Q(t) & \text{if } Q(t) = -1 \text{ and } x(t) \le w_1, \text{ or } Q(t) = 1 \text{ and } x(t) \ge w_2, \\ Q(t) & \text{else} \end{cases}, \quad (6) $$

Composed the above switching rule and equation (1)-(4), a deterministic piecewise-smooth system is obtained. Usually, the analysis of piecewise-smooth systems is quite complicated, while our model is a providential one for it has an explicit solution. Assume the point $(w_1, y)$ changes to $(w_1, f(y))$ in a whole training-forgetting period. By computation, we get $f(y) = \frac{Ay}{(A-1)y+1}$, where $A = [(\frac{w_1}{w_2})^{l_2}e^{w_2-w_1}]\beta[(\frac{w_1}{w_2})^{l_1}(\frac{1-w_1}{1-w_2})^{1-l_1}]^\alpha > 0$, and $\alpha = \frac{\alpha_2}{\alpha_1}, \beta = \frac{\beta_2}{\beta_1}$. So $f'(y) = \frac{A}{[(A-1)y+1]^2}$.

Consider the equilibrium point of map $f$, i.e. the point $y$ satisfies $f(y) = y$. The equilibrium point of $f$ indicates the period solution of the whole system, meaning that the collateral memory ability changes periodically with the main
memory ability. Solving the equation $f(y) = y$, we get $A = 1$ or $y = 0, 1$. For $y = 0$ and $y = 1$ are two trivial solutions, we focus on the case $A = 1$, then

$$ \alpha(1-l_1) \ln(1-w_1) + (\alpha l_1 + \beta l_2) \ln w_1 - \beta w_1 = \alpha(1-l_1) \ln(1-w_2) + (\alpha l_1 + \beta l_2) \ln w_2 - \beta w_2 $$

Note $g(w) = \alpha(1-l_1) \ln(1-w) + (\alpha l_1 + \beta l_2) \ln w - \beta w$, then $g(0) = g(1) = -\infty$.

The solution of $g'(w) = 0$ is $w_0 = l_2 - \frac{\sqrt{(\alpha+\beta-\beta l_2)^2+4\alpha\beta(l_2-l_1)-(\alpha+\beta-\beta l_2)}}{2\beta}$. A group of parameters $\alpha_i, \beta_i, l_i, i = 1, 2$ guaranteeing $A = 1$ may exist for $w_1, w_2$ satisfied $0 < w_1 < w_0 < w_2 < 1$ can be easily chosen. That means $f(y) \equiv y, \forall 0 < y < 1$, i.e., the collateral memory ability can keep a stable circle at any ability.

Otherwise if $A > 1$, then $f'(0) = A > 1$, so $y = 0$ is an unstable equilibrium point, while $f'(1) = 1/A < 1$, so $y = 1$ is a stable equilibrium point. That is to say, under this quantitative control switching rule, orbits start from every initial point except $x = 0$ will converge to $y = 1$ after sufficient times of training-forgetting cycling, i.e. the collateral memory ability will converge to the upper bound. Oppositely, if $A < 1$, $y = 0$ is a stable equilibrium point while $y = 1$ is an unstable equilibrium point. So the collateral memory vanishes no matter how the initial ability was.

## 3.2 Timing control switching rule

Another common switching rule is based on timing control. In this case, each time length of training and forgetting condition is fixed, noted as $\tau_1$ and $\tau_2$ separately. In most cases $\tau_2 > \tau_1 > 0$.

The Poincaré map of the system can also be computed explicitly in this case. The system starts from $(x_0, y_0)$ and changes to

$$ x(\tau_1+\tau_2) = \frac{e^{-\beta_1\tau_2}}{1 + (\frac{1}{x_0} - 1)e^{-\alpha_1\tau_1}} = x_0 \cdot \frac{1}{e^{\beta_1\tau_2}[x_0(1 - e^{-\alpha_1\tau_1}) - (e^{-\beta_1\tau_2} - e^{-\alpha_1\tau_1})] + 1} $$

and $y(\tau_1 + \tau_2) = \frac{M}{M+N(\frac{1}{y_0}-1)}$, where $M = e^{-\beta_2 l_2 \tau_2} - \frac{\beta_2}{\beta_1} x_1 e^{-\beta_1 \tau_2}$,

$$ N = \frac{\frac{\alpha_2}{\alpha_1} e^{\frac{\beta_2}{\beta_1} x_1}}{(1 + (\frac{1}{x_0} - 1)e^{-\alpha_1 \tau_1})^{\frac{\alpha_2}{\alpha_1}} e^{\alpha_2(1-l_1)\tau_1}} $$

So the equilibrium point is $x = 0$ or $x = \frac{e^{-\beta_1 \tau_2} - e^{-\alpha_1 \tau_1}}{1 - e^{-\alpha_1 \tau_1}}$. For the model located in $x \ge 0$, there is only one equilibrium point $x = 0$ and it is stable, when $\alpha_1 \tau_1 \le \beta_1 \tau_2$. There are one unstable equilibrium point $x = 0$ and one stable equilibrium point $x(0) = \frac{e^{-\beta_1 \tau_2} - e^{-\alpha_1 \tau_1}}{1 - e^{-\alpha_1 \tau_1}}$.

In this case, $y(0) = y(\tau_1 + \tau_2), \forall y(0)$ if and only if $M - N = 0$. Otherwise the system has two equilibrium points $y = 0$ and $y = 1$. If $M > N$, 0 is unstable and 1 is stable, while if $M < N$, 0 is stable and 1 is unstable.

## 3.3 Semi-timing control switching rule

Composing the above two rules, the semi-timing control switching rule is also widely used in actual. This rule fixed simultaneously the upper bound (or lower
bound) and the forgetting time length (or the training time length). The corresponding Poincare map is:

(1) The system starts from $(w_1, y_0)$, goes through a $\tau_1$ time training condition and reaches $(x_1, y_1)$, then goes back to $(w_1, y_2)$ through a forgetting condition. By computation, we get $y_2 = \frac{M}{M+N(\frac{1}{y_0}-1)}$, where $M = (w_1 + (1 - w_1)e^{-\alpha_1\tau_1})^{(\frac{\beta_2l_2}{\beta_1}+\frac{\alpha_2}{\alpha_1})}e^{\alpha_2(1-l_1)\tau_1}$, $N = e^{-\frac{\beta_2}{\beta_1}\frac{1}{w_1+(1-w_1)e^{-\alpha_1\tau_1}}}$.

(2) The system starts from $(w_2, y_0)$, goes through a $\tau_2$ time forgetting condition and reaches $(x_1, y_1)$, then goes back to $(w_2, y_2)$ through a training condition. Here $y_2 = \frac{M}{M+N(\frac{1}{y_0}-1)}$, where

$$M = (1+(\frac{e^{\beta_1\tau_2}}{w_2}-1)e^{-\alpha_1\tau_1})^{\frac{\alpha_2}{\alpha_1}}(\frac{1-w_2}{e^{\beta_1\tau_2}-w_2})^{-\frac{\alpha_2}{\alpha_1}(1-l_1)}, N = x_0^{-\frac{\alpha_2}{\alpha_1}}e^{\beta_2l_2\tau_2+\frac{\beta_1}{\beta_2}w_2(e^{-\beta_1\tau_2}-1)}.$$

In both two subcases, the dynamic property of $y$ is similar with the one in the timing control switching rule.

## 4 The ultimate stable states and bifurcation analysis

<figure><img src="image_3.png" /><figcaption>Fig. 1. The ultimate stable states under the perturbation of measurement error</figcaption></figure>

Although infinite multiple stationary states are existence in some cases of the above analysis, most of them are unstable in fact, for the observation is discrete. More specifically, throughout the above analysis, we assume that the system switches instantaneously whenever the main memorial ability reaches the preestablished value in Section 3.1 and 3.3. Actually, this instant transformation cannot be realized for the estimation of memorial ability must be discrete, furthermore, it costs time.

To describe the influence of the discrete observation, the switching rule can be rewritten as: (take the quantitative control switching rule in Section 3.1 for instance)

$$Q(t^+) = \begin{cases} Q(t) & \text{if } t_n \le t < t_{n+1}, \\ -Q(t) & \text{else} \end{cases}, \qquad (7)$$
where $t_i, i = 0, 1, ...$ is a group of switching time series. Usually they are determined mainly depending on a presupposed admissible error $\epsilon > 0$. And the system switches when the observed variable enters a presupposed neighborhood of the border line. That is to say, if $\Delta t$ is the observed interval time length,

$$t_i = k_i \Delta t, s.t. |x_i(k \Delta t) - w_1| < \epsilon, \text{ or } |x_i(k \Delta t) - w_2| < \epsilon, k_i \in \mathbb{N}.$$

Under this switching rule, the ultimate stable stationary states can be drawn by numerical simulation, as in Figure 1. Unlike the results given in Section 3.1, there are only finite stable stationary states, means only these states can exist under the discrete observation.

## 5 Conclusion

Mathematical model is established to discuss the relationship between the main memory ability and the relevance memory ability. Different switching rules are introduced and under each rule, equilibria and their stability are discussed. Although with some particular parameters, every orbit can be a periodic invariable one, but there can only be finite periodic orbits, due to the discrete observation, as explained concretely in Section 4. The results of dynamic analysis can be used for animal training. Further observational study can be done to demonstrate the models.

## 6 Acknowledgements

This study was funded by the National Natural Science Foundation of China (11072274).

## References

1. John Dewey. Democracy and Education: An Introduction to the Philosophy of Education. Macmillan Company. 1916.

2. Jean Piaget. The Psychology of Intelligence. Taylor and Francis Press. 2001.

3. Schmidt, Richard A. A schema theory of discrete motor skill learning. Psychological Review, Vol 82(4), 1975, 225-260

4. James M. Royer. Theories of the transfer of learning. Educational Psychologist. Vol 14(1). 1979, 53 - 69.

5. E. Thorndike. Animal Intelligence. Thoemmes Press. 1911.

6. Charles B. Fester B. F. Skinner Carl D. Cheney W. H. Morse P. B. Dews. Schedules of Reinforcement (B. F. Skinner Reprint Series). Copley Publishing Group. 1997

7. P Chance. Learning and Behavior: Active Learning Edition. Wadsworth Publishing Co Inc. 2008.

8. Burghes, D.N. and M.S. Borrie, Modelling with Differential Equations, Chichester, Horwood, 1981.

9. He mingfeng, Deng Chengrui, Feng lin, and Tian bowen. A cellular automata model for a learning process, Advances in Complex Systems 07, no. 03-04,2004, 433-439.

10. Kunze M., Non-smooth Dynamical Systems, Springer-Verlag, Berlin, Heidelberg, 2000.