# Mouse-Brain Topology Improved Evolutionary Neural Network for Efficient Reinforcement Learning

Xuan Han, Kebin Jia, Tielin Zhang

<figure><img src="image_1.png" /></figure>

## To cite this version:

Xuan Han, Kebin Jia, Tielin Zhang. Mouse-Brain Topology Improved Evolutionary Neural Network for Efficient Reinforcement Learning. 5th International Conference on Intelligence Science (ICIS), Oct 2022, Xi'an, China. pp.3-10, 10.1007/978-3-031-14903-0_1. hal-04666431

HAL Id: hal-04666431

https://hal.science/hal-04666431v1

Submitted on 1 Aug 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Mouse-Brain Topology improved Evolutionary
Neural Network for Efficient Reinforcement
Learning

Xuan Han¹,², Kebin Jia¹, and Tielin Zhang²

¹ Beijing University of Technology, Beijing, 100124, China.

² Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China.
kebinj@bjut.edu.cn, tielin.zhang@ia.ac.cn

**Abstract.** The brain structures are key indicators to represent the complexity of many cognitive functions, e.g., visual pathways and memory circuits. Inspired by the topology of the mouse brain provided by the Allen Brain Institute, whereby 213 brain regions are linked as a mesoscale connectome, we propose a mouse-brain topology improved evolutionary neural network (MT-ENN). The MT-ENN model incorporates parts of biologically plausible brain structures after hierarchical clustering, and then is tuned by the evolutionary learning algorithm. Two benchmark Open-AI Mujoco tasks were used to test the performance of the proposed algorithm, and the experimental results showed that the proposed MT-ENN was not only sparser (containing only 61% of all connections), but also performed better than other algorithms, including the ENN using a random network, standard long-short-term memory (LSTM), and multi-layer perception (MLP). We think the biologically plausible structures might contribute more to the further development of artificial neural networks.

**Keywords:** Evolutionary neural network · Reinforcement learning · Mouse-brain topology.

# 1 Introduction

The mammalian brain has been studied deeply in the past decades. The brain structures are key indicators for representing the learned knowledge from millions of years of evolution in different animals. Like many genetic evolutionary algorithms in machine learning, the biological brain is the best evolutionary outcome in a natural organism. However, it is still an open question whether new network structures copied from natural brains could contribute to the development of artificial neural networks, whereby the design of structures is usually considered more important than neuronal types. At present, the whole mouse brain has been widely examined, which contains around 213 brain regions, and the sparseness of the entire brain is about 36% [5]. Many types of research have been given on the topology of the mouse brain, and one of the most important
motivations is copying it to the conventional artificial neural networks (ANNs) for higher performance, or more energy efficiency [6].

Inspired by the biological networks and evolutionary algorithms, we propose a mouse-brain topology improved evolutionary neural network (MT-ENN), which incorporates biologically plausible brain structures and an evolutionary learning algorithm for efficient learning on two benchmark reinforcement learning (RL) tasks. The dynamic neurons contain 1st-order dynamics of membrane potentials supported by some additional key dynamic parameters. Then these neurons are connected by the copied topology from the biological brain and tuned by a global evolutionary algorithm. The two open-AI gym tasks, e.g., MountainCar-v2 and Half-cheetah-v2, are selected to verify the proposed MT-ENN. In addition, we also made a three-dimensional (3D) visual reconstruction of the entire network topology after network learning, which is convenient for identifying the connectome of different brain regions and might inspire back to the neuroscience researchers and answer the question that why some topology or brain regions are important for reinforcement learning.

## 2 Related works

A new continuous-time differential learning was proposed for RL tasks containing continuous dynamics [1]. Then, a hybrid learning framework was proposed by extending this new learning rule with a multi-scale dynamic coding [9]. The topology-focused algorithm was proposed by using a sub-network to replace a previous global network and achieved comparable performance, named as the lottery ticket hypothesis. A biological network using C.elegans topology was proposed to achieve higher performance than algorithms using random networks [3], which showed the efficiency of the biological topology.

These algorithms all performed well on RL tasks. However, they seldom use complicated network topology. One of the main motivations of this paper is that the further incorporation of network topologies, especially from smarter animals than C.elegans, will strengthen intelligent algorithms on cognitive functions to handle RL tasks. In addition, we think reducing the size of fully connected networks for better interpretability is at least as important as performance, which will also be further discussed in the following sections.

## 3 Methods

### 3.1 The Allen Mouse Brain Atlas

The Allen Institute for Brain Science has provided the public with the whole mouse brain atlas, containing neuronal types and network topology from some standard mouse brains. The network topology contains the directional mapping of 213 brain regions, which are further visualized by Houdini software for a better understanding. The overall connectome of the mouse brain is shown in Fig. 1A, where the 213 × 213 matrix is the connectomes between 213 brain regions. The
connectome is sparse, indicating that the bottom-up and top-down connections
are hierarchical and highly related to cognitive functions.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** The generation procedure of brain-inspired topology. (A) The mesoscale connectome of the Allen mouse brain atlas [5]. (B) An example of clustered topology from 213 brain regions. (C,D) The hierarchically clustered brain regions. The 213 mouse brain regions are split into 71 clusters at a cutting height of 0.8 (C) and the partition density (D). (E) The connectedness of different communities (clusters). (F) Schematic diagram of the connection of 46 brain regions in the 3D brain. (G) Schematic diagram of the connection of 46 brain regions in 2D visualization.</figcaption></figure>

3.2 The clustered hierarchical circuits

The connectivity matrix of the mouse brain atlas was then clustered into sub-clusters for easier analysis. It is a general challenge to find a proper clustering algorithm that could filter out trivial branches but leave out the key functional topology. A special hierarchical clustering method, i.e., Tanimoto coefficient algorithm, is selected for this clustering. The algorithm could generate multiple clusters given a directional topology. The detailed procedure of the clustering algorithm could be concluded as the following equations:
$$S(e_{ik}, e_{jk}) = \frac{a_i \cdot a_j}{|a_i|^2 + |a_j|^2 - a_i \cdot a_j}, \quad (1)$$

We divide 213 regions of the mouse brain into 71 clusters at a cutting height of 0.8 (relative value designed by experience) based on the links or connections between each two brain regions in both directions as well as weights or intensities of projections (Fig. 1C,D).

### 3.3 The neuron model

Here, we briefly describe the neuron and synapse model to design neural circuit dynamics [2] by using the following equation.

$$\left\{
\begin{aligned}
\dot{V}_i(t) &= [I_L + \sum I_C(t)]/C_m \\
I_L(t) &= \omega_L [E_L - V_{post}(t)] \\
I_C(t) &= \omega_C [E_C - V_{post}(t)]g(t) \\
g(t) &= 1/[1 + \exp(-\sigma(V_{pre}(t) - \mu))]
\end{aligned}
\right.
, \quad (2)$$

where $C_m$ is the membrane capacitance of the neuron, $I_C$ and $I_L$ are the input currents of the chemical channel and leakage channel, respectively. $E_C$ and $E_L$ are the corresponding reversal potentials. $V_{post}(t)$ and $V_{pre}(t)$ are the membrane potentials of post-synapses and pre-synapses, respectively. $g(t)$ is the membrane conductance, defining whether a synapse is excitatory or inhibitory by $E_C$. $\omega_C$ and $\omega_L$ stand for the conductances of the chemical channel and leakage channel. These equations show that the algorithms have a strong ability to model the time series reaching any time step.

### 3.4 Coping the biological circuits to artificial ones

According to the definition of these 213 brain regions in the Allen mouse brain atlas, we reorganized them into sensation, hidden, and motor brain regions, according to structural projection and physiological function. We also annotate the biological functions of the brain regions at each level of the communities of interest, which makes it possible to compare the hierarchical composition and functional circuitry of the community or groups of interest with the combination of the atlas.

Hence, some different types of topology are selected for the next-step simulation, including the whole 213 brain regions (directly copied from the biological atlas) and some other sub-brain topology (selected from 31, 46, and 49 brain regions based on the hierarchical clustering), which will be further discussed in the experimental sections.

### 3.5 The network learning

In this paper, a simple search-based algorithm is selected for network reinforcement learning [3], whereby the agent learns to make decisions after observing the
current state in an environment and then receives a timely or delayed reward. A fitness function is designed to collect these rewards and guide the direction of the random search. At the beginning of learning, the agent makes random decisions for exploration, and a good decision for a lower fitness function will be kept by saving the current parameters and focusing more on the exploitation. The search-based algorithm ARS can train a network by repeating two training strategies until convergence. First, $V_n$ values are obtained by calculating $\theta$ with fitness function $f$. Then the adaptive search algorithm calculates $E(R_\theta)$ by the average value of the worst $K$ samples among the $V_n$ values.

In addition, two benchmark Mujoco RL tasks are used to test the performance of the proposed algorithm. One important motivation is that the RL tasks are more related to the biological agent's learning procedure, which makes decisions after observation by maximizing its predicted rewards.

# 4 Experiments

## 4.1 The clustered brain regions

The 213 brain regions were clustered into multiple graphs, and parts of critical brain regions shared by different communities were recorded in Fig. 1B. It represents a sparse connection between different brain regions, congruent to the biological brain.

The community-61, community-65, and community-71 in 71 clusters were selected for subsequent experiments, given the cutting height of 0.8. It is impressive that the number of nodes in the largest connectedness in all clusters is less than 60 (Fig. 1E), indicating the network is very sparse. The community-61 contains 46 brain region nodes, which is called Circuit-46 (which means the clustering containing 46 brain regions). Similar to it, we give the names of community-65 and community-71 as Circuit-49 and Circuit-31, respectively. These circuits obtained after clustering will be used for the next-step RL experiments.

## 4.2 The network topology from biological mouse brain

The Circuit-46 is visualized in a standard 3D mouse brain common coordinate framework (CCF) from the Allen Brain Institute (Fig. 1F), where each point represents a brain region, and each link represents excitatory (blue ones) or inhibitory (red ones) connections between different regions (the total is 819 connections). We set 657 excitatory, and 162 inhibitory connections for Circuit-46, inspired by the biological discovery [8]. The connectivity strength lower than 0.05 was omitted and only left for those larger than the threshold for the ease of visualization (i.e., only 207 excitatory connections and 60 inhibitory connections were left after filtering).

In addition, the sensory, hidden, and motor brain regions are visualized in a 2D figure, including 8 sensory areas (red inverted triangles), 36 hidden areas (blue circles), and 2 motor areas (yellow triangles), as shown that in Fig. 1G.
## 4.3 Results with Circuit-46 and random networks

We conducted an experiment where random connections were given to a MT-ENN with the same number of neurons and synapses as those in the Circuit-46. The synapses' initial polarity was set randomly (excitatory or inhibitory). Only feedforward connections were given from sensory to hidden neurons, same as those in the hidden to motor neurons. The random circuits were then trained on RL tasks whose performances are reported in Fig. 2A,D. We observed that the performance of MT-ENN using mouse brain topology was higher than that using random connections, and the performance would be more significant for a simple (e.g., MountainCar-v2) instead of hard (e.g., Half-cheetah-v2) tasks. This result on two benchmark RL tasks demonstrates the usefulness of the biological structures.

<figure><img src="image_5.png" /><figcaption>**Fig. 2.** Performance of different algorithms on reinforcement learning tasks. (A,D) Performance of the MT-ENN using Circuit-46 and random structures on the MountainCar-v2 and Half-cheetah-v2 tasks, respectively. (B,E) Same to those in (A,D) but for the MT-ENNs using Circuit-31, Circuit-46, and Circuit-49. (C,F) Same to those in (A,D) but for the MT-ENNs using Circuit-46, random connections, MLP, and LSTM.</figcaption></figure>

## 4.4 Result comparison with different algorithms

We conducted a series of experiments to test the influence of the number of brain regions, including MT-ENNs using Circuit-31, Circuit-46, and Circuit-49, respectively. As shown in Fig. 2B,E, the MT-ENN using Circuit-46 is better than those using the other two structures. This result confirmed our hypothesis that Circuit-46 could be a winning ticket, containing better performance and lower sparsity. Hence, we selected Circuit-46 as MT-ENN's main biological structure basis for the subsequent experiments.
We further tested the performance of our MT-ENN with other SOTA algorithms, including long short-term memory (LSTM) and multi-layer perception (MLP). We keep the comparisons fair by using the same number of neurons, linear mapping functions, and learning algorithms. Other key parameters were also kept learnable. We chose the same number of cells (neurons) for the LSTM and MLP networks, equal to the size of the MT-ENN. The LSTM and MLP networks are fully connected, and the MT-ENN achieves 61% network sparsity. The result is shown in Table 1 and Fig. 2C,F.

As shown in Table 1, for the MountainCar-v2 task, our MT-ENN reached the score (mean reward) of 99.14, higher than other algorithms tuned with backpropagation. Given a more complex task, e.g., the Half-cheetah-v2 task, our MT-ENN reached a much higher performance (2,468 scores) than the state-of-the-art MLP (1,601 scores) and LSTM (1,009 scores). These results showed the efficiency of the biologically plausible brain circuits.

<table><caption>**Table 1.** The performance comparisons of the proposed MT-ENN with other SOTA algorithms on MountainCar-v2 and Half-cheetah-v2 tasks.</caption><thead><tr><th>Tasks</th><th>Architectures</th><th>Learning rules</th><th>Performance</th><th>Sparsity</th></tr></thead><tbody><tr><td rowspan="4">MountainCar-v2</td><td>LSTM [4]</td><td>BPTT</td><td>98.98±0.59</td><td>0%</td></tr><tr><td>MLP [7]</td><td>PPO</td><td>95.5±1.5</td><td>0%</td></tr><tr><td>Random</td><td>Search</td><td>49.59±49.59</td><td>61%</td></tr><tr><td><strong>MT-ENN (Ours)</strong></td><td><strong>Search</strong></td><td><strong>99.14±0.12</strong></td><td><strong>61%</strong></td></tr><tr><td rowspan="4">Half-cheetah-v2</td><td>LTSM [4]</td><td>BPTT</td><td>1009.51±641.95</td><td>0%</td></tr><tr><td>MLP [7]</td><td>PPO</td><td>1601.05±506.50</td><td>0%</td></tr><tr><td>Random</td><td>Search</td><td>1917.40±819.39</td><td>61%</td></tr><tr><td><strong>MT-ENN (Ours)</strong></td><td><strong>Search</strong></td><td><strong>2468.18±962.36</strong></td><td><strong>61%</strong></td></tr></tbody></table>

# 5 Discussion

Selectively copying biological structures into artificial neural networks is a short-cut for efficiently designing neural networks. In this paper, 213 mouse brain regions were clustered and analyzed to generate some sub-graph topology for the network design of MT-ENN. The 3D morphology helps us learn more about the neuron types [11], sparseness, and connectome of different brain regions during analysis. The comparisons of different sub-graph topologies will be extended for further discussion [10]. Combined with the biological understandings, these clustered results will help us select more topology that satisfies biological plausibility and efficiency. The experimental results have verified our hypothesis that mouse brain topology can improve evolutionary neural networks for efficient RL. In this paper, the brain region structure obtained by the joint action of mouse brain clustering and biological trust is better than the commonly used LSTM and MLP in RL tasks. In addition, the mouse brain structure also showed advantages in terms of sparsity. In the future, more biologically credible principles can
be borrowed from biological networks and applied to neural networks to achieve better integration of neuroscience and artificial intelligence and promote each other.

**Acknowledgements** This work was supported in part by the Shanghai Municipal Science and Technology Major Project (2021SHZDZX), the Youth Innovation Promotion Association CAS, and the National Natural Science Foundation of China under Grants No. 82171992 and 81871394.

## References

1. Doya, K.: Reinforcement learning in continuous time and space. Neural computation 12(1), 219-245 (2000)

2. Hasani, R., Lechner, M., Amini, A., Rus, D., Grosu, R.: Liquid time-constant networks. arXiv preprint arXiv:2006.04439 (2020)

3. Hasani, R., Lechner, M., Amini, A., Rus, D., Grosu, R.: A natural lottery ticket winner: Reinforcement learning with ordinary neural circuits. In: International Conference on Machine Learning. pp. 4082-4093. PMLR (2020)

4. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735-1780 (1997)

5. Oh, S.W., Harris, J.A., Ng, L., Winslow, B., Cain, N., Mihalas, S., Wang, Q., Lau, C., Kuan, L., Henry, A.M., et al.: A mesoscale connectome of the mouse brain. Nature 508(7495), 207-214 (2014)

6. Rubinov, M., Ypma, R.J., Watson, C., Bullmore, E.T.: Wiring cost and topological participation of the mouse brain connectome. Proceedings of the National Academy of Sciences 112(32), 10032-10037 (2015)

7. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

8. Wildenberg, G.A., Rosen, M.R., Lundell, J., Paukner, D., Freedman, D.J., Kasthuri, N.: Primate neuronal connections are sparse in cortex as compared to mouse. Cell Reports 36(11), 109709 (2021)

9. Zhang, D., Zhang, T., Jia, S., Xu, B.: Multiscale dynamic coding improved spiking actor network for reinforcement learning. In: Thirty-Sixth AAAI Conference on Artificial Intelligence (2022)

10. Zhang, T., Zeng, Y., Xu, B.: A computational approach towards the microscale mouse brain connectome from the mesoscale. J Integr Neurosci 16(3), 291-306 (2017)

11. Zhang, T., Zeng, Y., Zhang, Y., Zhang, X., Shi, M., Tang, L., Zhang, D., Xu, B.: Neuron type classification in rat brain based on integrative convolutional and tree-based recurrent neural networks. Sci Rep 11(1), 7291 (2021)