Self-Organization of Action Hierarchy and Compositionality
by Reinforcement Learning with Recurrent Neural Networks
1Cognitive Neurorobotics Research Unit
2Neural Computation Unit
Okinawa Institute of Science and Technology, Japan

Dongqi Han1, Kenji Doya2, Jun Tani1
dongqi.han@oist.jp

Introduction

The proposed RL framework: ReMASTER

A Hierarchy ofTimescales in Primate Cortex.

Recurrent Multi-level Actor-critic with STochasctic Experience Reply

Murray, John D., et al.
"A hierarchy of intrinsic timescales acrossprimate cortex."Nature neuroscience
17.12 (2014): 1661.

We are interested in studying the nerual mechnisms regarding the following questions:- How action primitives (options) that are useful for hierarchical control autonomously develop by RL?
- How the action primitives learned in a task can be re-used to facilitate learning novel tasks?
We seek the answers from the biological neural systems:- The cortex is full of recurrently connected neurons with highly stochastic firing behaviors.
- The primate cortex has a hierarchy of distinct intrinsic timescales, the higher the slower.
We propose ReMASTER, an RL framework in which useful actions primitives can self-organize:- Multiple levels of RNNs with hierachical timescales, the higher the slower. (2 levels in this work)- Stochasticity not only in the action outputs, but also in neuronal dynamics of all units.
- Only the lowest level receive observations and output actions, but each level learns a value function.

The Sequential Target-Reaching Task

The Consecutive Relearning Task

A two-wheeled robot agent (simulated) on a 2-dimensional
Large reward field is required to reach three target positions, in a seSmall reward
quence, red-green-blue, without receiving any signal indi3
cating which target to reach. The action is given by the rota1
1tion of its left and right wheels. At the beginning of each episode, the robot and targets are randomly placed on the field.
The robot has sensors detecting distances and angles from2
the three targets, as well as the current-step reward. When
Medium reward
it reaches a target in the correct sequence, it receives aone-step reward. The reward is given only if the agent followed the proper sequence, and is given only once for eachtarget. An episode terminates if the agent completes thetask or a maximum of 128 steps are taken. To successfully solve the task, the agent needs to develop
the cognitive capability to remember ``which target has been reached'', as well as to recognize thecorrect sub-goal (which can be considered as approaching a target in this task).
1

0.5

0.5

0.5

000

-0.5

-0.5

-0.5

-1-1

-1

PC1

1

1

1

low level

0.5

0.5

0.5

000-0.5

-0.5

-0.5

-1

-1-1

PC2

ReachGreenReachRed

10.50.50.5000-0.5-0.5-0.5-1-1-1510 15 20 25
time step

5Trajectory

1510 15
time step

10105555101500low level1515100010 15 20 25
time step

51015005100

2000

4000

(d)

Phase 21000

Phase 35001000
2000
episode6000
episode

8000

10000

(e)
10001000
2000
episodePhase 3 - LF

50001000
2000
episode(a) Illustration of the task. (b-d) Performance curves for all phases. ReMASTER-single V. is the case inwhich the higher level did not learn a corresponding value function. ReMASTER-det. stands for thecase in which all the neurons followed deterministic dynamics, and LSTM is the alternative using thesame algorithm but the network was a single-layer LSTM. (e) Performance curve of phase 3 with thelower-level synaptic weights frozen (Phase 3 - LF).

(c) high level
520ReMASTERReMASTER-single V
ReMASTER-det.
LSTM

500Start

Left Wheel Speed

MotorAction

1

success rate
100PC1150(c)(End)
ReachBlue

Blue->Green->RedPhase 1

success rate

12Green->Blue->Red

1000312Red->Green->Bluehigh level

1

12(b)

3

success rate(b)Episode 11900

31success rate

Episode 11800

PC2

Low-level
RNN output

High-level
RNN output(a)Episode 11700(a)Random Phase 1
Inheriting
Inheriting
Phase 2Phase 3weights 12,000 episodes weights 3,000 episodes weights 3,000 episodes

50-1-1

Manipulating
agent behaviors
by clamping
high-level RNN
states. All trajectories werefrom one agent
and each row
used samehigh-level RNN
states. Blacksquares and colored circles indicate the agent's
initial positions
and the target
positions, respectively.

510Right Wheel Speed

115(a) Three example episodes showing the behavior of a well-trained ReMASTER agent. The first 2 rows show
RNN output of two levels, where the vertical, dashed lines indicate the agent's reaching a target. The motor actions indicted by velocities of the two wheels are plotted in the third row. The fourth row is the robot's trajectories, where black squares indicate its starting positions and circles are target positions. (b) PCA for RNN outputs, using data from different episodes of one agent after convergence. Colors mean the agent is approaching
to the corresponding targets, where a deeper color means the agent is more closed to the target. Samples fromthe same episode are linked with black lines. (c) Similar to (b), but the colors indicate speed of the two wheels(see the colormap). The first 3 PCs were plotted.

Details
Importancesampleingratio

TD-error of
the lth level

Multiple Timescale Stochastic RNN (MTSRNN)

https://doi.org/10.1016/j.neunet.2020.06.002
GitHub.com/oist-cnru/ReMASTER

International Symposium on Artificial Intelligence and Brain ScienceTokyo, 10-12, October, 2020