Journal of Robotics, Networking and Artificial Life
Vol. 8(1); June (2021), pp. 6–9
DOI: https://doi.org/10.2991/jrnal.k.210521.002; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

A Multi-agent Reinforcement Learning Method forRole Differentiation Using State Space Filters with
Fluctuation Parameters
Masato Nagayoshi1,*, Simon J. H. Elderton1, Hisashi Tamaki2
Niigata College of Nursing, 240 shinnan-cho, Joetsu, Niigata 943-0147, Japan
Kobe University, 1-1 Rokkodai-cho, Nada-ku, Kobe, Hyogo 657-8501, Japan12

ARTICLE INFOABSTRACT

Article History

Recently, there have been many studies on Multi-agent Reinforcement Learning (MARL) in which each autonomous agent obtains
its own control rule by RL. Here, we hypothesize that different agents having individuality is more effective than uniform agents interms of role differentiation in MARL. We have previously proposed a promoting method of role differentiation using a waveformchanging parameter in MARL. In this paper, we confirm the effectiveness of role differentiation by introducing the waveformchanging parameter into a state space filter through computational examples using “Pursuit Game” as a multi-agent task.

Received 24 November 2020
Accepted 11 March 2021
Keywords
Reinforcement learningrole differentiation
meta-parameter
waveform changing
state space filter© 2021 The Authors. Published by Atlantis Press B.V.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
Engineers and researchers are paying more attention toReinforcement Learning (RL) [1] as a key technique for realizing computational intelligence such as adaptive and autonomous
decentralized systems. Recently, there have been many studies
on Multi-agent Reinforcement Learning (MARL) in which eachautonomous agent obtains its own control rule by RL. We hypothesize that different agents having individuality is more effective thanuniform agents in terms of role differentiation in MARL. Here, we
define “individuality” in this paper as being able to be externallyobserved, but not a difference that we are incapable observing, suchas a difference of internal construction.
We have been exploring the assertion that differences in interpretations of experiences in the early stages of learning have a great
effect on the creation of individuality of autonomous agents. To
produce differences in interpretations of the agents’ experiences,
we have utilized Beck’s “Cognitive distortions” [2], which is a formof cognitive therapy.
We then proposed a “fluctuation parameter” which is a waveform changing meta-parameter to realize “Disqualifying the positive” which is one of the “Cognitive distortions”, and a promoting
method of role differentiation using the fluctuation parameter inMARL [3].
In this paper, we introduce the “fluctuation parameter” into a state
space filter [4] to realize “Overgeneralizing” which is one of theCorresponding author. Email: nagayosi@niigata-cn.ac.jp*“Cognitive distortions”, and confirm the effectiveness of role differentiation by introducing the fluctuation parameter into the state
space filter through computational examples using “Pursuit Game”
as a multi-agent task.

2. Q-LEARNING
In this section, we introduce Q-Learning (QL) [5] which is one of
the most popular RL methods. QL works by calculating the qualityof a state-action combination, namely the Q-value, that gives theexpected utility of performing a given action in a given state. By
performing an action a ∈ AQ, where AQ ⊂ A is the set of available
actions in QL and A is the action space of the RL agent, the agent
can move from state to state. Each state provides the agent with areward r. The goal of the agent is to maximize its total reward.
The Q-value is updated according to the following Equation (1),when the agent is provided with the reward:
Q(s(t − 1), a(t − 1))← Q(s(t − 1), a(t − 1)) + a Q {r (t − 1) (1)+ g max Q(s(t ), b) − Q(s(t − 1), a(t − 1))}b ∈AQ

where Q(s(t – 1), a(t – 1)) is the Q-value for the state and the action
at the time step t – 1, aQ ∈ [0,1] is the learning rate of QL, γ ∈ [0,1]is the discount factor.
The agent selects an action according to the stochastic policy p (a|s),
which is based on the Q-value. p (a|s) specifies the probabilities oftaking each action a in each state s. Boltzmann selection, which isM. Nagayoshi et al. / Journal of Robotics, Networking and Artificial Life 8(1) 6–9

one of the typical action selection methods, is used in this research.
Therefore, the policy p (a|s) is calculated as Q( s , a) exp  t 	  p (a | s) =(2) Q(s, b) exp
∑ b∈AQ  t 
where t is a positive parameter labeled temperature. Here, hightemperatures cause random action. Low temperatures cause greedyaction.

3. STATE SPACE FILTER FOR
REINFORCEMENT LEARNING
We have proposed a state space filter based on the entropy which isdefined by action selection probability distributions in a state [4].
The entropy of action selection probability distributions using
Boltzmann selection in a state H(s) is defined by
H (s) = - (1 / log | A |) åp (a | s)log p (a | s) (3)aÎA

where p (a|s) specifies probabilities of taking each action a in eachstate s, A is the action space and |A| is the number of availableactions.

7

where A, tp, tpa, tps, l and ϕ is the amplitude, the phase, the dampedphase, the initial phase of damping, the wavelength, and the initial
phase parameter of the fluctuation, respectively. The phase tp, thedamped phase tpa, the initial phase of damping tps, and the wavelength l are needed to set proper units.

5. COMPUTATIONAL EXAMPLES
5.1. Pursuit GameThe effectiveness of the proposed approach is investigated in thissection. It is applied to the so-called “Pursuit Game” where three
RL agents move to capture a randomly moving target object in adiscrete 10 × 10 globular grid space. Two or more agents or anagent and the target object cannot be located at the same cell. At
each step, all agents simultaneously take one of the five possible
actions: moving north, south, east, west or standing still. A target
object is captured when all agents are located in cells adjacentto the target object and surrounding the target object in three
directions.
The agent has a field of view, and the depth of view is set at 3.
Therefore, the agent can observe the surrounding (3 × 2 + 1)2 − 1cells. The agent determines the state by information within the fieldof view.

The state space filter is adjusted by treating this entropy H(s) as anindex of a correctness of state aggregation in the state s. In particular, in case of mapping the input state space roughly to the state
space, a perceptual aliasing problem can occur. That is, the action
which an agent should select cannot be identified clearly. Thus, theentropy may not be small enough in a state and state space should
be divided. In this paper, sufficiency of the number of learning
opportunities is judged using a threshold value q L.

The positive reinforcement signal rt = 10 (reward) is given to allagents only when the target object is captured, and the positive
reinforcement signal rt = 1 (sub reward) is given to the agent only
when the agent is located in the cell adjacent to the target object,
and the reinforcement signal rt = 0 is given at all other steps. The
period from when all agents and the target object are randomly
located at the start point to when the target object is captured andall agents are given a reward, or when 100,000 steps have passed islabeled 1 episode. The period is then repeated.

Therefore, due to a perceptual aliasing problem having occurred, if
the entropy does not get smaller than a threshold value q H despite
the number of learning opportunities being sufficient, the state
space filter is adjusted by dividing the state.

5.2. Setting of Reinforcement
Learning Agents

Similarly, due to the states being too divided, if the entropy issmaller than q H in a state s and a different state mapping from atransited input state s′, and representative actions in each other’sstates are the same, the state space filter is adjusted by integrating
the states.

4. FLUCTUATION PARAMETER
Reinforcement learning has meta-parameters κ to determine how
RL agents learn control rules. The meta-parameters κ include thelearning rate a , the discount factor b, e of e -greedy which is one of
the action selection methods, and the temperature t of Boltzmann
action selection method.
In this paper, the following fluctuation parameter using damped
vibration function is introduced into this κ.
k (t p ) =k + A cos(2p (t p / l ) + f )(t pa < t ps )   k + A cos(2p (t / l ) + f ) × t / t (otherwise) (4)ppspa

All agents observe the target object only to confirm the effectiveness of role differentiation, e.g. moving east of the target object.
Therefore, the state space is constructed with a 1-dimensional
space.
Computational examples have been done with parameters asshown in Table 1. In addition, all initial Q-values are set at 5.0 as
the optimistic initial values, and qH was set referring to about 0.288:
the maximal value of the entropy when the highest selection probability for one action is 0.9.
Table 1 | Parameter setting ofQ-learning with a state space filter
Parameter

ValueaQgt
qHqL

0.1
0.9
0.1
0.3
1000

8

M. Nagayoshi et al. / Journal of Robotics, Networking and Artificial Life 8(1) 6–95.3. Example (A): Introducing the Fluctuation
Parameters into the Learning Rate
The effectiveness of role differentiation by introducing the threefluctuation parameters, in which the initial phase ϕ = 0, the amplitude A = 0.09, and the wavelength l = {50, 100, 500}[episode], into
the learning rate of QL with the state space filter (hereafter called“50”, “100”, and “500”, respectively) is investigated in comparison
with an ordinary QL with the state space filter without fluctuation parameter (hereafter called “constant”). Here, the fluctuation
parameters of all agents take the same value. The unit of the phase
tp is set [episode] which is the same as the wavelength l, the unitof the damped phase tpa is set [episode], and the initial phase ofdamping is set at tps = 250 [episode]. The range of values which thefluctuation parameter for αQ = 0.1 can take e.g. [0.01, 0.19] on thecondition of A = 0.09.
The average numbers of steps and the average size of the state spacerequired to capture the target object were observed during learning over 20 simulations with various wavelength parameters in thelearning rate, as described in Figures 1 and 2, respectively.
It can be seen from Figures 1 and 2 that, (1) “50”, “100” and “500”show a better performance than “constant” with regard to theobtained control rule, (2) “50”, “100” and “500” are smaller than“constant” with regard to the size of the state space.

5.4. Example (B): Introducing the Fluctuation
Parameters into the Temperature
The effectiveness of role differentiation by introducing the threefluctuation parameters, in which the initial phase ϕ = 0, theamplitude A = 0.09, and the wavelength l = {50, 100, 500}[episode],
into the temperature of Boltzmann action selection method (hereafter called “50”, “100”, and “500”, respectively) is investigated incomparison with an ordinary QL with the state space filter without
fluctuation parameter (hereafter called “constant”). Here, the fluctuation parameters of all agents take the same value. The unit of thephase tp is set [episode] which is the same as the wavelength l, theunit of the damped phase tpa is set [episode], and the initial phaseof damping is set at tps = 250 [episode]. The range of values which
the fluctuation parameter for t = 0.1 can take e.g. [0.01, 0.19] on
the condition of A = 0.09. If the temperature is zero, then action
selection of the agent is greedy and situations where agents cannot
capture the target object occur. Therefore, A is set at 0.09.
The average numbers of steps and the average size of the state spacerequired to capture the target object were observed during learning over 20 simulations with various wavelength parameters in thetemperature, as described in Figures 3 and 4, respectively.
It can be seen from Figures 3 and 4 that, (1) “50”, “100” and “500”show a better performance than “constant” with regard to theobtained control rule, (2) “50”, “100” and “500” are smaller than“constant” with regard to the size of the state space.
An example of obtained state space filters where wavelength l = 50
[episode] in 1000 episode is described in Figure 5. It can be seenfrom Figure 5 that each agent divided different respective area into
smaller parts.
Thus, the effectiveness of role differentiation by introducing thefluctuation parameter into the state space filter is confirmed. We
deem the effectiveness of role differentiation to be the result of“Overgeneralizing”.

Figure 1 | The average number of steps by various wavelength parametersin the learning rate (ϕ = 0 [rad]).

Figure 3 | The average number of steps by various wavelength parametersin the temperature (ϕ = 0 [rad]).

Figure 2 | The average size of the state space by various wavelengthparameters in the learning rate (ϕ = 0 [rad]).

Figure 4 | The average size of the state space by various wavelengthparameters in the temperature (ϕ = 0 [rad]).

M. Nagayoshi et al. / Journal of Robotics, Networking and Artificial Life 8(1) 6–9

9CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI Grant Number
JP19K04906.
Figure 5 | An example of obtained state space filters where wavelengthl = 50 [episode] in 1000 episode.

6. CONCLUSION
In this paper, we introduced a “fluctuation parameter” into a state
space filter in order to realize “Overgeneralizing” which is one of the“Cognitive distortions”. Through computational examples, we confirmed the effectiveness of role differentiation by introducing thefluctuation parameter into the state space filter. The effectiveness ofrole differentiation is deemed to be a result of “Overgeneralizing”.
Our future projects include applying the proposed method to realworld problems, such as improving work allocation tables producedby head nurses. The use of this model may assist head nurses in defining rules for making work allocation tables, and taking account into
the compatibility of different nurses and evolving working relationshipbetween nurses, value-functions which are difficult to set beforehand.

REFERENCES
[1]
[2]
[3]

[4]

[5]

R.S. Sutton, A.G. Barto, Reinforcement learning: an introduction,MIT Press, Cambridge, MA, 1998.
A.T. Beck, Cognitive therapy and the emotional disorders,International Universities Press, New York, 1976.
M. Nagayoshi, S.J.H. Elderton, H. Tamaki, A promoting methodof role differentiation using a learning rate that has a periodicallynegative value in multi-agent reinforcement learning, J. Robot.
Netw. Artif. Life 6 (2020), 221–224.
M. Nagayoshi, H. Murao, H. Tamaki, A state space filter for reinforcement learning in POMIDPs - application to a continuous
state space, 2006 SICE-ICASE International Joint Conference,IEEE, Busan, South Korea, 2006, pp. 6037–6042.
C.J.C.H. Watkins, P. Dayan, Technical note: Q-learning, Mach.
Learn. 8 (1992), 279–292.

AUTHORS INTRODUCTION
Dr. Masato Nagayoshi
He is an Associate Professor of Niigata
College of Nursing. He graduatedfrom Kobe University in 2002, andreceived Master of Engineering fromKobe University in 2004 and Doctor ofEngineering from Kobe University in2007. He is a member of IEEJ, SICE, andISCIE.

Mr. Simon J. H. Elderton
He is an Associate Professor of Niigata
College of Nursing. He graduated fromUniversity of Auckland with an Honours
Masters degree in Teaching English toSpeakers of Other Languages in 2010.
He is a member of JALT, Jpn. Soc. Genet.
Nurs., and JACC.

Dr. Hisashi Tamaki
He is a Professor of Graduate School ofEngineering, Kobe University. He graduated from Kyoto University in 1985,
and received Master of Engineering fromKyoto University in 1987 and Doctorof Engineering from Kyoto University
in 1993. He is a member of ISCIE, IEEJ,SICE, and ISIJ.