Journal of Robotics, Networking and Artificial Life
Vol. 6(4); March (2020), pp. 221–224
DOI: https://doi.org/10.2991/jrnal.k.200222.003; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

A Promoting Method of Role Differentiation using aLearning Rate that has a Periodically Negative Value inMulti-agent Reinforcement Learning
Masato Nagayoshi1,*, Simon J. H. Elderton1, Hisashi Tamaki2
Department of Nursing, Niigata College of Nursing, 240 Shinnan-cho, Joetsu, Niigata 943-0147, Japan
Department of Computer Science and System Engineering, Kobe University, 1-1 Rokkodai-cho, Nada-ku, Kobe, Hyogo 657-8501, Japan12ARTICLE INFOABSTRACT

Article History

There have been many studies on Multi-Agent Reinforcement Learning (MARL) in which each autonomous agent obtains
its own control rule by Reinforcement Learning (RL). Here, we hypothesize that different agents having individuality is more
effective than uniform agents in terms of role differentiation in MARL. In this paper, we propose a promoting method of roledifferentiation using a wave-form changing parameter in MARL. Then we confirm the effectiveness of role differentiation by thelearning rate that has a periodically negative value through computational experiments.

Received 31 October 2019
Accepted 16 December 2019
Keywords
Reinforcement learning
multi-agent
negative learning raterole differentiation© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

2. Q-LEARNING

Engineers and researchers are paying more attention toReinforcement Learning (RL) [1] as a key technique for realizing computational intelligence such as adaptive and autonomous
decentralized systems. Recently, there have been many studies
on Multi-Agent Reinforcement Learning (MARL) in which eachautonomous agent obtains its own control rule by RL. Then, we
hypothesize that different agents having individuality is more effective than uniform agents in terms of role differentiation in MARL.
Here, we define “individuality” in this paper as being able to be
externally observed, but not a difference that we are incapable
observing, such as a difference of internal construction.

In this section, we introduce Q-learning (QL) [3] which is one of
the most popular RL methods. QL works by calculating the qualityof a state-action combination, namely the Q-value, that gives theexpected utility of performing a given action in a given state. By
performing an action a ∈AQ, where AQ ⊂ A is the set of available
actions in QL and A is the action space of the RL agent, the agent
can move from state to state. Each state provides the agent with areward r. The goal of the agent is to maximize its total reward.

We consider that differences in interpretations of experiences inthe early stages of learning have a great effect on the creation ofindividuality of autonomous agents. In order to produce differences in interpretations of the agents’ experiences, we utilized
Beck’s “Cognitive distortions” [2], which is a cognitive therapy.
In this paper, we propose a “fluctuation parameter” which is a waveform changing meta-parameter in order to realize “Disqualifyingthe positive” which is one of the “Cognitive distortions”, and apromoting method of role differentiation using the fluctuation
parameter in MARL.
We then confirm the effectiveness of role differentiation by introducing the fluctuation parameter into the learning rate, especiallyhaving a periodically negative value, through computational experiments using “Pursuit Game” as one of the multi-agent tasks.
Corresponding author. Email: nagayosi@niigata-cn.ac.jp*The Q-value is updated according to the following Equation (1),when the agent is provided with the reward:
Q(s(t − 1), a(t − 1)) ← Q(s(t − 1), a(t − 1))   + a r (t − 1) + g max Q(s(t ), b) − Q(s(t − 1), a(t − 1)) Q{b ∈AQ}

(1)

where Q(s(t – 1), a(t – 1)) is the Q-value for the state and the action
at the time step t − 1, aQ ∈[0,1] is the learning rate of QL, g   ∈[0,1]is the discount factor.
The agent selects an action according to the stochastic policy p (a|s),
which is based on the Q-value. p (a|s) specifies the probabilities oftaking each action a in each state s. Boltzmann selection, which isone of the typical action selection methods, is used in this research.
Therefore, the policy p (a|s) is calculated as Q( s , a) exp  t 		 p (a | s) = Q(s, b)  exp∑ b∈AQ  t 
where t is a positive parameter labeled temperature.

(2)

M. Nagayoshi et al. / Journal of Robotics, Networking and Artificial Life 6(4) 221–224

222

3. FLUCTUATION PARAMETER
Reinforcement learning has meta-parameters k to determine how
RL agents learn control rules. The meta-parameters k include thelearning rate a, the discount factor b, e of e-greedy which is one of
the action selection methods, and the temperature t of Boltzmann
action selection method.
In this paper, the following fluctuation parameter using damped
vibration function is introduced into this k.
  tp (t pa < t ps )
k + Acos  2p   + f 
 l


    k (t p ) = k + Acos  2p  t p  + f  × t ps (otherwise) l
 t pa

Figure 2 | Range of the agent view (indicated by grayed area).
Table 1 | Parameters for Q-learning(3)

where A, tp, tpa, tps, l and f is the amplitude, the phase, the damped
phase, the initial phase of damping, the wavelength, and the initial
phase parameter of the fluctuation, respectively. The phase tp, thedamped phase tpa, the initial phase of damping tps, and the wavelength l are needed to set proper units.

4. COMPUTATIONAL EXAMPLESParameter

Value

aQ
gt

0.1
0.9
0.1

reinforcement signal rt = 1 (sub reward) is given to the agent only
when the agent is located in the cell adjacent to the target objectand the reinforcement signal rt = 0 at any other steps. The period
from when all agents and the target object are randomly located at
the start point to when the target object is captured and all agentsare given a reward, or when 100,000 steps have passed is labeled
1 episode. The period is then repeated.

4.1. Pursuit GameThe effectiveness of the proposed approach is investigated in thissection. It is applied to the so-called “Pursuit Game” where three
RL agents move to capture a randomly moving target object ina discrete 10 × 10 globular grid space. Two or more agents or anagent and the target object cannot be located at the same cell. At
each step, all agents simultaneously take one of the five possible
actions: moving north, south, east, west or standing still. A target
object is captured when all agents are located in cells adjacent to thetarget object and surrounding the target object in three directions
as shown in Figure 1.
The agent has a field of view, and the depth of view set at 3 as shownin Figure 2. Therefore, the agent can observe the surrounding(3 × 2 + 1)2 − 1 cells. The agent determines the state by information
within the field of view.
The positive reinforcement signal rt = 10 (reward) is given to allagents only when the target object is captured, and the positive

Figure 1 | Capture positions.

4.2. RL Agents
All agents observe the only target object in order to confirm
the effectiveness of role differentiation, e.g. moving east of thetarget object. Therefore, the state space is constructed with a one-­dimensional space.
Computational experiments have been done with parameters asshown in Table 1. In addition, all initial Q-values are set at 5.0 as
the optimistic initial values.

4.3. Example (A): Same Amplitude
The effectiveness of role differentiation by introducing four fluctuation parameters, in which the initial phase f = 0, l = 500 [step],
and the amplitude A = {0.1, 0.12, 0.15, 0.17}, into the learning rateof QL (hereafter called “0.1”, “0.12”, “0.15”, and “0.17”, respectively)
are investigated in comparison with an ordinary QL without fluctuation parameter (hereafter called “constant”). Here, the fluctuation
parameters of all agents take the same value. The unit of the phase
tp is set [step] which is the same as the wavelength l, the unit of thedamped phase tpa is set [episode], and the initial phase of damping
is set at tps = 1000 [episode]. The range of values which the fluctuation parameter for aQ = 0.9 can take e.g. [0.0, 0.2], [−0.05, 0.25]
on the condition of A = 0.1 and 0.15, respectively. If the unit of thephase tp is set [episode] and the learning rate is negative, then thecontrol rule which the agent obtains is random. This is the resultof negative learning at each step. Therefore, the unit of the phase
tp is set [step].
The average numbers of steps required to capture the target object
were observed during learning over 20 simulations with various
amplitude parameters in the learning rate, as described in Figure 3.

M. Nagayoshi et al. / Journal of Robotics, Networking and Artificial Life 6(4) 221–224

223

The average numbers of steps required to capture the targetobject were observed during learning over 20 simulations
with various shifting size of amplitude in the learning rate, asdescribed in Figure 4.
It can be seen from Figure 4 that, (1) “0.05” shows a better performance than any other methods with regard to promoting roledifferentiation. (2) “0.02” shows a worse performance than “0.1”.
(3) “0.07” shows a worse performance than any other method.
Thus, the moderate shifting size of amplitude in the learning rate
among the agents shows a better performance than the in the caseof same amplitude in the learning rate among the agent.
Figure 3 | Required steps of various amplitude parameters in the learning rate.

5. CONCLUSION
In this paper, we proposed a “fluctuation parameter” which
is a wave-form changing meta-parameter in order to realize“Disqualifying the positive” which is one of the “Cognitive distortions”, and a promoting method of role differentiation using thefluctuation parameter in MARL.
Through computational experiment using the “Pursuit Game”, weconfirmed the effectiveness of role differentiation by introducing
the fluctuation parameter into the learning rate, especially having aperiodically negative value.

Figure 4 | Required steps of various shifting size of amplitude in thelearning rate.

It can be seen from Figure 3 that, (1) “0.12” shows a better performance than any other methods and with regard to promoting roledifferentiation. (2) “0.17” shows a worse performance than “constant” with regard to promoting role differentiation.

Our future projects include evaluating the effectiveness of promoting role differentiation using our proposed fluctuation
parameter in order to realize “Jumping to conclusions”, “Making“must” or “should” statements”, and “Overgeneralizing” with astate space filter [4].

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

Thus, the learning rate that has a periodically a negative value only
slightly shows a better performance than the learning rate that has
any non-negative value. It could be considered that this is the resultof preventing over-fitting by having periodically a negative value.

ACKNOWLEDGMENT

4.4. Example (B): Various Amplitude

REFERENCES

In this section, the effectiveness of role differentiation by introducing
four fluctuation parameters, in which the initial phase f = 0, l = 500[step], and the shifting size of the amplitude: ±0.0, ±0.02, ±0.05, ±0.07
around A = 0.1, into the learning rate of QL (hereafter called “0.0”,“0.02”, “0.05”, and “0.07”, respectively) are investigated. For example,
the amplitudes of three agents are 0.05, 0.1, 0.15 on the condition of
the shifting size of the amplitude: ±0.05 around A = 0.1. The same as
Example (A), the unit of the phase tp is set [step] which is same as thewavelength l, the unit of the damped phase tpa is set [episode], andthe initial phase of damping is set at tps = 1000 [episode].

[1]

This work was supported by JSPS KAKENHI Grant Number
JP19K04906.

[2]
[3]
[4]

R.S. Sutton, A.G. Barto, Reinforcement learning: An introduction,
A Bradford Book, MIT Press, Cambridge, 1998.
A.T. Beck, Cognitive therapy and the emotional disorders,International University Press, New York, 1976.
C.J.C.H. Watkins, P. Dayan, Technical note: Q-learning, Mach.
Learn. 8 (1992), 279–292.
M. Nagayoshi, H. Murao, H. Tamaki, A state space filter for reinforcement learning in POMIDPs: application to a continuous
state space, 2006 SICE-ICASE International Joint Conference,Busan, South Korea, IEEE, 2006, pp. 6037–6042.

M. Nagayoshi et al. / Journal of Robotics, Networking and Artificial Life 6(4) 221–224

224

AUTHORS INTRODUCTION
Dr. Masato Nagayoshi
He is an Associate Professor of Niigata
College of Nursing. He graduatedfrom Kobe University in 2002, andreceived Master of Engineering fromKobe University in 2004 and Doctor ofEngineering from Kobe University in2007. He is a IEEJ, SICE, ISCIE member.

Mr. Simon J. H. Elderton
He is an Associate Professor of Niigata
College of Nursing. He graduated fromUniversity of Auckland with an Honours
Masters degree in Teaching English toSpeakers of Other Languages in 2010. Heis a JALT, Jpn. Soc. Genet. Nurs., JACC
member.

Dr. Hisashi Tamaki
He is a Professor of Graduate School ofEngineering, Kobe University. He graduated from Kyoto University in 1985,and received Master of Engineering fromKyoto University in 1987 and Doctorof Engineering from Kyoto Universityin 1993. He is a ISCIE, IEEJ, SICE, ISIJ
member.