(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

Cooperative Multi-Robot Hierarchical Reinforcement
Learning
Gembong Edhi Setyawan1, Pitoyo Hartono2, Hideyuki Sawada3
Department of Applied Physics, School of Advanced Science and Engineering1, 3
Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, Japan1, 3
School of Engineering, Chukyo University2
101-2 Yagoto Honmachi, Showa-ku, Nagoya, Aichi, 466-8666 Japan2

Abstract—Recent advances in multi-robot deep reinforcement
learning have made it possible to perform efficient exploration in
problem space, but it remains a significant challenge in many
complex domains. To alleviate this problem, a hierarchicalapproach has been designed in which agents can operate at many
levels to complete tasks more efficiently. This paper proposes anovel technique called Multi-Agent Hierarchical Deep
Deterministic Policy Gradient that combines the benefits ofmultiple robot systems with the hierarchical system used in Deep
Reinforcement Learning. Here, agents acquire the ability to
decompose a problem into simpler subproblems with varying
time scales. Furthermore, this study develops a framework toformulate tasks into multiple levels. The upper levels function tolearn policies for defining lower levels’ subgoals, whereas thelowest level depicts robot’s learning policies for primitive actionsin the real environment. The proposed method is implemented
and validated in a modified Multiple Particle Environment(MPE) scenario.
Keywords—Multi-robot
system;hierarchicalreinforcement learning; path-finding; task decomposition

I.

deep

INTRODUCTION

Multi-Robot System (MRS) research has attracted
significant attention recently due to its advantages over single
robots. The benefits include (1) the decrease in time and theimprovement in problem-solving efficiency due to the task
decomposition, (2) an increase in problem-solving reliability,
robustness, and resiliencies as the failures of single robots canbe [1]–[3]. Using MRS, numerous prospective applications
have been developed in which robots must be able to compete
and cooperate, such as formation coordination [4], hide andseek [5], exploration and search [6]–[8], object transportation[9], disaster detection [10], communication networks [11],[12], etc. This study focuses on using MRS technology for
exploration and search missions in unknown environments,
where robots must collaborate to find the optimal path.
As shown in [13], [14], Reinforcement Learning (RL) is
currently extensively employed as a robot learning algorithm
that can automatically handle exploration and search problems
in unknown environments, both in simulation and physicalenvironments. Multiple robots undertake search andexploration operations in large and complex environments,
such as in [15]–[17]. The objective of MRS is to distributetasks between many robots to increase efficiency. Nowadays,
applying RL is an important and challenging subject in MRS,This work was supported by JSPS Grants-in-Aid for Scientific Researchon Innovative Areas (Research in a proposed research area) 18H05473 and
18H05895.

where robots must learn and adapt based on their individual
strategies and collective behavior. Multi-Agent Deep
Deterministic Policy Gradient (MADDPG) algorithm was
proposed as an advancement of RL that may be used for multirobot learning in which robots can collaborate to solve
problems in unknown environments [18].
RL in complex environments is always challenging, hence
it serves as the primary motivation for the proposed approach.
This paper proposes Multi-Agent Hierarchical Deep
Deterministic Policy Gradient (MH-DDPG) as an extension of
MADDPG for solving search and exploration problems withmany robots in finding the optimal path for each robot invarious environmental complexities. Here, hierarchicallearning is adopted to learn in complex environments
efficiently [13].
The primary contribution of this paper is as follows: (1)MH-DDPG presents a framework that enables multiple robots
to learn by sharing environmental information collectively. (2)MH-DDPG proposes a hierarchical learning strategy byassigning different abstraction levels to the problem space,
where higher abstraction learnings supply the learning subgoal
for the lower ones, which consequently execute automatic task
decomposition.
The structure of this article is as follows: Section
II examines the previous research and theories relevant to the
proposed method. In Section III, the technical context andtheory are explained. The proposed MH-DDPG is discussed indetail in Section IV. Sections V and VI present the conducted
experiments and the validation of the results. Finally, section
VII provides conclusions and potential future works.
II. RELATED WORKS
Reinforcement Learning (RL) [19] is increasingly used aslearning method to address complicated problems like games[20], [21], and robotics [22], [23]. Q-Learning [24], SARSA[25], and Temporal Differences (TD) [26] are RL algorithms
that are predominantly applied to single agents within theMarkov Decision Process (MDP) mathematical modelingframework [27].
Traditional RL faces various problems when attempting tosolve real-world problems. This issue is known as the "curse ofdimensionality", in which data grows exponentially, andcomputations become costly. Deep Reinforcement Learning(DRL) was introduced to solve this issue, for example, in [20].

35 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

DRL models complex functions using the advantages of Neural
Networks (NN), for example, Deep Q-network (DQN) [20],[28]. However, implementing DQN for robots with manydegrees of freedom in action or continuous action spaceremains challenging. Policy Gradient (PG) [29] has been
proposed to solve these issues; nevertheless, PG has its
challenges, mainly that it requires extensive training. An actorcritic algorithm [25] combining DQN and PG's benefits was
proposed to mitigate these problems. In the actor-critic
algorithm, there are two networks: the actor-network, which
employs a policy gradient to create optimal policies, and thecritic-network, which contains DQN to evaluate the actor's
policies. Deep Deterministic Policy Gradient (DDPG) [30] isan actor-critic [31] based algorithm that has been proposed tosolve problems in continuous or high-dimensional action
spaces.
Multi-Robot System (MRS) is currently attracting muchinterest because it can solve complex problems that areprohibitively difficult for single-agent systems. Althoughseveral attempts have been made to apply the prior algorithmin MRS, it is still not as successful as in single-agent systems.
The recent development of MRS operates within the stochastic(Markov) games framework to model mathematical decisionmaking [32]. The difficulty of developing MRS is that thepolicies of one robot will impact a non-stationary environment
generated by the policies of another agent. Recently, Multiagent Deep Deterministic Policy Gradient (MADDPG) [18]was presented as a solution to this issue. For each robot, there
is a DDPG in the MADDPG. The MADDPG is able to manage
the problems of competitive as well as collaborative multiagent systems.
One attempt to alleviate the learning difficulty is to design
a mechanism for hierarchical learning. Hierarchical learning
involves decomposing large and complex problems into moremanageable subproblems. However, most algorithms need to
fix the sub-problems that may hinder problem-solving
flexibility. The Hierarchical Reinforcement Learning (HRL)
approach of a single robot with discrete action has been
proposed in [33], [34]; however, the subgoals need to be
assigned manually, while [35] has been designed to be able tofind subgoals automatically. Hierarchical Deep Reinforcement
Learning (HDRL), [36], [37] proposed systems that canoperate with continuous robot action. Algorithms in [33]–[37]work well with a single robot but are unsuitable for multi-robot
implementation. Studies in [38], [39] presented a multi-agent
hierarchy system with DRL for learning subgoals/skills at
higher levels. However, the higher-level environment was
manually defined using these approaches.
It is known that MADDPG can handle collaborative multirobot system problems in a simple environment. However, analgorithm that can improve the learning performance of multirobot systems is required for more complex environments. In
this study, we propose MH-DDPG by developing MADDPG toperform under a hierarchical system. The proposed method is
expected to be able to automatically discover subgoals,
allowing it to perform better in a high-complexity environment.

III. TECHNICAL PRELIMINARIESThis section highlights the theoretical basis for developinga hierarchical MADDPG method for cooperative multi-agent
learning in handling complex problems.
A. Markov Decision Process and Reinforcement Learning
The decision-making process of a single robot is often
based on Markov Decision Process (MDP). In RL, at each time
step t, the robot perceives a state 𝑆𝑡 from the environment's setof states 𝒮 (𝑆𝑡 ∈ 𝒮) and the robot selects an action 𝐴𝑡 from theset of actions 𝒜 that may be executed in the state 𝑆𝑡 (𝐴𝑡 ∈ 𝒜).
Here, the next state is decided based on transition probability 𝒫
under the learned policy , as shown in (1). The transition
brings the robot to the next state 𝑆𝑡+1 and gives reward 𝑅𝑡+1 .
(𝑠′, 𝑟|𝑠, 𝑎) = 𝒫(𝑆𝑡+1 = 𝑠 ′ , 𝑅𝑡+1 = 𝑟|𝑆𝑡 = 𝑠, 𝐴𝑡 = 𝑎)(1)

where 𝑠 ∈ 𝒮 and 𝑠′ ∈ 𝒮 are particular states that occur attime t and t+1, 𝑎 ∈ 𝒜 is the action taken by the robot at time t(𝑎 ∈ 𝒜), and (𝑟 ∈ ℛ) is the reward received by the robot attime t+1.
Typically, the Bellman Equation is used to optimize twofunctions in RL: the state (V*) and state-action (Q*) functions,for which the optimal equation is as follows:𝑎∗ ′𝑉 ∗ (𝑠) = 𝑚𝑎𝑥 𝑅(𝑠, 𝑎) + 𝛾 ∑𝑠′𝜖𝑆 𝒫𝑠𝑠′ 𝑉 (𝑠 )(2)𝑎∗𝑄 ∗ (𝑠, 𝑎) = 𝑅(𝑠, 𝑎) + 𝛾 ∑𝑠′𝜖𝑆 𝒫𝑠𝑠′ 𝑄 (𝑠′, 𝑎′)(3)𝜋Here, the discount factor (0 ≤ 𝛾 ≤ 1) is used to express thesignificance of the future reward value.
B. Stochastic (Markov) Games
In contrast to MDP, which is utilized for a single robot,
stochastic games are proposed as a mathematical framework
for modeling decision-making in Multi-robot Reinforcement
Learning. Stochastic games consist of N robots, a set of states
containing the state of all robots (𝒮), a set of actions 𝒜 from all
robots (A = A1 x A2 x... x AN), and a set of state transitions (T)which are the transition probability (𝒫) from the current state
to the next state for a robot based on the actions taken by all
robots (T: S x A1 x A2 x... x AN  𝒫 (S)). The reward obtainedby a robot depends on the actions taken by all robots (R: S x A1
x A2 x... x AN  ℛ). The reward function for each robot can beused to classify the type of games. For example, all robots
share the same reward function if they play cooperatively. In
contrast, when the robots play competitively, one robot aims tomaximize the reward while the other attempts to minimize it.
In stochastic games, the state value function (V) could be
written as follows:𝑘𝑉𝑖,𝜋 (𝑠) = 𝔼[∑∞
𝑘=0 𝛾 𝑟𝑖,𝑡+𝑘+1 | 𝑆𝑖,𝑡 = 𝑠𝑖 ](4)

C. Q-Learning and Deep Q-Network
Q-Learning is a RL mechanism based on Q-function.
Similar to (3), the Q-function is used to compute the expected
reward based on the action done by the robot in its current
state. The optimum policy is found by maximizing the value ofthe Q-function. The Q-value is learned iteratively, as follows:

36 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

𝑄(𝑠, 𝑎) ← 𝑄(𝑠, 𝑎) + 𝛼 (𝑅 + 𝛾 (max 𝑄(𝑠 ′ , 𝑎′ ) − 𝑄(𝑠, 𝑎)))𝑎′(5)where 0    1 denotes the learning rate.
The Q-value for all potential state-action combinations isstored in a Q-table, and thus high computational resources are
required in a large number of states and large action space. The
necessity for these costly computation resources can bealleviated by replacing the Q-table with a neural network for
estimating the Q-value as in Deep Q-Network (DQN).

𝐽(𝜃) = 𝔼𝑠~𝜌𝜋 [𝛻𝜃 𝜋𝜃 (𝑠)𝛻𝜃 𝑄𝜋 (𝑠, 𝑎)|𝑎=𝜋𝜃 (𝑠) ](10)

Deep Deterministic Policy Gradient (DDPG) is an actorcritical algorithm that combines DQN and DPG. DQN is forthe actors that operate in discrete action space, while DPG isfor the critics that work in continuous action space.

𝐿(𝑄) = 𝔼𝑠,𝑎,𝑟,𝑠′ [(𝑅(𝑠, 𝑎) + 𝛾 max 𝑄 ∗ (𝑠 ′ , 𝑎 ′ |𝜃̅ ) − 𝑄(𝑠, 𝑎|𝜃)) ] (6)F. Multi-Agent Deep Deterministic Policy GradientMulti-Agent Deep Deterministic Policy Gradient(MADDPG) is an expansion of DDPG that adopt an actorcritic algorithm as its fundamental structure. The MADDPG
contains multiple robots, each with its neural networks for theactors and the critics, while DDPG only uses a single robot.
Similar to DDPG, the actors in MADDPG receive input fromthe robot's local observations and produce executable action
recommendations for the robot. However, in contrast to thecritical network in DDPG, the input of critics in MADDPG
does come from not only the robot's local observations andactions but also other robots' observations and actions. The
critic's output is the Q-value, which is used to evaluate theactor's actions by considering other robots' observations andacts. The network of agents may therefore learn bothcooperative and competitive strategies.

where 𝜃̅ and  are the parameters for the target network and
the Q-network.

IV. MULTI-HIERARCHIES OF MULTI-AGENT DEEP
DETERMINISTIC POLICY GRADIENT

D. Policy Gradient
Policy Gradient (PG) is used to enhance DQN'sperformance in generating optimum policies. In DQN, therobot chooses an action with the maximum Q-value, while in
PG, the agent selects an action stochastically according to theprobability distribution generated in the output layer.

This study proposes Multi-Agent Hierarchical Deep
Deterministic Policy Gradient (MH-DDPG) as a new approachthat enables learning robots to decompose complex tasks into
more manageable subtasks at different time scales. Here, therobots train to learn several levels of policy, each of which hasa specific task for the agents to do in parallel.

Q-Network and target Network are the two neural networks
that constitute a DQN. The Q-network is used to train robots topredict the optimal Q-value, while the target network is used toforecast the next state based on the sample data and the optimal
Q-value from all potential actions in the next state. In addition,
DQN has a component called Experience Replay (ER) that
stores and generates training data for Q-Network.
The optimal policy for DQN is determined by minimizing
the Loss function in (6).
2𝑎′The PG consists of a neural network known as a policy
network, which predicts the probability distribution of actions
given the current state. Here, the optimal policy is determinedby maximizing the objective function defined as follows:
𝐽(𝜃) = ∑𝑠∈𝑆 𝑑𝜋 (𝑠) ∑𝑎∈𝐴 𝜋𝜃 (𝑎, 𝑠)𝑄𝜋 (𝑠, 𝑎)(7)where 𝑑𝜋 (𝑠) is the deterministic distribution of the stateson . Here, the objective function 𝐽(𝜃) can be maximized byadjusting the parameter  by gradient ∇𝜃 𝐽(𝜃 as follows:∇𝜃 𝐽(𝜃) = 𝔼𝑠~𝑑𝜋 ,𝑎~𝜋𝜃 [∇𝜃 log 𝜋𝜃 (𝑎, 𝑠)𝑄𝜋 (𝑠, 𝑎)](8)

E. Deterministic Policy Gradient and Deep Deterministic
Policy Gradient
The policy function in PG is always modeled as astochastic probability distribution of the agent's actions given
the current state. The Deterministic Policy Gradient (DPG) has
been proposed to model policy as a deterministic decision by
the agent in the current state. The objective function in DPG
can be written as follows:
𝐽(𝜃) = 𝔼𝑠~𝜌𝜋 [𝑅(𝑠, 𝜋𝜃 (𝑠))](9)

A. Architecture
MH-DDPG trains robots to hierarchically learn policies
based on the architecture shown in Fig. 1. Here, MH-DDPG iscomprised of DDPG and experience replay (ER). The numberof DDPG and ER depends on the number of agents andhierarchy. For example, suppose N and K indicate the numberof agents and hierarchies, respectively. Consequently, there are
N x K DDPG in MH-DDPG. Furthermore, the number of ER
will equal the number of hierarchies, K. The bottom level
represents the physical environment in which the robots
physically operate. While the higher level, robots are presented
by their abstraction.
Formally, the MH-DDPG with N agents and K levels aredefined by the set of state 𝒮 ; the set of joint action 𝒜 =𝓀𝓀𝑁⋃𝑁
𝑖=1 𝐴𝑖,𝑡 and the set of joint observations 𝒪 = ⋃𝑖=1 𝑂𝑖 , where𝓀𝓀𝐴𝑖,𝑡 and 𝑂𝑖,𝑡 are actions 𝐴𝑖 and observation 𝑂𝑖 for each agent 𝑖
at level 𝓀 and time 𝑡. Each agent will optimize their respective
policy at every level to estimate the transition probability 𝒫 for𝓀selecting an action at time 𝑡, such that 𝜋 = ⋃𝑁𝑖=1 𝜋𝑖,𝑡 , where 𝓀is the hierarchy level and 0 ≤ 𝓀 ≤ 𝐾 − 1.

where  is discounted state distribution. The gradient of
the objective function in DPG can be written as follows:

37 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

Fig. 1. Architecture of MH-MADDPG.

Fig. 2. Illustration Environment with Two Hierarchies.

For illustrating the dynamics of MH-DDPG multi-agent
particle environment (MPE) environments will be utilized [40].
One of the MPE scenarios, "simple spread” has been modified
here. For example, Fig. 2 depicts the problem that MH-DDPG
must address. Simple_spread is an environment with N robots
and M goals (landmarks). Robots are expected to cooperate toaccomplish a common objective while avoiding collisions with
one another. There are three robots (N=3), three goals (M=3),
and two hierarchies (K=2). First, the real problem of theenvironment is illustrated at level 0, where a blue circle
represents the actual robots, and a green rectangle represents
the goals. Then, at level 1, an abstraction of level 0, the robots
are referred to as abstract robots and symbolized by a red
circle. Abstract robots at level 1 possess actions with morecapabilities than those of actual robots at level 0. For instance,
the actual robots at level 0 have a maximum velocity of 1 pixel
per second, while the robot at level 1 is set with a maximum
velocity of 10 pixels per second. As seen on the right of Fig. 2,
the robot at level 1 has a greater range of distances than theactual robot for each action taken at each step.

The abstract robots are predicted to learn faster than theactual robots in achieving goals since they are less constrained
than actual robots (for example, more quickly and with no
obstacles). However, remember that abstract robots are only
imaginative robots with no capacity to execute physicalactions. The task of the abstract robot at level 1 is to learn how
to accomplish the main goal best, while at level 0, the task is tolearn how to achieve the subgoal optimally. MH-DDPG
implicitly assigns different objectives for each level, in which
the robots' objective at level 1 is to learn to achieve goals
optimally, while the robots' work at level 0 is to learn toachieve subgoals optimally. The subgoal at level 0 is
automatically determined from the higher level, which happens
when the abstract robot chooses the action 𝐴1𝑖,𝑡 based on the11policy 𝜋𝑖,𝑡in the current state 𝑆𝑖,𝑡
at time 𝑡. The robot will be
1transitioned to the next state 𝑆𝑖,𝑡+1 and will receive a reward
1
𝑅𝑖,𝑡+1. Then, the learning shifts to the bottom level, and thenext state at level 0 becomes a subgoal for the actual robots. In
addition, robots at level 0 engage in learning to achieve these0
subgoals. When the robots get 𝑆𝑖,𝑡
at level 0, the agent will pick00
the action 𝐴𝑖,𝑡 based on the policy of 𝜋𝑖,𝑡
. The robots then00transition to 𝑆𝑖,𝑡+1 and is rewarded with 𝑅𝑖,𝑡+1. The learning
process at level 0 will continue until the terminal criteria aresatisfied. A terminal condition is defined by manually setting
the maximum number of steps at level 0. If the terminal
requirements are satisfied, learning returns to level 1 to executethe next step at the top level.
B. Learning Dynamic
In MH-DDPG, the multiple robots learn in parallel at alllevels. The process of robot learning will start at the top leveland flows downward. Robot learning aims to provide optimum
policies for each robot at all levels. According to the RL
concept that the optimal policy is acquired by maximizing the
rewards received by each robot. The reward obtained in the

38 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

future by each robot i at level 𝓀 could be expressed by thefollowing V-function:𝓀𝓀𝑡 𝓀(𝑠) = 𝔼[∑∞𝑉𝑖,𝜋𝑡=0 𝛾 𝑟𝑖,𝑡+1 | 𝑆𝑖,𝑡 = 𝑠]

(11)

𝑎
𝓀
𝓀
′
(𝑠) = ∑𝑎𝜖𝐴 𝜋𝑖𝓀 (𝑎|𝑠)(𝑅𝑖𝓀 (𝑠, 𝑎) + 𝛾 ∑𝑠′𝜖𝑆 𝒫𝑠𝑠
𝑉𝑖,𝜋′ 𝑉𝑖,𝜋 (𝑠 )) (12)𝓀where 𝑟𝑖,𝑡+1
is the reward earned by agent 𝑖 at level 𝓀 at𝓀𝑡 + 1, and 𝜋𝑖 (𝑎|𝑠) is the agent policy. Here, the joint action isdesigned to make the robot's policy dependent on individual
policies and joint policies. The definition of the Q-function is
as follows:
𝓀
𝓀
𝓀
𝑡 𝓀
(𝑠, 𝑎) = 𝔼[∑∞𝑄𝑖,𝜋𝑡=0 𝛾 𝑟𝑖,𝑡+1 | 𝑆𝑖,𝑡+1 = 𝑠, 𝐴𝑖,𝑡 = 𝑎 ]𝓀(𝑠, 𝑎) = 𝑅𝑖𝓀 (𝑠, 𝑎) +
𝑄𝑖,𝜋
𝑎
𝓀
𝓀
′ ′
𝛾 ∑𝑠′𝜖𝑆 𝒫𝑠𝑠′ ∑𝑎𝜖𝐴 𝜋𝑖 (𝑎 |𝑠 ) 𝑄𝑖,𝜋 (𝑠′, 𝑎′)

(13)

(14)

The optimal policy is determined by maximizing the valueof all actions. According to the Bellman optimality equation,
the optimal V-value (V*) and Q-value (Q*) could be written as
follows:∗∗𝑎𝓀
𝑉𝑖𝓀 (𝑠) = 𝑚𝑎𝑥 𝑅𝑖𝓀 (𝑠, 𝑎) + 𝛾 ∑𝑠′𝜖𝑆 𝒫𝑠𝑠(𝑠 ′ )
′ 𝑉𝑖

(15)

𝜋𝑖

Algorithm 1. MH-MADDPG
Initialize: Actor-critic evaluation and target networks for each agent,
number of levels K, maximum step H, Replay buffer: For episode = 1 to max-episode, do:
For each agent i, set initial states (S) and goals (𝒢) for each agent:
Train (K-1, S, 𝒢): End for:: Function Train(𝓀 ::level, S ::state, 𝒢 ::goal):𝓀𝑠 ← 𝑆𝑖,𝑡 𝑆, 𝑔 𝓀  𝒢 (initial, t=0)
9 :10For t = 1 to H do:𝓀𝓀
For each agent n: select action (ai) where 𝑎𝑖 ← 𝐴𝑖,𝑡based on 𝜋𝑖,𝑡:11
12Execute actions 𝑎 = (𝑎1 , … , 𝑎𝑁 ) and observe reward r and new
13 :state s’:Store (s, a, r, s’) in replay buffer 𝒟
14 :If 𝓀 > 0:
15 :𝑔 𝓀−1 ← 𝑠′
16:
17
𝑇𝑟𝑎𝑖𝑛(𝓀 − 1, 𝑠, 𝑔 𝓀−1 )18 :End If19 :For agent i = 1 to N in level 𝓀 do:
Sample random minibatch of S samples (s, a, r, s’) from 𝒟 𝓀𝓀
20 :(𝑠 ′ , 𝑎1′ , … , 𝑎𝑁′ )| ′ ′,𝓀Set 𝑦 = 𝑟𝑖𝓀 + 𝛾𝑄𝑖,𝜋′
1
2
3
4
5
6
7
8

𝑎𝑖 =𝜋𝑖21

If the environment consists of N agents and K levels, thenthe policy set 𝜋 = {𝜋1𝓀 , 𝜋2𝓀 , … , 𝜋𝑁𝓀 } that is parameterized by
𝜃 = {𝜃1𝓀 , 𝜃2𝓀 , … , 𝜃𝑁𝓀 } , where 1 ≤ 𝑖 ≤ 𝑁 and 0 ≤ 𝓀 ≤ 𝐾 − 1 .
Then, the gradient of the expected return for each agent 𝑖 at
level 𝓀 could be expressed as follows:

22 :
23
:
:

(17)

𝓀
(𝑠, 𝑎𝑖 , … , 𝑎𝑁 ))𝑄𝑖,𝜋2

Update actor using:∇𝜃 𝓀 𝐽 =𝑖1𝓀(𝑠, 𝑎𝑖 , … , 𝑎𝑁 )|∑ ∇𝜃𝓀 𝜋𝑖𝓀 (𝑜𝑖 )∇𝑎𝑖 𝑄𝑖,𝜋
𝑆
𝑖

24
:

𝐽(𝜃𝑖𝓀 ) = 𝔼𝑥,𝑎~𝐷 [𝑅 (𝑠, 𝜋𝑖𝓀 (𝑠))]Update critic by minimizing the loss ℒ(𝜃𝑖𝓀 ) = ∑ (𝑦 −𝑆(16)𝑎𝑖 =𝜋𝑖𝓀 (𝑜𝑖 )End forUpdate target network parameters for each agent i in level 𝓀:
𝜃𝑖′,𝓀 ← 𝛼𝜃𝑖𝓀 + (1 − 𝛼)𝜃𝑖′,𝓀∇𝜃 𝓀 𝐽(𝜃𝑖𝓀 ) = 𝔼𝑥,𝑎~𝐷

End Function𝑖𝓀[∇𝜃 𝓀 log 𝜋𝑖𝓀 (𝑎𝑖𝓀 |𝒪𝑖𝓀 )𝑄𝑖,𝜋(𝑥 𝓀 , 𝑎1𝓀 , … , 𝑎𝑁𝓀 )| 𝓀

𝑎𝑖 =𝜋𝑖𝓀 (𝒪𝑖𝓀 )

𝑖

]

(18)

𝓀
where 𝑄𝑖,𝜋(𝑥, 𝑎1𝓀 , … , 𝑎𝑁𝓀 ) is the centralized Q-function atlevel 𝓀 that accepts as input all agent actions at level 𝓀 ,𝑎1𝓀 , … , 𝑎𝑁𝓀 , and observation 𝑥 at level 𝓀 of all agents, 𝑥 =(𝒪1𝓀 , … , 𝒪𝑁𝓀 ), with the output being the Q-value for each agent
𝑖 at level 𝓀 . Experience Replay buffer 𝒟 contains(𝑥, 𝑥 ′ , 𝑎1𝓀 , … , 𝑎𝑁𝓀 , 𝑟1𝓀 , … , 𝑟𝑁𝓀 ) where 𝑥′ is the next state obtained
after the agent took action while in state 𝑥. The centralize Q𝓀function 𝑄𝑖,𝜋will be updated by minimizing the following loss
function:𝓀ℒ(𝜃𝑖𝓀 ) = 𝔼𝑥,𝑎,𝑟,𝑥 ′ [(𝑄𝑖,𝜋(𝑥, 𝑎1𝓀 , … , 𝑎𝑁𝓀 ) − 𝑦 2 )]

(19)

′,𝓀
𝓀
′ ′,𝓀
𝑦 = 𝑟𝑖𝓀 𝑄𝑖,𝜋′,𝓀 (𝑥 , 𝑎𝑖 , … , 𝑎𝑁 )| ′,𝓀(20)′,𝓀 ′,𝓀𝑎𝑗 =𝜋𝑗 (𝒪𝑗 )𝑁

C. State, Observation, and Action SpaceConsider an environment with N robots and M goals.
Robots and goals have a physical entity represented by 𝑋 .
Based on the original MPE, 𝑋 is a two-dimensional object
characterized by its position and velocity. Furthermore, thestate contains polar coordinates that are utilized to identify the
robot's relative position to the goals and other robots. An
environment with N robots and M goals corresponds to a state
space with NxM polar coordinates of robots to the goals( 𝑑𝐺1,…,𝑁𝑥𝑀 ) and N-1 polar coordinates to other robots
𝐴(𝑑1,…,𝑁−1). However, it should be noted that the goals of thebottom level are the subgoals produced at the upper level.
Based on the preceding discussion, the state space 𝒮 is a𝓀𝓀mixture of each level state space: 𝒮 = ⋃𝐾−1
𝓀=0 𝑆 , where 𝑆 =𝐺𝐴{𝑋1,…,𝑁 , 𝑑1,…,𝑁𝑥𝑀, 𝑑1,…,𝑁−1}.

where 𝜋 ′,𝓀 = {𝜋𝜃′,𝓀 , … , 𝜋𝜃′,𝓀 } is the set of target policy
1

1

:

∗
𝑎
𝓀∗
𝑄𝑖𝓀 (𝑠, 𝑎) = 𝑅𝑖𝓀 (𝑠, 𝑎) + 𝛾 ∑𝑠′𝜖𝑆 𝒫𝑠𝑠′ 𝑄𝑖 (𝑠′, 𝑎′)with delayed parameter 𝜃𝑖′ at each level 𝓀. As the Q-function𝓀𝑄𝑖,𝜋for each agent 𝑖 is learned independently at all levels, thereward may be determined arbitrarily based on the issue. The
algorithm of MH-DDPG is shown in algorithm 1.

Then each agent can only observe their own state of theentire state, called observation. The observation space of each𝐺𝐴
agent at each level k is 𝑂𝑖𝓀 (𝑆) = {𝑋𝑖 , 𝑑𝑖,1,…,𝑀, 𝑑𝑖,1,…,𝑁−1},𝑡ℎwhere 𝑖 indicates the 𝑖 robot.

39 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

At 𝓀 = 0, the output layer of the actor networks generates
five outputs between 0 and 1 in which each one is associatedwith a particular action. The five outputs are denoted by 𝑢𝑛 , 𝑢𝑙 ,𝑢𝑟 , 𝑢𝑑 , and 𝑢𝑢 for no action, move left, right, down, and up,
respectively. At 𝓀 > 0, if the action of the abstract robot shouldhave more capabilities than the actual robot, the range 𝑢 isincreased multiplied by the sensitivity rate, therefore 𝑢 𝓀 =𝑢0 𝑥 𝜇, where 1 ≤ 𝓀 ≤ 𝐾 − 1 and 𝜇 is the sensitivity with avalue more than 1. The sensitivity of the upper level must belarger than the sensitivity of the lower level.

where 𝑑(𝐴, 𝐵) is the distance between two robots (A and B)
that can be calculated by (22) with i, j=1.

D. Reward Design
The reward is designed to correspond to the learning
objectives of the robot. The distance between objects
determines the reward design. Suppose that the positions of
two object types, A and B, in two dimensions are known. A
and B respectively add up to N and M, therefore 𝐴𝑖 =𝑗(𝑥1𝑖 , 𝑦1𝑖 ) and 𝐵𝑗 = (𝑥2 , 𝑦22 ), where 1 ≤ 𝑖 ≤ 𝑁 and 1 ≤ 𝑗 ≤ 𝑀.
The following formula may be used to compute the total
distance between two types of objects:

where 𝑑(𝐴, 𝑂) is the distance between robot A and obstacle
O that can be calculated by (22) with i,j=1.

𝑑(𝐴, 𝐵) = √(𝑥2 − 𝑥1 )2 + (𝑦2 − 𝑦1 )2

(21)
𝑗 2

3) Obstacle reward: Aims to encourage robots to avoid
obstacles. Due to the abstract robot's inability to detectobstacles, this reward is only applied at the lowest level.
Similar to other robot rewards, the robot must compute thedistance between itself and the obstacle to get reward.
−10; 𝑖𝑓 𝑑(𝐴, 𝑂) ≤ 𝐴𝑠𝑖𝑧𝑒 + 𝑂𝑠𝑖𝑧𝑒
𝑅𝑐 (𝐴, 𝐵) = {0; 𝑖𝑓 𝑑(𝐴, 𝑂) > 𝐴𝑠𝑖𝑧𝑒 + 𝑂𝑠𝑖𝑧𝑒(25)

E. Neural Network Models
Each robot at each level of the MH-DDPG consists of actor
and critic networks, structures of which are shown in Fig. 3.
Local observations are the inputs for the actor-network, whilerobot actions represent the output. Therefore, the critic-network
uses the observations and actions of all robots as inputs and Qvalue as outputs. The Q-value is then used as the training basis
for the actor networks. Every network employs the ADAM
optimizer with a learning rate () and a discount factor ().

𝑗 2

𝑀 √ 𝑖
𝑖
𝑑(𝐴1,...,𝑁 , 𝐵1,..,𝑀 ) = ∑𝑁
𝑖=1 ∑𝑗=1 (𝑥2 − 𝑥1 ) + (𝑦2 − 𝑦1 ) (22)The design of the reward differs between the bottom andupper levels. The term for the rewards at each level is
explained as follows:
1) The goal/subgoal reward: Designed to encourage
agents to achieve the Goal/Subgoal. This reward is available at
all levels. This reward is utilized at the highest level to promote
the abstract robot to accomplish the main goal and at the lowest
level to help the robot reach the subgoal. Reward calculations
will be based on the distance between all robots and goals
using (22) and (23).
𝑅(𝐴𝑔𝑒𝑛𝑡𝑖,…,𝑁 , 𝒢𝑗,…,𝑀 ) = −𝑑(𝐴1,...,𝑁 , 𝐵1,..,𝑀 )(23)

where A = Robot dan B = G (goal/subgoal).
2) Robot relative to other robots Reward: for avoiding
collisions between robots. This reward is only used at thelowest level because the abstract robot is unable to detect otherrobots. This reward term is calculated as follows:−1; 𝑖𝑓 𝑑(𝐴, 𝐵) ≤ 𝐴𝑠𝑖𝑧𝑒 + 𝐵𝑠𝑖𝑧𝑒
𝑅𝑐 (𝐴, 𝐵) = {0; 𝑖𝑓 𝑑(𝐴, 𝐵) > 𝐴𝑠𝑖𝑧𝑒 + 𝐵𝑠𝑖𝑧𝑒(a) Low-Complexity.

Fig. 3. Neural Network Model.

V. EXPERIMENTAL ENVIRONMENT
We conducted experiments for comparing the DDPG,
MADDPG, and MH-DDPG algorithms under the parameterslisted in Table I. The experimental environment is set withthree robots and three goals, with some obstacles to increase
the environment's complexity. Fig. 4 depicts the environment
for testing the proposed algorithm. The experiments wereconducted on three types of environments with various
complexities: low-complexity {Fig. 4(a)}, mid-complexity{Fig. 4(b)}, and high-complexity {Fig. 4(c)}.

(24)(b) Mid-Complexity.

(c) High-Complexity.

Fig. 4. Illustration of the Experimental Environment.

40 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

Fig. 5. Block Diagram Comparison for DDPG, MADDPG, and MH-DDPG Employing 3 Robots. Q, O, and a Represent the Q-Value, Observation, and Action,
Respectively.
TABLE I.

EXPERIMENT-SPECIFIC PARAMETERS FOR DDPG, MADDPG,AND MH-DDPG

Parameters

DDPG

MADDPG

MH-DDPG

The number of robots (N)

3

3

3

The number of levels (K)

-

-

2

The number of actors

3

3

6

The number of critics

3

3

6

The number of ERs

1

1

2

Sensitivity

555 (level 0),30 (level 1)

Specific Parameters

Actor and Critic Networks Parameters
Number of hidden layers

1

1

1

Number of hidden units

64

64

64

Activation Function

ReLU

ReLU

ReLU

Input Actor Network

Current
Observations

Current
Observations

Current
Observations

Output Actor Network

ActionActionActionInput Critic Network

Current
Observat
ion and
ActionCurrent
Observation
and Action

Current
Observation
and Action

Output Critic Network

Q-value

Q-ValueQ-value

Optimizer

ADAM

ADAM

ADAM

Learning rate ()

1e-2

1e-2

1e-2

Discount factor ()

0.97

0.97

0.97

Replay buffer size

10

6

6

106

Minibatch size

1256

1256

1256

Fig. 5 compares the block diagrams of DDPG, MADDPG,
and MH-DDPG, illustrating how the algorithm determines theparameter values for the specific parameters given in Table I,
except for sensitivity. The values for the sensitivity and thetraining parameters are empirically determined. From theexperiments, the determination of the training parameters in
different environments demonstrates that the algorithms are notexcessively sensitive to the chosen parameters.
Particularly in MH-DDPG, the designed environment candecompose into K levels. As a preliminary step in the proposed
algorithm, this research performs a two-level investigation(K=2). Where the bottom level (𝓀 = 0) is the real environment
used for actual robots learning, and the top level (𝓀 = 1) is theabstract environment used for abstract robot learning. In theenvironments, the robots must collaborate to accomplish thepredetermined goals. The robots' mission will be accomplished
if the robots can discover the optimal path for reaching all thegoals.
VI. RESULTS

Training parameters

10

The experiments compare the proposed algorithm against
MADDPG and DDPG. The first step is assessing the robots
learning performance based on the rewards obtained
throughout the learning process.
Fig. 6 depicts the learning curve based on the robot's
average reward in each episode. In this experiment, there were150000 episodes in each environment. The average reward therobot obtains in a low-complexity environment is greater thanin mid-complexity and high-complexity environments. A
greater average reward indicates that robots in simple
environments perform better than in other environments. A
low-complexity environment without obstacles makes it easierfor robots to reach their goals.
From Fig. 6, it can be observed that MH-DDPG converges
faster to the maximum rewards and produces larger reward ineach episode than DDPG and MADDPG, indicating that the
robots that were trained using MH-DDPG reaches the goals

41 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

faster. Fig. 6 also indicates that the superiority of MH-DDPG
over DDPG and MADDPG is consistent in complex
environments. In a high-complexity environment, the graphalso reveals that the average reward value is unstable for
DDPG and MADDPG, whereas MH-DDPG remains robust.
Fig. 7 depicts the robot's behavior during the learning
process compared to MADDPG and DDPG in a midcomplexity and high-complexity environment at t=0, 10, and
40. The blue circle indicates the actual robot, the green
rectangle represents the goals, and the red circle represents theabstract robot found exclusively on MH-DDPG. The abstract
robots will generate a subgoal for the lower level. At t=0, therobot begins its first step of learning. Here, the locations of theabstract robot are identical to the positions of the actual robots.
At t=10, robots are learning to achieve all goals. Here, on
MH-DDPG, a red circle indicates the presence of an abstract
robot. At each instant t, the abstract robot generates a subgoal
for the actual robot. In MH-DDPG, Actual robots will firstlearn to cover subgoals, but in DDPG and MADDPG, robots

will learn to cover main goals straight away since there are no
subgoals. Abstract robots that cannot detect obstacles mightoccasionally be located in the same area as the obstacle, as
shown in the high-complexity environment at time t=10. Thiscondition is sometimes harmful to the actual robot when it
learns to achieve the subgoal since the actual robot cannot
reach the subgoal properly, which consequently decreases theassociated state-action values.
Finally, at t=40, the final step of learning in a single
episode occurs. In DDPG, it is evident that the robots havedifficulty cooperating to reach the goal in both mid-complexityand high-complexity environments, which is also consistent
with the successful rates shown in Fig. 8. In the midcomplexity environment, the robot tends to go toward one of
the goals. Therefore the targeted objectives to cover all goals
are often not achieved. This is because the robots work
independently and do not share information. In highcomplexity, obstacles tend to hinder the robot's ability toachieve the subgoals.

Fig. 6. Reward Graph.

Fig. 7. Comparison between DDPG, MADDPG, and MH-DDPG on the Mid-complexity and High-complexity Environment.

42 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022

Fig. 8. Success Rate.

From Fig. 7, for MADDPG, it can be observed that the
robots can work collectively to achieve goals in a midcomplexity environment, whilein high-complexity
environments, due to the complex configurations of the
obstacles, the robots faced difficulty in the early phase of thelearning process as apparent from the fluctuating graph, but
gradually stabilize as the learning progresses. In MH-DDPG,
subgoals increase the robot's problem-solving ability as they
are guided by intermediate objectives that consequently
contrained the problem space.
Fig. 8 depicts the average success rate for the respective
algorithm in each environment during the learning process. In
all environments, the robots with MADDPG and MH-DDPG
were able to cooperate and learn policies to achieve their goals,
while the robots with DDPG failed to do so. The failure of therobots in DDPG is due to the lack of information sharing
between robots; hence, the robots only learn independently andmay repeat the failure of other robots. In the low-complexity
environment, the average success rates for MADDPG and MHDDPG are 74.18% and 82.15%, respectively. As thecomplexity of the environment increases, the average successrate for MADDPG and MH-DDPG decreases, as seen from thegraphs for mid and high-complexity environments. MADDPG
and MH-DDPG had respective success rates of 56.08% and73.18% in a mid-complexity environment, while in a highcomplexity environment, these success rates were 44.18% and72.99%, respectively. The results show that MH-DDPG has agreater success rate than MADDPG. This indicates that
decomposing the problem environment into many levels is
advantageous for maximizing robot learning performance.
VII. CONCLUSION
MH-DDPG is proposed as a novel framework for multirobot learning with hierarchical Deep Reinforcement Learning.
Here, the robots collectively learn by sharing information aboutstate-action values from their individual runs. In addition, theproposed MH-DDPG provides a mechanism for creating multilevel abstraction, in which higher-level abstraction space allow
the robots to execute a kind of “image training” where they
may virtually explore the problem space without considering
the physical constrains in real-world space. The virtual
experiment in abstract space allows the robot to discover thereal robots' intermediate goals rapidly. The intermediate goals

helps to limit the exploration for the real robots, thus
alleviating the curse of dimensionality.
Through some empirical experiments, it can be observedthat the proposed MH-DDPG outperforms DDPG and
MADDPG in learning efficiency and success rate.
The weakness of the MH-DDPG is that an abstract robot at
higher levels is incapable of detecting obstacles. Hence nonrealistic subgoals are sometimes produced. This is the cost thatneeds to be paid for removing the physical constraints in the
abstract space. In this preliminary experiment, the abstract
robots are given higher speed but are constrained by theirinability to detect obstacles, but it does not have to be so. In thefollowing study, experiments will be conducted with various
constrained conditions at the higher abstraction levels.
Immediate future research topics include investigating theeffect of the number of levels and the number of robots in MHDDPG. In addition, implementing the proposed learning
method into physical robots is also of interest.
[1]
[2]

[3]

[4]

[5]

[6]

[7]

[8]

REFERENCES
Z. Yan, N. Jouandeau and A. A. Cherif, “A survey and analysis of multirobot coordination,” Int. J. Adv. Robot. Syst., vol. 10, pp. 1–18, 2013.
J. Song and S. Gupta, “CARE: Cooperative Autonomy for Resilienceand Efficiency of robot teams for complete coverage of unknown
environments under robot failures,” Auton. Robots, vol. 44, no. 3–4, pp.
647–671, 2020.
Y. Rizk, M. Awad and E. W. Tunstel, “Cooperative heterogeneous
multi-robot systems: A survey,” ACM Comput. Surv., vol. 52, no. 2,
2019.
Z. A. Ali, Z. Han and R. J. Masood, “Collective motion and selforganization of a swarm of uavs: A cluster-based architecture,” Sensors,
vol. 21, no. 11, pp. 1–19, 2021.
B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powel, B. McGrew
and I. Mordatch, “Emergent tool use from multi-agent autocurricula,”
2020.
M. Bakhshipour, M. Jabbari Ghadi and F. Namdari, “Swarm robotics
search & rescue: A novel artificial intelligence-inspired optimization
approach,” Appl. Soft Comput. J., vol. 57, pp. 708–726, 2017.
J. P. Queralta, J. Taipalmaa, B. C. Pullinen, V. K. Sarker, T. N. Gia, H.
Tenhunen, M. Gabbouj, J. Raitoharju and T. Westerlund, “Collaborative
multi-robot search and rescue: Planning, coordination, perception, andactive vision,” IEEE Access, vol. 8, pp. 191617–191643, 2020.
D. S. Drew, “Multi-Agent Systems for Search and Rescue
Applications,” Curr. Robot. Reports, vol. 2, no. 2, pp. 189–200, 2021.

43 | P a g e
www.ijacsa.thesai.org

(IJACSA) International Journal of Advanced Computer Science and Applications,Vol. 13, No. 9, 2022
L. Hawley and W. Suleiman, “Control framework for cooperative object
transportation by two humanoid robots,” Rob. Auton. Syst., vol. 115, pp.
1–16, 2019.
[10] X. Zhou, W. Wang, T. Wang, Y. Lei and F. Zhong, “Bayesian
Reinforcement Learning for Multi-Robot Decentralized Patrolling inUncertain Environments,” IEEE Trans. Veh. Technol., vol. 68, no. 12,
pp. 11691–11703, 2019.
[11] N. Naderializadeh, J. J. Sydir, M. Simsek and H. Nikopour, “Resource
Management in Wireless Networks via Multi-Agent Deep
Reinforcement Learning,” IEEE Trans. Wirel. Commun., vol. 20, no. 6,
pp. 3507–3523, 2021.
[12] J. Sheng, X. Wang, B. Jin, J. Yan, W. Li, T.-H. Chang, J. Wang and H.
Zha, “Learning structured communication for multi-agent reinforcement
learning,” Auton. Agent. Multi. Agent. Syst., vol. 36, no. 2, 2022.
[13] F. Niroui, K. Zhang, Z. Kashino and G. Nejat, “Deep Reinforcement
Learning Robot for Search and Rescue Applications: Exploration in
Unknown Cluttered Environments,” IEEE Robot. Autom. Lett., vol. 4,
no. 2, pp. 610–617, 2019.
[14] S. M. Sombolestan, A. Rasooli and S. Khodaygan, “Optimal pathplanning for mobile robots to find a hidden target in an unknown
environment based on machine learning,” J. Ambient Intell. Humaniz.
Comput., vol. 10, no. 5, pp. 1841–1850, 2019.
[15] J. Hu, H. Niu, J. Carrasco, B. Lennox and F. Arvin, “Voronoi-BasedMulti-Robot Autonomous Exploration in Unknown Environments via
Deep Reinforcement Learning,” IEEE Trans. Veh. Technol., vol. 69, no.
12, pp. 14413–14423, 2020.
[16] R. J. Alitappeh and K. Jeddisaravi, “Multi-robot exploration in task
allocation problem,” Appl. Intell., vol. 52, no. 2, pp. 2189–2211, 2022.
[17] T. Fan, P. Long, W. Liu and J. Pan, “Distributed multi-robot collision
avoidance via deep reinforcement learning for navigation in complex
scenarios,” Int. J. Rob. Res., vol. 39, no. 7, pp. 856–892, 2020.
[18] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel and I. Mordatch, “Multiagent actor-critic for mixed cooperative-competitive environments,”
Adv. Neural Inf. Process. Syst., vol. 2017-Decem, pp. 6380–6391, 2017.
[19] R. S. Sutton and A. G. Barto, Reinforcement Learning: an Introduction,
2nd ed. London, England: The MIT Press Cambridge, Massachusetts,
1998.
[20] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S.
Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D.
Wierstra, S. Legg and D. Hassabis, “Human-level Control through Deep
Reinforcement Learning,” Nature, vol. 518, no. 7540, pp. 529–533,
2015.
[21] X. Wang, J. Song, P. Qi, P. Peng, Z. Tang, W. Zhang, W. Li, X. Pi, J.
He, C. Gao, H. Long and Q. Yuan, “SCC: an efficient deep
reinforcement learning agent mastering the game of StarCraft II,” 2021.
[22] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A.
Irpan, B. Eysenbach, R. Julian, C. Finn and S. Levine, “Actionable
Models: Unsupervised Offline Reinforcement Learning of Robotic
Skills,” 2021.
[23] M. Dalal, D. Pathak and R. Salakhutdinov, “Accelerating Robotic
Reinforcement Learning via Parameterized Action Primitives,” in 35th
Conference on Neural Information Processing Systems (NeurIPS 2021),
2021, no. NeurIPS.
[9][24] C. J. C. H. Watkins and P. Dayan, “Q-Learning,” Mach. Learn., vol. 8,
pp. 279–292, 1992.
[25] G. A. Rummery and M. Niranjan, On-Line Q-Learning Using
Connectionist Systems. Technical Report CUED/F-INFENG/TR 166,
Cambridge University Engineering Department, 1994.
[26] R. S. Sutton, “Learning to Predict by the Methods of Temporal
Differences,” Mach. Learn., vol. 3, no. 1, pp. 9–44, 1988.
[27] M. L. Puterman, Markov Decision Processes: Discrete Stochastic
Dynamic Programming. New Jersey: John Wiley & Sons, Inc., 1994.
[28] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D.
Wierstra and M. Riedmiller, “Playing Atari with Deep Reinforcement
Learning,” vol. arXiv prep, 2013.
[29] R. S. Sutton, D. McAllester, S. Singh and Y. Mansour, “Policy Gradient
Methods for Reinforcement Learning with Function Approximation,” inProceedings of the 12th International Conference on Neural Information
Processing Systems, 1999, pp. 1057–1063.
[30] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D.
Silver and D. Wierstra, “Continuous control with deep reinforcement
learning,” 4th Int. Conf. Learn. Represent. ICLR 2016 - Conf. Track
Proc., 2016.
[31] V. R. Konda and J. N. Tsitsiklis, “Actor-critic algorithms,” Adv. Neural
Inf. Process. Syst., pp. 1008–1014, 2000.
[32] M. L. Littman, “Markov games as a framework for multi-agent
reinforcement learning,” Mach. Learn. Proc. 1994, pp. 157–163, 1994.
[33] R. S. Sutton, D. Precup and S. Singh, “Between MDPs and semi-MDPs:
A framework for temporal abstraction in reinforcement learning
Richard,” Artif. Intell., vol. 112, no. 1, pp. 181–211, 1998.
[34] A. Bai and S. Russell, “Efficient reinforcement learning with hierarchiesof machines by leveraging internal transitions,” in IJCAI International
Joint Conference on Artificial Intelligence, 2017, vol. 0, pp. 1418–1424.
[Online].
[35] G. E. Setyawan, H. Sawada and P. Hartono, “Combinations of MicroMacro States and Subgoals Discovery in Hierarchical Reinforcement
Learning for Path Finding,” Int. J. Innov. Comput. Inf. Control, vol. 18,
no. 2, pp. 447–462, 2022.
[36] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D.
Silver and K. Kavukcuoglu, “FeUdal networks for hierarchicalreinforcement learning,” 34th Int. Conf. Mach. Learn. ICML 2017, vol.
7, pp. 5409–5418, 2017.
[37] Z. Yang, K. Merrick, S. Member, L. Jin, H. A. Abbass and S. Member,“Hierarchical Deep Reinforcement Learning for Continuous Action
Control,” IEEE Trans. Neural Networks Learn. Syst., vol. 29, no. 11, pp.
5174–5184, 2018.
[38] Z. Liang, J. Cao, W. Lin, J. Chen and H. Xu, “Hierarchical Deep
Reinforcement Learning for Multi-robot Cooperation in PartiallyObservable Environment,” Proc. - 2021 IEEE 3rd Int. Conf. Cogn.
Mach. Intell. CogMI 2021, pp. 272–281, 2021.
[39] J. Yang, I. Borovikov and H. Zha, “Hierarchical cooperative multi-agent
reinforcement learning with skill discovery,” Proc. Int. Jt. Conf. Auton.
Agents Multiagent Syst. AAMAS, vol. 2020-May, pp. 1566–1574,
2020.
[40] H. Clark and S. Brennan, “Emergence of grounded compositional
language in multi-agent populations,” 32nd AAAI Conf. Artif. Intell.
AAAI 2018, pp. 1495–1502, 2018.

44 | P a g e
www.ijacsa.thesai.org