Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games
Authors: Shengyi Huang
Santiago Ontanon
Venue:  (2021)
Keywords: reinforcement learning
real-time strategy games
sparse rewards
shaped rewards
policy gradient
sample-efficiency

Abstract: Training agents using Reinforcement Learning in games with sparse rewards is a challenging problem, since large amounts of exploration are required to retrieve even the first reward. To tackle this problem, a common approach is to use reward shaping to help exploration. However, an important drawback of reward shaping is that agents sometimes learn to optimize the shaped reward instead of the true objective. In this paper, we present a novel technique that we call action guidance that successfully trains agents to eventually optimize the true objective in games with sparse rewards while maintaining most of the sample efficiency that comes with reward shaping. We evaluate our approach in a simplified real-time strategy (RTS) game simulator called $\mu$RTS. 

Under review as a conference paper at ICLR 2021
ACTION GUIDANCE: GETTING THE BEST OF SPARSE REWARDS AND SHAPED REWARDS FOR RIAL STRATEGY GAMES
Anonymous authors Paper under double- blind review
ABSTRACT
Training agents using Reinforcement Learning in games with sparse rewards is a challenging problem, since large amounts of exploration are required to retrieve even the first reward. To tackle this problem, a common approach is to use reward shaping to help exploration. However, important drawback of reward shaping is that agents sometimes learn to optimize the shaped reward instead of the true objective. In this paper, we present novel technique that we call action guidance that successfully trains agents to eventually optimize the true objective in games with sparse rewards while maintaining most of the sample efficiency that comes with reward shaping. We evaluate our approach in a simplified real-time strategy (RTS) game simulator called pRTS.
Training agents using Reinforcement Learning with sparse rewards is often difficult (Pathak et al., 2017]. First, due to the sparsity of the reward, the agent often spends the majority of the training time doing inefficient exploration and sometimes not even reaching the first sparse reward during the entirety of its training. Second, even if the agents have successfully retrieved some sparse rewards, performing proper credit assignment is challenging among complex sequences of actions that have led to theses sparse rewards. Reward shaping (Ng et al. 1999) is widely-used technique designed to mitigate this problem. It works by providing intermediate rewards that lead the agent towards the sparse rewards, which are the true objective. For example, the sparse reward for game of Chess naturally +1 for winning. -1 for losing, and 0 for drawing, while possible shaped reward might be +1 for every enemy piece the agent takes. One of the critical drawbacks for reward shaping that the agent sometimes learns to optimize for the shaped reward instead of the real objective. Using the Chess example, the agent might learn to take as many enemy pieces as possible while still losing the game. good shaped reward achieves a nice balance between letting the agent find the sparse reward and being too shaped (so the agent learns to just maximize the shaped reward), but this balance can be difficult to find.
In this paper, we present novel technique called action guidance that successfully trains the agent to eventually optimize over sparse rewards while maintaining most of the sample efficiency that comes with reward shaping. It works by constructing main policy that only learns from the sparse reward function R.M and some auxiliary policies that learn from the shaped reward func- tion RA,, RAy.........................RR.. ......... R.A.. During training, we use the same rollouts train the main and auxiliary policies and initially set high-probability of the main policy to take action guidance from the aux- iliary policies, that is, the main policy will execute actions sampled from the auxiliary policies. Then the main policy and auxiliary policies are updated via off- off-policy policy gradient. As the training goes on, the main policy will get more independent and execute more actions sampled from its own policy. Auxiliary policies learn from shaped rewards and therefore make the training sample- efficient, while the main policy learns from the original sparse reward and therefore makes sure that the agents will eventually optimize over the true objective. We can see action guidance combining reward shaping to train auxiliary policies interlicaved with a sort of imitation learning to guide the main policy from these auxiliary policies.
We examine action guidance in the context of a real-time strategy (RTS) game simulator called μRTS for three sparse rewards tasks of varying difficulty. For each task, we compare the performance of training agents with the sparse reward function RM, a shaped reward function RA,, and action guidance with singular auxiliary policy learning from R.A,- The main highlights are:

---

Under review as a conference paper at ICLR 2021
Action guidance is sample-efficient. Since the auxiliary policy learns from R., and the main policy takes action guidance from the auxiliary policy during the initial stage of training, the main policy is more likely to discover the first sparse reward more quickly and learn more efficientlyy Empirically, action guidance reaches almost the same level of sample efficiency as reward shaping in all of the three tasks tested.
The true objective is being optimized. During the course of training, the main policy has never seen the shaped rewards. This ensures that the main policy, which is the agent we are really interested in, is always optimizing against the true objective and is less biased by the shaped rewards. As an example, Figure shows that the main policy trained with action guidance eventually learns to win the game as fast as possible, even though it has only learned from the match outcome reward (+1 for winning, -1 for losing, and for drawing). In contrast, the agents trained with reward shaping learn more diverse sets of behaviors which result in high shaped reward.
To support further research in this field, we make our source code available at GitHut as well as all the metries, logs, and recorded videos
RELATED WORK
In this section, we briefly summarize the popular techniques proposed address the challenge of sparse rewards.
Reward Shaping, Reward shaping is a common technique where the human designer uses domain knowledge to define additional intermediate rewards for the agents. Ng et al. (1999) show that a slightly more restricted form of state-based reward shaping has better theoretical properties for preserving the optimal policy.
Transfer and Curriculum Learning. Sometimes learning the target tasks with sparse rewards too challenging. and it is more preferable to learn some easier tasks first. Transfer learning leverages this idea and trains agents with some easier source tasks and then later transfer the knowledge through
value function (Taylor al. 2007) or reward shaping (Svetlik al. (017). Curriculum learning further extends transfer learning by automatically designing and choosing a full sequences of source tasks (i.e. curriculum) (Narvekar & Stone 2018).
Imitation I Learning, Alternatively, it is possible to directly provide examples of human demonstra- tion or expert replay for the agents to mimic via Behavior Cloning (BC) (Bain & Sammut] 1995), which uses supervised learning to learn a policy given the state-action pairs from expert replays. Al- ternatively, Inverse Reinforcement Learning (IRL) (Abbeel Ng 2004) recovers reward function from expert demonstrations to be used to train agents.
Curiosity-driven Learning. Curiosity driven learning seeks to design intrinsic reward func- tions (Burda et al. 2019) using metrics such as prediction errors (Houthooft et al. (2016) and "visit counts" (Bellemare et al. 2016, Lopes et al. 2012). These intrinsic rewards encourage the agents to
explore unseen states.
Goal-oriented Learning. In certain tasks, it is possible to describe a goal state and use it in conjunction with the current state input (Schaul et al.) 2015). Hindsight experience replay (HER) (Andrychowicz et al., 2017) develops better utilization existing data in experience re- play by replaying each episode with different goals. HER is shown to be an effective technique in sparse rewards tasks.
Hierarchical Reinforcement Learning (HRL). If the target task is difficuut to learn directly, it also possible to hierarchically structure the task using experts' knowledge and train hierarchical agents, which generally involves main policy that learns abstract goals, time, and actions, well as auxiliary policies that learn primitive actions and specific goals (Dietterich) 2000). HRL is especially popular in RTS games with combinatorial action spaces (Pang al. 2019 Ye al. 2020).
The most closely related work is perhaps Scheduled Auxiliary Control (SAC-X) Riedmiller et al. 2018 which is an HRL algorithm that trains auxiliary policies to perform primitive actions with
https: /
dance
Hlinded for peer review
2

---

Under review as a conference paper at ICLR 2021
6000 d. 20
4640 /19a 1900
(a) shaped reward
(b) action guidance
(https: /tttr///aabeaaabeee.e.m wwtp://ww.................oooo/m
www.coo//hhc...o. hatp/hhabbp
Figure 1: The screenshot shows the typical learned behavior of agents in the task of DefeatRan- domEnemy. (a) shows that agent trained with some shaped reward function RA, learns many helpful behaviors such as building workers (grey circles), combat units (blue circles), and barracks (grey square) or using owned units (with red boarder) to attack enemy units (with blue border), but does not learn to win as fast as possible (i.e. it still does not win at internal time step = 6000). In contrast, (b) shows learns to win as fast as possible (i.e. about to win the game at = 440), with its main policy learning from the match outcome reward function R& and singular auxiliary policy learning from the same shaped reward function RAi- Click on the link below figures to see the full videos of trained agents.
agent trained with action guidance optimizes
the match outcome and
shaped rewards and a main policy to schedule the use of auxiliary policies with sparse rewards. However, our approach differs in the treatment of the main policy. Instead of learning to schedule auxiliary policies, our main policy learns to act in the entire action space by taking action guidance from the auxiliary policies. There are two intuitive benefits to our approach since our main policy learns in the full action space. First, during policy evaluation our main policy does not have to commit a particular auxiliary policy to perform actions for a fixed number of time steps like is usually done in SAC-X. Second, learning in the full action space means the main policy will less likely suffer from the definition of hand-crafted sub-tasks, which could be incomplete or biased.
BACKGROUND
We consider the Reinforcement Learning problem in Markov Decision Process (MDP) denoted as (S, A, P, Po: r. 7, T), where S is the state space, A is the discrete action space, P: SxAxS S + [0, is the state transition probability, Pu : S - [0, 1] is the the initial state distribution, r S x A + is the reward function, y is the discount factor, and T is the maximum episode length. A stochastic policy Te : S x A + [0, 1]. parameterized by parameter vector 6, assigns a probability value to an action given a state. The goal is to maximize the expected discounted return of the policy:
T-
where is the trajectory (so, Oo, ro, 81, 8T -r-1,, and sp Pa. P(- a 1). a ~ Ta(-1s)), r (st, oz))
E,
Σ
Policy Gradient Algorithms. The core idea behind policy gradient algorithms is to obtain the policy gradient V of the expected discounted return with respect to the policy parameter 0. Doing gradient ascent 0 = + VeJ therefore maximizes the expected discounted reward. Earlier work proposes the following policy gradient estimate to the objective J (Sutton & Barto 2018):
9policy.0 0 ErANo
Ve log To (s)ss
3

---

Under review as a conference paper at ICLR 2021
where G2 Ek=n 7 r+++ denotes the discounted return following time This gradient estimate, however, suffers from large variance (Sutton & Barto 2018) and the following gradient estimate is suggested instead:
Spolicy. e
E
V8
log (a(o())))A(T] V.t)
t=0
where A(T, V) is the General Advantage Estimation (GAB) (Schulman et al. 2015), which mea- sures "how good is a, compared to the usual actions", and V + IK is the state-value function.
ACTION GUIDANCE
The key idea behind action guidance is to create main policy that trains on the sparse rewards, and creating some auxiliary policies that are trained on shaped rewards. During the initial stages of training. the main policy has a high probability to take action guidance from the auxiliary policies, that is, the main policy can execute actions sampled from the auxiliary policies, rather than from its policy. As the training goes on, this probability decreases, and the main policy executes more actions sampled from its own policy. During training, the main and auxiliary policies are updated via off-policy policy gradient. Our use of auxiliary policies makes the training sample-efficient, and our use of the main policy, who only sees its own sparse reward, makes sure that the agent will eventually optimize over the true objective of sparse rewards. In a way, action guidance can be seen as training agents using shaped rewards, while having the main policy learn by imitating from them.
Specifically, let us define M as the MDP that the main policy learns from and A {Ai, A2, ..., As) be set of auxiliary MDPs that the auxiliary policies learn from. In our constructions, M and A share the same state, observation, and action space. However, the reward function for M is RM which is the sparse reward function, and reward functions for A are RAi.... RA which are the shaped reward functions. For each of these MDPs S {MJUA above, us initialize a policy To, parameterized by parameters 0g, respectively. Furthermore, let us TS {Tee 3 S} to denote the set of these initialized policies.
At each timestep t, let us some exploration strategy S that selects policy T. Ts sample an action a given 8,. At the end of the episode, each policy ma E as can be updated via its off-policy policy gradient (Degris et al. 2012 Levine et al. 2020):
T-1
mo (ar)st)
T-1
E BrInT,
II
Σ
Vo log no (agls:) A(T, V,t)
To, (as (t))st)
When Te, the gradient in Equation [ wise, the objective means off- off-policy policy gradient update for mg.
means on-policy policy gradient update for иT. Other-
3.1 PRACTICAL ALGORITHM
The gradient in Equation [ is unbiased, but its product of importance sampling ratio
xo(a.]s.) (IT--) Ne, (0rl))) known to cause high variance (Wang et al. 2016). In practice, we clip the gradient the same way Proximal Policy Gradient (PPO) (Schulman et al. 2017):
T-1
(LIP((0) = E
Vamin (p,(0)((T,) V, clpp( A(T, V.D)
(2)
1 11 if p: < 1-E
Ty (a. (sc) To, (arls:))
(0)
clip (p),
1 ite if p( > 1+E
(i) otherwise
During the optimization phase, the agent also learns the value function and maximize the policy's entropy. We therefore optimize the following joint objective for each ιо E Ts.
CLIP L CLIP (0)
L (0) c2S[.
where C₁ C2 are coefficients, is an entropy bonus, and VF is the squared error loss for the value function associated with my done by SShhlman et al. (201)). Although action guidance can be
4

---

Under review as a conference paper at ICLR 2021
configured leverage multiple auxiliary policies that learn diversified reward functions, we only use one auxiliary policy for the simplicity of experiments. In addition, we use é-greedy as the exploration strategy S for determining the behavior policy. That is, at each timestep the behavior e and T0p for 'D E A with probability (note that is is different from the clipping coefficient of PPO). Additionally, is set to be constant 0.95 at start for some period of time steps (c.g. 800,000), which we refer to as the shift period (the time it takes to start "shifting" focus away from the auxiliary policies), then is to linearly decay to eend for some period of time steps (e.g. 1,000,000), which we refer to as the adaptation period (the time takes for the main policy fully "adapt" and become more independent). Lastly, we included a pseudocode of action guidance in Algorithm Tat at the Appendix.
policy is selected to be n with probability
3.2 POSITIVE LEARNING OPTIMIZATION
During our initial experiments, we found the main policy sometimes did not learn useful policies. Our hypothesis is that this was because the main policy is updated with too many trajectories with zero reward. Doing a large quantities of updates of these zero-reward trajectories actually causes the policy converge prematurely, which is manifested by having low entropy in the action probability distribution.
To mitigate this issue of having too many zero-reward trajectories. use preliminary code-level optimization called Positive Learning Optimization (PLO). After collecting the rollouts, PLO works by skipping the gradient update for Tec Ts and its value function if the rollouts contains no reward according to Rg. Intuitively, PLO makes sure that the main policy learns from meaningful experience that associated with positive rewards. To confirm its effectiveness, provide an ablation study of PLO in the experiment section.
4 EVALUATION
We use PRTS as our testbed, which a minimalistic RTS game maintaining the core features that make RTS games challenging from an Al point of view: simultaneous and durative actions, large branching factors and real-time decision making. To interface with JRTS, we use gym- micrort (Huang & Ontañón 2020) to conduct experiments. The details of gym-microrts as a RL interface can be found at Appendix A. A.
4.1 TASKS DESCRIPTION
We examine the three following sparse reward tasks with range of difficulties. For each task, compare the performance of training agents with the sparse reward function RM, a shaped reward function RA, and action guidance with a single auxiliary policy learning from RA,, Here are the descriptions of these environments and their reward functions.
1. Ltttaa In this task, the agent's objective is learn move the other side of the map where the enemy units live and start attacking them. Its RM gives +1 reward for each valid attack action the agent issues. This is of sparse reward because the action space so large: the agent could have build a barracks produce unit; it is unlikely that the agents will by chance issue lots of moving actions (out of 6 action types) with correct directions (out of 4 directions) and then start attacking. Its R,A, gives the difference between previous and current Euclidean distance between the enemy base and closet unit owned by the agent as the shaped reward in addition to R-M-
2. ProduceCombatUnits: In this task, the agent's 's objective is to learn to build as many combat units possible. Its R. gives +1 reward for each combat unit the agent produces. This is a more challenging task because the agent needs to learn 1) harvest resources, 2) produce barracks, 3) produce combat units once enough resources are gathered, 4) move produced combat units out of the way so as to not block the production of new combat units. Its RA. gives +1 for constructing every building (c.g. barracks), +1 for harvesting resources, +1 for returning resources, and +7 for each combat unit it produces.
5

---

Under review as a conference paper at ICLR 2021
3. DefeutaandomEnem:: In this task, the agent's objective is to defeat biased random bot of which the attack, harvest and return actions have 5 times the probability of other actions. Additionally, the bot subjects to the same gym-microrts limitation (See Appendix (A.2) as the agents used in our experiments. Its RM gives the match outcome as the reward (-1 on loss, 0 on draw and +1 a win). This is the most difficult task we examined because the agent is subject to the full complexity of the game, being required to make both macro- decisions (e.g. deciding the high-level strategies to win the game) and micro-decisions (e.g. deciding which enemy units to attack. In comparison, its RA, gives +5 for winning, +1 for harvesting one resource, for returning resources,
for producing one worker, +0.2 for constructing every building, +1 for each valid attack action it issues, +7 for each combat unit it produces, and +0.2 * d) where is difference between previous and current Euclidean distance between the enemy base and its closet unit owned by the agent.
4.2 AGENT SETUP
We use PPO (Schulman et al. 2017) as the base DRL algorithm to incorporate action guidance. The details of the implementation, neural network architecture, hyperparameters, proper handling of jRTS'ss action space and invalid action masking (Huang & Ontañón 2020) can be found in Appendix B We compared the following strategies:
1. Sparse reward (first baseline). This agent is trained with PPO on RM for each task.
2. Shaped reward (second baseline). This agent is trained with PPO on RA, for each task.
3. Action guidance- long adaptation. The agent is trained with PPO + action guidance with shift 2, 000, 000 time steps, adaplation = 7, 000, 000 time steps, and <end = 0.0
4. Action guidance short adaptation. The agent is trained with PPO action guidance with shift = 800, 000 time steps, adaptation = 1, 000, 000 time steps, and Cend = 0.0
5. Action guidance- mixed policy. The agent trained with PPO + action guidance with shift = 2. 000, 000 time steps and adaptation = 2, 000, 000 time steps, and Cend = 0.5. We call this agent "mixed policy" because it will eventually have 50% chance to sample actions from the main policy and 50% chance to sample actions from the auxiliary policy. It is effectively having mixed agent making decisions jointly.
Although it is desirable to add SAC-X to the list of strategies compared, it was not designed to handle domains with large discrete action spaces. Lastly, we also toggle the PLO option for action guidance long adaptation, action guidance short adaptation. action guidance- mixed policy, and sparse reward training strategies for a preliminary ablation study.
4.3 EXPERIMENTAL RESULTS
Each of the 6 strategies is evaluated in tasks with 10 random seeds. We report the results in Table [ From here on, we use the term "sparse return" to denote the episodic return according to R.M and "shaped return" the episodic return according to RAI- All the learning curves can be found in Appendix C
Below are our observations.
reward shaping. Since the auxiliary policy learns from RA, and the main policy takes a lot of action guidance from the auxiliary policy during the shift period, the main policy is more likely discover the first sparse reward more quickly and learn more efficiently. As an example, Figure demonstrates such sample-efficiency in Produce- CombatUnits where the agents trained with sparse reward struggle to obtain the very first reward. In comparison, most action guidance related agents are able to learn almost as fast as the agents trained with shaped reward.
Action guidance almost sample-efficient
Action guidance eventually optimizes the sparse reward. This is perhaps the most important contribution our paper. Action guidance eventually optimizes the main policy over the true ob- jective, rather than optimizing shaped rewards. Using the ProduceCombatUnic task as example, the agent trained with shaped reward would only start producing combat units once all the resources have been harvested, probably because the +1 reward for harvesting and returning resources are easy to retrieve and therefore the agents exploit them first. Only after these resources are exhausted would the agents start searching for other sources of rewards then learn producing combat units.

---

Under review as a conference paper at ICLR 2021
10
Episode Reward
8
4
Episode
2
0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
1c7
shaped reward
Time Steps
1e7
action guidance long adaptation w/
action guidance short adaptation w/ PLO
action gui dandecee short adaptation action guidance- short adaptation PLO
sparse reward
Figure 2: The faint lines are the actual sparse return each seed for selected strategies Produce- CombatUnits; solid lines their means. The left figure showcase the sample-efficinncy of action guidance; the right figure is motivating example for PLO.
Table I: The average sparse return achieved by each training strategy in each task over 10 random seeds.
LearnToAtaac
ProduceCombatUnit
DefeadRandomEneny
sparse reward (first baseline)
3.30 1 5.04
0.00 0.01
-0.07 1: 0.03
sparse reward w/ PLO
0.00 - 0.00
0,00 + 0.01
-0.05 1: 0.03
shaped reward (second baseline)
10.00 1 0.00
9.57 0.30
0.08 1: 0.17
action guidance long adaptation
11.00 0.00
8.31 # 2.62
0.11 0.35
action guidance long adaptation w/ PLO
11.00 = 0.01
6.96 4.04
0.52 = 0.35
action guidance mixed policy action guidance mixed policy w/ PLO
11.00 = 0.00
9.67 = 0.17
0.40 = 0.37
10.67 + 0.12
9.36 0.35
0.30 = 0.42
action guidance- short adaptation
11.00 0.01
2.95 4.48
-0.06 + 0.04
action guidance- short adaptation w/ PI.O
11.00 + 0.00
9.48 0.51
-0.05
In contrast, the main policy of action guidance short adaptation w/ PLO are initially guided by the shaped reward agent during the shift period. During the adaptation period, we find the main policy starts to optimize against the real objective by producing the first combat unit as soon possible. This disrupts the behavior learned from the auxiliary policy and thus cause a visible degrade in the main policy's performance during 1M and 2M timesteps as shown in Figure 2 As the adaption period comes to an end, the main policy becomes fully independent and learn to produce combat units and harvest resources concurrently. This behavior matches the common pattern observed in professional RTS game players and is obviously more desirable because should the enemy attack early, the agent will have enough combat units to defend.
In the DefcaaRaandmmEemy task, the agents trained with shaped rewards learn a variety behav- iors; some of them learn to do worker rush while others learn focus heavily harvesting resources and producing units. This is likely because the agents could get similar level of shaped rewards despite having diverse set of behaviors. loogg adaptation w/ PLO would start optimizing the sparse reward after the shift period ends; almost always learns do a worker rush, which an efficient way to win against a random enemy as shown in Figure !
comparison, the main policy of action guidance
The hyper-parametter adaptation and shift matter: Although the agents trained with action guid- ance short adaptation w/ PLO learns the more desirable behavior, they perform considerably worse the harder task of DefeatRandomEnemy. adaptation should be set. However, in ProduceCombatUnigs agents trained with action guidance long adaptation w/ PLO exhibits the same category of behavior as agents trained with shaped reward, where the agent would only start producing combat units once all the resources have been harvested. reasonable explanation is that higher adaptation gives more guidance to the main policy to consis- tently find the sparse reward, but it also inflicts more bias on how the task should be accomplished; lower adaption gives less guidance but increase the likelihood for the main policy to find better ways to optimize the sparse rewards.
suggests the harder that task is perhaps the longer
7

---

Under review as a conference paper at ICLR 2021
1:145 9.9 1823 11400.0
MNG F.0 Las 21 2100000
(a) shaped reward
(b) action guidance
hhttps://streamne htt.:lltt.cooytptt70000
hhttps://strcamate
The screenshot shows the typical learned behavior of agents in the task of ProduceCom- batUnits. (a) shows an agent trained with shaped reward function RA, learn to only produce combat units once the resources are exhausted (i.e. it produces three combat units at = 1410). In contrary, (b) shows agent trained with action guidance learn to produce units and harvest resources con- currently (i.c. it produces three combat units at = 890). Click on the link below figures to see the full videos of trained agents.
Figure 3:
Positive Learning Optimization results show large variance. We found PLO to be interesting yet sometimes effective optimization in stabilizing the performance for agents trained with action guidance. However, the results show large variance: PLO either significantly helps the agents or make them much worse. As a motivating example, Figure 2 showcases the actual sparse return of 10 seeds in ProduceCombatUnnc PLOO seem to always converge while agents trained without PLO would only sometimes converge. However, PI.O actually hurt the performance of action guidance long adaptation in ProduceCom- batUnits by having a few degenerate runs as shown in Figure 2 It is also worth noting the PLO does not help the sparse reward agent at all, suggesting PLO an optimization somewhat unique to action guidance.
where agents trained with action guidance short adaptation and
Action guidance- mixed policy is viable. According Table I agents trained with action guid- ance mixed policy with or without PLO seem to perform relatively well in all three tasks examined. This is interesting discovery because suggests action guidance could go both ways: the auxiliary policies could also benefit from the learned policies of the main policy. An alternative perspective is to consider the main policy and the auxiliary policies as a whole entity that mixes different reward functions, somehow making joint decision and collaborating to accomplish common goals.
5 CONCLUSIONS
In this paper, we present novel technique called action guidance that successfully trains the agent to eventually optimize over sparse rewards yet does not lose the sample efficiency that comes with reward shaping, effectively getting the best of both worlds. Our experiments with DefeatRandomEn- emy in particular show it is possible to train a main policy on the full game μRTS using only the match outcome reward, which suggests action guidance could serve promising alternative to the training paradigm of AlphaStar (Vinyals et al. 2019) that uses supervised learning with human replay data bootstrap an agent. As part of future work, we would like scale up the approach to defeat stronger opponents.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Picter Abbeel, and Wojeiech Zaremba. Hindsight experience play. In Advances in neural information processing systems, pp. 5048-5058, 2017.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15. pp. 103-129, 1995.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in neural information processing systems, pp. 1471-1479, 2016.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pond'e e de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. ArXiv, abs/1912.066800, 2019.
Greg Brockman, Vicki Cheung. Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01544 2016.
Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity- learning. In ICLR, 2019.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint arXiv:1205.4883 2012.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. 2017.
https:
// //github.cco htt.uub...oo//peaa//aaaseeeeeeee
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. Journal of artificial intelligence research, 13:227-303. 2000.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras. Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study ppo and trpo. In International Conference on Learning Representations, 2019.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Curiosity-
driven exploration in deep reinforcement learning via bayesian neural networks. 2016.
Shengyi Huang and Santiago Ontañón. A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006.14177 14171, 2020.
Shengyi Huang and Santiago Ontañón. Comparing observation and action representations for deep reinforcement learning in prts. 2019.
Anssi Kanervisto, Christian Scheller, and Ville Hautamäki. Action space shaping in deep reinforce- ment learning, arXiv preprint arXiv:2004. 009800 2020.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 198869442..6900 2014.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Manuel Lopes, Tobias Lang, Marc Toussaint, and Pierre-Y Oudeyer. Exploration in model- based reinforcement learning by empirically estimating learning progress. Advances in neural information processing systems, pp. 206-214,200 2012.

---

Under review as a conference paper at ICLR 2021
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th interaational conference on machine learning (ICML-10), pp. 807-814, 2010.
Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. arXiv preprint arXiv:/812.00285. 2018.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. 1999.
Zhen-Jia Pang, Ruo-Ze Liu, Zhou- Yu Meng, Yi Zhang, Yang Yu, and Tong Lu. On reinforcement learning for full-length game of starcraft. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 4691 10991,99,,.9 2019.
Deepak Pathak, Pulkit Agrawal, Alexci A. Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In ICML, 2017.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint arXiv: 10567, 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference machine learning, pp. 1312-13200 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 00w::1006002438. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06777 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Maxwell Svetlik, Matteo Leonetti, Jivko Sinapov, Rishi Shah, Nick Walker, and Peter Stone. Au- tomatic curriculum graph generation for reinforcement learning agents. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Matthew E. Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for tem- poral difference learning. J. Mach. Learn. Res., 8:2125-2167, 2007.
Oriol Vinyals, Timo Ewalds. Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets,
Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint
2017.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnekk, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforeement learning. Nature, 575(7782)350-33555 2019.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint arXiv:1611.01224 2016
Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforce- ment learning. In AAAI, pp. 6672-6679, 2020.
10


Reviews:
[{"review_id": "l-XbIXd6Ccb", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-----------------------------------------\nPost-Rebuttal\n-----------------------------------------\nI believe both the strong points and the weaknesses I pointed out in my review remain valid. Therefore I am keeping my score unchanged.\nIn case of acceptance, I suggest to the authors to carefully address all the comments from my review and the other ones in the camera-ready version (especially the issues related to clarity.).\n\n----------------------------------------\nThe authors propose a reward shaping method to learn faster a reinforcement learning task. Their method consists of learning multiple policies, one optimized on the real reward, while each of the other policies is optimized on a different reward shaping function.\n\n---------------------\n# Pros\n\n- relevant research topic\n- paper is well-written and generally easy to understand\n- novel approach - I can't remember another approach that learns multiple policies from multiple shaping rewards and balance between them\n- Experimental results seem good\n- Codification freely available\n\n--------------------------------------\n# Cons\n\n- Could have included more related works in the experimental evaluation\n\n\n----------------------------\n# Further comments\n\n- I wouldn't say that you train multiple \"agents\", one from each reward shaping function. A more appropriate name would be saying that you train multiple \"policies\", each of them maximizing each of the \"reward signals\". Then, the connection with Probabilistic Policy Reuse would become more obvious (btw, you should discuss the relation of your work with it)\n\nFernández, Fernando, and Manuela Veloso. \"Probabilistic policy reuse in a reinforcement learning agent.\" Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems. 2006.\n\nUsually, when you are trying another \"agent\" you are implying that this agent will explore by itself, which is not true for your method. You should read more about that (and discuss the relation with it) in the survey below:\n\nSilva, Felipe Leno, and Anna Helena Reali Costa. \"A survey on transfer learning for multiagent reinforcement learning systems.\" Journal of Artificial Intelligence Research 64 (2019): 645-703.\n\n- Section 1 could be improved by more explicitly discussing the difference between the current paper and each category of related works. In special, I missed a more comprehensive discussion of reward shaping approaches and the inclusion of state-of-the-art reward shaping algorithms in the experimental evaluation.\n\n- Why did you include a graph in Figure 2 in which \"shaped reward\" performed better than all others? You should perhaps also included the number of time steps used for each approach to converge to the best performance because sample complexity is as important as asymptotic performance.\n\n- It is not very clear to me exactly how the agent selects which policy to follow at every step. Maybe it would be better to have an algorithm in the manuscript.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gO2kB4_BZJM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an approach called action guidance, made to address issues in more standard applications of reward shaping. The main idea of their approach is that there are two different kinds of agents, one (auxiliary agents) that learn from shaped reward functions alone and the other (main agent(s)) that learn only from the actual sparse rewards. The authors made use of a simplified RTS domain and demonstrated that their approach outperformed a more naive shaped reward approach. In addition they demonstrated an ablation study on positive learning optimization. \n\nThe basic idea of this paper is simple and elegant, and the paper is well-written. The results are somewhat messy, but overall represent a positive signal for this approach. The paper is also well-written and the video examples are effective at conveying the results. \n\nI have a number of concerns with this current paper draft. First, it’s unclear to me why the authors chose this RTS environment or why the authors didn’t try several different environments to show the generality of this approach. Further, the evaluation results seem somewhat inconsistent. The action guidance agents overall do quite well but the shaped reward baseline has comparable performance on the first two tasks. This further leads me to believe that this may not be the best environment to test this approach. It’s also somewhat disappointing that the PLO results are inconclusive, this indicates that the work is perhaps still somewhat immature. \n\nOverall I lean slightly towards acceptance. I think that the basic idea here is potentially very impactful. However, my concerns listed above hold me back from stronger support for the paper in its current state. \n\nQuestions for the authors: \n1. Why did the authors choose this RTS environment?\n2. How did the authors account for the performance of the shaped reward baseline for the first two tasks?\n3. Do the authors not have a clearer sense of why the PLO results were inconclusive? Or what experiments could be run to delve into this further?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fzRqAN1nl0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed an algorithm called action guidance that trains the agent to eventually optimize over sparse rewards while maintaining most of the sample efficiency that comes with reward shaping. The authors examine three sparse reward tasks with a range of difficulties to prove the effectiveness of action guidance.  However, I don't think this paper is fully prepared for submission as the method is not novel enough and there exist some possible issues that need to be discussed and resolved. \n\nBelow are the detail comments. \n\nAbout the method. The key idea behind action guidance is to create a main agent that trains on the sparse rewards, and creating some auxiliary agents that are trained on shaped rewards. And the main agent follows the instructions of auxiliary agents in the initial stage and the probability of it decreases during the following training. A concern is that if there exist several auxiliary agents, how do you arrange the shaped rewards to each auxiliary agent? If there is a conflict between the shaped rewards for the training and guidance of the agent, will the main agent still be trained well? Besides，the method itself is like using imitation learning to obtain initial policy parameters and continues to optimize using sparse reward, the novelty of the method is not sufficient enough.\n\nAbout the experiments. The baselines use PPO to train agents with sparse rewards or shaped rewards respectively and there are no other SOTA methods designed for sparse rewards compared in the experiments, which is not convinced. Besides, in the environment ProduceCombatUnits, the shaped rewards include the reward for each combat unit the agent produces, which is exactly the sparse reward. Is it means that the agent using shaped rewards has the same optimization direction as the one using sparse rewards? I'm not sure if this is fair enough as the effectiveness of action guidance is not clear in this setting. Lastly, the random opponents in the experiments are not strong, I'm wondering about the agent's performance in a harder setting.\n\nAbout the writing. The paper is well-written and self-contained. However, the figures to show the typical learned behavior of agents are not clear enough. For example, it's a little bit hard to recognize the enemy units as the blue borders are too thin.\n\nOverall, I vote for a  rejection. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6pCUr0ciBO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an approach for learning policies across multiple MDPs and using those policies to improve learning performance on the task that the agent designer cares about. The approach assumes that a set of MDPs are provided to the learning agent, and that all of the MDPs have the same underlying task but with different reward densities (i.e., some of these MDPs have shaped rewards, and thus are faster to learn from). The approach operates by training the main agent to imitate the actions chosen by the other agents that are trained on the MDPs with shaped reward functions.\n\nOverall, the approach is interesting and can be applicable to multi-task learning benchmarks, even though in its current presentation the authors do not focus on those settings.\n\nPros:\nThe paper is well-written.\nThe presentation of the idea is clear.\nThe experiment section and the results are easy to understand.\n\nCons:\nIt seems like the overall contribution seems small, as the approach assumes access to many MDPs with different reward densities. Given such an assumption, it is natural to understand why the current approach works. \nThe scope of the current work is limited to RTS domains. \nA concise description of the algorithm seems to be missing, making it difficult to understand the overall algorithm.\n\nQuestions: \n1.The approach relies on having access to many MDPs. It would be useful to describe the MDPs that the authors have considered for their experiments?\n2. It seems like the different policies can be different only if their corresponding reward functions are different. Is that right? If so, in many domains, the approach relies on careful design of different MDPs to get the approach to work. The authors need to discuss these assumptions made by their approach.\n3. From the experiments, it seems like the baseline agent with shaped rewards produces the same asymptotic performance as that of the agent with action guidance. If so, then the authors need to justify the use of additional computation for learning the many policies for action guidance.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary:
This paper proposes an interesting idea where additional auxiliary tasks allow an agent to more quickly learn in a sparse reward task.

Comments: 
* The authors may want to look at "Parallel Multi-Environment Shaping Algorithm for Complex Multi-step Task"
https://www.sciencedirect.com/science/article/abs/pii/S092523122030655X
as it has a somewhat related idea and is also in RTS games.
* It wasn't clear to me how the auxiliary tasks were generated/selected, or how the algo would work if a poor auxiliary task was used.
* I wasn't sure why SAC-X wasn't empirically compared to in a domain where it and this method could both apply.

Discussion:
The reviewers agreed that this paper could be significantly improved in multiple dimensions.

Recommendation:
I recommend we reject this paper. However, I encourage the authors to work to improve it as I'd really like to understand where and why this method is successful --- I would eventually like to incorporate it into my own work.
Decision: Reject