Question: How does shaper learn to tackle "Red-herrings?
Evidence: Reinforcement learning (RL) offers the potential for autonomous agents to learn complex behaviours without the need for human intervention. Despite the notable success of RL in a variety domains, enabling RL algorithms to learn successfully in numerous real-world tasks remains a challenge. A key obstacle to the success of RL algorithms is the requirement of a rich reward signal that can guide the agent towards an optimal policy. In many settings of interest such as physical tasks and video games, rich informative signals of the agent's performance are not readily available. For example, in the video game Super Mario, the agent must perform sequences of hundreds of actions while receiving no rewards for it to successfully complete its task. In this setting, the sparse reward provides infrequent feedback of the agent's performance. This leads to RL algorithms requiring large numbers of samples (and high expense) for solving problems. Consequently, there is great need for RL techniques that solve these problems efficiently.
Various biological systems have mechanisms that produce rewards for activities that present little or no direct biological utility. Such mechanisms, which have been fashioned over millions of years are designed to provide intrinsic rewards to promote behaviours that improve chances of outcomes with biological utility. In RL, reward shaping (RS) is a tool to introduce intrinsic rewards known as shaping rewards that supplement environment reward signals. These rewards can encourage exploration and insert structural knowledge in the absence of informative environment rewards which can improve learning outcomes. In general however, RS algorithms assume hand-crafted and domainspecific shaping functions whose construction is typically highly labour intensive. This runs contrary to the aim of autonomous learning. Moreover, poor choices of shaping rewards can worsen the agent's performance. To resolve these issues, a useful shaping reward must be obtained autonomously.
Inspired by naturally occurring systems, we tackle the problem of sparse and uninformative rewards by developing a framework that autonomously constructs shaping rewards during learning. Our framework, ROSA, works by introducing an additional RL agent, Shaper, that adaptively learns to construct shaping rewards by observing Controller (the agent whose goal is to solve the environment task) while Controller learns to solve its task. This generates tailored shaping rewards without the need for domain knowledge or manual engineering. The shaping rewards supplement the environment reward and promote effective learning, our framework therefore addresses the key challenges in RS.
The resulting framework is a two-player nonzero-sum Markov game (MG) -an extension of Markov decision process (MDP) that involves two independent learners with distinct objectives. In our framework, the two agents that have distinct learning agendas, cooperate to achieve Controller's objective. This MG formulation confers various advantages:
1) The shaping-reward function is constructed fully autonomously. The game also ensures the shaping reward improves Controller's performance unlike RS methods that can lower performance.
2) By learning the shaping-reward function while Controller learns its optimal policy, Shaper learns to adaptively facilitate Controller's learning and improve Controller's performance.
3) Both learning processes converge so Controller learns the optimal value function for its task. 4) ROSA learns subgoals to decompose complex tasks and promote complex exploration patterns.
An integral component of ROSA is a novel combination of RL and switching controls. This enables Controller to quickly determine useful states to learn to add and calibrate shaping rewards (i.e. the states in which adding shaping rewards improve Controller's performance) and disregard others. This is in contrast with an RL controller (i.e. Controller) which must learn its best actions at every state. This leads to Shaper quickly finding shaping rewards that guide Controller's learning process toward optimal trajectories (and away from suboptimal trajectories, c.f. Experiment 1).
For our two-player framework to succeed we have to overcome several obstacles. Solving MGs involves finding a stable point in which each player responds optimally to the actions of the other. In our MG, this stable point describes a pair of policies for which Shaper introduces a shaping reward that improves performance and, with that, Controller executes an optimal policy for the task. Tractable methods for solving MGs are rare with convergence of MG methods being seldom guaranteed except in a few special cases. Nevertheless, using special features in the design of our game, we prove the existence of a stable point solution of our MG. We then prove the convergence of our learning method to the solution of the game and show that the solution coincides with the solution of the Controller's problem. This ensures Shaper learns a shaping-reward function that improves Controller's performance and that Controller learns the optimal value function for the task.
The resulting framework is a two-player nonzero-sum Markov game (MG) [34] – an extension of Markov decision process (MDP) that involves two independent learners with distinct objectives. In our framework, the two agents that have distinct learning agendas, cooperate to achieve Controller’s objective. 1) The shaping-reward function is constructed fully autonomously. The game also ensures the shaping reward improves Controller’s performance unlike RS methods that can lower performance. 2) By learning the shaping-reward function while Controller learns its optimal policy, Shaper learns to adaptively facilitate Controller’s learning and improve Controller’s performance. 3) Both learning processes converge so Controller learns the optimal value function for its task. 4) ROSA learns subgoals [28] to decompose complex tasks and promote complex exploration patterns.
Answer: Thanks to the reviewer for this question. This highlights a major benefit of the two-player framework. As we explain on Pg. 2, shaper has its own distinct objective which seeks to improve the return for controller and, its own policy. This enables Shaper to introduce shaping rewards that promote complex exploration patterns. Crucially, the switching control aspect of Shaper’s intervention means that it can choose not to add shaping rewards at states that are not helpful.