Unsupervised Control Through Non-Parametric Discriminative Rewards

1 INTRODUCTION . Currently , the best performing methods on many reinforcement learning benchmark problems combine model-free reinforcement learning methods with policies represented using deep neural networks ( Horgan et al. , 2018 ; Espeholt et al. , 2018 ) . Despite reaching or surpassing human-level performance on many challenging tasks , deep model-free reinforcement learning methods that learn purely from the reward signal learn in a way that differs greatly from the manner in which humans learn . In the case of learning to play a video game , a human player not only acquires a strategy for achieving a high score , but also gains a degree of mastery of the environment in the process . Notably , a human player quickly learns which aspects of the environment are under their control as well as how to control them , as evidenced by their ability to rapidly adapt to novel reward functions ( Lake et al. , 2017 ) . Focusing learning on mastery of the environment instead of optimizing a single scalar reward function has many potential benefits . One benefit is that learning is possible even in the absence of an extrinsic reward signal or with an extrinsic reward signal that is very sparse . Another benefit is that an agent that has fully mastered its environment should be able to reach arbitrary achievable goals , which would allow it to generalize to tasks on which it wasn ’ t explicitly trained . Building reinforcement learning agents that aim for environment mastery instead of or in addition to learning about a scalar reward signal is currently an open challenge . One way to represent such knowledge about an environment is using an environment model . Modelbased reinforcement learning methods aim to learn accurate environment models and use them either for planning or for training a policy . While learning accurate environment models of some visually rich environments is now possible ( Oh et al. , 2015 ; Chiappa et al. , 2018 ; Ha & Schmidhuber , 2018 ) using learned models in model-based reinforcement learning has proved to be challenging and model-free approaches still dominate common benchmarks . We present a new model-free agent architecture of Discriminative Embedding Reward Networks , or DISCERN for short . DISCERN learns to control an environment in an unsupervised way by learning purely from the stream of observations and actions . The aim of our agent is to learn a goal-conditioned policy πθ ( a|s ; sg ) ( Kaelbling , 1993 ; Schaul et al. , 2015 ) which can reach any goal state sg that is reachable from the current state s. We show how to learn a goal achievement reward function r ( s ; sg ) that measures how similar state s is to state sg using a mutual information objective at the same time as learning πθ ( a|s ; sg ) . The resulting learned reward function r ( s ; sg ) measures similarity in the space of controllable aspects of the environment instead of in the space of raw observations . Crucially , the DISCERN architecture is able to deal with goal states that are not perfectly reachable , for example , due to the presence of distractor objects that are not under the agent ’ s control . In such cases the goal-conditioned policy learned by DISCERN tends to seek states where the controllable elements match those in the goal state as closely as possible . We demonstrate the effectiveness of our approach on three domains – Atari games , continuous control tasks from the DeepMind Control Suite , and DeepMind Lab . We show that our agent learns to successfully achieve a wide variety of visually-specified goals , discovering underlying degrees of controllability of an environment in a purely unsupervised manner and without access to an extrinsic reward signal . 2 PROBLEM FORMULATION . In the standard reinforcement learning setup an agent interacts with an environment over discrete time steps . At each time step t the agent observes the current state st and selects an action at according to a policy π ( at|st ) . The agent then receives a reward rt = r ( st , at ) and transitions to the next state st+1 . The aim of learning is to maximize the expected discounted return R = ∑∞ t=0 γ trt of policy π where γ ∈ [ 0 , 1 ) is a discount factor . In this work we focus on learning only from the stream of actions and observations in order to forego the need for an extrinsic reward function . Motivated by the idea that an agent capable of reaching any reachable goal state sg from the current state s has complete mastery of its environment , we pose the problem of learning in the absence of rewards as one of learning a goal-conditioned policy πθ ( a|s ; sg ) with parameters θ . More specifically , we assume that the agent interacts with an environment defined by a transition distribution p ( st+1|st , at ) . We define a goal-reaching problem as follows . At the beginning of each episode , the agent receives a goal sg sampled from a distribution over possible goals pgoal . For example , pgoal could be the uniform distribution over all previously visited states . The agent then acts for T steps according to the goal-conditioned policy πθ ( a|s ; sg ) receiving a reward of 0 for each of the first T − 1 actions and a reward of r ( sT ; sg ) after the last action , where r ( s ; sg ) ∈ [ 0 , 1 ] for all s and sg 1 . The goal achievement reward function r ( s ; sg ) measures the degree to which being in state s achieves goal sg . The episode terminates upon the agent receiving the reward r ( sT ; sg ) and a new episode begins . It is straightforward to train πθ ( a|s ; sg ) in a tabular environment using the indicator reward r ( s ; sg ) = 1 { s = sg } . We are , however , interested in environments with continuous highdimensional observation spaces . While there is extensive prior work on learning goal-conditioned policies ( Kaelbling , 1993 ; Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Held et al. , 2017 ; Pathak et al. , 2018 ) , the reward function is often hand-crafted , limiting generality of the approaches . In the few cases where the reward is learned , the learning objective is typically tied to a pre-specified notion of visual similarity . Learning to achieve goals based purely on visual similarity is unlikely to work in complex , real world environments due to the possible variations in appearance of objects , or goal-irrelevant perceptual context . We now turn to the problem of learning a goal achievement reward function rφ ( s ; sg ) with parameters φ for high-dimensional state spaces . 3 LEARNING A REWARD FUNCTION BY MAXIMIZING MUTUAL INFORMATION . We aim to simultaneously learn a goal-conditioned policy πθ and a goal achievement reward function rφ by maximizing the mutual information between the goal state sg and the achieved state sT as shown in ( 1 ) . I ( sg , sT ) = H ( sg ) + Esg , sT∼p ( sg , sT ) log p ( sg|sT ) ( 1 ) Note that we are slightly overloading notation by treating sg as a random variable distributed according to pgoal . Similarly , sT is a random variable distributed according to the state distribution induced by running πθ for T steps for goal states sampled from pgoal . The prior work of Gregor et al . ( 2016 ) showed how to learn a set of abstract options by optimizing a similar objective , namely the mutual information between an abstract option and the achieved 1More generally the time budget T for achieving a goal need not be fixed and could either depend on the goal state and the initial environment state , or be determined by the agent itself . state . Following their approach , we simplify ( 1 ) in two ways . First , we rewrite the expectation in terms of the goal distribution pgoal and the goal conditioned policy πθ . Second , we lower bound the expectation term by replacing p ( sg|sT ) with a variational distribution qφ ( sg|sT ) with parameters φ following Barber & Agakov ( 2004 ) , leading to I ( sg , sT ) ≥ H ( sg ) + Esg∼pgoal , s1 , ... sT∼πθ ( ···|sg ) log qφ ( sg|sT ) . ( 2 ) Finally , we discard the entropy term H ( sg ) from ( 2 ) because it does not depend on either the policy parameters θ or the variational distribution parameters φ , giving our overall objective ODISCERN = Esg∼pgoal , s1 , ... sT∼πθ ( ···|sg ) log qφ ( sg|sT ) . ( 3 ) This objective may seem difficult to work with because the variational distribution qφ is a distribution over possible goals sg , which in our case are high-dimensional observations , such as images . We sidestep the difficulty of directly modelling the density of high-dimensional observations by restricting the set of possible goals to be a finite subset of previously encountered states that evolves over time ( Lin , 1993 ) . Restricting the support of qφ to a finite set of goals turns the problem of learning qφ into a problem of modelling the conditional distribution of possible intended goals given an achieved state , which obviates the requirement of modelling arbitrary statistical dependencies in the observations.2 Optimization : The expectation in the DISCERN objective is with respect to the distribution of trajectories generated by the goal-conditioned policy πθ acting in the environment against goals drawn from the goal distribution pgoal . We can therefore optimize this objective with respect to policy parameters θ by repeatedly generating trajectories and performing reinforcement learning updates on πθ with a reward of log qφ ( sg|sT ) given at time T and 0 for other time steps . Optimizing the objective with respect to the variational distribution parameters φ is also straightforward since it is equivalent to a maximum likelihood classification objective . As will be discussed in the next section , we found that using a reward that is a non-linear transformation mapping log qφ ( sg|sT ) to [ 0 , 1 ] worked better in practice . Nevertheless , since the reward for the goal conditioned-policy is a function of log qφ ( sg|sT ) , training the variational distribution function qφ amounts to learning a reward function . Communication Game Interpretation : Dual optimization of the DISCERN objective has an appealing interpretation as a cooperative communication game between two players – an imitator that corresponds to the goal-conditioned policy and a teacher that corresponds to the variational distribution . At the beginning of each round or episode of the game the imitator is provided with a goal state . The aim of the imitator is to communicate the goal state to the teacher by taking T actions in the environment . After the imitator takes T actions , the teacher has to guess which state from a set of possible goals was given to the imitator purely from observing the final state sT reached by the imitator . The teacher does this by assigning a probability to each candidate goal state that it was the goal given to the imitator at the start of the episode , i.e . it produces a distribution p ( sg|sT ) . The objective of both players is for the teacher to guess the goal given to the imitator correctly as measured by the log probability assigned by the teacher to the correct goal .

Summary: In this paper, the authors address the problem of learning to achieve perceptually specified goals in a fully unsupervised way. For doing so, they simultaneously learn a goal-conditioned policy and a goal achievement reward function based on the mutual information between goals sampled from an a priori distribution and states achieved using the goal-conditioned policy. These two learning processes are coupled through the mutual information criterion, which seems to result in efficient state representation learning for the visual specified goal space. A key feature is that the resulting metrics in the visual goal space helps the agent focus on what it can control and ignore distractors, which is critical for open-ended learning.