Evaluating Agents Without Rewards

1 INTRODUCTION Metric Reward Correlation Task Reward 1.00 Human Similarity 0.67 Input Entropy 0.54 Information Gain 0.49 Empowerment 0.41 Metric Human Correlation Human Similarity 1.00 Input Entropy 0.89 Information Gain 0.79 Task Reward 0.67 Empowerment 0.66 Deep reinforcement learning ( RL ) has enabled agents to solve complex tasks directly from high-dimensional image inputs , such as locomotion ( Heess et al. , 2017 ) , robotic manipulation ( Akkaya et al. , 2019 ) , and game playing ( Mnih et al. , 2015 ; Silver et al. , 2017 ) . However , many of these successes are built upon rich supervision in the form of manually defined reward functions . Unfortunately , designing informative reward functions is often expensive , time-consuming , and prone to human error ( Krakovna et al. , 2020 ) . Furthermore , these difficulties increase with the complexity of the task of interest . In contrast to many RL agents , natural agents generally learn without externally provided tasks , through intrinsic objectives . For example , children explore the world by crawling around and playing with objects they find . Inspired by this , the field of intrinsic motivation ( Schmidhuber , 1991 ; Oudeyer et al. , 2007 ) seeks mathematical objectives for RL agents that do not depend on a specific task and can be applicable to any unknown environment . We study three common types of intrinsic motivation : Despite the empirical success of intrinsic motivation for facilitating exploration ( Bellemare et al. , 2016b ; Burda et al. , 2018b ) , it remains unclear which family of intrinsic objectives is best for a given scenario , for example when task rewards are sparse or unavailable , or when the goal is to behave similarly to human actors . Moreover , it is not clear whether different intrinsic objectives offer similar benefits in practice or are orthogonal and should be combined . To spur progress toward better understanding of intrinsic objectives , we empirically compare the three objective families in terms of their correlation with human behavior and with the task rewards of three Atari games and Minecraft Treechop . The goal of this paper is to gain understanding rather than to propose a new intrinsic objective or exploration agent . Therefore , there is no need to estimate intrinsic objectives while the agents are learning , which often requires complicated approximations . Instead , we train several well-known RL agents on three Atari games and Minecraft and store their lifetime datasets of experience , resulting in a total of 2.1 billion time steps and about 9 terabytes of agent experience . From the dataset of each agent , we compute the human similarity , input entropy , empowerment , and infogain using simple estimators with clearly stated assumptions . We then analyze the correlations between these metrics to understand how they relate to another and how well they reflect task reward and human similarity . The key findings of this paper are summarized as follows : • Input entropy and information gain both correlate better with human similarity than task reward does . This implies that to measure how similar an agent ’ s behavior is to human behavior , input entropy is a better approximation than task reward . • Simple implementations of input entropy , information gain , and empowerment correlate well with human similarity . This suggests that they can be used as task-agnostic evaluation metrics when human data and task rewards are unavailable . • As a consequence of the these two findings , task-agnostic metrics can be used to measure a different component of agent behavior than is measured by the task rewards of the reinforcement learning environments considered in our study . • Input entropy and information gain correlate strongly with each other , but to a lesser degree with empowerment . This suggests that optimizing input entropy together with either of the two other metrics could be beneficial for designing exploration methods . This paper is structured as follows . Section 2 describes the games and agents used for the study . Section 3 details the experimental setup and estimators used to implement the metrics . Section 4 discusses quantitative and qualitative results . Section 5 summarizes key take-aways and recommendations . 2 BACKGROUND . To validate the effectiveness of our metrics for task-agnostic evaluation across a wide spectrum of agent behavior , we retrospectively computed our metrics on the lifetime experience of well-known RL agents . Thus , we first collected datasets of a variety of agent behavior on which to compute and evaluate our metrics . Environments We evaluated our agents in three different Atari environments provided by Arcade Learning Environment ( Bellemare et al. , 2013 ) : Breakout , Seaquest , and Montezuma ’ s Revenge , and additionally in the Minecraft Treechop environment provided in MineRL ( Guss et al. , 2019 ) . Breakout and Seaquest are relatively simple reactive environments , while Montezuma ’ s Revenge is a challenging platformer requiring long-term planning . Treechop is a 3D environment in which the agent receives reward for breaking and collecting wood blocks , but has considerable freedom to explore the world . We chose these four environments because they span a range of complexity , freedom , and difficulty , as detailed in Appendix E. Agents The seven agent configurations represented in our dataset include three RL algorithms and two trivial agents for comparison . We selected RL agents spanning the range from extrinsic task reward only to intrinsic motivation reward only . Additionally , we included random and no-op agents , two opposite extremes of trivial behavior . Our goal was to represent a wide range of behaviors : playing to achieve a high score , playing to explore the environment , and taking actions without regard to the environment . Specifically , we used the PPO agent ( Schulman et al. , 2017 ) trained to optimize task reward , and the RND ( Burda et al. , 2018b ) and ICM ( Pathak et al. , 2017 ) exploration agents using PPO as an optimizer , which can incorporate both an intrinsic reward signal , and an extrinsic reward signal which can be enabled for task-specific behavior , or disabled for task-agnostic behavior . We evaluate RND and ICM in both of these configurations . Each agent is summarized in Appendix F . 3 METHOD . The goal of this paper is to evaluate agents using metrics other than task reward . For this , we collected 100 million frames on each of the three Atari environments with each of seven agents : random , no-op , PPO , and RND and ICM with and without task reward . Minecraft was evaluated for 12 million frames per agent because the simulation is slower than the Atari games , and five agents rather than seven were used , excluding both configurations of ICM . To evaluate the agents , we first preprocessed the data and then computed our metrics in aggregate over the entire lifetime of each agent/environment configuration ( yielding one number per metric-agentenvironment ) . In this section , we describe our preprocessing method and introduce our estimators for the five considered metrics . 3.1 PREPROCESSING . To efficiently compute the metrics in a robust manner , we discretize the agent ’ s input images so that they can be represented by integer indices . This allows us to summarize each collected datasets as a probability tensor that holds the probability of each transition . Figure 1 visualizes the preprocessing for the three environments . Discretization We first convert the RGB images to grayscale as they were seen by the agents . After that , we bilinearly resize them to 8 × 8 pixels . We discretize these low-resolution images to four possible values per pixel , with thresholds chosen as the brightness percentiles 25 , 50 , and 75 across all unique values of the corresponding pixel in the environment across all agents . Aggregation The unique compactified images are enumerated and summarized into a tensor of counts for each agent and environment combination . For an image index 1 ≤ i ≤ |X| , and action index 1 ≤ j ≤ |A| , and a successor image index 1 ≤ k ≤ |X| , where X is the set of inputs and A the set of actions , the count tensor is defined as , Nijk . = number of transitions from image bucket i and action j to image bucket k. ( 1 ) Normalizing the count tensor N yields a probability tensor P that stores the probability of each transition in the agent ’ s dataset . Under the assumption of a Markovian environment and agent , the probability tensor fully describes the statistics of the preprocessed dataset , P . = 1∑ ijkNijk N , so that ∑ ijk Pijk = 1 . ( 2 ) The probability tensor P describes the joint probability of transitions for each agent and environment and thus allows us to compute any marginals and conditionals needed to compute the metrics . 3.2 METRICS . We compare two task specific metrics , task reward and human similarity , as well as three task-agnostic metrics : input entropy , information gain , and empowerment . The task-agnostic metrics cover the different types of objectives identified by Hafner et al . ( 2020 ) . We compute a single value of each of these metrics on each agent-environment dataset . Task Reward The reward provided by reinforcement learning environments measures success at a specific task . The environments we use have only one predefined task each , despite the wide range of conceivable objectives in Montezuma ’ s Revenge and Minecraft in particular . This is true of many RL environments , and limits one ’ s ability to analyze the behavior of an agent in a general sense within one environment . While there are multi-task benchmarks , they often include a distinct environment for each task rather than multiple tasks in the same environment ( Yu et al. , 2019 ) . This would make it difficult to evaluate the agent ’ s ability to globally explore its environment independent of the task . Human Similarity Task reward captures only the agent ’ s success at the specific task defined via the reward function . This may not match up with a human observer ’ s definition of intelligence . We suggest that a more general measure of intelligence may relate to similarity between the agent ’ s behavior and human behavior in the same environment , i.e . using human behavior as a “ ground truth ” . Hence , we propose a human similarity metric that approximates the overlap between the inputs observed by a human player and an RL agent . To compute this , we used the Atari-HEAD dataset ( Zhang et al. , 2019 ) and preprocessed the data as described in Section 3.1 . We compute human similarity as the Jaccard index , or intersection over union , between input images encountered in the human dataset and those encountered by the artificial agent . It is simpler than but related to inverse reinforcement learning objectives in the vein of Ziebart et al . ( 2008 ) ; Klein et al . ( 2012 ) . For this , we first compute the empirical visitation probabilities of inputs from the probability tensors P agent and P human of the artificial agent and the human player , respectively . The human similarity is then computed as the fraction of non-zero entries , S . = ∣∣ { i : p ( i ) > 0 ∧ q ( i ) > 0 } ∣∣∣∣ { i : p ( i ) > 0 ∨ q ( i ) > 0 } ∣∣ , where p ( i ) .=∑ jk P agentijk , q ( i ) . = ∑ jk P humanijk . ( 3 ) Note that while we use input images from recorded human behavior to compute human similarity , we are not able to compare the human and agent behavior directly , as the RL agents play in an environment with sticky actions , while the humans did not . More generally , human similarity would be challenging to compute in some environments , such as high-dimensional continuous control . Even where it is possible , large human datasets are expensive to collect . Thus , we consider three taskagnostic metrics , which do not require environment-specific engineering or human demonstrators . Input entropy Input entropy measures how improbable the agent ’ s inputs are under a trained density model ( Schmidhuber , 1991 ) . It has been used in RL to model an agent ’ s success at exploring its environment , where a higher input entropy score implies a wider variety of states observed , for instance by Oudeyer et al . ( 2007 ) ; Bellemare et al . ( 2016a ) ; Burda et al . ( 2018b ) . In general , input entropy is the cross entropy of future inputs under a density model trained alongside the agent , but as we retrospectively compute the metric over the agent ’ s lifetime , we only need to compute the probability vector over inputs once . Specifically , we use the marginal entropy over individual inputs , computed by summing the probability tensor over the second and third axes as , C . = − ∑ i p ( i ) log p ( i ) , where p ( i ) .= ∑ jk P agentijk . ( 4 ) Empowerment Empowerment measures the agent ’ s influence over its environment ( Klyubin et al. , 2005 ) , which has been applied to reinforcement learning by Mohamed and Rezende ( 2015 ) ; Karl et al . ( 2017 ) . We use the interpretation of empowerment that measures the agent ’ s realized influence on the world , rather than its potential influence , formalized as the mutual information between actions and sensory inputs ( Salge et al. , 2014 ; Hafner et al. , 2020 ) . We compute empowerment mutual information as the difference between the entropy of actions given the preceding input , before and after observing the following input , E . = ( ∑ ijk p ( i , j , k ) log p ( i , j , k ) ) − ( ∑ ij q ( i , j ) log q ( i , j ) ) , where p ( i , j , k ) .= Pijk∑ j′ Pij′k , q ( i , j ) . = ∑ k Pijk∑ j′k Pij′k . ( 5 ) Information gain Information gain is a measure of how much the agent learns from its observations Lindley et al . ( 1956 ) . It is the mutual information between observations and the agent ’ s representation of the environment . Information gain has led to successful exploration in reinforcement learning ( Sun et al. , 2011 ; Houthooft et al. , 2016 ; Shyam et al. , 2018 ) . To measure the amount of information gained , we need a way to represent the agent ’ s knowledge about its environment . Preprocessing the agent ’ s inputs into discrete classes enables us to represent its knowledge as a belief over the transition matrix M . The total information gain of over agent ’ s lifetime is the entropy difference of its beliefs at the beginning and end of the dataset , I . = H [ M ] −H [ M | dataset ] = E [ log p ( M | dataset ) − log p ( M ) ] . ( 6 ) We choose the belief to be a vector of Dirichlet distributions as in ( Sun et al. , 2011 ; Friston et al. , 2017 ) , each of which has one concentration parameter , initialized at 1 , for each possible subsequent input . For each transition that occurs in the dataset , the corresponding concentration parameter is set to 2 , which we empirically found to be more effective than varying the parameter based on the number of occurrences . This is explained by the limited amount of stochasticity in the environments even after pre-processing ; more than 80 % of all state/action pairs transition to only one unique state in Breakout , Seaquest , and Minecraft , so that seeing a transition once establishes that the transition is possible , and seeing it multiple times should not increase the model ’ s confidence by very much . The entropy of a Dirichlet distribution is given by , H [ D ] = logB ( α ) + ( |X|∑ k=1 α− |X| ) ψ ( |X|∑ k=1 α ) − |X|∑ k=1 ( αk − 1 ) ψ ( αk ) , ( 7 ) where α is the vector of concentration parameters , ψ is the digamma function , and B is the incomplete beta function ( Lin , 2016 ) . The entropy of the distribution over M is then given by adding up the entropy of each of the Dirichlet distributions .

Summary: This work studies four task-agnostic metrics for evaluating reinforcement learning agents: human similarity, curiosity, empowerment and information gain. Experiments were conducted with three selected RL algorithms (PPO, ICM and RND) on selected atari games. The results show that a combination of task reward and curiosity better explain human behavior and some non-reward metrics correlate better with human behavior than task reward. The authors propose that such task-agnostic can be used as intrinsic signals for training RL agents when task reward and human data are not available in an environment.