Evaluating Agents Without Rewards
Authors: Brendon Matusch
Jimmy Ba
Danijar Hafner
Venue:  (2021)
Keywords: reinforcement learning
task-agnostic
agent evaluation
exploration
information gain
empowerment
curiosity

Abstract: Reinforcement learning has enabled agents to solve challenging control tasks from raw image inputs. However, manually crafting reward functions can be time consuming, expensive, and prone to human error. Competing objectives have been proposed for agents to learn without external supervision, such as artificial input entropy, information gain, and empowerment. Estimating these objectives can be challenging and it remains unclear how well they reflect task rewards or human behavior. We study these objectives across seven agents and three Atari games. Retrospectively computing the objectives from the agent's lifetime of experience simplifies accurate estimation. We find that all three objectives correlate more strongly with a human behavior similarity metric than with task reward. Moreover, input entropy and information gain both correlate more strongly with human similarity than task reward does.

Under review as a conference paper at ICLR 2021
EVALUATING AGENTS WITHOUT REWARDS
Anonymous authors
Paper under double-bling review
ABSTRACT
Reinforcement learning has enabled agents solve challenging control tasks from raw image inputs. However, manually crafting reward functions can be time consuming, expensive, and prone to human error. Competing objectives have been proposed for agents to learn without external supervision, such as artificial input entropy, information gain, and empowerment. Estimating these objectives can be challenging and it remains unclear how well they reflect task rewards or human behavior. We study these objectives across seven agents and three Atari games. Retrospectively computing the objectives from the agent's ' lifetime of experience simplifies accurate estimation. We find that all three objectives correlate strongly with human behavior similarity metric than with task reward. Moreover, input entropy and information gain both correlate more strongly with human similarity than task reward does.
INTRODUCTION
Deep reinforcement learning (RL) has enabled agents to solve complex tasks directly from high-dimensional image inputs, such as locomotion (Heess et al., 2017), robotic manipulation (Akkaya al., 2019), and game playing (Mnih al., 2015; Silver al., 2017). However, many of these successes are built upon rich supervision in the form of manually defined reward functions. Unfortunately, designing informative reward functions is often expensive, time-consuming. and prone to human error (Krakovna et al., 2020). Furthermore, these difficulties increase with the complexity of the task of interest.
Metric Reward Correlation
Task Reward Human Similarity
1.00
0.67
Input Entropy Information Gain
0.54
0.49
Empowerment
0.41
Metric Human Correlation
In contrast to many RL agents, natural agents generally learn without externaly provided tasks, through intrinsic objectives. For example, children explore the world by crawling around and playing with objects they find. Inspired by this, the field of intrin- sic motivation (Schmidhuber, 1991; Oudeyer et al., 2007) seeks mathematical objectives for RL agents that do not depend on a specific task and can be applicable to any unknown environment. We study three common types of intrinsic motivation:
Human Similarity
1.00
Input Entropy Information Gain
0.89
0.79
Task Reward Empowerment
0.67
0.66
Table 1: Correlation coefficients between each metric and task re- ward or human similarity. The 3 task-agnostic metrics correlate more strongly with human simi- larity than with task reward. This suggests that typical RL tasks may not be a sufficient proxy for intelligent behavior seen in hu- mans playing the same games.
Input entropy encourages encountering rare sensory inputs, measured by a learned density model (Schmidhuber, 1990; Bellemare et al., 2016b; Pathak al., 2017; Burda et al., 2018b).
Information gain, or infogain for short, rewards the agent for discovering the rules of its environment (Lindley et al., 1956; Houthooft et al., 2016; Shyam al., 2018; Sekar et al., 2020).
Empowerment measures the agent's influence it has over its sensory inputs or environment (Klyubin et al., 2005; Mo- hamed and Rezende, 2015; Karl et al., 2017).
Despite the empirical success of intrinsic motivation for facilitating exploration (Bellemare et al., 2016b; Burda et al., 2018b), it remains unelear which family of intrinsic objectives is best for given scenario, for example when task rewards are sparse or unavailable, or when the goal is to behave similarly human actors. Moreover, it not clear whether different intrinsic objectives offer

---

Under review as a conference paper at ICLR 2021
similar benefits in practice or are orthogonal and should be combined. To spur progress toward better understanding of intrinsic objectives, we empirically compare the three objective families in terms of their correlation with human behavior and with the task rewards of three Atari games and Minecraft Treechop.
The goal of this paper is to gain understanding rather than to propose a new intrinsic objective or exploration agent. Therefore, there is no need to estimate intrinsic objectives while the agents are learning, which often requires complicated approximations. Instead, we train several well-known RL agents three Atari games and Minecraft and store their lifetime datasets of experience, resulting in a total of 2.1 billion time steps and about 9 terabytes of agent experience. From the dataset of each agent, we compute the human similarity, input entropy, empowerment, and infogain using simple estimators with clearly stated assumptions. We then analyze the correlations between these metrics to understand how they relate to another and how well they reflect task reward and human similarity.
The key findings of this paper are summarized as follows:
Input entropy and information gain both correlate better with human similarity than task reward does. This implies that to measure how similar an agent's behavior is to human behavior, input entropy is better approximation than task reward.
Simple implementations of input entropy, information gain, and empowerment correlate well with human similarity. This suggests that they can be used as fask-agnostic evaluation metrics when human data and task rewards are unavailable.
As a consequence of the these two findings, task-agnostic metrics can be used to measure a different component of agent behavior than is measured by the task rewards of the reinforcement learning environments considered in our study.
Input entropy and information gain correlate strongly with each other, but to a lesser degree with empowerment. This suggests that optimizing input entropy together with either of the two other metrics could be beneficial for designing exploration methods.
This paper is structured as follows. Section 2 describes the games and agents used for the study. Sec- tion 3 details the experimental setup and estimators used to implement the metrics. Section 4 discusses quantitative and qualitative results. Section summarizes key take-aways and recommendations.
BACKGROUND
To validate the effectiveness of our metrics for task-agnostic evaluation across a wide spectrum of agent behavior, we retrospectively computed our metrics on the lifetime experience of well-known RL agents. Thus, we first collected datasets of variety of agent behavior which to compute and evaluate our metrics.
We evaluated our agents in three different Atari environments provided by Arcade Learning Environment (Bellemare et al., 2013): Breakout, Seaquest, and Montezuma's 's Revenge, and additionally in the Minecraft Treechop environment provided MineRL (Guss et al., 2019). Breakout and Seaquest relatively simple reactive environments, while Montezuma's Revenge is a challenging platformer requiring long-term planning. Treechop is a 3D environment in which the agent receives reward for breaking and collecting wood blocks, but has considerable freedom to explore the world. We chose these four environments because they span a range of complexity, freedom, and difficulty, detailed in Appendix E.
Environments
The seven agent configurations represented in our dataset include three RL algorithms and two trivial agents for comparison. We selected RL agents spanning the range from extrinsic task reward only intrinsic motivation reward only. Additionally, we included random and no-op agents, two opposite extremes of trivial behavior. Our goal was to represent a wide range of behaviors: playing to achieve a high score, playing to explore the environment, and taking actions without regard to the environment. Specifically, we used the PPO agent (Schulman et al., 2017) trained to optimize task reward, and the RND (Burda et al., 2018b) and ICM (Pathak et al., 2017) exploration agents using PPO
Agents
optimizer, which can incorporate both intrinsic reward signal, and an extrinsic reward signal which can be enabled for task-specific behavior, or disabled for aggnstin behavior. We evaluate RND and ICM in both of these configurations. Each agent is summarized in Appendix F.
2

---

Under review as a conference paper at ICLR 2021
1820
K3PSS
(a) Breakout
(b) Scaquest
(d) Minecraft
(c) Montezuma
Figure 1: Preprocessing used assign input images buckets. Similar to Go- Go-Explore (Ecoffet et al., 2019), we resize the images to 8 x pixels and discretize each of the resulting cells to one of values. The examples show that this procedure preserves positions of objects in the game, such the player, ball, fish, and skull. We enumerate the compactified images to represent each unique frame by an integer index to compute discrete probability tensors for the environments.
3 METHOD
The goal this paper to evaluate agents using metrics other than task reward. For this, we collected 100 million frames each of the three Atari environments with each of seven agents: random, no-op, PPO, and RND and ICM with and without task reward. Minecraft was evaluated for million frames per agent because the simulation is slower than the Atari games, and five agents rather than seven were used, excluding both configurations ICM.
To evaluate the agents, we first preprocessed the data and then computed our metrics in aggregate over the entire lifetime of each agent/environment configuration (yielding number per metric-agent agent- environment). In this section, we describe our preprocessing method and introduce our estimators for the five considered metrics.
3.1 PREPROCESSING
To efficiently compute the metrics in a robust manner, we discretize the agent's input images so that they can be represented by integer indices. This allows us summarize each collected datasets probability tensor that holds the probability of each transition. Figure visualizes the preprocessing for the three environments.
Discretization We first convert the RGB images to grayscale as they were seen by the agents. After that, bilinearly resize them to 8 pixels. We discretize these wwwresolution images to four possible values per pixel, with thresholds chosen as the brightness percentiles 25, 50, and 75 across all unique values of the corresponding pixel in the environment across all agents.
Aggregation The unique compactified images are enumerated and summarized into a tensor of counts for each agent and environment combination. For an image index 1 < < [X], and action index < j < |A], and successor image index the set of actions, the count tensor is defined as,
[X], where is the set inputs and A
Nijk = number of transitions from image bucket i and action j to image bucket k.

---

Under review as a conference paper at ICLR 2021
Normalizing the count tensor N yields a probability tensor P that stores the probability of each transition in the agent's 's dataset. Under the assumption of a Markovian environment and agent, the probability tensor fully describes the statistics of the preprocessed dataset,
1 N,
so that Σ 1
Pask 1.
Eisk Nisk
(2)
The probability tensor P describes the joint probability of transitions for each agent and environment and thus allows us to compute any marginals and conditionals needed to compute the metrics.
3.2 METRICS
We compare two task specific metrics, task reward and human similarity, well as three task-agnostic metrics: input entropy, information gain, and empowerment. The task-agnostic metrics cover the different types of objectives identified by Hafner al. (2020). We compute a single value of each of these metrics on each agent- enveroonnvr dataset.
Task Reward The reward provided by reinforcement learning environments measures success at specific task. The environments we use have only one predefined task each, despite the wide range of conceivable objectives in Montezuma's Revenge and Minecraft in particular. This is true of many RL environments, and limits one's 's ability to analyze the behavior of an agent in a general sense within one environment. While there are multi-task benchmarks, they often include distinct environment for each task rather than multiple tasks in the same environment (Yu et al., 2019). This would make it difficult to evaluate the agent's ss ability to globally explore its environment independent of the task.
Human Similarity Task reward captures only the agent's 's success at the specific task defined via the reward function. This may not match up with human observer's 's definition of intelligence. We suggest that a more general measure of intelligence may relate to similarity between the agent's s behavior and human behavior in the same environment, i.e. using human behavior as "ground truth". Hence, we propose a human similarity metric that approximates the overlap between the inputs observed by a human player and an RL agent. To compute this, used the Atar dataset (Zhang et al., 2019) and preprocessed the data as described in Section 3.1.
We compute human similarity as the Jaccard index, or intersection over union, between input images encountered in the human dataset and those encountered by the artificial agent. It is simpler than but related to inverse reinforcement learning objectives in the vein of Ziebart et al. (2008); Klein et al. (2012). For this, first compute the empirical visitation probabilities of inputs from the probability tensors pagent and phuman of the artificial agent and the human player, respectively. The human similarity is then computed the fraction of non-zero entries.
{i p(1) 0A (1) > 0}| [ p()
S
where p(t)=
Σ jk
pagent ijk
(10== jk
phuman 12k
(3)
0 ov > 0
Note that while we use input images from recorded human behavior to compute human similarity, not able to compare the human and agent behavior directly, the RL agents play in an environment with sticky actions, while the humans did not. More generally, human similarity would be challenging to compute in some environments, such high-dimensional continuous control. Even where it is possible, large human datasets are expensive to collect. Thus, we consider three task- agnostic metries, which do not require environment-specion engineering or human demonstrators.
Input entropy Input entropy measures how improbable the agent's inputs are under trained density model (Schmidhuber, 1991). It has been used in RL to model an agent's success at exploring its environment, where a higher input entropy score implies a wider variety of states observed, for instance by Oudeyer et al. (2007); Bellemare et al. (2016a); Burda et al. (201 8b).
In general, input entropy is the cross entropy of future inputs under density model trained alongside the agent, but as we retrospectively compute the metric over the agent's lifetime, we only need to compute the probability vector over inputs once. Specifically, we use the marginal entropy over individual inputs, computed by summing the probability tensor over the second and third axes as,
C -
Σ p(i) log logr(i), where p(1) =
Σ j
Aggnt
(4)
ijk
4

---

Under review as a conference paper at ICLR 2021
Empowerment Empowerment measures the agent's influence over its environment (Klyubin et al., 2005), which has been applied to reinforcement learning by Mohamed and Rezende (2015); Karl al. (2017). We use the interpretation of empowerment that measures the agent's 's realized influence on the world, rather than its potential influence, formalized as the mutual information between actions and sensory inputs (Salge al., 2014; 4; Hafner et al., 2020).
We compute empowerment mutual information as the difference between the entropy of actions given the preceding input, before and after observing the following input.
E' <E p(i, p) loggl
-(2 a(i,5)
Eg(t, log log
(5)
Pyk
Ex Pojk EyxPoy
where p(i.j.)) =
Ey Piyk g(L.))
Information gain Information gain is measure of how much the agent learns from its observations Lindley et al. (1956). It is the mutual information between observations and the agent' 's representation of the environment. Information gain has led to successful exploration in reinforcement learning (Sun et al., 201 I; Houthooft et al., 6; Shyam et al., 2018)
To measure the amount information gained, we need a way to represent the agent's knowledge about its environment. Preprocessing the agent's 'ss inputs into discrete classes enables us to represent its knowledge as a belief over the transition matrix M. The total information gain of over agent's lifetime is the entropy difference of its beliefs at the beginning and end of the dataset,
[[[] - [[ dataset] =
log dataset) - log (oM](
(6)
We choose the belief to be vector of Dirichlet distributions as in (Sun et al., 2011; 1; Friston et al., 2017), each which has one concentration parameter, initialized at 1, for each possible subsequent input. For each transition that occurs in the dataset, the corresponding concentration parameter is set to 2, which we empirically found to be more effective than varying the parameter based on the number of occurrences. This is explained by the limited amount of stochasticity in the environments even after pre- pre-processing more than 80% of all state/action pairs transition to only one unique state in Breakout, Seaquest, and Minecraft, that seeing transition once establishes that the transition is possible, and seeing it multiple times should not increase the model's confidence by very much.
The entropy of a Dirichlet distribution is given by,
X
H [] log B(a) + Σ a 1X|
|X|
(az ()xk(
a
A=1
where is the vector of concentration parameters, 1 is the digamma function, and B is the incomplete beta function (Lin, 2016). The entropy of the distribution over M then given by adding up the entropy of each of the Dirichlet distributions.
4 ANALYSIS
We conduct wide range of analyses to understand how the three task-agnostic metrics relate to another and to the supervised metrics of task reward and human similarity. We first compare the agents included in our RL datasets based on their values of the two supervised and three task-agnostic metrics we consider. Next, we analyze correlations between our task-agnostic and supervised metrics. Finally, we discuss correlations between the three task-agnostic metrics themselves. Figure 3 shows correlation matrices of the five metrics, Table 2 contains the tables of all computed metric values, and further visualizations can be found in the supplementary material. We use the OpenAl implementations of ICM (Burda et al., 2018a) and RND (Burda al., 2018b) and the Stable Baselines implementation of PPO (Hill et al., 2018). We will release the source code for replicating our analyses and the collected datasets and metrics upon publication spur further work unsupervised evaluation of RL. agents.
5

---

Under review as a conference paper at ICLR 2021
Breakout
Seaquest
Montezuma
Minecraft
Combined
E
C
E
C
E
C
E
C
E
(reward)
(reward)
(reward)
Breakout
Seaquest
Montezuma
Minecraft
Combined
E
C
E
E
E
E
ICM (unsup)
(unsup)
Figure Overview of agent performance under the five metrics. Task Reward (R) and Human Similarity (S) are supervised metrics. Input Entropy (C), Empowerment (E), and Information Gain (I) are task-agnostic metrics. The two task-specific exploration agents achieve the highest task reward and human similarity average across Atari environments, and RND without reward in Minecraft. ICM or RND each achieve the highest input entropy and infogain value in three out of four environments according ICM achieve high empowerment, even in Montezuma where they achieve low task reward. The no-op agent achieves the lowest scores in all metrics does not show up in the normalized coordinates.
metrics. More unexpectedly,
find that PPO and task-agnostic
4.1 EVALUATION OF AGENTS
Task reward Comparing the mean episode score of task-specific RND and ICM in agent datasets (Appendix D) with Taīga et al. (2020), we find similar performance of the two agents. Our agents perform better in Breakout, ICM has higher reward than RND in Seaquest which is the reverse in Taiga et al. (2020). We observe Table that fask-specific RND or ICM achieves the highest task reward per time step in all environments, showing that agents benefit from explicit exploration objectives in all three environments.
Human similarity Human similarity is the highest for task- ICM and RND in Seaquest and Montezuma respectively, but for task task-agnostic ICM Breakout and task-agnostic RND Minecraft. We find that exploration agents achieve the highest human similarity in all three environments, as expected.
Input entropy We find that task-agnostic ICM and RND obtain the highest input entropy in all environments, except Minecraft where the random agent achieves highest input entropy. These results may be related to the fact that ICM and RND maximize input entropy (Appendix F), and would suggest that using extrinsic reward "distracts" the task apeecifiiic agents from maximizing input entropy. Minecraft is outlier among our environments in that random actions can explore a wide range of states by looking around. The no-op agent is always the lowest, and Breakout and Seaquest all RI. agents achieve higher input entropy than the random agent.
Information gain Information gain highest for the random agent in Breakout, for PPO in Seaquest, for the two configurations of RND in Montezuma, and for the random agent in Minecraft. In all four environments, the agent achieving the highest input entropy also achieves the highest infogain, confirming our observation that input entropy and infogain are related objectives.
Empowerment
The agents exhibiting high and low empowerment vary more between environments than for input entropy. We observe that in three Atari environments, the random agent has relatively high empowerment; higher than all of the RI agents in Breakout (Table 2). This may be related the

---

Under review as a conference paper at ICLR 2021
Breakout
Seaquest
Montezuma
Minecraft
Combined
R
C
C-
C
I
I
I
E
R
R
R
R
S
C
E
Pearson correlation coefficients between the five lifetime metrics considered in this study: Task Reward (R), Human Similarity (S), Input Entropy (C), Empowerment (E), Information Gain (I). The metrics were computed for the lifetime of each agent and the correlation is taken across agents. Aggregated across environments, all metrics correlate positively. Human similarity correlates substantially with both task reward (0.67) and the task-agnostic metrics (0.89, 0.79, 0.66). Reward only correlates weakly with the task-agnostic metrics (0.54, 0.49, 0,41). As per-environment correlations are over 7 data points in Atari environments and in Minecraft, we do report them as statistically significant results.
Figure 3:
fact that our simple preprocessing method is not semantically meaningful, i.e. it does not provide for generalization across similar but non-identical images. Thus, empowerment may distinguish well between behavior in which the agent learns over time, and random behavior resulting in many subtly different episodes. This suggests that learning good input representations may key exploration.
4.2 EVALUATION OF TASK-AGNOSTIC METRICS
Correlation with task reward We find that task reward correlates relatively well (0.67) with human similarity. None of the task-agnostic metries correlate strongly with reward, the highest being input entropy at 0.54. Considering that human similarity itself correlates well with reward, this suggests that input entropy and task reward capture different "components" of agent behavior.
Correlation with human similarity Multiple metries correlate well with human similarity, the strongest once again being input entropy at 0.89. Human similarity exhibits stronger correlations with the aask-agnostic metrics consider than does task reward. correlates the most strongly with both task-specific metrics, as is visually evident in Figure 4. It would be worth considering correlations with human similarity on more open-ended environments in the future, where human players are not necessarily optimizing task reward. This would help determine whether the strong correlations observed with human similarity are a property of the task-agnostic metrics in general, of the human behavior on Atari games specifically.
worth noting that input entropy
Comparison between environments In 23 out of 24 cases in Figure 3, we find that the three lask-agnosin metrics correlate positively with task reward and human similarity. The correlations are especially strong in Breakout and Seaquest. Input entropy correlates strongly with reward in Breakout (0.85). Similarly, empowerment is correlated with reward in Seaquest (0.61). We attribute this to the fact that Breakout and Seaquest are reactive games that require fast paced action choices to achieve high task rewards. In Montezuma, the task-agnostic metrics correlate less strongly with reward; fact, empowerment does not correlate at all (0.00). We suggest this may related to the very difficuut exploration in the game, where the agent is free to take many different courses of action, some of which result in high empowerment, without obtaining any task reward. Minecraft is an open-world environment, but the Treechop task limits the agent breaking rather than placing blocks, so this specific task less open-ended. We find that input entropy and infogain correlate very strongly with task reward (0.98, 0.91) and all three task-agnostic metrics correlate with human similarity (0.89, 0.78, 0.74) in Minecraft.
Combining metrics for task reward or human similarity than any individual metric, we sum up different task-agnostic metrics. No summation of two three task-agnostic metrics correlates more strongly with human similarity or task reward than input entropy alone. Additionally, we find that a linear model of input entropy, infogain, and empowerment of 0.55 and 0.91 respectively. This is only slightly better than the correlation with input entropy in the human similarity case (0.89).
To analyze whether combination of task-agnostic metrics accounts better
predict task reward and human similarity with correlations
7

---

Under review as a conference paper at ICLR 2021
R 0= p=0.004
R= R=0.49; p=0.039
R=0.41; -00::==00111
2
2
Task Reward
Task Reward
Task Reward
0
0
-2
-2
-2
-2
-2
0
-2
Input Entropy
Information Gain
Empowerment
R=0.89; p<0.001
R=0.79; p<0.001
R=0.66; p<0.001
Human Similarity
Human Similarity
Human Similarity
0
-2
-2
-2
-2
-2
0
-2
Input Entropy
Information Gain
Empowerment
No-op
Random
ICM,
RND,
PPO
ICM,
RND,
Figure 4: Scatter plots showing correlations between the three task-agnostic metrics (X axis) and the two task-specific metrics (Y axis), normalized for each environment separately. It is visually evident that the task-agnostic metrics show significant correlations with human similarity but are only weakly correlated with task reward. Additionally, note that the no-op configurations are clearly distinguished from the other agents along the input entropy, empowerment, infogain, and human similarity axes, while yielding similar task reward to the random agent and some RL agent configurations. This shows that input entropy, empowerment, and infogain capture aspects of behavior not captured by task reward, in particular distinguishing between the random and no-op agents that behave very qualitatively differently despite achieving similar task reward.
4.3 COMPARISON AMONG TASK-AGNOSTE METRICS
Correlation among metrics In Figure 3, input entropy and information gain are shown to correlate strongly with one another (0.95). Input entropy also correlates positively with both of them, though less so (0.66 with input entropy and 0.55 with information gain). This suggests that empowerment explores in a different manner than input entropy and information gain. This finding could guide the design of future exploration methods by suggesting that, while combining metries does not seem to be beneficial in our environments, empowerment and input entropyyinformation gain are distinct objectives and so combination of task-agnostic metrics could be applicable in other cases.
5 DISCUSSION
In this paper, we have collected large and diverse datasets of agent behavior, computed three task- agnostic metrics on each dataset, and analyzed the correlations of the task-agnostic objectives with task reward and with a human similarity metric. We have found implementation of input entropy shows promise task-agnostic objective for RI. agent evaluation, and that input entropy and information gain correlate more strongly with human similarity than task reward does.
simple probability tensor
Reliability of results We confident in proposing input entropy as task-agnostic exploration metric, given that it correlates the most with task reward and human similarity. We are also confident in human similarity metric supervised baseline. As the human data does not sticky
8

---

Under review as a conference paper at ICLR 2021
actions, are limited to using heuristic that does not rely the action dynamies. Despite this, human similarity correlates strongly with task reward as expected.
Limitations and future work
Our downscaling and discretization method is simple and transparent preprocessing method, but may not be optimal. More semantically meaningful representations, potentially including deep learning embeddings, may have the potential uncover additional correlations and/or increase the low overlap observed for our human similarity metric.
The human dataset used is limited in quantity compared to agent data (roughly 250K frames per environment, versus 100M frames agent data the Atari games). This is possible reason that our human similarity metric, overlap between agents and human inputs, is low; see Table 2. Access to more human data would be helpful for future work.
When RL agents are trained to optimize extrinsic task reward, it is clear what task the agent is trying to accomplish. The same is not necessarily true of human data, especially in more open-ender games like Montezuma's ' Revenge, where players may choose explore pursue an objective other than that defined by the reward function. Use of a human similarity metric with respect to distinct human datasets with different tasks for the players could yield some insight how closely the human concept of exploration aligns with the task in commonly used environments. An example of such a dataset is is the MineRL. human dataset (Guss et al., 2019).
Summary of insights Task reward, while useful measure of agent intelligence, may not be complete. We propose input entropy as promising task-agnostic metric for agent evaluation, finding that it correlates more strongly with human similarity than does task reward. We also find that input entropy and information gain correlate strongly with each other to a lesser degree with empowerment, and thus recommend future research into combining empowerment with input entropy or information gain, in a variety of environments.

---

Under review as a conference paper at ICLR 2021
REFERENCES
I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's s cube with a robot hand. arXiv preprint arXiv: 1910.0111.. 2019.
Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, pages 1930-2941 2018.
M. Bellemare, Srinivasan, Ostrovski, T. Schaul, D. Saxton, and Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016a.
M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016b.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.00 04355, 20.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint 1810. 12894, 2018b.
A. Ecoffet, J. Huizinga, Lehman, K. O. Stanley, and J. Clune. Go-explore: a new approach for hard-exploration problems. arXiv preprint arXiv:1901.10999 2019.
K. J. Friston, M. Lin, C. D. Frith, G. Pezzulo, J. A. Hobson, and S. Ondobaka. Active inference, curiosity and insight. Neural computation, 2911)):2633-266 2017.
W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov. MineRL: large-scale dataset Minecraft demonstrations. Sreett-Eighhlt International Joint Conference on Artificial Intelligence, 2019. URL http: htttt://minerl.... hwmm/minnnenne
D. Hafner, P. A. Ortega, J. Ba, T. Parr, K. Friston, and N. Heess. Action and perception as divergence minimization. arXiv preprint wxXiv:2009.01991 2020.
N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, M. Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint 2017.
A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu. Stable baselines. ht tttps: //tithggttt//////// com/hill a/stable bas.oieessseeeee,,,eeeeeeee 2018.
R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pages 1109-1117, 2016.
M. Karl, M. Soelch, P. Becker Bhnker-Ehmnk, D. Benbouzid, P. van der Smagt, and J. Bayer. Unsupervised real-time control through variational empowerment. arXiv preprint arXiv:1710.05101 2017.
E. Klein, M. Geist, B. Piot, and O. Pietquin. Inverse reinforcement learning through structured classification. F. Pereira, C. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Ad- vances in Neural Information Processing Systems, volume 25, pages 1007-1015. Curran As- sociates, Inc., 2012. URL https: http://ppocccccnnncncecee www.ineuu.... wttppppp.cpppppp/22.... /fliee/ pdf.
10

---

Under review as a conference paper at ICLR 2021
A. S. Klyubin, D. Polani, and C. L. Nehaniv. Empowerment: A universal agent-centrie measure of control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pages 128-135, IEEE, 2005.
V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg. Specification gaming: the flip side of ai ingenuity, 2020. URL
com/ httll///ppefficattee ficat httppppeeeeaaiiiooaaaaaainneeee htt.a//wwo...ggg.o..oooaooaaaaaa
J. Lin. On the dirichlet distribution. Master's thesis, Queen's University, 2016. URL https:
/ maa.t..uueett.......
D. V. Lindley et al. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 27(4):986-100) 1956.
M. C. Machado, M. G. Bellemare, E. Talvitie, J. Veness, M. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Rexearch, 61:523-562, 2018.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pages 2125-2133, 2015.
P.-Y. Oudeyer, F. Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286, 2007.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self sulf-super prediction. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition Workshops, pages 16-17, 2017.
C. Salge, C. Glackin, and D. Polani. Changing the environment based on empowerment as intrinsic motivation. Entropy, 1665):289--2819, 2014.
J. Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. 1990.
J. Schmidhuber. Curious model- buiell-u control systems. In [Proceedings) 1991 IEEE International Joint Conference on Neural Networks, pages 1458-1463. IEEE, 1991.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 707.06637, 2017.
R. Sckar, O. Rybkin, K. Danilidis, P. Abbecl, D. Hafner, and D. Pathak. Planning to explore via self-supervised world models. arXiv preprint w0Xi::2005.05660 2020.
P. Shyam, W. Jaškowski, and F. Gomez. Model-based active exploration. arXiv preprint arXiv:1810.12162 2018
D. Silver, J. Schrittieserr, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676): 354, 2017.
Y. Sun, F. Gomez, and Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International Conference Artificial General Intelligence, pages 41 41-51. Springer, 2011.
A. A. Taiga, W. Fedus, M. C. Machado, A. C. Courville, and M. G. Bellemare. On bonus-based exploration methods in the arcade learning environment. 2020.
T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for mulli-task and meta reinforcement learning. arXiv preprint arXiv:1910.100 10897, 2019.
11

---

Under review as a conference paper at ICLR 2021
R. Zhang, Z. Liu, L. Guan, L. Zhang, M. M. Hayhoe, and D. H. Ballard. Atari-head: Atari human eye-tracking and demonstration dataset. ArXiv, abs//1903.0654, 2019.
B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Proc. AAAI, pages 1433-1438, 2008
12

---

Under review as a conference paper at ICLR 2021
A
LIFETIME METRICS
Breakout
Symbol
Metric
No-op
Random
ICM,
RND,
PPO
ICM,
RND,
R
Task Reward
0.0000
0.0071
0.0695
0.0786
0.0576
0.1302
0.0743
S
Human Sim.
0.0000
0.0247
0.0346
0.0346
0.0317
0.0346
0.0263
C
Input Entropy
0.0000
7.9303
16.2240
15.9964
15.0695
16.6389
14.9995
Infogain
0.0000
0.0203
0.3458
0.3213
0.2770
0.3533
0.2596
E
Empowerment
0.0000
0.4039
0.4028
0.3907
0.3479
0.3548
0.2770
Seaquest
Symbol
Metric
No-op
Random
ICM,
RND,
PPO
ICM,
RND,
R
Task Reward
0.0000
0.1546
0.1789
0.1424
0.7638
1.7379
1.0993
S
Human Sim.
0.0001
0.0031
0.0034
0.0035
0.0015
0.0037
0.0032
C
Input Entropy
6.7031
12.4245
16.6600
17.0014
12.6777
14.6161
15.7433
I
Infogain 0.0000
0.2209
0.3700
0.3835
0.1941
0,2779
0.3319
E
Empowerment
0.0000
0.6110
0.4013
0.4008
0.8661
0.6806
0.5975
Montezuma
Symbol
Metric
No-op
Random
ICM,
RND,
PPO
ICM,
RND,
R
Task Reward
0.0000
0.0003
0.0000
0.2163
0.0003
1.0620
4.2374
S
Human Sim.
0.0001
0.0069
0.0063
0.0081
0.0070
0.0075
0.0120
C
Input Entropy
1.9008
7.1812
7.9123
8.4405
7.1757
7.1073
7.4942
I
Infogain
0.0000
0.0120
0.0165
0.0229
0.0113
0.0126
0.0125
E
Empowerment
0.0000
0.1326
0.2629
0.1743
0.1436
0.1869
0.1373
Minecraft Symbol
Metric
No-op
Random
RND,
PPO
RND,
R
Task Reward
0.0000
0.0012
0.0013
0.0012
0.0011
S
Human Sim.
2.2 x 10-6
2.8 10-5
4.0 10
3.2 x 10-5
3.2 10-5
Input Entropy
9.4112
16.2270
14.8039
15.3758
15.0612
Infogain
0.0004
0.0583
0.0345
0.0502
0.0477
E
Empowerment
0.0000
0.0770
0.2885
0.1077
0.3444
Table 2: Lifetime values of each metric for all agents and environments, with the highest value of each row in bold. all three environments, the highest task reward achieved by task-specific RND or ICM, which maximize both task reward and different implementations of input entropy; and the highest input entropy achieved by task-agnostic RND or ICM, which is be expected as these agents maximize input entropy alone. Agents with the highest human similarity, empowerment, and information gain vary by environment. Note that the random agent achieves high reward in Minecraft; this may be related the shorter run-time of 12 million frames, which necessary because the Minecraft environment is slower than the Atari games. Nonetheless, the random agent in Minecraft has high input entropy and infogain, as expected due the correlation those metries with reward.
13

---

Under review as a conference paper at ICLR 2021
B INFORMATION GAIN VARIANT CORRELATIONS
Shared Discretizations Human Sim.
Unshared Discretizations Human Sim.
Information Gain Implementation
Task Reward
Task Reward
Dirichlet of transition counts
-0.44
-0.36
0.05
0.40
Dirichlet of unique transitions
0.49
0.79
0.35
0.71
Logarithm transition counts
0.55
0.84
0.37
0.78
Square root of transition counts
0.52
0.84
0.38
0.78
Table 3: Correlations of four information gain implementations with task reward and human similarity, We compare the implementations for two forms of preprocessing, which use the same discretization across agents or use different discretization for each agent based on its data only. We find that the Dirichlet distribution of unique transitions and logarithm and square root of transition counts exhibit greater correlations when using shared discretizations, and that all of the said three methods correlate strongly with human similarity and more weakly with task reward.
C
HUMAN SIMILARITY VARIANT CORRELATIONS
Human Sim. Implementation
Task Reward
Input Entropy
Information Gain Empowerment
Jaccard Similarity
0.67
0.89
0.79
0.66
Lensen-Shannon Divergence 0.26
0.77
0.67
0.66
Table 4: Correlations of two human similarity implementations with task reward and the three task-agnostic metrics. We compare the Jaccard similarity (intersection over union) of the set of states visited by the human player and those visited by the RI. agent, with the Jensen-Shannon divergence between the two We find that Jaccard similarity correlates much more strongly with task reward, slightly more strongly with input entropy and information gain, and near-equally with empowerment, compared Censen-Shannol divergence. The two implementations have correlation 0.78 with each other.
D EPISODE RETURNS
Naive
Task Agnostic
Task-Specife
No-op
Random
ICM
RND
PPO
ICM
RND
Breakout
0.0000
1.6513
92.9397
132.7004
105.6570
240.3896
147.6850
Seaquest
0.0000
98.9152
330.5743
349.1177
1713.8523
4493.0141
2700.6128
Montezuma 0.0000
0.6422
0.0130
489.0153
0.7339
238.7989
5186.5665
Table 5: For our analysis we normalized all metrics by the number of time steps in the agent's 's dataset. For comparison with prior work, this table shows the unnormalized episode returns of all agents. These verify that the agents were trained correctly. Note that they each correspond to only one random seed.
14

---

Under review as a conference paper at ICLR 2021
E ENVIRONMENTS
We consider three Atari environments: Breakout, Seaquest, and Montezuma's is Revenge. All three games are 2D environments with backgrounds fixed relative to the screen. The player is free to move around within the screen on one axis in Breakout, and two axes in the Seaquest and Montezuma. In Montezuma, the player can additionally navigate from one room into another.
Breakout Breakout is game in which the agent controls paddle at the bottom of the screen with the objective of bouncing a ball between the paddle and the blocks above, which disappear upon contact with the ball. The game is nearly deterministic: the only source of randomness other than sticky actions the initial direction of the ball after starting the game or losing a life. Breakout is the simplest of our three environments, the player only being free to move paddle in one dimension.
Seaquest Seaquest is a game in which the player controls a submarine, with the objective of defending oneself against sharks and other submarines which appear frequently and randomly at both sides of the screen. Additionally, the agent is tasked with picking up divers, which also appear at random, and bringing them to the top of the screen. Because of the random appearance of sharks and divers, the game can difficult to predict, and made more so by sticky actions. It more challenging that Breakout, as the player moves along two dimensions and enemies appear at random; however, the agent's 's task is reactive, with no long-term planning required.
Montezuma Montezuma's Revenge is difficult platformer game with large first level consisting of many rooms, which necessitates long-term planning. The player must navigate ladders, ropes, and various hazards such as moving skulls and lasers. Rewards are very sparse, and given only when the player completes an objective such finding key opening door, which often require complex and specific action sequences. For this reason, intrinsic rewards (Burda et al., 2018b) or human demonstrations (Aytar et al., 2018) are important to succeed the game.
Minecraft Treechop MineRL (Guss et al., 2019) is a set of environments Minecraft, a block- based 3D game in which the player can explore, build, and mine within procedurally generated world. The Treechop environment provides the player with an axe and restricts the action space such that the player collecting wood blocks from trees. Though the goal is clearly defined, there are a wide range of possible activities the agent can pursue.
walk around and break blocks, being given task reward for breaking and
We follow the standard evaluation protocol (Machado al., 2018). The Atari games yield 210 160 x 3 images, which are converted to grayscale and rescaled to 84 x 84 before being input to the agent; MineRL returns 64 64 images which are input the agent directly. The agent chooses one of set of discrete actions: 4 in Breakout, and 18 in Seaquest and Montezuma. While the effects of the actions in the original games are deterministic, we use "sticky actions", meaning that the agent's action is ignored with 25% chance and instead the previous action is repeated.
15

---

Under review as a conference paper at ICLR 2021
F AGENTS
Random An agent that uniformly samples random actions from the available action space.
No-op The three environments we consider have a no-op action which does nothing, though the environment still continues update. We consider an agent which always takes this no-op action.
PPO Proximal Policy Optimization (Schulman al., 2017) trained on extrinsic rewards only. PPO is a commonly used policy gradient algorithm that optimizes the task reward on-policy. It optimizes task reward while preventing the policy from changing too much on each training step, so that the learning process is stable.
ICM Intrinsic Curiosity Module (Pathak et al., 2017) is exploration agent that maximizes input entropy addition to the task reward signal. Image embeddings are created that incorporate only those aspects of the image that affect the agent, by training network predict the agent's action given the preceding and following input images, A second network is trained to predict an input image embedding given the preceding input and action. Its error is used as intrinsic objective.
Random Network Distillation (Burda et al., 2018b) is an agent based on PPO that maximizes input entropy, implemented as the prediction error of model trained to distill random function. It makes use of randomly initialized and fixed neural network which predicts an embedding from agent's input images. Another network is trained on the outputs of the random network, and its error, which models the agent's ' "familiarity" with the inputs in question, are used as an intrinsic objective.
RND
16


Reviews:
[{"review_id": "wxH0-OWfJN2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies four task-agnostic metrics for evaluating reinforcement learning agents: human similarity, curiosity, empowerment and information gain. Experiments were conducted with three selected RL algorithms (PPO, ICM and RND) on selected atari games. The results show that a combination of task reward and curiosity better explain human behavior and some non-reward metrics correlate better with human behavior than task reward. The authors propose that such task-agnostic can be used as intrinsic signals for training RL agents when task reward and human data are not available in an environment.\n\nPros:\nTask-agnostic metrics are useful for evaluating RL agents without access to task reward; measuring behavior similarity with human data also provides insights into different behavior of RL algorithms;\nThe insights from analyzing the three task-agnostic metrics’ correlation with both task reward and human behavior similarity are useful for designing new RL algorithms as indicated by the authors;\nThe paper is well written with clarity and includes all experimental details for reproducibility.\n\nCons:\nThe intrinsic metrics studied in this paper are not novel and have been used in various existing RL algorithms alongside task reward for training agents; to demonstrate the acclaimed usefulness of the proposed metrics, it is desired to see experiments training RL agents with only task-agnostic metrics;\nThe experiments conducted are on agent’s life-time data; to gain better understanding of the learning dynamics of RL algorithms, it would be useful to see evaluation of the data at different learning stages.\n\n--------------------------------------\n\n**Update**: After reading the assessment of other reviewers and the referenced papers in the intrinsic reward literature, I am reassured that the methods/metrics proposed in this paper are not novel and, as pointed out by other reviewers, have been studied under other terminologies in different prior works. The analysis of these metrics' correlation with human data is still an interesting piece of result but is not significant enough to become the sole contribution of an ICLR paper. Therefore, I move my initial assessment of 6 to 4.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dJeMKJEcAs7", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for this paper. The curiosity and exploration is an important topic for RL research and we need more in-depth analysis of existing methods. The paper as it stands, provide useful, but expected insights. The difficulty I've with the paper is that it's not clear what exactly you're after here. \n\n\"We find that all three objectives correlate more strongly with human behavior than with the task reward. Moreover, task reward with curiosity better explains human behavior than task reward alone.\": If the idea is to convey the message that humans display curiosity as measured by your interpretation and way of measuring it, then there is a large body of text on human curiosity that already discusses these topics. Additionally, for this you don't need to train artificial agents.\n\n\"Simple implementations of curiosity, empowerment, and information gain correlate substantially with human similarity. This suggests that they can be used as task-agnostic evaluation metrics when human data and task rewards are unavailable.\": following from above comments, all the research on intrinsic reward uses this intuition already, so it's not clear what is added extra here. In addition, as discussed in the notes below, the empowerment and info gain the simplistic way that they are implemented are not actually good measures as a random agent is able to score strongly on those without having any intelligence.\n\nNotes:\n- Table 1 is misplaced on page 1.\n- Section 3.1, discretisation: What is the effect of the choice of 8x8 on the overall results? What would've happened with 16x16 for example? Maybe explore these kind of choices that will impact your results.\n- Section 3.2, human similarity: the sentence: \"We suggest that a more general measure of intelligence may relate to similarity between the agent’s behavior and human behavior in the same environment, i.e. using human behavior as a “groundtruth”.\" overstates the originality of this suggestion as this is not the first time that similarity or imitating human behavior is suggested as a measure of intelligence. Perhaps, you may want to restrict this to certain papers that you feel take a different task oriented approach.\n- Eq 3: I would've thought the human similarity measure to capture the distribution of actions in a particular state as the primary measure than the probability of being at the same state (expressed by the discretised image). While due to previous actions, an agent or human will end up in a certain state, the proposed measure captures the action similarity implicitly rather than explicitly.\n- Eq 3: Any particular reason for using Jaccard index with positive probability thresholds as the measure of similarity? I think a probabilistic measure such as KL-Div would be a more appropriate way to work with distributions of states than thresholded Jaccard similarity.\n- Figure 2: it seems that random agent scores highly in Empowerment and Information Gain metrics. This is very counter intuitive, since (1) the agent doesn't learn from experience, its information gain should be zero; (2) and high score in empowerment may suggest empowerment as computed here is not a good metric for measuring intelligence.\n- Table 2: this is an important table, but has been placed in Appendix, making it not only hard to read the paper, but also I would think is put there to meet the paper limits as otherwise, it would've been located where the results are being discussed. I suggest either to find a way to include it in the main text or remove direct discussion about it from the main results. There is a lot of repetition in the text so it should be possible to be brief and concise but add important results to the main text.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "F93qR5b0tE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThe goal of this paper is to improve our understanding of reward-agnostic metrics drawn from the literature through comparison with human behaviour and task reward. This paper compares two intrinsic reward methods against three baselines on three Atari environments on five metrics, including task reward, a simple metric for human similarity, and three information-theoretic assessments of aggregated observation counts drawn from the literature, which they call task-agnostic metrics. The authors report the correlation between the different metrics.\n\n**Strengths and Weaknesses**\n\nConstructing a comparative understanding of the many methods for exploration, intrinsic motivation, and curiosity is a vastly underdeveloped area. I think that this paper's goal is to do some of that work, which I see as a strength. However, the experiments are not appropriately designed to provide reliable results and the paper includes substantial errors in understanding the existing literature, and as the paper is essentially an empirical survey, appropriately representing the other literature is critical. \n\nVisually inspecting Figure 4, it appears that the results would be completely different if the no-op agent was excluded (and to a lesser extent, the random agent). My concern is that these baselines are categorically different from the agents we are actually interested in and appear to strongly affect the results. For example, without the no-op agent, it appears that the correlation between Human Similarity and Empowerment would be much weaker, and might actually be negative.\n\nThe Human Similarity metric does not seem to be a meaningful metric for what it is designed to measure. This is of particular concern to me because much of the interpretation of the data relies on comparison with the Human Similarity metric, so using such a simplified metric doesn't seem sufficient. The Human similarity data only considers which observations an agent shares with the human data, without regard for how many times each one visits a particular state. A human might make exactly one observation in a given bucket, and an agent making only one observation in that bucket would receive the same score for it as an agent that returns to that state millions of times. The generalization between state observations created by the preprocessing seems like it can only exacerbate the issue.\n\nA similar concern arises when looking at the curiosity metric. Using entropy of sensory input visitation as a metric measures uniformity of visits to states, rather than measuring the ability of the agent to visit as many states as possible. In particular, you can construct examples in which visiting a small subset of states with uniform frequencies results in higher performance on this metric than covering more states, but with less uniform distributions. In principle, most researchers designing algorithms to improve exploration algorithms would care about this distinction. Intuitively, actually visiting a state and ensuring that the agent has observed what is there is important for ensuring the agent can find the optimal parts of the world.\n\nThe use of the word curiosity in this paper is problematic overall. Using the word curiosity to refer to both a metric and a set of methods leaves quite a bit of room for confusion for the reader. In particular, the methods and their metric are not as closely related as the authors suggest in the paper. While the authors appear to have the misconception that methods like ICM and RND are designed to increase the entropy over observations (stated on page 6), this is not the case. Importantly, these rewards are designed to be consumable, so they eventually no longer shape the behaviour of the agent and the agent is left to pursue (typically external) goals. That could result in visit frequencies being highly non-uniform.\n\nThe word curiosity has been used in the realm of reinforcement learning to refer to many very different methods, not necessarily methods that measure probability under a trained density model, and it isn't appropriate to provide this blanket definition of the word curiosity without some language to tell the reader that the word curiosity is simply a shorthand in this paper, in particular, to refer to methods that fall under the given definition.\n\nWhile this paper makes clear calls to the foundation of ideas from the literature that are employed in this paper (e.g., work on curiosity, information gain, empowerment, human performance on Atari, etc.), there is no discussion of related kinds of comparative work that already exists in the literature. Neither the literature comparing multiple intrinsic reward agents nor the literature comparing the exploratory behaviour of RL agents with that of humans is discussed. \n\n**Recommendation**\n\nI am recommending that this paper be rejected on the basis of lack of appropriate evidence for their claims and inappropriate use of language to describe curiosity, a word with a diverse history in the literature. \n\n**Specific Examples of Issues**\n\nThe characterization \"Curiosity encourages encountering rare sensory inputs, measured by a learned density model\" (p. 1) does not capture the definition of curiosity used as a metric: \"the cross entropy of future inputs under a density model trained alongside the agent\" (p. 4)\nThe characterization is inherently contradictory, as if curiosity is \"successful\" what does it mean for a sensory input to be rare? The characterization might be better captured by a definition that requires visiting many states.\n\nThe Go-Explore algorithm by Ecoffet et al. (2019) is explicitly not an intrinsic motivation algorithm (for example, see the paragraphs devoted to contrasting Go-Explore with IM methods on page 2 of Ecoffet et al., 2019) and the paper provides little evidence of the empirical success of IM methods, so citing the paper for such evidence does not appear appropriate. \"Despite the empirical success of intrinsic motivation for facilitating exploration ...\" (p. 1) \n\n**Additional Feedback (Here to help, not necessarily part of decision assessment)**\n\nI found myself trying to come up with a more appropriate name for the metric you call curiosity, and I think that \"Observation entropy\" might capture the mathematical definition appropriately.\n\nMore data might improve the quality of the results of your experiments; if you are interested in including other intrinsic-reward methods into future experiments, a list of fifteen different intrinsic rewards is included in https://arxiv.org/abs/1906.07865\n\nCan you clarify what preprocessing is done for the images fed to the agents? This information belongs somewhere prior to \"We first convert the RGB images to grayscale as they were seen by the agents.\" (p. 3)\n\nI can't find the definitions of A (likely the action set?) and X (likely the set of possible 8x8 discretized images?) (used on p. 3) and it would be helpful to have these notations defined explicitly.\n\n\"has enable agents\" (p. 1) Typo.\n\n\"Atari Learning Environment\" (p. 2) I this was meant to be \"Arcade Learning Environment\"\n\n\"task-agnostic metric\" (p. 5) Typo.\n\n\"human similarity it correlates\" (p 8) Typo.\n\n\"For this reason, intrinsic rewards (Burda et al., 2018b) or human demonstrations (Aytar et al., 2018) are important to succeed at the game.\" (p. 12) Rather than \"are important\" I would suggest \"have been important\" since there is no evidence that there doesn't exist some method of another category that succeeds in Montezuma's Revenge that hasn't been published yet.\n\n\"chooses one of a set\" (p. 12) reads a little strangely, since the agent is choosing an action, not a set.\n\nICM is not designed to be a complete agent (as it \"can potentially be used with a range of policy learning methods,\" Pathak et al., 2017, p. 16) and so the phrase \"is an exploration agent\" (p. 12) is not accurate. I understand that you are using a PPO agent augmented with ICM, following Burda et al. (2018a), but that would be helpful information to include in your description of the agents in the appendix (perhaps along with a reminder to the reader about where to find the OpenAI implementations that you are using).\n\nIn Appendix D, the explanation of ICM (p. 12) would benefit from explaining what learning algorithm/agent architecture is used to optimize the intrinsic (or intrinsic + extrinsic) reward, to parallel the description given for PPO.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EYQs2jNgT7a", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "------------------------------------        \n**Summary:**\n\nThis paper proposes to study three types of intrinsic motivations: curiosity, empowerment and information gain. They propose to compute these measures on the lifetime experience of RL agents and to use them as behavioral metrics. To evaluate these metrics, they perform a correlation study with respect to two traditional behavioral metrics: the task reward and human similarity.\n\n------------------------------------        \n\n**Strong points:**\n\nThe paper is clearly written and well organized. \nI believe it is important to conduct studies that do not present a novel algorithm but try to gain understanding on existing approaches.\nDesigning new behavioral metrics for RL agents, especially ones that do not require rewards is indeed a good idea and will be useful to the community.\nAll details required for reproducibility are present and the code will be released.\n\n------------------------------------        \n**Weak points:**\n\nHere I list weak points, in order of increasing importance.\n* *Downsampling:*\nThese task-agnostic metrics rely on the downsampling of the frame. I feel like this would not work well for Minigrid, Nethack, Mujoco etc. Can you discuss that point?\n* *About the choice of environments:*\nThis paper investigates the evaluation of agents without rewards, in three environments that are explicitly reward-based with well defined rewards. The justification of this choice is also not really discussed except from “we chose these environments because they span a range of complexity, freedom and difficulty”. Breakout and  Seaquest barely require any exploration (breakout is arguably close to a dense reward problem). I wish this study involved more environments, especially environments designed specifically to study exploration issues like NetHack or Minigrid. It would be nice to study the correlation of task-agnostic metrics and human similarity in environments without rewards (which are the target environment of such metrics in the first place).\n* *Human similarity measure:*\nAs I understood it, the human similarity measure is the fraction of downsampled states that are visited by both the RL agent and the human demonstrator over the size of the union of these states sets. \n * We might consider the human coverage as the target coverage. Then the human similarity metric is nothing else than a coverage metric. Can you compute the discrete state coverage metric of RL agents and the correlation to the human similarity metric? \n * I believe a more relevant metric would evaluate whether agents select the same actions when presented the same states. I don’t think the sticky actions are a problem here: one could compute matrices Q of size (|X|, |A|) that empirically estimate the probability of selecting any action in any state for both the human and the RL agent. Whenever the environment decides to use a sticky action, the agent’s policy is still selecting an action that we can use instead of the sticky one. Once we have these matrices Q, then we can compute their average term-by-term difference, and use the opposite as a human similarity measure. Do you have an opinion on that?\n* *Methodology:*\nI am concerned about the validity of the results presented in this paper, as the method is not very rigorous. “correlate substantially” is highly subjective. Usually one would use “correlate significantly”, and support this claim by statistical evidence of the significance of the correlations.\nPlease report which correlation measure is being performed (pearson, spearman, kendall) and report the p-value of the associated statistical test (scipy returns it automatically with the coefficient). This is important to assess whether the evidence is sufficient to claim that there is a correlation.\n * In Fig 3. correlations measures are reported over 7 points, this is quite low and requires statistical tests to be interpretable.\n *  Table 3: measures are episodic returns for 1 seed. This should be said clearly and one should be very cautious with the interpretation of these results.\n * When testing multiple hypotheses in parallel, a good practice is to implement the family-wise error rate correction of the confidence level. If you test for one correlation with confidence level alpha=5% (probability to observe a correlation where there is not stays below 5%), then testing N correlations results in a higher chance of observing a false positive (let us say N*5%). For this reason, the FWER correction proposes to decrease the confidence level of each test by a factor N so that the overall confidence level of the multiple tests remains alpha. This means that, to test for correlations in Figure 3 (10 correlations by graph), we may want to require p-values below alpha / 10 (e.g. 0.005 for an overall 5% confidence level). Theoretically, this should be done for all three environments (so / 30). An alternative is to formulate hypotheses a priori instead of searching for correlations in the wild.\n * “we find that a linear model of curiosity, empowerment, and infogain can predict task reward and human similarity with correlations of 0.36 and 0.86 respectively”. I’m not sure this is a legitimate approach. I’m not entirely sure so it’s open for discussion. This boils down to training a prediction model from task-agnostic metrics to the human similarity score and to evaluate its performance (correlation) on the same training data. Usually you would have an hypothesis (a particular linear combination of these) that you would evaluate (compute the correlation) and test the corresponding significance. \n* *The no-op condition:*\nI see no clear reason to introduce a no-op agent in this study. This agent does nothing, which by construction results in the minimization of all metrics studied here. I think the three points introduced by the no-op agent (in each of the three environments) are the main reason explaining the correlations in Fig 4. If you remove them, then I believe most correlations disappear, some might even become anti-correlated (human sim vs infogain and human sim vs empowerment). Please report the correlation measures (and significance) without these points. \n\n------------------------------------        \n**Recommendation and justification:**\n\nIn the present state of the paper I recommend a rejection (score 4). I think the topic of research is important and the authors should pursue in that direction. However, the methodology of the current version of this paper is not good enough. The introduction of the no-op agent may be explaining most of the correlation discussed in the results. No statistical test has been conducted to show evidence for the significance of the results. \n\nIn order to update my score, I would need a more rigorous correlation study that asserts the significance of the correlations (using corrections). I also think the no-op condition should be removed. The correlation between the curiosity score and human similarity score might still show but it is probably that most of the others would not. The introduction of a human-similarity metric that evaluates the similarity in decision making instead of state visitation might however bring interesting results. \n\n------------------------------------        \n**Feedback to improve the paper (not part of assessment):**\n\n* In the abstract, “compute the objectives” sounds weird, here they are behavioral metrics, although some RL algorithms can be designed to optimize them (in which case they are objectives).\n* What do you mean by ‘estimate intrinsic objectives while the agents are learning, which often requires complicated approximations”? Which complicated approximations?\n* What is a “complete” or “optimal” measure of agent intelligence? I think using “agent intelligence” is vague and not well defined. \n* I am curious, what is the size |X| for the three environments?\n* Task reward: is it the mean over the lifetime, the sum? Is it computed during training episodes, including exploration noise (e.g. epsilon greedy)?\n* Figure 2: what are the axes? Can you explain how you normalize the scores? I’m guessing it’s normalized between to [0,1] by the range across different environments?\n* Colormap for correlation plots is not ideal, it’s difficult to appreciate the colors, maybe pick something with more different colors (not just a gradient between two). \n* How do you normalize and aggregate task reward and curiosity into a unique metric?\n* “human similarity exhibits stronger correlations with the task-agnostic metrics we consider than does task reward” → not true for empowerment (0.57 < 0.6).\n* What is the set of states the curiosity measure is computed on? If it is the set of states visited by the agent, then having a uniform exploration of a very small set of states would result in a high curiosity score. I feel this is not what we want, we want uniformity, but also coverage.\n* The term “curiosity” is quite general and has been used for many purposes in the litterature. For this reason, I think it is not the best term to use here state-entropy would be much more descriptive. When defining curiosity via “a higher curiosity score implies a wider variety of states observed”, the authors cite Oudeyer et al. 2007. I just checked it, and this paper actually presents the classification of several principles to implement the concept of “curiosity” or “intrinsic motivations”. It also presents an algorithm that maximizes the agent’s learning progress. This is different from the diversity-maximization approaches this paper refers to.\n* It would have been interesting to present algorithm optimizing for empowerment and information gain (here the two algorithms both optimize for curiosity). So far the random agent is the one maximizing these metrics. One would hope that algorithms guided by these objectives would do better. This result would support the intuition of the authors towards algorithms that mix infogain/empowerment objectives with curiosity objectives.\n\n\n**Typos:**\n\n* “across a wide spectrum of agent behavior” → “behaviors”.\n* “well known RL agents” → “well known RL algorithms” ?\n* “We first collected datasets of a variety of agent behavior on which to compute and evaluate our metrics” → This sounds weird to me. “After collecting learning trajectories for various RL agents, we can compute behavioral metrics” ?\n* “the total information gain of over agent’s lifetime” → “...gain computed over the agent’s lifetime”.\n* “may key to exploration” → “may be key to a good exploration”.\n* “In 17 out 18 cases” → “In 17 out of 18”.\n\n\n\n\n\n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper, in its current form, is not strong enough to allow for publication.  There are specific weaknesses that need to be tackled: a better correlation study; a clearer relationship to existing literature (and improvement on the novelty); clearer, more precise use of descriptions.

The authors are encouraged to continue with their work and submit a more mature manuscript.
Decision: Reject