Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: **Summary**

The goal of this paper is to improve our understanding of reward-agnostic metrics drawn from the literature through comparison with human behaviour and task reward. This paper compares two intrinsic reward methods against three baselines on three Atari environments on five metrics, including task reward, a simple metric for human similarity, and three information-theoretic assessments of aggregated observation counts drawn from the literature, which they call task-agnostic metrics. The authors report the correlation between the different metrics.

**Strengths and Weaknesses**

Constructing a comparative understanding of the many methods for exploration, intrinsic motivation, and curiosity is a vastly underdeveloped area. I think that this paper's goal is to do some of that work, which I see as a strength. However, the experiments are not appropriately designed to provide reliable results and the paper includes substantial errors in understanding the existing literature, and as the paper is essentially an empirical survey, appropriately representing the other literature is critical. 

Visually inspecting Figure 4, it appears that the results would be completely different if the no-op agent was excluded (and to a lesser extent, the random agent). My concern is that these baselines are categorically different from the agents we are actually interested in and appear to strongly affect the results. For example, without the no-op agent, it appears that the correlation between Human Similarity and Empowerment would be much weaker, and might actually be negative.

The Human Similarity metric does not seem to be a meaningful metric for what it is designed to measure. This is of particular concern to me because much of the interpretation of the data relies on comparison with the Human Similarity metric, so using such a simplified metric doesn't seem sufficient. The Human similarity data only considers which observations an agent shares with the human data, without regard for how many times each one visits a particular state. A human might make exactly one observation in a given bucket, and an agent making only one observation in that bucket would receive the same score for it as an agent that returns to that state millions of times. The generalization between state observations created by the preprocessing seems like it can only exacerbate the issue.

A similar concern arises when looking at the curiosity metric. Using entropy of sensory input visitation as a metric measures uniformity of visits to states, rather than measuring the ability of the agent to visit as many states as possible. In particular, you can construct examples in which visiting a small subset of states with uniform frequencies results in higher performance on this metric than covering more states, but with less uniform distributions. In principle, most researchers designing algorithms to improve exploration algorithms would care about this distinction. Intuitively, actually visiting a state and ensuring that the agent has observed what is there is important for ensuring the agent can find the optimal parts of the world.

The use of the word curiosity in this paper is problematic overall. Using the word curiosity to refer to both a metric and a set of methods leaves quite a bit of room for confusion for the reader. In particular, the methods and their metric are not as closely related as the authors suggest in the paper. While the authors appear to have the misconception that methods like ICM and RND are designed to increase the entropy over observations (stated on page 6), this is not the case. Importantly, these rewards are designed to be consumable, so they eventually no longer shape the behaviour of the agent and the agent is left to pursue (typically external) goals. That could result in visit frequencies being highly non-uniform.

The word curiosity has been used in the realm of reinforcement learning to refer to many very different methods, not necessarily methods that measure probability under a trained density model, and it isn't appropriate to provide this blanket definition of the word curiosity without some language to tell the reader that the word curiosity is simply a shorthand in this paper, in particular, to refer to methods that fall under the given definition.

While this paper makes clear calls to the foundation of ideas from the literature that are employed in this paper (e.g., work on curiosity, information gain, empowerment, human performance on Atari, etc.), there is no discussion of related kinds of comparative work that already exists in the literature. Neither the literature comparing multiple intrinsic reward agents nor the literature comparing the exploratory behaviour of RL agents with that of humans is discussed. 

**Recommendation**

I am recommending that this paper be rejected on the basis of lack of appropriate evidence for their claims and inappropriate use of language to describe curiosity, a word with a diverse history in the literature. 

**Specific Examples of Issues**

The characterization "Curiosity encourages encountering rare sensory inputs, measured by a learned density model" (p. 1) does not capture the definition of curiosity used as a metric: "the cross entropy of future inputs under a density model trained alongside the agent" (p. 4)
The characterization is inherently contradictory, as if curiosity is "successful" what does it mean for a sensory input to be rare? The characterization might be better captured by a definition that requires visiting many states.

The Go-Explore algorithm by Ecoffet et al. (2019) is explicitly not an intrinsic motivation algorithm (for example, see the paragraphs devoted to contrasting Go-Explore with IM methods on page 2 of Ecoffet et al., 2019) and the paper provides little evidence of the empirical success of IM methods, so citing the paper for such evidence does not appear appropriate. "Despite the empirical success of intrinsic motivation for facilitating exploration ..." (p. 1) 

**Additional Feedback (Here to help, not necessarily part of decision assessment)**

I found myself trying to come up with a more appropriate name for the metric you call curiosity, and I think that "Observation entropy" might capture the mathematical definition appropriately.

More data might improve the quality of the results of your experiments; if you are interested in including other intrinsic-reward methods into future experiments, a list of fifteen different intrinsic rewards is included in https://arxiv.org/abs/1906.07865

Can you clarify what preprocessing is done for the images fed to the agents? This information belongs somewhere prior to "We first convert the RGB images to grayscale as they were seen by the agents." (p. 3)

I can't find the definitions of A (likely the action set?) and X (likely the set of possible 8x8 discretized images?) (used on p. 3) and it would be helpful to have these notations defined explicitly.

"has enable agents" (p. 1) Typo.

"Atari Learning Environment" (p. 2) I this was meant to be "Arcade Learning Environment"

"task-agnostic metric" (p. 5) Typo.

"human similarity it correlates" (p 8) Typo.

"For this reason, intrinsic rewards (Burda et al., 2018b) or human demonstrations (Aytar et al., 2018) are important to succeed at the game." (p. 12) Rather than "are important" I would suggest "have been important" since there is no evidence that there doesn't exist some method of another category that succeeds in Montezuma's Revenge that hasn't been published yet.

"chooses one of a set" (p. 12) reads a little strangely, since the agent is choosing an action, not a set.

ICM is not designed to be a complete agent (as it "can potentially be used with a range of policy learning methods," Pathak et al., 2017, p. 16) and so the phrase "is an exploration agent" (p. 12) is not accurate. I understand that you are using a PPO agent augmented with ICM, following Burda et al. (2018a), but that would be helpful information to include in your description of the agents in the appendix (perhaps along with a reminder to the reader about where to find the OpenAI implementations that you are using).

In Appendix D, the explanation of ICM (p. 12) would benefit from explaining what learning algorithm/agent architecture is used to optimize the intrinsic (or intrinsic + extrinsic) reward, to parallel the description given for PPO.
review_title: Initial review for Submission 1867

final_score: rating: 4: Ok but not good enough - rejection
review_content: Thanks for this paper. The curiosity and exploration is an important topic for RL research and we need more in-depth analysis of existing methods. The paper as it stands, provide useful, but expected insights. The difficulty I've with the paper is that it's not clear what exactly you're after here. 

"We find that all three objectives correlate more strongly with human behavior than with the task reward. Moreover, task reward with curiosity better explains human behavior than task reward alone.": If the idea is to convey the message that humans display curiosity as measured by your interpretation and way of measuring it, then there is a large body of text on human curiosity that already discusses these topics. Additionally, for this you don't need to train artificial agents.

"Simple implementations of curiosity, empowerment, and information gain correlate substantially with human similarity. This suggests that they can be used as task-agnostic evaluation metrics when human data and task rewards are unavailable.": following from above comments, all the research on intrinsic reward uses this intuition already, so it's not clear what is added extra here. In addition, as discussed in the notes below, the empowerment and info gain the simplistic way that they are implemented are not actually good measures as a random agent is able to score strongly on those without having any intelligence.

Notes:
- Table 1 is misplaced on page 1.
- Section 3.1, discretisation: What is the effect of the choice of 8x8 on the overall results? What would've happened with 16x16 for example? Maybe explore these kind of choices that will impact your results.
- Section 3.2, human similarity: the sentence: "We suggest that a more general measure of intelligence may relate to similarity between the agent’s behavior and human behavior in the same environment, i.e. using human behavior as a “groundtruth”." overstates the originality of this suggestion as this is not the first time that similarity or imitating human behavior is suggested as a measure of intelligence. Perhaps, you may want to restrict this to certain papers that you feel take a different task oriented approach.
- Eq 3: I would've thought the human similarity measure to capture the distribution of actions in a particular state as the primary measure than the probability of being at the same state (expressed by the discretised image). While due to previous actions, an agent or human will end up in a certain state, the proposed measure captures the action similarity implicitly rather than explicitly.
- Eq 3: Any particular reason for using Jaccard index with positive probability thresholds as the measure of similarity? I think a probabilistic measure such as KL-Div would be a more appropriate way to work with distributions of states than thresholded Jaccard similarity.
- Figure 2: it seems that random agent scores highly in Empowerment and Information Gain metrics. This is very counter intuitive, since (1) the agent doesn't learn from experience, its information gain should be zero; (2) and high score in empowerment may suggest empowerment as computed here is not a good metric for measuring intelligence.
- Table 2: this is an important table, but has been placed in Appendix, making it not only hard to read the paper, but also I would think is put there to meet the paper limits as otherwise, it would've been located where the results are being discussed. I suggest either to find a way to include it in the main text or remove direct discussion about it from the main results. There is a lot of repetition in the text so it should be possible to be brief and concise but add important results to the main text.

review_title: This paper compares a few RL agents trained with different task oriented and exploration objectives to human baselines and finds that human baseline correlates with both task and curiosity metrics.

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work studies four task-agnostic metrics for evaluating reinforcement learning agents: human similarity, curiosity, empowerment and information gain. Experiments were conducted with three selected RL algorithms (PPO, ICM and RND) on selected atari games. The results show that a combination of task reward and curiosity better explain human behavior and some non-reward metrics correlate better with human behavior than task reward. The authors propose that such task-agnostic can be used as intrinsic signals for training RL agents when task reward and human data are not available in an environment.

Pros:
Task-agnostic metrics are useful for evaluating RL agents without access to task reward; measuring behavior similarity with human data also provides insights into different behavior of RL algorithms;
The insights from analyzing the three task-agnostic metrics’ correlation with both task reward and human behavior similarity are useful for designing new RL algorithms as indicated by the authors;
The paper is well written with clarity and includes all experimental details for reproducibility.

Cons:
The intrinsic metrics studied in this paper are not novel and have been used in various existing RL algorithms alongside task reward for training agents; to demonstrate the acclaimed usefulness of the proposed metrics, it is desired to see experiments training RL agents with only task-agnostic metrics;
The experiments conducted are on agent’s life-time data; to gain better understanding of the learning dynamics of RL algorithms, it would be useful to see evaluation of the data at different learning stages.

--------------------------------------

**Update**: After reading the assessment of other reviewers and the referenced papers in the intrinsic reward literature, I am reassured that the methods/metrics proposed in this paper are not novel and, as pointed out by other reviewers, have been studied under other terminologies in different prior works. The analysis of these metrics' correlation with human data is still an interesting piece of result but is not significant enough to become the sole contribution of an ICLR paper. Therefore, I move my initial assessment of 6 to 4.
review_title: Useful analysis of non-task metrics for RL agents; more experiments on training with the metrics would be better

final_score: rating: 4: Ok but not good enough - rejection
review_content: ------------------------------------        
**Summary:**

This paper proposes to study three types of intrinsic motivations: curiosity, empowerment and information gain. They propose to compute these measures on the lifetime experience of RL agents and to use them as behavioral metrics. To evaluate these metrics, they perform a correlation study with respect to two traditional behavioral metrics: the task reward and human similarity.

------------------------------------        

**Strong points:**

The paper is clearly written and well organized. 
I believe it is important to conduct studies that do not present a novel algorithm but try to gain understanding on existing approaches.
Designing new behavioral metrics for RL agents, especially ones that do not require rewards is indeed a good idea and will be useful to the community.
All details required for reproducibility are present and the code will be released.

------------------------------------        
**Weak points:**

Here I list weak points, in order of increasing importance.
* *Downsampling:*
These task-agnostic metrics rely on the downsampling of the frame. I feel like this would not work well for Minigrid, Nethack, Mujoco etc. Can you discuss that point?
* *About the choice of environments:*
This paper investigates the evaluation of agents without rewards, in three environments that are explicitly reward-based with well defined rewards. The justification of this choice is also not really discussed except from “we chose these environments because they span a range of complexity, freedom and difficulty”. Breakout and  Seaquest barely require any exploration (breakout is arguably close to a dense reward problem). I wish this study involved more environments, especially environments designed specifically to study exploration issues like NetHack or Minigrid. It would be nice to study the correlation of task-agnostic metrics and human similarity in environments without rewards (which are the target environment of such metrics in the first place).
* *Human similarity measure:*
As I understood it, the human similarity measure is the fraction of downsampled states that are visited by both the RL agent and the human demonstrator over the size of the union of these states sets. 
 * We might consider the human coverage as the target coverage. Then the human similarity metric is nothing else than a coverage metric. Can you compute the discrete state coverage metric of RL agents and the correlation to the human similarity metric? 
 * I believe a more relevant metric would evaluate whether agents select the same actions when presented the same states. I don’t think the sticky actions are a problem here: one could compute matrices Q of size (|X|, |A|) that empirically estimate the probability of selecting any action in any state for both the human and the RL agent. Whenever the environment decides to use a sticky action, the agent’s policy is still selecting an action that we can use instead of the sticky one. Once we have these matrices Q, then we can compute their average term-by-term difference, and use the opposite as a human similarity measure. Do you have an opinion on that?
* *Methodology:*
I am concerned about the validity of the results presented in this paper, as the method is not very rigorous. “correlate substantially” is highly subjective. Usually one would use “correlate significantly”, and support this claim by statistical evidence of the significance of the correlations.
Please report which correlation measure is being performed (pearson, spearman, kendall) and report the p-value of the associated statistical test (scipy returns it automatically with the coefficient). This is important to assess whether the evidence is sufficient to claim that there is a correlation.
 * In Fig 3. correlations measures are reported over 7 points, this is quite low and requires statistical tests to be interpretable.
 *  Table 3: measures are episodic returns for 1 seed. This should be said clearly and one should be very cautious with the interpretation of these results.
 * When testing multiple hypotheses in parallel, a good practice is to implement the family-wise error rate correction of the confidence level. If you test for one correlation with confidence level alpha=5% (probability to observe a correlation where there is not stays below 5%), then testing N correlations results in a higher chance of observing a false positive (let us say N*5%). For this reason, the FWER correction proposes to decrease the confidence level of each test by a factor N so that the overall confidence level of the multiple tests remains alpha. This means that, to test for correlations in Figure 3 (10 correlations by graph), we may want to require p-values below alpha / 10 (e.g. 0.005 for an overall 5% confidence level). Theoretically, this should be done for all three environments (so / 30). An alternative is to formulate hypotheses a priori instead of searching for correlations in the wild.
 * “we find that a linear model of curiosity, empowerment, and infogain can predict task reward and human similarity with correlations of 0.36 and 0.86 respectively”. I’m not sure this is a legitimate approach. I’m not entirely sure so it’s open for discussion. This boils down to training a prediction model from task-agnostic metrics to the human similarity score and to evaluate its performance (correlation) on the same training data. Usually you would have an hypothesis (a particular linear combination of these) that you would evaluate (compute the correlation) and test the corresponding significance. 
* *The no-op condition:*
I see no clear reason to introduce a no-op agent in this study. This agent does nothing, which by construction results in the minimization of all metrics studied here. I think the three points introduced by the no-op agent (in each of the three environments) are the main reason explaining the correlations in Fig 4. If you remove them, then I believe most correlations disappear, some might even become anti-correlated (human sim vs infogain and human sim vs empowerment). Please report the correlation measures (and significance) without these points. 

------------------------------------        
**Recommendation and justification:**

In the present state of the paper I recommend a rejection (score 4). I think the topic of research is important and the authors should pursue in that direction. However, the methodology of the current version of this paper is not good enough. The introduction of the no-op agent may be explaining most of the correlation discussed in the results. No statistical test has been conducted to show evidence for the significance of the results. 

In order to update my score, I would need a more rigorous correlation study that asserts the significance of the correlations (using corrections). I also think the no-op condition should be removed. The correlation between the curiosity score and human similarity score might still show but it is probably that most of the others would not. The introduction of a human-similarity metric that evaluates the similarity in decision making instead of state visitation might however bring interesting results. 

------------------------------------        
**Feedback to improve the paper (not part of assessment):**

* In the abstract, “compute the objectives” sounds weird, here they are behavioral metrics, although some RL algorithms can be designed to optimize them (in which case they are objectives).
* What do you mean by ‘estimate intrinsic objectives while the agents are learning, which often requires complicated approximations”? Which complicated approximations?
* What is a “complete” or “optimal” measure of agent intelligence? I think using “agent intelligence” is vague and not well defined. 
* I am curious, what is the size |X| for the three environments?
* Task reward: is it the mean over the lifetime, the sum? Is it computed during training episodes, including exploration noise (e.g. epsilon greedy)?
* Figure 2: what are the axes? Can you explain how you normalize the scores? I’m guessing it’s normalized between to [0,1] by the range across different environments?
* Colormap for correlation plots is not ideal, it’s difficult to appreciate the colors, maybe pick something with more different colors (not just a gradient between two). 
* How do you normalize and aggregate task reward and curiosity into a unique metric?
* “human similarity exhibits stronger correlations with the task-agnostic metrics we consider than does task reward” → not true for empowerment (0.57 < 0.6).
* What is the set of states the curiosity measure is computed on? If it is the set of states visited by the agent, then having a uniform exploration of a very small set of states would result in a high curiosity score. I feel this is not what we want, we want uniformity, but also coverage.
* The term “curiosity” is quite general and has been used for many purposes in the litterature. For this reason, I think it is not the best term to use here state-entropy would be much more descriptive. When defining curiosity via “a higher curiosity score implies a wider variety of states observed”, the authors cite Oudeyer et al. 2007. I just checked it, and this paper actually presents the classification of several principles to implement the concept of “curiosity” or “intrinsic motivations”. It also presents an algorithm that maximizes the agent’s learning progress. This is different from the diversity-maximization approaches this paper refers to.
* It would have been interesting to present algorithm optimizing for empowerment and information gain (here the two algorithms both optimize for curiosity). So far the random agent is the one maximizing these metrics. One would hope that algorithms guided by these objectives would do better. This result would support the intuition of the authors towards algorithms that mix infogain/empowerment objectives with curiosity objectives.


**Typos:**

* “across a wide spectrum of agent behavior” → “behaviors”.
* “well known RL agents” → “well known RL algorithms” ?
* “We first collected datasets of a variety of agent behavior on which to compute and evaluate our metrics” → This sounds weird to me. “After collecting learning trajectories for various RL agents, we can compute behavioral metrics” ?
* “the total information gain of over agent’s lifetime” → “...gain computed over the agent’s lifetime”.
* “may key to exploration” → “may be key to a good exploration”.
* “In 17 out 18 cases” → “In 17 out of 18”.









review_title: An important topic of research but the methodology lacks rigor

Meta-review: The reviewers agree that the paper, in its current form, is not strong enough to allow for publication.  There are specific weaknesses that need to be tackled: a better correlation study; a clearer relationship to existing literature (and improvement on the novelty); clearer, more precise use of descriptions.

The authors are encouraged to continue with their work and submit a more mature manuscript.
Decision: Reject