Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors study what they refer to as ecological reinforcement learning, defined as the interaction between properties of the environment and the reinforcement learning agent. They introduce environments with characteristics that reflect natural environments: non-episodic learning, uninformative reward signals, and natural dynamics that cause the environment to change. These factors are shown to significantly affect the learning progress of RL agents and, unexpectedly, the agents can sometimes learn more efficiently in these more challenging conditions.

Clarity:

The paper seems to be clearly written. The code will be made publicly available.

Novelty:

The main contribution from the paper seems to be two novel benchmark problems with characteristics that reflect natural environments and an exhaustive evaluation of the performance of standard algorithms on them. While the experimental results show some light about the performance of existing methods in the proposed environment, the paper does not contain any methodological contributions. Because of this, it is hard to assess the novelty of the work.

Quality and significance:

While the paper makes some interesting points, I feel that the proposed environments are too few, and too simple and unrealistic when compared to real-world problems. Because of this, one cannot know how general and significant the conclusions obtained are.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper discussed several properties of environments used in reinforcement learning research experiments. The main conclusion is the environment should be dynamic and non-episodic, and the environment shaping is introduced to be effective. In general, the idea is well-presented and easy to follow. However, I have some concerns about the proposed method and experiments:

----
1. In general, I think the arguments in the paper are still a bit vague to be demonstrated using only experimental results. One improvement could be using some mathematical formulation to describe the argument and conducting some analysis. For instance, the paper can formulate the environment shaping and reward shaping concretely and prove that environment shaping could replace reward shaping. 


2. For the experiment comparing the reward shaping and environment shaping: the environment shaping method is designed and more complicated than the reward shaping, I think it could be more convincing if the authors investigate and develop more approaches for reward shaping. Otherwise, it is a bit hard to argue the environment shaping could outperform reward shaping a lot. 

3. To argue that the non-episodic environment is better than episodic ones, I think the paper should consider more tasks besides the two mentioned in the experiment section. From figure 3, the non-episodic dynamic environment is not very clearly better than episodic one from all scenarios. 

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary

This paper discusses the value of creating more challenging environments for training reinforcement learning agents. Specifically, the paper focuses on three characteristics of the environment that the paper claims are necessary for developing intelligent agents. The first of these properties is stochasticity in the environment transitions, specifically stochasticity that is independent of the action taken by the agent. The next is sparsity of rewards; discontinuing the use of reward shaping to define desired behavior. Finally the paper argues that environments should not be episodic, that natural environments are continuing tasks so research focus should be around solving continuous tasks.

Review

My primary concern about this paper is the largely insufficient literature review. Many of the claims made in the motivation of this paper are not novel to this paper and are, in fact, incredibly vibrant sub-communities of study within the field of Reinforcement Learning. A more careful literature review should have easily found these communities and more nuanced claims could have been made. I will give concrete examples of the most important cases of missing literature in the following paragraphs, but this list is not exhaustive.

The paper claims that most standard RL environments include detailed reward functions that unnecessarily shape learning and inject bias into the learning process. While I agree that this is problematic, I disagree that this paper provides any novel insights towards this problem. The problem of learning from sparse rewards is well-known in the RL community and is a hot-topic of study. Even in the standard environments cited by the paper we have Montezuma's Revenge and Pitfall, two environment notorious for their difficulty due to sparse reward; each of which with its own host of literature surrounding only the single environment (for instance, I encourage the authors to investigate the highly controversial Go-Explore paper by Uber and its references). Other environments not considered by this paper include the Malmo (Minecraft) learning environment, around which NeurIPS 2019 hosted an extremely sparse reward competition. Another (overlapping) community of RL research interested in the sparse reward setting is the intrinsic reward community, one such paper being Riedmiller and Hafner et al. 2018.

This paper claims that all standard environments are episodic. Of the environments listed as "standard" by this paper, this claim does not even hold. However, there is a large chunk of the RL community that is not represented here. The continual learning and life-long learning communities are focused exclusively on the problem of non-episodic learning. Some example environments used by the community include Malmo, MuJoCo, DeepMind's Lab environment, and many smaller toy domains designed to showcase individual problems including Cart-Pole, RiverSwim, Pendulum, and Acrobot; with the smaller environments from OpenAi Gym cited by this paper. Another smaller community to investigate would be the average reward formulation of the RL problem, which fairly exclusively focuses on the continual learning problem.

Finally, this paper seems vaguely reminiscent of a few particular environments that I have seen in the literature previously. For example, Berkeley's robot task discovery playpen (see for example Singh, Yang, Hartikainen, Finn, Levine 2019). Or an even more similar simulated environment being the Playroom environment by Singh, Barto, Chentanez 2005. Finally, Malmo has been used in a similar way as the tool-building examples mentioned in this paper.

There were a few key issues with the experiments discussed in this paper. The first of which being "Hypothesis 1" which states: "Non-episodic learning is more difficult than episodic learning because the agent must handle a non-stationary learning problem." This hypothesis alone does not appear to be uniformly true. In fact, imagine a simple 5-state random walk markov chain environment without termination. Each state is visited infinitely many times, so the chain is ergodic and there are no non-stationary points. The empirical section uses meta-parameters that were ill-motivated with no discussion about meta-parameter selection. It is critical to point out that the stepsize used for an episodic problem will likely not be the optimal stepsize for the corresponding non-episodic problem, as the magnitude and variance of the considered returns are necessarily different (in response to Figure 3). Further, evaluating over 3 random seeds simply is not sufficient to make any statistically significant claims when comparing any of these curves (for instance in Figure 5).

Additional Comments (do not influence rating)

For a paper exclusively introducing a new control environment, it is critical to include a discussion about the exploration problem. At the very least, I would appreciate seeing sensitivity curves for values of epsilon.

This paper makes many strong claims about the nature of intelligence that are neither supported in the work or are accepted in the community. While it is intuitive that the environment plays a critical role in developing intelligence, the lack of universal definition of intelligence makes this a non-falsifiable claim. Although I appreciate the point the authors are trying to make, which is that RL research frequently is done in the realm of toy simulated domains, I do not think that this paper includes the appropriate supporting evidence to validate such lofty claims.

It would be interesting to change the exploration method from epsilon-greedy to sampling according to the softmax action distribution. This can have dramatically improved performance on non-adversarial exploration problems, and reduces the need for scheduled epsilon decay. It additionally reduces the need for two extra meta-parameters, allowing the empirical claims to be made more strongly without performing some sweep over parameters.

--------------------
Edit after reading discussion, rebuttal, and edits to manuscript.

The paper's intended contribution was different than I had realized during the initial review phase. I appreciate the effort the author put into the response and the changes made to the literature review section of the paper. I think these help to demonstrate the scope and placement of the work considerably.

I still believe the paper over-states the novelty of the results and I still find it difficult to understand their utility. I believe that the entirety of the paper falls under the domain of exploration in RL, but this is not made clear through the introduction or related works section (though the updates to the related works section help considerably). 

Stochasticity in the environment helping the agent to learn is not a surprising finding at all if the exploration method is insufficient. To give a small example, imagine an agent wandering in a tabular gridworld. If the agent has no method of exploration, and the environment has sparse reward, then it is not surprising that the agent would get stuck in a corner. If the environment was modified so that each transition had a 5% chance of randomly going another direction (e.g. the "right" action has a 5% chance of becoming an "up" action), then we've effectively encoded epsilon-greedy exploration through the environment dynamics. All of this comes down to say, it is difficult to separate the dynamics of the environment from the agent's exploration method. I think it would require a careful study that considers these aspects more explicitly.

To summarize, I think the paper could easily be accepted to a future conference, but I think it is important to:
- Make connection between the insights and exploration clear, specifically designing the introduction, lit review, and experiments around this connection.
- Make sure the contributions are extremely clear in the writing. Demonstrate those contributions directly in the empirical section.
- Tone down the claims in the writing. Many of the claims about the state of the field in RL are demonstrably incorrect. I agree with the sentiment trying to be expressed, but the absolutism makes it difficult to separate that sentiment from a deep understanding of the current RL literature with only the most important papers cited, or a fundamentally insufficient lit review. To make claims about the state of a field the lit review should be rather extensive.
review_title: Official Blind Review #2

Meta-review: This paper investigates how the properties of an environment affect the success of reinforcement learning, and in particular finds that random dynamics and non-episodic learning makes learning easier, even though these factors make learning more difficult when applied individually. The paper was reviewed by three experts who gave Reject, Weak Reject, and Weak Reject recommendations. The main concerns are about missing connections to related work, overstating some contributions, and experimental details. While the author response addressed many of these issues, reviewers felt another round of peer review is really needed before this paper can be accepted; R2's post-rebuttal comments give some specific, constructive, concrete suggestions for preparing a revision.
Decision: Reject