Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 1. Summary

The authors introduce a simulator (JBW) with the goal of supporting continual learning. They demonstrate that RL agents struggle with a lot of the tasks in JBW. The majority of the paper describes the technical details of JBW, and show that RL agents can struggle to solve continually changing tasks in JBW.

2. Decision (accept or reject) with one or two key reasons for this choice.

I'm borderline. It is valuable to have environments that support continual learning, although the experimental investigation into different forms of non-stationarity would be more informative. 

Re new implementations: the continual learning setting is certainly important and interesting, but existing environments (see BabyAI, https://arxiv.org/abs/1810.08272 (focus on NLP)) do feature multiple tasks and it is not hard to augment these to run `forever'.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Jelly Bean World: A Testbed for Never-Ending Learning
This work introduces a domain for evaluating and experimenting with algorithms for never-ending learning. These are variants of grid worlds which have multi-task, multi-modal, dynamic settings and can lead to interesting learning challenges. 

I’m referring to never-ending learning as NEL throughout. 

Introduction
Comment: It’s good to tell the reader as early as possible why never ending learning is different than multi-task or continual or lifelong learning? Those are more commonly used terms in the community so it should be situated properly. All of this stuff about NEL seems very similar to continual learning/lifelong learning and we should really reference it and describe the difference?

I didn’t find “In order to more formally describe general intelligence, we posit that there is an underlying measure of complexity of the environment E such that: (i) highly specialized and non-general learning algorithms can perform well in environments with low complexity, but (ii) environments with high complexity require successful learning agents to possess more general learning capabilities” to provide much clarity. Can we either remove it or stick to the later formalism?

In never-ending learning, we explicitly disallow the learning agentπfrom learning across multiple episodes or in multiple environments, which is closer to humanlearning. -> Is this just the same as saying you’re reset free and in a single environment? This sentence is a bit confusing. 

One potential criticism of using simplified simulated worlds like JBW is why should we believe that insights that we get from JBW would carry over to the real world natural environments that NEL ideally cares about. Why is this actually representative of the real world? Because that is really what we care about with NEL. There is merit to simulation in this setting but only if we believe that either insights, algorithms or policies also hold in the real world and we can representatively model the worlds complexity. Can we verify this somehow? 

Design
Does the user have control over all the agents? Or how are they programmed

I would move the details of procedural generation to the appendix, they’re a bit distracting from the point. 

I would also tell the readers why things like scent, intensity, interaction etc are important early on, otherwise it’s confusing what their purpose is. 

In general I quite like the setup, it seems like it has the sufficient amount of complexity in modality, interaction and multi-agent systems to be useful. I wonder if it’s also useful to introduce autonomous self-powered agents which move on their own in the environment and introduce dynamic non stationarities. 

A little more description of the multi-agent, multi-task, curriculum stuff would be useful in the design section. 

The reward functions are all sparse? Or do they need guidance to get to objects as well?

I’m still a bit confused about the interactions functions. Could those be described a bit further?

Perhaps a practical question is how does this relate to the work described in the BabyAI/Minigrid stuff from MILA and other simulated gridworld style environments with multiple agents and such. 

Experiments: 
In the case studies, are things multi-agent?
I wonder if in the reset-free experiment, if we just use dynamic agents in a multi-agent setup, would this just work?

Is it a little odd that the without occlusion performance comes back down to around the same as with occlusions?

Is the scent just perhaps misconfigured/too hard to learn from coz it never seems like it’s doing well with scent?

Overall, I like the paper and the introduced environment. I think it’s important to study scenarios such as the ones described here and this provides a tractable way to start. I am however concerned that the environments are too simplistics and perhaps too far from the real world for the insights to carry over to more realistic scenarios. Some suggestions would be to try and make the environment a bit more realistic and less toy so that insights might also more easily transfer to real world scenarios. But I think with some of the clarifications above and a bit more description, this would be a valuable contribution on topics which are not thought about enough in RL. I also think that actual visuals and videos on an actually accessible website would make it easier for the reviewers/readers to understand the importance of this. I'm currently listing it as a weak accept but I would like the authors to better clarify some of the points mentioned above, discuss how realistic the setting is and also provide us with videos of the environment to better gauge things. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary

This paper introduces a new environment for testing lifelong or never-ending learning. The goal of the environment is to act as a new benchmark testbed for challenging existing agents and models across areas of research, encouraging and pushing new research towards solving challenges in curriculum learning, exploration, representation learning, and continual learning. The contributions in this paper extend upon previous work by building an easily controllable environment generator with key necessary features for lifelong learning including: non-stationarity, multiple task specification, and multiple sets of observable features.

Review

The paper highlights many key characteristics of an environment that are challenging to current RL models. This focus on building a benchmark upon which further research can measure performance is important. I find the proposed environment to be incredibly intriguing and would find it valuable to the field of lifelong learning (or continual learning or never-ending learning, etc.). I think the size and scope of the environment generator is impressive, showing a considerable amount of engineering effort has gone into its design.

The largest overarching issue that I would like to point out is the limited study of modelling choices. I am not an expert on applied Reinforcement Learning, so I can make very few claims about the validity of the chosen network architecture or use of the PPO policy-gradient algorithm for this environment. However, it is critical, in my view, that a paper introducing a new environment studies these effects itself; demonstrating how various degrees of learning capacity or wider ranges of learning algorithms behave in the given environment. If a slightly larger network architecture trivially solves each task in this environment, can this still be considered a benchmark task? A key result in the paper that I would like to see further investigated (even with only a different network architecture) would be Figure 6, the comparison between scent, vision, and vision+scent. It is unclear to me why the scent features would be so challenging to learn from and specifically why they would harm the representation so permanently. A deeper study using only the scent features would be valuable to me. In its current state, it appears that these feature provide no additional information and are thus not necessary to include in the environment; breaking one of the primary motivating features of JBW: the multi-modality. I recognize that the paper comments on the orthogonality of the scent playing a role, and notes that further results are included on a not-yet-available website (presumably to maintain anonymity). However, I would like to see these results included in the appendix of the paper so I could better assess the utility of the scent features. Perhaps an additional result showing the average reward versus the cosine distance (or other measure of orthogonality) between "jellybeans" and "onions" would additionally motivate the utility of the scent features.

The paper empirically investigates the use of curriculum learning to accelerate learning for a particular task. The paper then claims that curriculum learning improves learning speed, but ultimately does improve final performance. This demonstration is intended to showcase the use of the proposed environment (JBW) for curriculum learning. However, there are few key issues with this empirical study. First, the paper shows the reward rate of 3 different curricula but does mention the metric used to compare the agents during the time the curricula is active. It is implied that the metric is the reward rate of each individual agent; however, each agent has a unique reward function making comparisons between agents impossible. Curriculum #2 can only receive positive rewards while Curriculum #1 can only receive negative rewards. Naturally this means that Curriculum #2 must have strictly greater or equal reward rate over Curriculum #1. Even in the case that the final objective specifies the metric used, these are still highly non-comparable entities. A suggestion to improve this result would be to run each curricula for 100k as a "pretraining" phase, then to restart the agents to the same state in the environment and measure their performance from there.

The case study measuring the effects of non-stationarity of the rewards does not provide sufficient evidence that the proposed environment contributes a novel ability to investigate non-stationarity. First, the given study of non-stationarity focuses solely on an alternating reward function, clearly demonstrating the problem of catastrophic forgetting. While this is a motivating demonstration, it is not novel and the issue of catastrophic forgetting in our models has been known since at least the 90s (e.g. French 1999 and related). Carefully and scientifically investigating such an issue is best done in a far less complex environment where more precise results can be drawn. Further, the ability to oscillate a reward function in this way is not unique to this environment and can be trivially done in most environments. Secondly, it is unclear if JBW allows for non-stationarity in the transition probabilities in the MDP. This is a critical component to non-stationarity and would be a necessary feature for me to claim non-stationarity is widely supported in the environment.

The paper starts with a motivating conversation about environment complexity, with interesting insights into measuring the complexity of an environment based on the complexity of the policy used to solve that environment. However this conversation is ignored until the conclusion of the paper, where the paper claims to have built an environment of greater complexity than already existing environments. Without any supporting evidence in the body of the paper, it is impossible to verify the validity of this statement, and it is still an open question to me whether this claim is even falsifiable in the first place. As a concrete counter-claim, I would claim that the Minecraft environment (Malmo) has similar or higher complexity to the proposed environment in most aspects. Minecraft has a far greater diversity of objects, a third dimension of movement, adversarial components, hunger and health, etc. each of which adding a large level of complexity not achievable in the proposed environment. This is not to say that I expect the proposed environment to contain these features, but rather to point out that claims of greater complexity may be ill-founded.

Additional Comments (not affecting score)

I do slightly question if ICLR is the appropriate venue for such work. While I recognize that the scope of this conference has shifted considerably over the past few years, this paper (as written) does not further understanding or study of learning representations. I believe a more careful demonstration of the representation induced by characteristics of the environment is within easy reach of the paper, but is not currently presented.

-----------

After the author response, reading other reviews/responses, and looking at the edited draft:

I am convinced of the utility of the domain, the scope of the engineering effort put into building, and the ease with which it can be configured by the user to test many applicable settings (partial observability, stochasticity in transitions and rewards, etc.). I remain slightly skeptical of the amount of benefit the proposed provides over the Malmo environment for any of the settings discussed in the case-studies.

I specifically feel my concerns about the stochasticity in the transitions and environment complexity have been well addressed. My concerns about the curriculum learning demonstration are partially addressed to a point where I am satisfied. My concerns about the modeling choice are also partially satisfied, with one lingering concern. I am unclear if the environment is trivially solvable by using more computation resources (e.g. bigger networks). However, after reconsideration I decided this concern bares less weight than I previously considered.

All this considered, I am changing my rating from 3 -> 6.
review_title: Official Blind Review #1

Meta-review: This paper proposes a flexible environment for studying never ending learning. During the discussion period, all reviewers found the paper to be borderline.

Pros:
- we don't have good lifelong or never-ending RL environments, and this paper seems to provide one
- includes a number of interesting features such as multiple input modalities, non-episodic interactions, flexible task definitions

Cons:
- procedurally generated, toy environment
- unclear if the environment reflects the characteristics of real world NEL problems

In the balance, I think the environments add value to the RL community, and being presented at ICLR would increase its visibility.
Decision: Accept (Poster)