Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript presents a concrete formalization of the "Autonomous Reinforcement Learning" (ARL) problem: this problem refers to a scenario where the agents has no (or very few) resets during learning of the task(s). The paper also unifies previous works into two evaluation scenarios: (a) Deployed Policy Evaluation, and (b) Continuing Policy Evaluation. The manuscript, also, provides a benchmark: "Environments for Autonomous Reinforcement Learning (EARL)". This benchmark contains diverse environments from simple manipulation tasks to locomotion and complex manipulation. Using this benchmark the authors provide an analysis of the crucial factors that affect the performance of current state-of-the-art algorithms for the ARL problem.

The main contributions of the paper are:

- Concrete formalization of the "Autonomous Reinforcement Learning" (ARL) problem
- A novel benchmark for ARL with diverse tasks
- Novel evaluation settings (Deployment Setting and Continuing Setting) for ARL
- Insights on the important factors for effective ARL

main_review: Strengths
-------------

- The paper provides a nice formalization of an important problem (that of Autonomous Reinforcement Learning)
- I like the distinction into "Deployment Setting" and the "Continuing Setting". Usually people when hearing of ARL, they have in mind the "Continuing Setting", but the "Deployment Setting" is also very useful. More importantly, they are different problems. In any case, the distinction can make things easier to talk about and compare methods/ideas.
- Nice benchmark environments capturing a wide variety of tasks/scenarios
- Thorough analysis providing useful insights (mostly validating our intuition, but this is important!)

Weaknesses
------------------

- The ARL problem is strongly related to the autonomous skill discovery problem (see for example {1,2,3}). For instance, without proper exploration and skill distillation (either automatic or with some heuristic), ARL is not really a solvable problem without many resets. I would definitely like to see a nice discussion about this and other similarities. This would make the paper complete.


References
-----------------
{1}: Campos, V., Trott, A., Xiong, C., Socher, R., Giró-i-Nieto, X. and Torres, J., 2020, November. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning (pp. 1317-1327). PMLR.

{2}: Gregor, K., Rezende, D.J. and Wierstra, D., 2016. Variational intrinsic control. arXiv preprint arXiv:1611.07507.

{3}: Pong, V.H., Dalal, M., Lin, S., Nair, A., Bahl, S. and Levine, S., 2020, November. Skew-fit: State-covering self-supervised reinforcement learning. In International Conference on Machine Learning. PMLR.


summary_of_the_review: The manuscript was a nice and interesting read. The formalism is solid, the benchmark is going to be useful to many researchers and the insights are interesting and although somewhat expected from intuition, it is good to see them result from experiments and written down nicely.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper describes a new framework called Autonomous Reinforcement Learning (ARL) to facilitate studies on continual real-world embodied learning, such as that performed by humans and animals. Most reinforcement learning (RL) studies use episodic benchmark tasks. The paper argues that conducting studies on non-episodic tasks is crucial. After defining ARL, the authors introduce simulated benchmarks. 
main_review: The strength of this paper is that they clearly argue that the studies on continual learning are important and give a clear notion about ARL.
The weakness of this paper is the lack of theoretical novelty and new proposal except for the argument around ARL and its benchmark tasks. 

summary_of_the_review: I totally agree with the authors' argument about the importance of continual reinforcement learning. I also agree that the RL culture that makes agents' life "episodic" tends to prevent us from tackling the problem. I also think the "non-episodic" feature is one of the important ones, although the real-world embodied learning, such as that performed by humans and animals, has many other essential features, e.g., the reward function is not unique and environmental dynamics are changing over time even in an episode, 
Therefore, I like the argument of the authors, especially in the Introduction. 

However, it's questionable if the paper has a sufficient paper as an ICLR conference paper though it has great importance in its aspect of "position paper." 

1) Theoretical contribution around ARL. 
To my understanding, the theoretical notion of ARL is not new. The general and classical framework of RL does not necessarily have the assumption of episodic segmentation. Therefore, the novelty of the authors' proposal about ARL is limited in a theoretical viewpoint though it is quite meaningful in the "cultural" viewpoint.

2) Benchmarks.  
The authors described some benchmarks. However, to my understanding, even though they showed the pre-existing RL algorithms underperformed in the ARL settings, they are not providing a suitable task for developing ARL algorithms. This is because they only showed the pre-existing methods tend to underperform in the ARL settings. 
Some more constructive proposal is expected. 

The paper is very good from the viewpoint of position/short journal paper. However, I am not sure if the paper is suitable for the ICLR conference paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper highlights the important concept of autonomous RL, inspired by real-world scaling constraints of robot learning experiments. Hiding the cost of environmental resets is unrealistic and hinders progress in embodied RL. This incentivises the search for algorithms which score better trade-offs between expensive environment resets and average return of learned policies, using the two definitions given in the paper. Clear conceptual problem formulations are given which reproduce challenges observed during real-world RL experiments with robots, but in tractable settings using simulation.
main_review: Pros:

Excellent example of carefully chosen augmentations of a standard problem formulation in order to factor in hidden costs with substantial practical implications in several experimental settings of interest, such as real-world robotics experiments.

Well designed toy domains which emulate some such difficulties for quick experimentation in a comparable way.


Cons:

It is not clear whether the proposed methodology really deals with issues of environment resets in the fullness of time. Will following this methodology result in much fewer resets and superior learning if tasked to achieve the same level of performance as expensive and impractical full episodic reset methodology, at least for tasks of interest with real-world robots? While algorithms developed for ARL may indeed require fewer interventions in a fixed amount of experiment time, this does not mean that convergence to an optimal policy will not take considerably more experimentation time than episodic reset approaches. If total experimentation time is also a factor, then the trade-off of autonomous operation in the real-world vs. cost of human resets seems to be still an open problem in need of task specific trade-offs.

Perhaps there is no choice but to resort to ARL, but it is imho not clear that ARL doesn’t introduce learning issues of its own, since learning from biased data streams is known to be particularly challenging and empirically leads to suboptimal learning, as per the continual learning literature. This view is not necessarily invalidated by experimental evidence presented in the paper, at least as far as I can tell.

More discussion and standardization may be needed in order to reduce variance of ARL experiments when reporting results. Without the aid of environment resets, ARL data sampling is even more contingent on learned policies and learning dynamics of particular algorithms and function approximators. Hence, standard convergence assumptions and hopes may be even more ephemeral, especially with non-linear function approximation. Also, sensitivity to implementation details may be increased, e.g. hyperparameters, software libraries and their default settings, even model initializations, etc.

summary_of_the_review: This paper gives two convincing definitions of autonomous reinforcement learning and introduces clear evaluation methodologies accessible for the community. Relevant baseline results are provided, such that the proposed benchmarks are good starting points for future research. While some conceptual challenges linger, this could be resolved by followup works.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes a framework for benchmarking episodic and non-episodic algorithms. The authors provide both a mathematical formalisation and a description of an experimental framework composed of existing environments. It is quite difficult to see clear contributions in this paper since the theoretical framework does not really introduce novel concepts and the experimental framework is essentially a collection of existing environments. 
main_review: Even if I always commend papers that propose benchmarks and evaluation framework, I found this work quite problematic. First of all, the authors use the expression autonomous reinforcement learning, which is per se already quite "charged". In fact, the concept of autonomy is rather well defined in our community. It is unclear why the behavior described in this work is considered "autonomous". Usually, the concept of autonomy is defined with respect to the concept of human intervention in setting goals, etc. The authors do not consider this type of autonomy. Instead, they differentiate systems according to learning based on episodes or not. It seems to me that the actual classification used by the authors is not really distant from the existing separation between continual and non continual learning.

The authors also discuss a distinction between systems that are trained using episodic training and then deployed as continual learning systems, but the reviewer believes that it is not really necessary to devise new benchmarks, etc. since existing ones can be considered sufficient to deal with these situations.

The definition of deployed policy evaluation (Definition 1) is rather unclear. How are you going to derive that? It seems to me that in practical terms, the calculation of such a value is not possible.

Definition 2 ("continual policy evaluation") appears to me as the classic definition of a RL problem in a non-episodic case.

The reviewer also does not see the need of introducing the concept of "irreversible states". At the end of the day, a situation where an agent is stuck in a certain situation might happen. The reviewer is not sure why this case should be considered separately. The reviewer also wonders if there is any specific requirement for this assumption.

The models used in the benchmarks are not novel (see Section 5), but they are essentially existing ones. The contributions of the authors with respect these existing environments is unclear.

The evaluation appears to be a sort of standard analysis of the performance of the environments under some specific conditions. The reviewer was not able to see a novel contribution in it.

Minor points:

- The authors say: "Current episodic settings typically provide an environment reset every 100 to 1000 steps, corresponding to ε ∈ (1e-3, 1e-2) and an autonomous operation time of typically a few seconds to few minutes depending on the environment.". I am not sure about this claim of the authors. The reset really depends on the actual experiments.

- The authors say: "In essence, the policy output from the algorithm A trained in the autonomous training setting should do well on the deployment setting as quickly as possible." I am not sure about this - I think it really depends on the environment, if it is stationary, etc. Also, the authors are assuming here that learning happens at deployment time as well and this is not always the case.

summary_of_the_review: The contributions are rather limited in my opinion since the theoretical framework essentially essentially proposes re-definitions of existing concepts. The experimental framework is composed of existing benchmarks so it is difficult to see a contributions there.
review_title: null

Meta-review: This paper formalizes the setting where an autonomous RL agent operates with zero or very few resets, and provides a novel benchmark for this setting with diverse environments ranging from simple manipulation to complex manipulation/locomotion. The paper then uses this benchmark to analyze current methods and provide insight into those crucial factors that affect performance in this setting. The insights into current methods especially are appreciated. As one reviewer stated, "This paper isolates one problematic assumption in the way of [progress in RL], the environment reset problem, and provides the groundwork for [such] progress, i.e. baselines, clear metrics, etc. I believe the community is much better off with this paper published, since prior works don't seem to have used compatible methodologies."
Decision: Accept (Poster)