Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper considers the non-episodic RL setting (no environment resets) where an agent is given experience data from a source task and needs to efficiently learn to maximize rewards. They give discussions on why current RL approaches like SAC and GAIL will struggle in this setting (mainly due to distributional shifts from a source task). They then propose QWALE, an "adverserial imitation learning" approach (similar to GAIL) that uses Q-values (pretrained) to weight a discriminator that is learned and used as auxiliary rewards during training. They show experimentally that previous works (including GAIL) indeed struggle in this setting and are outperformed by QWALE.
strengths_and_weaknesses: This is a very interesting paper that proposes a novel algorithm (QWALE) to learn policies in non-episodic discounted settings. I believe this is very relevant to the RL community, and the contributions are significant relative to previous works. Mainly,
- They propose the single life RL setting, which is a novel more realistic non-episodic setting where the agent is given data collected (and policies/value functions learned) from previous experience.
- They provide intuitive explanations for why previous works can struggle in this setting.
- They provide a number of experiments in multiple domains (including complex continuous domains) demonstrating how QWALE out performs previous RL methods like GAIL in the non-episodic setting.

However, I have a couple important concerns on whether the claims made in the paper are well supported. If I understand correctly, the specific main contributions of this work are the new discriminator update rule (eqn 1) and the reward shaping approach (line 7 in algorithm 1). However 
- No theory is given about them. For example, how does the weightings added to the discriminator update rule affect the theoretical analysis of GAIL [26]. How does the proposed reward shaping affect optimality?
- No explanation was given for the specific choice of the weightings ($exp(Q(s, a) − b$) and $exp(-(Q(s, a) − b))$. Why is there an exponential in there and what is the effect of b theoretically or experimentally?
- b is claimed to incentivize the agent to move towards states with higher value than its current state. This was not supported theoretically nor experimentally.

questions: - Missing reference in line 248: "Appendix ??"
- $q_\theta$ appears in algorithm 1 and line 243, is critical to the approach, but is never defined. Did you mean $q_\phi$?
- How is $D_{online}$ updated in Alg 1?
- In line 618 of the Appendix you give a different reward function used to speed up training (different from the one in Algorithm 1). What exact reward function was used for training? Was the same reward function used for the GAIL baseline?
- In figure 5 (or the Appendix), please include the trajectories for SAC and GAIL (and compare them with QWALE)
- How sensitive is QWALE to the hyper-parameters involved? A hyper-parameter sweep plot would be useful (since there are no theoretical contributions to strengthen the claims)
limitations: The authors have adequately addressed most of the limitations (except where mentioned in my review above) and potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces and tackles the single-life reinforcement learning (SLRL) problem, in which there are no episodic resets available, and the agent has to solve the task in one-shot. The agent has access to prior data which comes from a similar but not the same environment, and this data might not be expert data. The authors propose QWALE, Q-Weighted Adversarial Learning, which extends Adversarial Imitation Learning methods to the general setting in which  the data might not be optimal for the current environment, by weighting the experiences with a pretrained Q-function.
strengths_and_weaknesses: Strength
- It introduces and describes single-life reinforcement learning as a specific set of tasks in which an agent has to learn how to recover from mistakes on its own without relying on episodic reset
- Good comparisons to other methods
Weaknesses
- I am not sure how clear it is the importance of SLRL problems. The examples described of a robot exploring another planet or a rescue robot are closely related to the concept of safe reinforcement learning or safe exploration [1], where some constraints have to be respected. Moreover, I would think that similar problems are addressed in classical robotics motion planning literature.

[1] Garcıa, Javier, and Fernando Fernández. "A comprehensive survey on safe reinforcement learning." Journal of Machine Learning Research 16.1 (2015): 1437-1480.
questions: - Figure 2 positioning is not good. It’s hard to understand what the point of Figure 2 is until you reach Section 7.
- Both PointMass and TableTop training/testing environment sound very similar to the test bed environments used in meta learning papers, where the dynamics or the goals change slightly in the environment. For this reason, I would expect that Meta Learning algorithms like MAML would perform well in environments like tabletop, have the authors tried something similar?
- Figure 5 is never linked, I guess it should be linked somewhere around section 7.3?

limitations: Overall, this paper introduces SLRL and shows how current algorithms are not capable of autonomously recovering in an environment in which they have to reach the goal but are not provided with resets from bad states. It evaluates current algorithms and proposes a new weighted adversarial learning approach to overcome this problem. QWALE is sound and the idea of weighting the examples based on the Q-function trained in the source environment makes sense.
The main limitation of this work is that it’s very similar to other approaches like the ICLR 2022 paper Autonomous Reinforcement Learning [2] which introduces a similar setting in which environment resets are rare or not available at all, and I am not sure how useful it is to the community.

[2] Sharma, Archit, et al. "Autonomous Reinforcement Learning: Formalism and Benchmarking." ICLR 2022.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper motivates single life RL setup involves pre-training on prior experience and generalizing another setting with novel states and that it is different from episodic or reset-free RL. As the objective is to perform well in single life on novel setting, the paper proposes to bias the exploration at test time to the known distribution at training time. For this, GAIL with the discriminator weighted with Q values while training on prior data is used. QWALE trained policy generalizes to new initial position of mug in Tabletop organization, wind in pointmass, hurdles in half cheetah and new combination of task in Franka-Kitchen environment.
strengths_and_weaknesses: The paper is well-written and structured. The paper discusses the The Q-values from prior dataset allow that generalization to novel unseen scenarios at test time.  

However, unlike the motivating example of finding water on Mars where the water would not be found at the same place as a desert on Earth, the experiment setup does not discuss if the goal position would be changed, different from that seen in the prior dataset. The goal remains the same in prior data and “novel” test setting, for example, in pointmass environment at (100, 0), in Franka kitchen with both microwave and cabinet closed, etc.
questions: - SLRL is claimed to be a special case of continual or lifelong learning. SLRL focuses to explore as shown in the prior data and avoids exploring unknowns in the novel setting so that it doesn’t go out of training distribution. How does SLRL method compares to any continual lifelong RL approaches? A comparison could demonstrate scenarios where focussing on the prior data for exploration clearly beneficial (or detrimental) to the performance of the agent.
- Is a fundamental assumption in SLRL that the reward distribution and state which represent goal should have same distribution as the shown in prior trajectories?  It seems that weighting on Q-values from prior data will induce this assumption and limit the possible scenarios in which the policy will generalize to new settings.
limitations: The authors have discussed the potential limitations and future scope of SLRL. Some aspects of potential negative societal impact of the  work should be included, like if SLRL is used in home or space-probing robot, what the scenarios will be that it could not handle.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces the setup of single-life reinforcement learning (SLRL), where the algorithm is given offline data from some environment and then is deployed in a related but slightly different environment; it needs to perform as good as possible in a single trial, leveraging online learning as well as offline data. This differs from both no-reset RL (because we care only about a single trial) and 0-shot generalization in RL (because we can learn along the way).

Then the authors propose QWALE, an algorithm based on GAIL, to tackle the setup of SLRL. They provide an empirical evaluation in 3 different continuous control environments. They compare their approach to several baselines, such as vanilla GAIL, RND, and vanilla SAC. They demonstrate that typically success rate is improved and, more visibly, the speed of reaching the solution is higher.

strengths_and_weaknesses: Strengths:

[S1] The introduced setup of SLRL seems important, describing a quite natural real-world scenario.

[S2] The paper is well written and easy to follow.

[S3] The empirical evaluation is sound and demonstrates improvements brought by the presented approach (see the Questions section for the discussion about baselines though).

Weaknesses:

[W1] The approach assumes that although there is a shift between source and target environments, the state space is essentially the same. If e.g. visual observations were used and the environment shifted from Earth to Mars, then the discriminator would learn to discriminate based on visual cues, which is not what we want in this approach. I think this is fine, it should be clearly stated as a limitation though.

[W2] The novelty of the method itself is limited. It is based on GAIL and adds a weighting based on the critic network, which is reminiscent of critic filtering in CRR ( https://arxiv.org/abs/2006.15134 ). However, it is used in a novel setup.

questions: [Q1] I have suggestions for additional baselines:
- just run SAC in the source environment and evaluate in the target one (without online learning); this is just a sanity check to verify that the shift is challenging enough - the performance should be very poor
- just train SAC on the target environment, completely from scratch
- fine-tune SAC, and add a behavioral cloning loss of the offline data as a regularization; this is another way to “anchor” the solution in the prior data. I conjecture this should help if only start states are changed (and not the dynamics), and otherwise would probably not work well.

[Q2] I’d suggest providing the details about the algorithm clearly in the main text, including the baseline term b described as “implementation detail” and details of normalization of Q values.

[Q3] I’d suggest citing the CRR work (mentioned in W2, section Weaknesses).

limitations: Please see W1 (section Weaknesses) and address this in the text.
review_title: null

Meta-review: metareview: The paper introduces a new formulation for single life reinforcement learning which is interesting. Moreover, an algorithm is presented for solving this RL scenario. The paper was evaluated positively by all reviewers. The 2 borderline reviews main concerns were:
- missing theoretical evidence / motivation for the algorithm (Riviewer muVA): This concern has been mostly addressed by the authors. They motivate their choice of the weights, but how to incorporate the weights into the algorithm is clear to me on an intuition level, but not so much backed on theory. 
- the algorithm was not illustrated for scenarios with changing goals (Reviewer jw3X): This concern was addressed by the rebuttal.

Unfortunately, the two reviewers with the borderline scores  did not respond to the rebuttal, but I think their concerns have been mostly addressed and they should have raised their score. Hence, I recommend accepting the paper. 
confidence: Less certain
Decision: Accept