You Only Live Once: Single-Life Reinforcement Learning
Authors: Annie S Chen
Archit Sharma
Sergey Levine
Chelsea Finn
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
autonomous reinforcement learning
adversarial imitation learning

Abstract: Reinforcement learning algorithms are typically designed to learn a performant policy that can repeatedly and autonomously complete a task, usually starting from scratch. However, in many real-world situations, the goal might not be to learn a policy that can do the task repeatedly, but simply to perform a new task successfully once in a single trial. For example, imagine a disaster relief robot tasked with retrieving an item from a fallen building, where it cannot get direct supervision from humans. It must retrieve this  object within one test-time trial, and must do so while tackling unknown obstacles, though it may leverage knowledge it has of the building before the disaster. We formalize this problem setting, which we call single-life reinforcement learning (SLRL), where an agent must complete a task within a single episode without interventions, utilizing its prior experience while contending with some form of novelty. SLRL provides a natural setting to study the challenge of autonomously adapting to unfamiliar situations, and we find that algorithms designed for standard episodic reinforcement learning often struggle to recover from out-of-distribution states in this setting. Motivated by this observation, we propose an algorithm, Q-weighted adversarial learning (QWALE), which employs a distribution matching strategy that leverages the agent's prior experience as guidance in novel situations. Our experiments on several single-life continuous control problems indicate that methods based on our distribution matching formulation are 20-60% more successful because they can more quickly recover from novel states.

You Only Live Once: Single-Life Reinforcement Learning
Annie Chen', Archit Sharma', Sergey Levine", Chelsea Finn
Stanford University", UC Berkeley²
asc80stanford. edu
Abstract
Reinforcement learning algorithms are typically designed to learn a performant policy that can repeatedly and autonomously complete task, usually starting from scratch. However, in many real-world situations, the goal might not be to learn a policy that can do the task repeatedly, but simply to perform new task successfully once in single trial. For example, imagine a disaster relief robot tasked with retrieving an item from fallen building, where it cannot get direct supervision from humans. It must retrieve this object within one test-time trial, and must do while tackling unknown obstacles, though it may leverage knowledge has of the building before the disaster. We formalize this problem setting, which we call single-life reinforcement learning (SLRL), where an agent must complete a task within single episode without interventions, utilizing its prior experience while contending with some form of novelty. SLRL provides a natural setting to study the challenge of autonomously adapting to unfamiliar situations, and we find that algorithms designed for standard episodic reinforcement learning often struggle to recover from out-of-distribution states in this setting. Motivated by this observation, we propose an algorithm, Q-weighted adversarial learning (QWALE), which employs distribution matching strategy that leverages the agent's 's prior experience as guidance in novel situations. Our experiments on several single- life continuous control problems indicate that methods based our distribution matching formulation are 20-60% more successful because they can more quickly recover from novel states.
Introduction
Agents operating in the real world must often contend with novel situations that differ from their prior experience. For example, cearch-and-rescuee disaster relief robot may encounter novel obstacles while traversing a building. The vast range of situations that could be encountered at test time cannot be anticipated in advance during training. and therefore agents that can adapt test time will be better equipped to cope with new situations. Reinforcement learning (RL) algorithms in principle provide suitable paradigm for such adaptation. However, much of the work in RL focuses learning optimal policy that can repeatedly solve a given task, whereas on-the-fly adaptation simply requires the agent to complete the task set of challenges. In this work, we formalize this problem setting as single-life reinforcement learning (SLRL), where the agent evaluated on its ability to successfully and autonomously adapt to a novel scenario within single trial.
without human interventions supervision, presenting a different
Given some prior experience, SLRL induces the challenge of contending with novelty (c.g., new initial state distribution or new dynamics) at test time without episodic resets. A key problem that makes this difficult is that the agent will need to recover from unfamiliar situations without any
Project website:
life-rl
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Episodic reinforcement learning
Reset-free reinforcement learning
Single life reinforcement learning
train
train
prior
:
test
test
Figure 1: We study the single-life reinforcement learning (SLRL) problem, where given prior data, an agent must complete task autonomously in single trial in domain with novel distribution shift.
interventions, which can be especially challenging when informative shaped rewards are unavailable, as is often the case in the real world. In fact, task rewards sometimes only provided at the end of robot may only receive reward signal after maneuvers to find a
the life, c.g. the search-and-reseee desired object and may therefore get stuck behind unknown debris. In episodic RI., the agent can rely on a reset to recover from an unfamiliar state. In contrast, in SIRL, the agent only has its prior good state distribution. Consequently, we find that fine-tuning a pre-trained value function via online RL will struggle in SLRL settings, as it does not explicitly encourage the agent to return to its known distribution and complete the desired task.
experience to leverage in order to find its way back
To better handle the SLRL setting, we need a way to provide intermediate shaped rewards that guide the adaptation process, and help the agent recover when it gets stuck. One potential option for providing the desired guidance is to use reward shaping towards the agent's distribution of prior experience. Adversarial imitation learning (AIL) approaches such GAIL (Ho and Ermon (2016) can potentially provide such reward shaping, but using existing AlL methods naively may not give the intended behavior in the SLRL setting due two main shortcomings. First, such methods assume expert demonstrations are given as prior data, but in SLRL, we may be given suboptimal offline prior data. Second, AIL methods train the agent to match the entire distribution of prior data, which may be key to learning an optimal policy, but may be a drawback in our setting, as the agent might not be consistently guided towards task completion. To address these shortcomings, we propose Q Q-weighted adversarial learning (QWALE), in which different states in the prior data are weighted different amounts by their estimated Q-value. QWALE incentivizes the agent move towards states in the prior data with higher values, so that agent is more consistently guided towards states closer to task completion.
Our contributions are as follows. We formalize the SLRL problem setting, which provides a natural setting in which to study autonomous adaptation to novel situations in reinforcement learning. Moreover, we believe SLRL is a useful framework for modeling many situations in the real world. We explore reward shaping through distribution matching as one potential choice of methods for SLRL and identify challenges that uniquely arise in the SLRL setting with existing distribution matching approaches. We propose a new approach, Q-weighted adversarial learning (QWALE), which is less sensitive to the quality of prior data available and provides the agent with a shaped reward towards completing the desired task a single time. Through our experiments, we find that QWALE can meaningfully guide the agent to recover to the prior data distribution to complete the desired task 20-60% more successfully on four separate domains compared to existing distribution matching approaches and RL fine-tuning. QWALE provides a strong baseline for developing algorithms that can better adapt to novelty online and recover from --i---i-itritttionn states.
2 Related Work
Autonomous RL. In the context of deep RL, agents typically (but not always) are trained in episodic
setting and are evaluated on the quality of the learned policy. Several recent works have developed
algorithms that can learn without episodic resets (Han et al., [2015] Eysenbach et al.) 2017 Zhu et al.
2020a [harma et al. 202 2022 Gupta et al. 2021 2022 Kim et al. 2022 Sharma et al.) 202 Like our work, such methods aim to make it possible to learn without any episodic resets, but are typically still focused on acquiring an effective policy that can perform the task repeatedly, typically by training some auxiliary controller to enable the policy to "retry" the task multiple times without

---

resets. In contrast, our aim is to develop an algorithm that can solve the task once, but as quickly as possible, which introduces a unique set of challenges as we discussed above.
Continual RL. There is a rich literature on reinforcement learning in the continuing setting Ma--
hadevan 1996 Schwartz 1993 Sutton and Barto 2018 Xu and Zhu 2018, RRlnick et al. 2019
Lomonaco et al. 2020 Wei et al. 2020] that considers maximizing the average reward accumulated
over an infinite horizon without episodic resets. Such works often also consider regret minimization as the objective. SLRL is closely related, and can be viewed as special case where the agent has access a prior offline dataset and aims to solve single task as quickly as possible in new domain with distribution shift. While the focus in continual learning is on general "lifelong" methods or on exploration, our focus is on effectively leveraging prior data in a setting that is meant to be reflective of real-world tasks (for example, in robotics).
Leveraging offline data in online RL. Learning expert policies given prior interaction data has
been extensively studied in imitation learning [Argall et al.) 2009 Ross et al.
2011
Ghasemipour
ct al. 2020], inverse RL (Ng et al.] 2000 [Finn al. [0166] Ziebart et al., 2008
2010
RL for sparse
reward settings [Brys et al. 2015 Nair et al., 20118 Rajeswaran et al. 2017
Hester et al.
2018
Vecerik al.) 2017] and offline RL (Wang et al. 2020b, Levine al.
2020
Kumar et al.
2020
Kidambi et al. 2020 Wu et al. 2019a Nair et al. 2018). Across all these diverse topics, the goal is
to learn a competent policy that can solve the task efficiently whereas the objective in this work is to complete the task in a single trial as quickly as possible. To this end, we build on recent ad adversarial
2016 Fu et al. 2017 [Singh et al. 2019 Torabi et al. 2019 Kostrikov et al.) 2018 Zhu et al. [202bb to encourage agent's s state visitation towards expert prior data where the agent is likely to be successful. Prior methods have also studied adversarial inverse RL and imitation learning with non-expert data [Wang et al., 2018 [unn and Ma 2019 Wu et al.) 2019b] Wang et al. 2020a 202 Cao et al.] [022,] Beliaev et al.] 2022]. However, as we discuss in Section these approaches need to be adapted for the SLRL setting to be efficient at completing the task and to handle novelty, for example when the dynamics may have changed.
approaches inverse RL (Ho and Ermon
Transfer and adaptation in RL. Many prior works have studied the problem of adapting in pres- ence of shifts between train and test settings, often
specific problem setting such as sim2real
transfer (Sadeghi and Levine 2016,] Tobin et al. 2017 Peng et al. Mehta et al. 2020 or fast
adaptation via meta-learning Finn et al. 2017] J. Nichol et al. 2018 J. Nagahandi et al. 2018 Zintgraf et al. [219)) |Finn et al. [2019]. A common theme in these works is that the algorithm can often train in preparation for adaptation at test-time, thus affecting the prior experiences may collect. In contrast, the SLRL setting lays algorithmic emphasis
online exploration and adaptation, as the agent has access to a fixed prior dataset of experiences. Other transfer learning approaches adapt the weights of the policy to new environment or task, either through rapid zero-shot adaptation Hansen et al. [2020), , Yoneda et al. [202T] or through extended episodic online training Khetarpal et al. 2020]. Rusu et al. [216)), Eysenbach et al. [2020). Xie et al. (2020), Xie and Finn [2021). Unlike the latter, we focus on adaptation within a single episode, but, unlike the former, with a focus on extended
exploration and learning over tens of thousands of timesteps. This problem setting leads to unique challenges, namely that the agent must autonomously recover from mistakes, hence requiring a distinct approach.
3 Preliminaries
In this section, we describe some preliminaries before formalizing our problem statement in the following section. We consider an agent that operates in a Markov decision process (MDP) consisting of the tuple M AS P, R, 7), where S is the state space, A is the agent's 's action space, P(set1l8e. ax) represents the environment's transition dynamics, R. : S -> TR indicates the reward function, Po S - denotes the initial state distribution, and 7 E [0, 1) denotes the discount factor. In typical reinforcement learning, the objective is find policy π that maximizes J J(T)) E,~x D>> Rs))s
Although our method is a reward-driven RL. algorithm, we utilize concepts from imitation learning to overcome sparse rewards, utilizing potentially suboptimal prior data. To this end, we build on adversarial imitation learning (AIL), which uses prior data D Pprior in the form of expert demonstrations (we will relax this requirement) to recover the expert's policy.
One such method is GAII. (o and Ermon 2016), which finds a policy uo that minimizes the Jensen- Shannon divergence between its stationary distribution and the expert data. It does so by
3

---

training a discriminator network D: S x A (0, 1), alternating updates with updates to the policy T. Concretely, D and π are learned by optimizing the following:
min max E.p Dog( ())) [, D(s, a)]] -X((=) DE(O,I)SXA
min Dis(p (s. (5 a)) X²((T),
(I)
where Dis is the Jensen- Saasen-Shannon divergence and p² (s. a) is the optimal state-action occupancy measure of the expert Te (represented by demonstrations), In Section we will see that AlL-style discriminator-base approaches can be adapted to the SLRL problem setting without demonstration data, and will therefore form the basis of our method.
4 Single-Life Reinforcement Learning
In this section, we formalize the single-life reinforcement learning (SLRL) problem. The defining characteristic of SLRL that the agent given a single "life" (i.e., one long trial) to complete a desired task, with the trial ending when the task is completed. The agent must complete the task autonomously, without access to any human interventions or resets. Furthermore, as the case in many real-world situations, when faced with situations where task must be completed once, informative shaped rewards are not easily obtained, and so the single life might have reward only once after the task has been completed. However, an agent typically has some prior knowledge. E.g., an agent tasked with retrieving a valuable from a burning building may have experience finding items in that building before the fire. We will therefore assume access to offline prior data of some sort that the agent may use for pretraining and during its single life. In many cases such as the disaster relief example, we may not have expert prior data of the desired task in the desired environment. Hence, we emulate this setting by providing the agent with prior data from a related environment and deploying its single life domain with distribution shift.
We can formalize this setting as follows. We are given prior data prri which consists of transitions from some source MDP Moouree The agent will then interact with a target MDP defined by Munget (S,A, P, R. Po: 7)- We assume that the the target MDP has an aspect of novelty not present in the source MDP, such as different dynamics P(s St. ag) or a different initial state distribution p. Naturally, the more similar the domains are, the easier the problem becomes, and the effectiveness of any algorithm will be strongly dependent on the degree of similarity, though formalizing precise assumption on similarity between the source and target domain is difficult. The reward between the two MDPs the same, meaning the agent is still trying to accomplish the same task in the target domain as in the source.
Maintaining the same notation as the previous section, the SLRL problem aims to maximize J E where h the trial horizon, which may be oc. In general, we expect the task reward to be such that learning only from task rewards during the single life is difficult, as the reward may be very sparse or even awarded only upon successful completion of the task. We assume that there are no sink states beyond terminal success state, such that it is possible for the agent to autonomously recover from any mistake. Note that this setup is essentially the same as the widely studied regret minimization problem in exploration [Mahadevan] 1996). However, while regret minimization is typically studied in the context of RL exploration theory, our aim with SLRL is to study a particular special case of the more general regret minimization framework that is meant to reflect realistic setting in real-world RL (e.g., robotics) where an agent with prior experience must solve a task in a single (potentially long) trial.
As we analyze in Section [6] algorithms designed for episodic policy learning do not perform well in the SLRL problem setting, even when the policy and replay buffer are pre-trained and seeded with the prior data, because they do not quickly recover from mistakes to get back onto a good state distribution. In Section 5 we will discuss an approach based on distribution matching that attempts to address this issue.
5 Q-weighted Adversarial Learning (QWALE)
In this section, we will present our method for addressing SLRL, which we call QWALE. The key insight in QWALE is to utilize the prior data Dprior to efficiently complete the task, especially when the reward information is sparse in the target domain. The framework of AIL provides a reasonable starting point, though it is not sufficient by itself: existing AIL methods assume that the prior data

---

consists of expert demonstrations, and they train the agent to match the entire demo distribution. Since our goal is not learn a policy that repeatedly performs the task, but rather to solve it as quickly as possible once, we do not actually want to learn to imitate prior data, but rather to seek out states that resemble the best states in the prior data, with better states being more preferred. This is especially important when the prior data is not actually optimal, but might consist of arbitrarily suboptimal states. We will discuss how this can be accomplished with a modification of AIL which, instead of treating prior data as equally desirable, preferentially drives the agent toward states that resemble the best states in the prior data.
5.1 Algorithm Description
Our algorithm's 's desired behavior is to lead the agent towards the distribution of prior data, therefore helping it recover when it falls into out-of- offiistibibutiinn states, while also pushing it towards task completion. GAIL (Ho and Ermon 2016) offers a framework shape the state-action distribution towards that of an expert. In particular, GAIL minimizes Dis(p"(s, a) p"(ss, a)) (ignoring the causal entropy H()))). In SLRL, we relax this assumption, as we may only have access suboptimal offline prior data. As such, we desire an algorithm that will be agnostic to the quality of prior data. Hence, instead of learning a policy that aims to uniformly match the entire state-action distribution of the prior data, we want to match the statenaction pairs that will lead to task completion.
The problem of matching the state-action distribution has been studied in stochastic optimal control, particularly in REPS [Peters et al., 2010]. Roughly, this problem can be framed as the RL problem with an additional constraint encouraging the learned distribution to be close to the distribution of the prior data. The solution to this optimization problem gives us that our desired target state- action distribution has the form (s, a) x (s 0) exp (OP(QQTTaa (Tuge (s, a) - V/uua (s)), where the corresponding policy can be derived by using Taareet = Prange (8, a)/ SA Piunge (s, a d')da' using [yedd
ct al. 2008 For formal description of the problem, see Appendix A.I.
Thus, replacing Purget as the target distribution instead of p", we propose Q weighted adversarial learning (QWALE), which minimizes Djs(p" || Puurge)) as follows:
minDis (p= (8, a) ll Plageals, (8, a)) = min max E. pipppii [log D(s, + Eeaamp [log(1 - ((s.a)).
D
exp(Q TurQQueeee (8, a) V (s))
min max D
E aacope
E [exp(( @apusa Musse (s, a) Viages (s)] log D(s, + E
[log(1 D(s,
min max E. where the constant E.
[exp (QTTm (s, VTuae (s)) log Dss, a)) + Esg [log(1 D(s,a))).
pa [cxp(Q (s, a) Musat (s)) can be ignored.
Hence, QWALE trains weighted discriminator using a fixed Q-function Q(s, a) trained in the source MDP to distinguish between useful transitions and ones that may be less useful. This Q-function may be obtained through RL pretraining, which is what we use for our experiments, other ways, such as offline RL or Monte Carlo estimation. We train the discriminator in a similar manner as GAIL, where the positives come from offfine data and negatives from online experience.
variety of
When training the discriminator, we weight the positive states s by exp(Q(s, a) b). where b an implementation detail treated as a hyperparameter. Following /Abdolmaleki et al. [2018]. we exclude Viuges (s) in the weighting term in our practical implementation. Including the term changes the bias on the discriminator, which in effect just adds constant to the reward. In practice, to avoid having to tune b, we just use the value of the most recent state as b, i.e. b= Q(5t, az). We include additional discussion in the Appendix. We normalize the Q-values to be between 0 and by subtracting the min and dividing by the (max- min) and train the Q-weighted discriminator in alternating updates that finetune the policy and critic in an AIL fashion. In this manner, QWALE extends AIL to the general setting with any prior data.
Intuitively, the goal of our weighted discriminator training procedure is to obtain a discriminator that, when used as a reward, will drive the agent toward states that it believes would lead to better outcomes than its present state, based on the prior data. The Q-function quantifies the agent's belief from the prior data that a particular state will lead to high reward, making this a natural choice for estimating how desirable a state is at any given time. Hence, using Q-values to weight the examples for the discriminator will cause the discriminator to prefer states that are closer to the goal over states that are further away. This is significantly different from the behavior we would expect to see if we
5

---

Figure 2: We evaluate in four different domains, including Tabletop-Organizatiin Pointmass, HalfChectah, and a Franka-Kiin Kitchen environment with microwave and cabinet. At test time, an aspect of novelty is introduced in each environment initial mug positions for Tabletop. wind for Pointmass, hurdles for the HaltCheetah, and combination of tasks for the Kitkae.Ktcheen.
were simply imitating the optimal policy, which would give equal weight to all of the transitions along an optimal path.
5.2 Practical Implementation
Algorithm 1 Q-WEIGHTED ADVERSARIAL LEARNING (QWALE)
1: // Single Trial Deployment
2: Require: peiur. test MDP Mies. pretrained critic (ss, a). and (optionally) policy m:
3: Initialize: replay buffer for online transitions Dealiei parameters for discriminator go (prior 8;), timestep
==0
4: while task not complete do
5: Take ~ a.. st), and observe and
6:
Dontine D online U [(.,, des 1]]
7:
Φ Φ V (() Update discriminator according to Eq
8:
r((s)= = r(s)) log(1 9o (prior sc))
9:
Q(s,a), x + (AC(Q(s, a), Dpace U Dontine, r')
10:
Increment
We optimize our objective using maximum entropy off-policy RL with the SAC algorithm (Haarnoja al. (2018), modified similar manner as we did with GAIL, in the previous section. In particular, we learn an additional discriminator go (prior st), optimized using standard cross-entropy loss, which is weighted accordingly.
((Ø)
(exp((
b) log (prior |}] D (log go (online [)]]
The discriminator is used to modify the rewards when updating off- off-policy from all experience prior and online. At single trial test time, the actor and critic are optionally initialized with the pretrained weights, and the replay buffer is initialized with the offline data. For details such as network architecture and hyperparameters, see Appendix A We present the full algorithm in Algorithm
6 Experiments
The goal of our experiments is to answer the following questions: (1) How does QWALE compare to prior reinforcement learning and distribution matching approaches in single-life RL settings? (2) Do distribution matching approaches help agents learn to recover from novel situations in single-life RL? (3) How does QWALE compare to different variants of adversarial imitation learning, with different prior datasets?
6.1 Experimental Setup
To answer the above questions, we construct four single-life RL domains with varying prior datasets and sources of novelty, and then measure performance both in terms of speed of task completion and overall single-life success. In this subsection, we describe this experimental set-up detail.
Environments. We consider the following four problem domains. First, in the Pireetoo-Oraniizatiion environment from the EARL benchmark [Sharma et al. (2021b), the agent tasked with bringing mug to one of four different locations designated by goal coaster. The prior data always has new
the same starting position of the mug. In the target environment, the starting position location unseen in the prior data. Second, the Pointmass setting tasks agent to move in 2D from

---

its starting location at the origin (0,0) to the point (100, 0). The target environment introduces a dynamics shift in the form of strong "wind", where the agent is involuntarily pushed upward in the y-direction each step. Third, we construct a modified HalfCheetah environment, in which it is difficult but feasible for the cheetah to recover when flipped over. The target environment includes hurdles that the cheetah must jump over, as the prior data does not include these obstacles. Finally, we evaluate on a modified Franka- Franka-Kitchen Kitchen environment, adapted from Gupta et al.) [2019], where the task is to close microwave and hinged cabinet. The prior data only contains trajectories of closing the microwave and the hinged cabinet separately, so the agent must figure out online how to complete both tasks in row. In other words, both objects are open at the start of single- life RL, and the agent has only previously seen instances where only one is open. For the latter two environments, dense rewards are given, and the discriminator-bate based reward is added to the extrinsic reward during single-life training. These environments are shown in Figure 2
Further details on the environments
are given in Appendix A
Comparisons. To answer question (1), we compare QWALE to three alternative methods: (a) SAC fine-tuning, which pre-trains policy and value function in the source setting and fine-tunes for single, long episode in the target environment, (b) SAC-RND, which additionally includes an RND exploration bonus [urda et al. 2018 during single-life fine-tuning, and (c) GAIL-s, which runs generative adversarial imitation learning Ho and Ermon (2016), Kostrikov et al. (018) where the discriminator only operates on the current state s. We choose for the discriminator to only look at so that it is less susceptible to dynamics shift between the source data and target environment. We additionally compare GAIL-sa, which passes both the current state and action to the discriminator. All methods use soft actor-critic (SAC) Haarnoja et al. (0188) as the base RI. algorithm. We present additional comparisons in the Appendix.
Prior datasets. For all four environments, we evaluate SLRL using data collected through RL as our prior data. More specifically, we run SAC in the source MDP in the standard episodic RL setting for K steps and take the last 50,000 transitions as the prior data. K is chosen such that the prior data contains some good transitions but has not converged to optimal policy yet. While we are able to run episodic RL in the source MDP, this is not a requirement for SLRL, as long as prior data in the source MDP is available. For all methods, including QWALE, GAIL variants, and SAC fine-tuning variants, the policy and value function are pretrained in this manner for the initialization of single-life RL. We note that AlL methods like GAIL typically assume that the prior data consists of expert demonstrations but we apply the algorithm only using mixed quality prior data, unless otherwise noted. In particular, Section 6.4 further evaluates AIL methods using demos as prior data, using 10 demonstrations for the Tabletop environment and 3 demonstrations for the Pointmass domain. We include such experiments to answer question (3), i.e. investigate how the quality of prior data may affect performance.
Evaluation Metrics. To evaluate each method in each environment, report the average and median number of steps taken before task completion across 10 seeds along with the standard error and success rate (out of 10). During single-life life RL, for all environments, the agent is given maximum of 200,000 steps to complete the task. If it has not completed the task after 200k steps, then 200k is logged the total number of steps, and the run is marked as unsuccessful.
6.2 Results using mixed data as prior data
In this subsection, we aim to answer our first experimental question and study how QWALE performs compared prior reinforcement learning and distribution matching approaches in single-life RL settings. As seen in Table I we find that QWALE achieves the lowest average and median number of steps as well as highest number of successes on all four domains. In particular, QWALE takes 20-40% fewer steps on average as the next best performing method. GAII-S and GAIL-sa also outperform finetuning SAC across three of the four domains. These results demonstrate the suitability of beabuutiooommtctiiin approaches over RL finetuning in the SLRL setting. We see that guidance particularly towards a good state distribution is important, as we compare to finetuning SAC with an exploration bonus through random network distillation (Burda et al. (2018). From Table [] although RND may improve performance. particularly in the Tabletop domain, it generally does not perform as well as the distribution matching approaches, especially QWALE, showing that simply increasing exploration is not enough. Furthermore, these results show that the additional shaping provided by weighting the prior data by Q-value when training the discriminator can significantly improve guidance towards the goal. While GAIL. gives equal weight to all transitions along an optimal path, the agent in QWALE is consistently guided towards states in the prior data with higher
7

---

Method
Aug
Std crror
Success/
Median
Method
Avg
Sid
Success / 10
Median
Tabletop
GAIL-s
83.2k +: 23.8k
8
75.6k
Cheetah
GAILs
99.2k 23 23.0k
7
77.4k
GAIL-sa
61.5k 1: 28.7k
2.4k
GAIL-sa
102.0k
19.3k
85.6k
128.9k
23.9k
SAC-RNI>
94.8k
26.9k
51.4k
SAC-RND
158.4
20.5k
200.0k
QWALE (ours)
44.4k
24.6k
8.9k
QWALE: (ours)
9.5k 74.3k
10
68.9k
Pointmaas
GAIL-s
100.9% 1: 33.0%
5
101.90
Kitchen
GAILs
111.3k 2: 27.9K
122.1k
GAIL-sa
140.4% 1. 30.33
200.0k
GAIL-sa
127.8k 26.9k
189.1k
198.1k
112.5k
SAC-RNI>
184.11 1k 15.9%
200
SAC-RND
132.7k 26.2k
200.0k
QWALE (ours)
61.0k 28.8K
1.7k
QWALE (ours)
79.9 21.7k
59.33
We evaluate the performance of QWALE to finetuning SAC and GAIL in our four environments using mixed data collected through RL as prior data. We omit the results of Behavior Cloning (BC), as it is unsuccessful completing the task in every domain due the distribution shift. We find that GAIL outperforms finetuning SAC and SAC-RND 3 4 domains, and QWALE outperforms both GAIL variants on domains on the average number of steps before task completion. All methods are evaluated over 10 seeds.
Table I:
Q-value, leading to more efficient and reliable single-life task completion. Additional comparisons are presented the Appendix.
6.3 Recovering from novel situations in SLRL
Next, answer question (2), we ana- lyze how QWALE helps agents learn to recover from novel situations SLRL. First, we analyze the state visitation of SAC finetuning in the online phase for
SAC Fine-Tuning
QWALE
1000
videce
Tabletop
223800
1000c0
the Tabletop and Pointmass domains. key challenge of SLRL (and also fully autonomous RL in general) is that the agent falls off of the dis- tribution, it cannot rely resets to get back on track. Since there a gap between the source and target do- mains, the agent will inevitably find itself in states that out of distri- bution from the prior data. A value function pre-trained the source data could in principle be used to evalu- states and guide the agent back towards good states, but it will be inaccurate on states outside of the prior data Fu et al. [2019]; then, when the value of some of those out-of- distributions states is overestimated, the value function may misguide the policy away from good states. Hence, fine-tuning a pre-trained value func- tion via online RI. will not explicitly encourage the agent to get back on distribution, especially in sparse reward envs. As a result, the agent may spend a lot of time (perhaps infinite time) drifting once it falls out of distribution, which
15000
335ec0
Jstece
7000
Pointmass
330000
sosc
4000
2200
23000
1000
Goal
Figure 3: We visualize the online state visitation of the mug in the Tabletop (top) and the agent the Pointmass (bottom) environments of single-life trial using SAC finetuning (left) and QWALE (right). The expert states are purple, and we color the trajectories green to blue according to timestep for both methods. See Section for plots colored according to reward. In both environments. the SAC fine-tuning agent fails recover from novel states, but guided towards task completion with QWALE.
occurs in Figure 3
On the other hand, distribution-matchin methods like QWALE will explicitly encourage the agent to get back on distribution, by giving higher rewards on distribution than off distribution. In Figure we visualize QWALE's state visitation in the Tabletop and Pointmass domains throughout a single lifetime. We color the trajectories according to timestep for both methods, and include the trajectories colored by reward (discriminator score) Figure 5 The coloring in the timestep-colored plots highly correlated with that in the reward-colored plots, showing how the reward helps the agent recover from unfamiliar states and gradually guides the agent towards the goal. In particular, when the agent is out of distribution, the agent is incentivized explore states that will lead it closer back

---

Average Steps Task Completion
Median Steps to Task Comsletion
Successes (out of 10)
200000
10
Number Steps
1.50000
Number Stepps
150000
8-
100000
100000
Number
50000
50000
21
Tabletop
Polittmss
Tabletop
Pointmass
Tabletop
Pointmass
Environment
Environment
Environment
GAIL-sa
QWALE
Figure 4: Discriminator based approaches to single- life RL with expert demonstrations as the prior data. Given demonstration data, in both the Tabletop and Pointmass domains, QWALE matches significantly outperforms both GAII. variants all metrics.
to the prior state distribution, and when the agent within distribution, is incentivized to move to states closer to the goal, leading efficient task completion.
6.4 Using demos as prior data
Finally, we evaluate the performance of different discriminator-base approaches in the two SLRL problem Bettings-Tabbette and Pointmass-winn where demonstration data is available prior data. We compare using AIL with a state-only discriminator (GAIL-s) as well as with a state-action discrimina- tor (GAIL-sa) to our proposed Q-weighted discriminator method (QWALE). With the latter method, we have access to the same Q-function pretrained when collecting prior data using standard RL for the mixed data experiments above, but we do not initialize any of the algorithms test time with the pretrained policy and critic weights.
From Figure 4 the GAIL variants are both able to consistently solve the task in the Tabletop domain, but unsurprisingly, GAIL--sa does especially poorly in the Pointmass domain, where the dynamics have changed at test time. Compared to the two GAIL variants, QWALE gives significant improvement in both domains. These results demonstrate how more detailed reward shaping towards the completion of the desired task can be helpful in the SLRL setting even with demonstrations as prior data. Moreover, comparing these results with those Table | while access to expert data as prior data unsurprisingly improves the performance of GAIL methods, it can also improve the performance of QWALE.
Conclusion
In this paper, we formalized and studied the problem setting of single-life reinforcement learning: setting where an agent needs to autonomously complete a task once in single trial while drawing upon prior experience from related environment. We found that standard fine-tuning via RL is ill- suited for this problem because the algorithm struggles to recover from mistakes and novel situations. We hypothesized that this happens since resets in episodic RL prevent algorithms from needing to recover, whereas single-life RL and continuing settings in general do demand the agent to find its way back to good states on its own. We then postulated that distribution matching methods that aim to match the distribution of related prior data may help agents recover via reward shaping, and presented a new distribution matching method, QWALE, that weights examples by their Q Qaluue Our experiments verified that distribution matching approaches indeed do make better use of prior data, and that QWALE outperforms prior distribution matching methods on four single-life RL problems.
While QWALE can efficiently complete novel target tasks in a single episode without any interventions, important limitations remain. No algorithm, including QWALE, was able to complete the target task with 100% success, indicating that future works should aim to improve an algorithm's ability to solve tasks consistently. Moreover, the methods that we evaluated all used pre-trained policy and value function from the source domain, which may be difficult to obtain in some source scenarios, as opposed only obtaining some demonstrations or offline data. Finally, would be interesting to explore problems with greater degrees of novelty between the source and target environments. In our experimental domains, the reward and state space between the source and target MDPs were assumed to stay the same. Eventually, we hope that this work may be extended to the case where the reward may different the online trial from the prior data. We expect that such settings would place even greater importance autonomy and exploration, requiring sophisticated strategies for adapting

---

online to novelty. We hope that future work can explore these interesting questions and continue to make progress on allowing RL agents to autonomously complete tasks within a single lifetime.
Acknowledgments
We thank members of the IRIS lab for helpful discussions on this project. This research was supported by Google and the Office of Naval Research (ONR) via grants N00014-21-1-2685 and N00014-20-1: 2675. Annie Chen is supported by an NSF graduate student fellowship. Chelsea Finn is CIFAR fellow.
References
Abbas Abdolmalcki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint 1rXiv:1806, 0896,06920, 2018.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-48,, 2009.
Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. arXiv preprint arXiv:2202.0/288, 2022.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor. and Ann Nowé. Reinforcement learning from demonstration through shaping. In Twenty-fourth international joint conference on artificial intelligence, 2015.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018.
Zhangjie Cao, Zihan Wang, and Dorsa Sadigh. Learning from imperfect demonstrations via adversar- ial confidence transfer. arXiv preprint arXiv:2202.02907. 2022.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint arXiv: 2017.
Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.1399 2020.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta- letanlea In International Conference on Machine Learning, pages 1930-1930. PMLR, 2019.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- 2017.
ment learning. arXiv preprint arXiv:1710.110
Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine Diagnosing bottlenecks in deep q-learning algorithms. In International Conference on Machine Learning, pages 2021-2030. PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective imitation learning methods. In Conference on Robot Learning, pages 1259-1277. PMLR, 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving hong-horizon tasks via imitation and reinforcement learning. arXiv preprint axXiv:1910.11956. 2019.
Abhishek Gupta, Justin Yu, Tony ZZaa Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 6664-6671. IEEE, 2021.
10

---

Abhishek Gupta, Corey Lynch, Brandon Kinman, Garrett Peake, Sergey Levine, and Karol Haus- man. Bootstrapped autonomous practicing via multi-task reinforcement learning. arXiv preprint arXiv:2203. 15755, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pages 1861-1870. PMLR, 2018.
Weiqiao Han, Sergey Levine, and Pieter Abbeel. Learning compound multi-step controllers under unknown dynamics. In 2015 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 6435-6442. IERE, 2015.
Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyå, Pieter Abbeel, Alexei A Efros, Lerrel Pinto, and Xiaolong Wang. Self-supervised policy adaptation during deployment. arXiv preprint 0.Xiv:2007.04309. 2020.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. arXiv preprint arXiv:2012.13490, 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Jigang Kim, J hyeon Park, Daesol Cho, and H Jin Kim. Automating reinforcement learning with example-based resets. IEEE Robotics and Automation Letters, 2022.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Diseriminator-actor aat.oor--cttooetttiic Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint 1 2018.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33: 179-1191, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.0/6 2020.
Vincenzo Lomonaco, Karan Desai, Eugenio Culurciello, and Davide Maltoni. Continual reinforce- ment learning in 3d non-stationary environments. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
Sridhar Mahadevan. Average reward reinforcement learning: Foundations, algorithms, and empirical
results. Machine learning, 22(1):159-195, 1996.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. Conference on Robot Learning, pages 1662-1176 PMLR, 2020.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reiiforeeement learning, arXiv preprint arXiv: 1803. 2018.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on roboties and automation (ICRA), pages 6292-6299. IEEE, 2018.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. Ieml, volume 1, page 2, 2000.

---

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta- meta-learning algorithms. arXiv preprint mttp::i830.09999 2018.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Picter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth
AAAI Conference on Artificial Intelligence, 2010.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 1709. 10087, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Gamett, editors, Advances Neural Information Processing Systems, volume 32. /tatttts://proeeeeee http:///toceeeen..e/ee wwwgggs...eurigg ceeurrpp.cc/ppper//011//// fa7 a Paper pdf
Curran Associates, Inc., 2019,
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627 635. JMLR Workshop and Conference Proceedings, 2011.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.04671, 2016.
Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without a single real image. arXiv preprint arXiv:1611.04201, 2016.
Anton Schwartz. A reinforcement learning method for maximizing undiscounted rewards. In Proceedings of the tenth international conference on machine learning, volume 298, pages 298- 305, 1993.
Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous reinforcement learning via subgoal curricula. In M. Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 18474 18486. Curran Associates, Inc., 2021a. URL UtL.httpss//rrrcediin .df
neurips.
Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Formalism and benehmarking. arXiv preprint arXiv:2112.09005, 2021b.
Archit Sharma, Rehaan Ahmad, and Chelsea Finn. A state-distribution matching approach to non-episodic reinforcement learning. arXiv preprint 2.XXi::205.052222 2022.
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv:1904.0783 2019.
Mingfei Sun and Xiaojuan Ma. Adversarial imitation learning from incomplete demonstrations. arXiv preprint arXiv: 12311195..111,,,. 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear program- ming. In Proceedings of the 25th international conference Machine learning, pages 1032-1039, 2008.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEFF/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
12

---

Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial imitation learning from state-only demonstrations. Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2229-2233 2019.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint arXiv: 1707.08817, 2017.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Ruohan Wang, Carlo Ciliberto, Pierluigi Amadori, and Yiannis Demiris. Support- adversarial imitation learning. arXiv preprint arXiv:2002.0880) 2020a.
Yunke Wang, Chang Xu, and Bo Du Robust adversarial imitation learning via adaptively-scleted demonstrations. In Zhi-Hua Zhou, editor. Proceedings of the Thirtieth International Joint Confer- ence on Artificial Intelligence, IJCAI-21, pages 3155-3161. International Joint Conferences on Artificial Intelligence Organization, 2021a.
Yunke Wang, Chang Xu, Bo Du, and Honglak Lee. Learning to weight imperfect demonstrations, In International Conference Machine Learning, pages 10961-10970. PMLR, 2021b.
Ziyu Wang, Alexander Novikov, Konrad Zoina, Josh S Merel, Jost Tobias Springenberg. Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7773 2020b.
Chen-Ya Yu Wei, Mehdi Jafarnia Jahromi, Haipeng Luo, Hiteshi Sharma, and Rahul Jain. Model- free reinforcement learning in infinite-horizon Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10170-10180. PMLR, 13-18 Jul 2020.
reward Markov decision processes. In
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1911::9111.13611 2019a.
Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imita- tion learning from imperfect demonstration. In International Conference on Machine Learning, pages 6818 8-877. PMLR, 2019b.
Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences. arXiv preprint arXiv:2 1et.:2/090098888 2021.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity. arXiv preprint arXiv:2006. 10701, 2020.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Basac-Bincch,,,, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL https:/::://pro http://ppceeeeig..../ www.eediinnn..urips....s.s.eesss neurips cc/
Paper pdf
Takuma Yoneda, Ge Yang, Matthew R Walter, and Bradly Stadie. Invariance through inference. arXiv 22. preprint 1tt::i::2122.08222
2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 2017.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of roal-world robotic reinforcement learning. arXiv preprint arXiv:2004. 12570, 2020a.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. In the fourt Annual Conference on Neural Information Processing Systems (NeurIPS 2020). 2020b.
13

---

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 143--133.. Chicago, IL, USA, 2008.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In ICML, 2010.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08344, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you diseuss any potential negative societal impacts of your work? [N/A] We do not expect any negative societal impacts that are unique to this work compared to other works on reinforcement learning.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Included in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Included in the supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See experiments in Section ??.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Included the supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "j44HPvaKBo7", "reviewer": "Reviewer_jw3X", "summary": "The paper motivates single life RL setup involves pre-training on prior experience and generalizing another setting with novel states and that it is different from episodic or reset-free RL. As the objective is to perform well in single life on novel setting, the paper proposes to bias the exploration at test time to the known distribution at training time. For this, GAIL with the discriminator weighted with Q values while training on prior data is used. QWALE trained policy generalizes to new initial position of mug in Tabletop organization, wind in pointmass, hurdles in half cheetah and new combination of task in Franka-Kitchen environment.", "questions": "- SLRL is claimed to be a special case of continual or lifelong learning. SLRL focuses to explore as shown in the prior data and avoids exploring unknowns in the novel setting so that it doesn’t go out of training distribution. How does SLRL method compares to any continual lifelong RL approaches? A comparison could demonstrate scenarios where focussing on the prior data for exploration clearly beneficial (or detrimental) to the performance of the agent.\n- Is a fundamental assumption in SLRL that the reward distribution and state which represent goal should have same distribution as the shown in prior trajectories?  It seems that weighting on Q-values from prior data will induce this assumption and limit the possible scenarios in which the policy will generalize to new settings.", "limitations": "The authors have discussed the potential limitations and future scope of SLRL. Some aspects of potential negative societal impact of the  work should be included, like if SLRL is used in home or space-probing robot, what the scenarios will be that it could not handle.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and structured. The paper discusses the The Q-values from prior dataset allow that generalization to novel unseen scenarios at test time.  \n\nHowever, unlike the motivating example of finding water on Mars where the water would not be found at the same place as a desert on Earth, the experiment setup does not discuss if the goal position would be changed, different from that seen in the prior dataset. The goal remains the same in prior data and “novel” test setting, for example, in pointmass environment at (100, 0), in Franka kitchen with both microwave and cabinet closed, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "eKcMAjUBAMu", "reviewer": "Reviewer_bBp1", "summary": "This paper introduces and tackles the single-life reinforcement learning (SLRL) problem, in which there are no episodic resets available, and the agent has to solve the task in one-shot. The agent has access to prior data which comes from a similar but not the same environment, and this data might not be expert data. The authors propose QWALE, Q-Weighted Adversarial Learning, which extends Adversarial Imitation Learning methods to the general setting in which  the data might not be optimal for the current environment, by weighting the experiences with a pretrained Q-function.", "questions": "- Figure 2 positioning is not good. It’s hard to understand what the point of Figure 2 is until you reach Section 7.\n- Both PointMass and TableTop training/testing environment sound very similar to the test bed environments used in meta learning papers, where the dynamics or the goals change slightly in the environment. For this reason, I would expect that Meta Learning algorithms like MAML would perform well in environments like tabletop, have the authors tried something similar?\n- Figure 5 is never linked, I guess it should be linked somewhere around section 7.3?\n", "limitations": "Overall, this paper introduces SLRL and shows how current algorithms are not capable of autonomously recovering in an environment in which they have to reach the goal but are not provided with resets from bad states. It evaluates current algorithms and proposes a new weighted adversarial learning approach to overcome this problem. QWALE is sound and the idea of weighting the examples based on the Q-function trained in the source environment makes sense.\nThe main limitation of this work is that it’s very similar to other approaches like the ICLR 2022 paper Autonomous Reinforcement Learning [2] which introduces a similar setting in which environment resets are rare or not available at all, and I am not sure how useful it is to the community.\n\n[2] Sharma, Archit, et al. \"Autonomous Reinforcement Learning: Formalism and Benchmarking.\" ICLR 2022.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- It introduces and describes single-life reinforcement learning as a specific set of tasks in which an agent has to learn how to recover from mistakes on its own without relying on episodic reset\n- Good comparisons to other methods\nWeaknesses\n- I am not sure how clear it is the importance of SLRL problems. The examples described of a robot exploring another planet or a rescue robot are closely related to the concept of safe reinforcement learning or safe exploration [1], where some constraints have to be respected. Moreover, I would think that similar problems are addressed in classical robotics motion planning literature.\n\n[1] Garcıa, Javier, and Fernando Fernández. \"A comprehensive survey on safe reinforcement learning.\" Journal of Machine Learning Research 16.1 (2015): 1437-1480.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KBDgrML8Zqr", "reviewer": "Reviewer_bMAN", "summary": "The paper introduces the setup of single-life reinforcement learning (SLRL), where the algorithm is given offline data from some environment and then is deployed in a related but slightly different environment; it needs to perform as good as possible in a single trial, leveraging online learning as well as offline data. This differs from both no-reset RL (because we care only about a single trial) and 0-shot generalization in RL (because we can learn along the way).\n\nThen the authors propose QWALE, an algorithm based on GAIL, to tackle the setup of SLRL. They provide an empirical evaluation in 3 different continuous control environments. They compare their approach to several baselines, such as vanilla GAIL, RND, and vanilla SAC. They demonstrate that typically success rate is improved and, more visibly, the speed of reaching the solution is higher.\n", "questions": "[Q1] I have suggestions for additional baselines:\n- just run SAC in the source environment and evaluate in the target one (without online learning); this is just a sanity check to verify that the shift is challenging enough - the performance should be very poor\n- just train SAC on the target environment, completely from scratch\n- fine-tune SAC, and add a behavioral cloning loss of the offline data as a regularization; this is another way to “anchor” the solution in the prior data. I conjecture this should help if only start states are changed (and not the dynamics), and otherwise would probably not work well.\n\n[Q2] I’d suggest providing the details about the algorithm clearly in the main text, including the baseline term b described as “implementation detail” and details of normalization of Q values.\n\n[Q3] I’d suggest citing the CRR work (mentioned in W2, section Weaknesses).\n", "limitations": "Please see W1 (section Weaknesses) and address this in the text.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n[S1] The introduced setup of SLRL seems important, describing a quite natural real-world scenario.\n\n[S2] The paper is well written and easy to follow.\n\n[S3] The empirical evaluation is sound and demonstrates improvements brought by the presented approach (see the Questions section for the discussion about baselines though).\n\nWeaknesses:\n\n[W1] The approach assumes that although there is a shift between source and target environments, the state space is essentially the same. If e.g. visual observations were used and the environment shifted from Earth to Mars, then the discriminator would learn to discriminate based on visual cues, which is not what we want in this approach. I think this is fine, it should be clearly stated as a limitation though.\n\n[W2] The novelty of the method itself is limited. It is based on GAIL and adds a weighting based on the critic network, which is reminiscent of critic filtering in CRR ( https://arxiv.org/abs/2006.15134 ). However, it is used in a novel setup.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G3k2E3nkTzq", "reviewer": "Reviewer_muVA", "summary": "This paper considers the non-episodic RL setting (no environment resets) where an agent is given experience data from a source task and needs to efficiently learn to maximize rewards. They give discussions on why current RL approaches like SAC and GAIL will struggle in this setting (mainly due to distributional shifts from a source task). They then propose QWALE, an \"adverserial imitation learning\" approach (similar to GAIL) that uses Q-values (pretrained) to weight a discriminator that is learned and used as auxiliary rewards during training. They show experimentally that previous works (including GAIL) indeed struggle in this setting and are outperformed by QWALE.", "questions": "- Missing reference in line 248: \"Appendix ??\"\n- $q_\\theta$ appears in algorithm 1 and line 243, is critical to the approach, but is never defined. Did you mean $q_\\phi$?\n- How is $D_{online}$ updated in Alg 1?\n- In line 618 of the Appendix you give a different reward function used to speed up training (different from the one in Algorithm 1). What exact reward function was used for training? Was the same reward function used for the GAIL baseline?\n- In figure 5 (or the Appendix), please include the trajectories for SAC and GAIL (and compare them with QWALE)\n- How sensitive is QWALE to the hyper-parameters involved? A hyper-parameter sweep plot would be useful (since there are no theoretical contributions to strengthen the claims)", "limitations": "The authors have adequately addressed most of the limitations (except where mentioned in my review above) and potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a very interesting paper that proposes a novel algorithm (QWALE) to learn policies in non-episodic discounted settings. I believe this is very relevant to the RL community, and the contributions are significant relative to previous works. Mainly,\n- They propose the single life RL setting, which is a novel more realistic non-episodic setting where the agent is given data collected (and policies/value functions learned) from previous experience.\n- They provide intuitive explanations for why previous works can struggle in this setting.\n- They provide a number of experiments in multiple domains (including complex continuous domains) demonstrating how QWALE out performs previous RL methods like GAIL in the non-episodic setting.\n\nHowever, I have a couple important concerns on whether the claims made in the paper are well supported. If I understand correctly, the specific main contributions of this work are the new discriminator update rule (eqn 1) and the reward shaping approach (line 7 in algorithm 1). However \n- No theory is given about them. For example, how does the weightings added to the discriminator update rule affect the theoretical analysis of GAIL [26]. How does the proposed reward shaping affect optimality?\n- No explanation was given for the specific choice of the weightings ($exp(Q(s, a) − b$) and $exp(-(Q(s, a) − b))$. Why is there an exponential in there and what is the effect of b theoretically or experimentally?\n- b is claimed to incentivize the agent to move towards states with higher value than its current state. This was not supported theoretically nor experimentally.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept