SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning
Authors: Marco Bagatella
Sammy Joe Christen
Otmar Hilliges
Venue: Accepted by TMLR ()

Abstract: Efficient exploration is a crucial challenge in deep reinforcement learning. Several methods, such as behavioral priors, are able to leverage offline data in order to efficiently accelerate reinforcement learning on complex tasks. However, if the task at hand deviates excessively from the demonstrated task, the effectiveness of such methods is limited. In our work, we propose to learn features from offline data that are shared by a more diverse range of tasks, such as correlation between actions and directedness. Therefore, we introduce state-free priors, which directly model temporal consistency in demonstrated trajectories, and are capable of driving exploration in complex tasks, even when trained on data collected on simpler tasks. Furthermore, we introduce a novel integration scheme for action priors in off-policy reinforcement learning by dynamically sampling actions from a probabilistic mixture of policy and action prior. We compare our approach against strong baselines and provide empirical evidence that it can accelerate reinforcement learning in long-horizon continuous control tasks under sparse reward settings.

Published in Transactions on Machine Learning Research (08/2022)
SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning
Marco Bagatella
marco.bagatellaaaii winf.eth..ch
Deportment of Computer Science
ETH Zürich
Sammy Christen
sow.my.chrrislee@@nn....
Department of Computer Science
ETH Zürich
Otmar Hilliges
ofmar ddmar.hilkgeesiinf.fce eth.:/e.ssii.f.hh......
Department of Computer Science
ETH Zürich
Reviewed on OpenReview: https: // openreview. net/ net/forum?
forum? ? id= FfWfuFCX9
Abstract
Efficient exploration is crucial challenge in deep reinforcement learning. Several methods, such as behavioral priors, are able to leverage offline data in order to efficiently accelerate reinforcement learning on complex tasks. However, the task at hand deviates excessively from the demonstrated task, the effectiveness of such methods limited. In our work, we propose learn features from offline data that are shared by more diverse range of tasks, such correlation between actions and directedness. Therefore, introduce state-free priors, which directly model temporal consistency in demonstrated trajectories, and are capable of driving exploration in complex tasks, even when trained on data collected on simpler tasks. Furthermore, we introduce a novel integration scheme for action priors in off- policy reinforcement learning by dynamically sampling actions from a probabilistic mixture of policy and action prior. We compare our approach against strong baselines and provide empirical evidence that can accelerate reinforcement learning lorg-horizon continuous control tasks under sparse reward settings.
1 Introduction
Exploration is a fundamental issue in reinforcement learning (RL): in order for an agent to maximize its reward signal, needs adequately cover its state space and observe the outcome of actions. This becomes increasingly difficult when dealing with large, continuous state and action spaces, as is the case in many real world applications. Despite large and fruitful body of research exploration (Bellemare et al.,
2016
Osband a.. 2016
Azizzadenesheli et al. 2018 Burda et al, Dabney et al. 2021 Ecoffct et al. 2021 Vulin et al. 2021 most genoral-purpose algorithms remain
Tang et al., 2017 Osband a.. 2018
2019
based on e-greedy exploration (Muih et al. 2015) or ent Gaussian policies (Haarnoja el al., In the absence of an informative reward signal, both methods rely on uniformly sampling actions from the action space, independently of the history of the agent. Unfortunately, in sparse reward settings, achieving positive returns by uncorrelated exploration becomes exponentially less likely as the horizon length increases (Dabney et. al. 2021
2018

---

Published in Transactions on Machine Learning Research (08/2022
Offline
State-fre Prior s(al
History H
tralu
Task-ngnostic Trajectories
Mixture
secppl
Domain
Gap
Policy n(ala]
State
Acton
Downstrsam Environment
Figure 1: SFP: A state-free prior is trained on weakly informative trajectories, such as reaching random goals reaching task (top left). Actions then sampled from dynamic mixture between the state-free prior and the policy in downstream learning of more complex tasks. Our method works with both vector- vector-based and image- based state impuls.
A promising approach achieve efficient exploration is that of using a behavioral prior to guide the policy Perrssh al., 2020; Tirumala et al. 2020; Siingh al.) 2021]. Typically, this is learned from expert trajectories state-conditional action distribution. Behavioral priors able to foster directed exploration (Singh et al. 2021), by assuming strong similarity between the agent and expert tasks. However, agent should idcally be able to produce efficient explorative behaviors across broad range of diverse tasks, even if they are rather different from those demonstrated (Parisi et al. 2021). Similarly, it should be possible lo leverage information collected on simpler tasks to enable learning in more complex scenarios (Florensa et al. 2017 Gehring et al. 2021
Let us, for instance, consider surveying robot that trained from scratch in simple room, and needs to be redeployed to complex construction site: even if observations in the second environments drastically different from those previously received (e.g. due to different lighting conditions and terrain patterns), relevant information could still be recovered from past experiences. We argue that such information often includes correlation and directedness in the robot's 's behavior. In particular, these characteristics of the robot behavior are not necessarily related in their entirety to the state space it operates in.
In this paper, we thus propose to focus on the temporal structure of demonstrations rather than on task- specific strategies. We find that this choice enables knowledge transfer from simple offline trajectories to diverse family of more complex tasks. Our method, which dub State-frce Priors for exploration in Off-policy Reinforcement Learning (SFP), introduces state-free priors slate-independenl non-Markovian action distributions π Ta=:aap)) modeling promising actions conditioned on past actions. We find that this class of priors is sufficient for capturing desirable propertics for exploration, such as dircctedness and temporal correlation. This information often learnable from few expert trajectories collected in simple environments (e-g- reaching uniformly sampled end-effector positions for a robotic arm, see Figure T While we assume that such trajectories display qualities such as correlation and directedness, and in particular reflect good exploration strategies for the task hand, do not require them to explicitly encode state-action de- pendencies specific to downstream tasks. State-free priors can then be used to guide exploration in more complex tasks, in which observed states are not guaranteed to belong to the expert state distribution, and behavioral priors are, as a result, inherently limited.
Furthermore, we propose a principled manner of integrating action priors into the Soft Actor Critic framework (Haarnoja al. 2018). In downstream Icarning of more complex tasks, our method samples actions from dynamic mixture between the policy and state-free prior. Crucially, we derive an update rule for learning the mixing coefficient by directly optimizing (max-entropy) RL objective. Our dynamic integration allows the policy to receive strong guidance when needed, while crucially retaining the ability to explore diverse actions when the prior is no longer beneficial.
In our experiments, we first analyze the choice of conditioning variables for action priors by measuring their effectiveness accolerating downstream RL. We then focus on stato-frce priors and verify their capability to produce correlated and directed behavior. Most importantly, we provide empirical evidence that SFP can leverage weakly informative trajectories to accelerate learning. For this purpose, our method is evaluated
2

---

Published in Transactions on Machine Learning Research (08/2022)
against several baselines in complex long-horizon control tasks with sparse rewards. Finally, present further applications of SFP, including leveraging offline state-based trajectories to (a) accelerate visual RL. or (b) guide exploration when the task is corrupted by biased observations.
Our contributions can be organized as follows:
1. We propose learnable non-Markovian action priors conditioned on past actions. We show that sampling from these priors produces directed and correlated trajectories.
We introduce a principled manner of integrating action priors into the Soft. Actor Critic framework (Haarnoja et al. 2018).
3. We show how state-free priors can be learned from few expert trajectories on simple tasks (c.g. reaching) and used to improve exploration efficiency in more complex, unseen tasks (e.g. opening a window), even across fundamentally different settings (e.g., from non-visual to visual RL).
After discussing our method's novelty and related literat ure in Section 2 we introduce our setting in Section The method is described in Section | while empirical evidence of its effectiveness is reported in Section Finally, Section 6 contains a brief closing discussion of our work. Code and video are available on our project page]
2
Related Work
While we limit this section to essential topics, further discussion can be found in Appendix
Teextoally--Exteeddee Exploration Several works have attempted to directly address the inability of traditional methods, such as e-greedy or uniform action sampling (Lillicrap et al. 2016 Haarnoja et al, 2018), to produce correlated trajectories. A recent study (Dabney et al.) [021) highlights this issue and shows how repeating random actions for multiple steps is sufficient to significantly accelerate Rainbow (Hessel et al. 2018) the Atari benchmark (Bellemare et al. 2013). Similarly, Amin et al. (2027) propose non-learned policy inspired by the theory of freely-rotating chains in polymer physics to collect initial explorative trajectories in continuous control tasks. Both methods pinpoint a fundamental issue, but rely on scripted policies which are hand-crafted for a particular family of environments. On the other hand, our prior is learned, and does not require engincering an explorer, which can be unfcasible for complex tasks. A similar approach was previously proposed by |Bogdanovic & Righetti (2019), who also leverage simple trajectories to accelerate learning in new tasks by designing learned exploration model (LEP) conditioned sequence of recent states. Our method further explores this direction by removing the assumption of shared state space between tasks, and enables knowledge transfer a wider range of tasks. Moreover, compared to previous works, our integration of priors into off-policy RL is not limited to initial trajectories and dynamically adjusts the likelihood of sampling actions from the prior.
Hierarchical Reinforcement Learning
Another approach to tackle exploration-hard tasks is to rely on a hicrarchical decom positions into different levels of temporal and functional abstraction (Parr & Russell 1998
Dietterich 2000 Sutton et al. 1999: Dayan Hinton 2000). For instance, tasks can be decomposed into high level planning and a set of low-level policies, often referred to as skills (Konidaris & Barto 2007 Eysenbach
et al. 2019) options (Sutton et al.
1999 Bacon et al. 2017). This effectively reduces the planning horizon
and allows efficient solving of complex tasks [Bacon et al. 2017 Vezhnevets et al. 2017 Nachum et al. 2018
Levy et al. 2019 Christen et al. 2021). In general, such approaches only target temporal correlation within skills, as each skill is generally selected independently from the last one. Incidentally, our method is not designed to achieve temporal abstraction, but can be interpreted in a hicrarchical framework (Schäfer et al. 2021 in which a high-level criterion (the mixing function) governs a probabilistic choice between an explorer (state-free prior) and an exploiter (policy). Our method's application is also inherently related to the works of Florensa et al. [2017] Gehring et al. (2021). which propose to extract skills on simple training tasks and
https://eth-aif www.ttigtuub..iistbt
3

---

Published in Transactions on Machine Learning Research (08/2022)
deploy them in more complex scenarios. In contrast with these methods, training state-free prior does not require access to the training environment, but only to offline trajectories.
Behavioral Priors Behavioral priors generally represent state-conditional action distributions, modeling promising actions for the current state of the environment. Such priors can be learned jointly with the policy in the context of KL.rroullarzed RL (Tirumala et al.] 2019:
2020 An important line of work deals with information asymmetry (Galashov et al., 2019 Salter et al.
2022 which consists in restricting the
information available to the prior. As a result, learned priors can more easily generalize different settings, or be trained jointly accelerate policy learning. In this context, Tirumala et al. (2020) briefly mention the possibility to condition behavioral priors exclusively on vector of previous actions. Our work can be seen as broad empirical exploration in this direction, coupled with an integration scheme that is better suited to this class of priors. A second approach consists in learning behavioral priors from expert policies on related tasks. This is the case for several works (Peng et al.) 2019; Pertscc et al., 2020 2021 Ajay et al., 2021 which adopt a Gaussian behavioral prior in a latent skill-space. In particular, Pertsch et al. (2020) report that prior is crucial to guiding a high-level actor an HRI. framework. An important contribution to the field is made by PARROT (Singh et al.) 2021). which focuses on a visual setup and introduces flow-based transformation of the action space to allow arbitrarily complex prior distributions. We exlend this idea to prior action distributions that are not conditioned on the current state part thereof, but rather past actions, and are therefore non-Markovian. Morcover, we proposc a novel, more flexible integration of the prior distribution into the learning algorithm. Finally, we overcome the reliance on a strong similarily between states observed by the expert and the agent.
3 Background
3.1 Setting
Reinforcement learning (RL) is the problem that an agent faces when learning to interact with a dynamic enviromment. Albeit with slightly different definition, we formalize the environment as a coal-conditioned Markov Decision Process (gc-MDP) (Nasiriany et al. 2019). that consists of a 6-tuple (S, A, 9, R, T),, where is the state space, A is the action space, G C S is the goal space, R : S x g + R a scalar reward function, T A II(S) probabilistic transition function that maps state-action pairs to distributions over S and, finally, y is a discount factor. Assuming goals be drawn from a distribution pg, the objective of an RL agent can then be expressed over a time horizon T as finding a probabilistic policy X² arg m g EE= 7'R(st, g), with 8, ~ T(x--1, (r-1) and a:1 x (8)-1,9). In order to simplify notation, from this point on, we will implicitly include the goal into the state at each time step: + (s:.g).
We focus on long-hov horizon control problems with contimuous state and action spaces and sparse rewards, i.e., non-zero only after lask completion. Although our method can be generally applied to off-policy RL methods, we build upon Soft Actor Critic (Haarnoja al. 2018), due to its wide adoption in these settings,
Finally, in contrast with several behavioral prior approaches (Galashov et al., 2019 Pertsch et al. 2020: Singh et al. )021), we adopt more general and challenging setting. First, we do not assume that prior information on the structure of the state space is available. Second, while we also assume access to a collection of trajectories Dexp = {(sbs db. of oj.................. ......,,,. s/y, (2 we do not require trajectories to be collected on task that is similar to the one at hand. From this point on, we refer to the task used for collecting data as the training task, and to the task the RI. agent has to complete as the downstream task.
3.2 Behavioral Priors
A behavioral prior #(ass) (Pertsch et al. 2020 Singh et al. 2021 is state-conditional probability distri- bution over the action space (ef. Figure 2). Behavioral priors can be trained to assign high probability to useful actions with respect to the current state, and hence be used to accelerate RL. A behavioral prior can only guide the policy effectively as long as prior information on the structure of the state space available (Galashov et al. 2019). or the prior has been trained on data collected on a closely related task (Pertsch)
et al. 2020 Singh al. 2021
4

---

Published in Transactions on Machine Learning Research (08/2022)
In our settings, the structure of observations is unknown and expert trajectories Dexp weakly informative
or collected on simple tasks. As a consequence, the distribution of training states might not match the distribution of states produced by downstream tasks: behavioral priors will then be evaluated on out-of- distribution samples and their performance will degrade drastically, as shown in Scction 5.1
4
Method
Our method relies on the integration of a learned state-free prior into an off-policy RL algorithm. Thus, we first define and discuss the class of action priors of interest, and later describe how they can be integrated into existing off-policy algorithms.
4.1 State-free Priors
Within the settings outlined in Section 3.1 it is still possible to extract and transfer knowledge from the expert dataset Dexp to the agent. In this case, conditioning an action prior on the current state might be both insufficient and counterproductive, as a state-conditional action prior would receive out-of-distribution samples as inputs, due to the distribution shift between states from the training and downstream task. For this reason, we shift our focus towards modeling the temporal correlation of expert trajectories.
In order to recover this information, we thus propose to learn state-free prior. A state-free prior is a non-Markovian, state-independent action prior, representing a probability distribution over the action space that is conditioned on past actions: म ((a))46-)) (see Figure 2 c). By renouncing the informativeness of states with respect to expert actions, state-free free priors retain the ability to model temporal correlation, which crucially enables knowlodge transfer to tasks with a fundamentally different state distribution with respect to demonst rations. This is indeed viable strategy in hard-exploration setting, when no prior information is available on the state space and no reward is observed: in this case, extracting any information from the state remains challenging. Our empirical evidence suggests that the simplest form of state-free priors, i.e., it ((a/aa (see Figure 2P) is surprisingly competitive with variants conditioned on multiple past actions (cf. Section 5.1 and is sufficient to capture non-trivial temporal relations.
Independently of the conditioning variables, state-free priors can conveniently be modeled as conditional generative models and trained through empirical risk minimization. For the purpose of this paper, we choose to the conditional variant of the Real Non Value Preserving Flow (Dinh et al. 2017 Ardizzone ct al. 2019), which is well suited for Euclidean action spaces (Singh et al. 2021 Morcover, in the SAC [ramework allows arbitrarily complex prior distributions, which Real NVP Flows are in principle able to capture.
intogration
In the context of Real NVP Flows, training samples are actions a E A, paired with conditioning variables af d61 thus the learned mapping is a = fo(2; af-1), with ~ A. Since So invertible, analytically compute the likelihood of single training pair (a,, ") and maximize its expected value through standard meudient-bbsedd optimization techniques. An empirical justification of this choice is found in Appendix B while implementation details are reported in Appendix K For complete introduction to Real NVP Flows, we refer the reader to Dinh et al. (2017).
possible to
One final concern regards the nature of the data for training the state-free prior. The main require- ments for the training data are two: (1) the task on which the data is collected needs to share the same action space of the downstream task and (2) the training trajectories should display the desired qualitics of corrclation and directness. In general, we adopt weakly informative expert trajectories gen- erated by achieving random goals in simple environ- ments. Such simple trajectories can be produced from scratch using standard RL or generated by scripted policy. We remark that, in contrast with
Behavioral Prior
State-free Priors (our)
50
50
a) H (a,
b) (at 0g
(ac (c-1)
Figure 2: Graphical models of different priors: from left to right, a behavioral prior, general state-free prior and single-step state-free prior. Solid arrows represent the environment's transition function T, while dashed arrow indicate conditional modeling.
5

---

Published in Transactions on Machine Learning Research (08/2022)
existing approaches PPertsch et al. 2020 Singh et al. 2021), this framework poses very weak requirements on the similarity between the task used for data collection and the target task. As we show in Section 5.4 this allows our method to transfer knowledge across different tasks and settings, such as from a simple reaching task with access to the true state of the system to a window-closing task in a visual RL sctting. Limitations of our method with respect to training data are discussed Section 6
4.2 Soft Actor Critic with SFP
The main challenge introduced by state-free priors stems from their non- Markoviantty, which renders existing integrations schemes unsuitable Singh al. (2021) Tirumala et al. (2020) (sce Section 5.2 for an empirical evaluation and a broader discussion). For this reason, we introduce a novel method to integrate action priors in an off-policy RL framework. While our method can be formally derived for Markovian priors, it can also be generalized to non- Markovioo action distributions by allowing further bias in Q-value estimates. In the following derivations, we will consider a general state-dependent non-Markovian prior [( H2) with H, = #+s(,,, a ), but the integration remains applicable to state-free or behavioral priors alike (for briefness, we will interchangeably use the compact notation (a;)). The key strategy consists of sampling actions from a dynamically weighted mixture between the policy and a fixed prior distribution. We demonstrate as an integration into the SAC framework.
SAC is an off-policy actor critic algorithm that trains stochastic policy T and state-action value function esti- mator Qe in an off-policy way; its objective is designed to pursue rewards while maximizing the entropy of its policy.
A Algorithm SAC with SFP
1: Train action prior 7(a, H+)
2: Initialize parameters 0, d,w
When prior knowledge on the structure the environment or (ask is available, simply sampling actions from a maxi- mal entropy policy Ts may not be optimal. On the other hand, blindly sampling from fixed action prior स prevents exploitation of reward signals well as any behavior which is not encoded in the prior. Ideally, it is desirable to con- trol the degree to which actions are sampled from the prior. We propose to achieve this in a natural way by sampling actions from a mixture स between the policy To and the prior ::
for each iteration do Observe 80
5: Initialize history Ho (so)
6:
for each environment step do
T:
d = (s))))
8:
a ~ ă = (1 (1-)) {.)a(ae(ss) + (,,
9:
8 ~ T
10:
H1+1 H, (14))
11:
end for
12: for each gradient step do
04 ~ #-1a+) = (1- - Ae)ma(-1s)) + AeF(-),
(1)
13: Update 0 ,,w
14: end for
where the mixing parameter > is bounded (0 5 Ae 1) and computed dynamically at each step f.
15: end for
In principle, it is desirable to bias the mixture स towards the prior मे or the policy म, depending on which of the two is more likely to reach the goal. Interestingly, the mixing weight is computed through a parameterized function of states X, = (st), this behavior can be naturally recovered by maximizing the maximum entropy objective with respect to the mixture म:
75 (R(s),))) + ((H((T((-ss))) ]
arg max E
(2)
1=0
We note that, while this expectation is computed over trajectories sampled from the mixture म, the entropy term (-T)()ls:) is computed on the policy n alone, as it is not desirable to incentivize exploration of states according to the prior It' 's entropy.
Through straightforward derivations (see Appendix A it is possible to derive learning objectives for the Q-estimator Q and for the policy Tæ. as well as for mixing function Au- Given a distribution D of observed transitions, the loss functions for the Q-estimator can be defined as:
(Q (s, u((s,,,s)))
Joz
E (s,D,)))~
(3)
6

---

Published in Transactions on Machine Learning Research (08/2022)
where the target for the Q-value is computed as
a 5') R(s,a) + + (2(,'')))) a") - a To with a' =(1ss',, TT(-1s))
On the other hand, the policy T and the mixing function A can be trained to maximize the value function (Haarnoja et al., (018) for the mixture म. The value function be decomposed in expectations over actions sampled from the prior π and the policy T derived in Appendix A
(*))= (-))())
(2(() +
E arota(-1s)
[20(),0)]
log((o(als))) amma(-11)
E
(5)
where À = (s) is the mixing weight computed on the current state s. This decomposed value function can be maximizod by minimizing the two following losses with respect the policy's and the mixing function's 's parameters:
Js. E (5)~D)
(s) lo
with ~ Td(-1s).
(6)
- - (5>~D
JA. = E
with ā a(),a~ (-(1s)
(7)
The three objectives can be empirically estimated and minimized through standard procedures, as reported in Haarnoja al. (2018) and in Appendix A We further note that, as expected, Equation encourages high mixing weights X = (s:) in case Q-values for actions sampled from the prior स higher than those for actions sampled from the policy Tø. and vice versa (see Appendix F for empirical analysis the evolution of À during training). We remark that the resulting mechanism is similar in spirit to the Q-filter presented in Nair et al. (2018). which is however originally only applied as a binary signal for weighting a bchavioral cloning loss.
A Algorithum 1 summarizes (in blue) the necessary modifications for integrating the action prior into the SAC framework. Namely, actions are sampled from a mixture (line 8) weighted according to the output of a mixing function (line 7). Finally, the history of the agent needs to be initialized (line 5) and updated cach step (line 10). Updale rules for 0, and w (line 13) are computed by minimizing the objectives in Equations
and 7
5 Experiments
We evaluate our method in series of experiments to empirically validate our contributions. First, in Section 5.1 we compare the effectiveness of different action priors to justify the choice of state-independent conditioning. Then, in Section 5.2 we evaluate
integration against existing ones in the context of state- free priors. In Section 5.3 we verify that sampling from a stato-free prior can produce correlated and state- covering behavior, without the need to hand-craft an exploration policy. Nexi, we show how our met can improve exploration efficiency in unseen long- horizonrizon tasks by comparing against various baselines in state-based RL (Section 5.4). Finally, we present a proof of concept of different applications for stato-free priors, i.c., in visual RL and when dealing with biased observations (Section 5.5). Ablations for our method and baselines provided in Appendix

---

Published in Transactions on Machine Learning Research (08/2022)
Training Environment
Dowastream Enviromments
Training Environment
Downstream Enviromments
meta-wordd
point-maze
Figure 3: Non-exhaustive overview environments used in our experimental validation.
Baselines We now present a brief introduction to the baselines consider, while implementation details are provided in Appendix K.3
SAC: vanilla Soft Actor Critic [Haarnoja et al. 2018
SAC+BC: SAC with warm-started policy through behavior cloning.
SAC-PolyRL: SAC with locally self-avoiding walks {Amin et al. 2021).
PARROT-state: flow-based behavioral prior enforced through a transformation of the action space (Singh et al. 2021p. We benchmark a state- based variant in non-visual settings. We found other methods based on behavioral priors to perform similarly, while being arguably more complex (sce Appendix H for additional baseline).
We evaluate our method on three types of domains, namely robotic manipulation, maze navigation and robolic locomotion:
Environments
meta-world: robot manipulation tasks from the MT10 benchmark in the publicly available meta-world suite (Yu et al. 2020).
point-maze: qualitatively different 2D maze structures, navigated by point-like agent Pitis et al.
(2020)
gym-mujoco: widely adopted continuous control enviromments for locomotion of simulated robots
Brockman et al. (0166) Todorov et al. (2012).
The first two suites define diverse tasks that share similar underlying dynamics (e.g. navigating mazes with different structures, or interacting with different objects), and are therefore used to benchmark the ability to transfer knowledge from offline trajectories to complex downstream tasks (Section 5.1
In this case, offline trajectories are collected in simple training tasks that involve reaching uniformly sampled goals in an emply environment (reach for meta-world and room for point-mazze evaluated on wider range of downstream tasks, involving object manipulation, such as opening window, or navigation in more complex mazes. While our main results focus on priors trained on weakly-informative data, we note that the performance of state free priors with respect behavioral priors dependent on the choice of training task. We further elaborate on this topic by providing an additional empirical analysis on more complex, object-interaction training task in Appendix D
5.3
5.4]
RL agents are then trained and
On the other hand, gym-mujoct environments do not offer straightforward way to define tasks, and are therefore used to evaluate a proof-of-concept experiment on biased observations (Section 5.5).
More details can be found in Appendix J while visual examples of environments are provided in Figure 3
cumulative returns per test episode. For each task, we average this metric over 10 random seeds (excluding gym-mujoce experiments and visual RL experiments, which are respectively limited to 5 and 2 seeds for computational reasons). Unless stated explicitly, when aggregating results for different tasks (e.g. in plots marked downstream) we perform simple mean aggregation due to its statistical efficiency, as per-task scores are not normalized, and in the same range for all lasks. Allernatively, we report results for a more robust. aggregation scheme (i.e. IQM (Agarwal et al. 2021)) in Appendix M Uncertainty is quantified through 95% stratified basic bootstrap confidence intervals, as suggested Agarwal et al. (2021). Across the experimental section, all priors are trained on 4000 trajectories of 500 steps each. An ablation on the amount of training data is reported in Appendix C Further details be found in Appendix K.1
Metrics and Training Data Our main metric for Sections 5.1
5.2
5.4
5.5
8

---

Published in Transactions on Machine Learning Research (08/2022)
meta-world (training)
meta-world (downstream)
100
R(
Test Return
Test Return
400
=(adoF33)
R(adloL-))
200-
200
0-
0
3
2
Steps
x x10
Steps
x10
Figure 4: Comparison of downstream performance of action priors with different conditioning variables, reported the training task (left) and averaged over downstream tasks (right)..
5.1 Conditioning Variables for Action Priors
We now empirically show how the effectiveness of an action prior depends on the conditioning variables and on the similarity between the training and the downstream tasks. For this purpose, we train several variants of flow-based action priors on offline reaching trajectories and integrate them into SAC learner as described in Section 4 In particular, we compare state-frce priors conditioned on action soquences of different lengths (1, 2, 5, 10), non-Markovian priors conditioned on the previous state-action pair and a behavioral prior (conditioned on the state alone). In Figure ! we report learning curves for the training task (reach), as well averaged learning curves over all downstream tasks in meta-world. We observe that all priors are capable of guiding downstream RL as long the task at hand matches the training task. In this case, state-conditional priors achieve slightly faster exploration by leveraging state-related and task-specific information. However, we find that including the state in the conditioning variables (as done in न a(a st) and i (a, St. ac-1)) can jeopardize the ability to transfer knowledge to a different task. We hypothesize that this is due to a mismatch between the state distribution used for training the prior and that generated by the downstream task.
On the other hand, state-free priors not conditioned states by design, prove to be capable alternative across both settings and are able to transfer knowledge to unseen tasks. While conditioning longer action sequences can improve performance, we note that single-action-coniitoon models (( are sufficient for capturing non-trivial temporal dependencies within our settings. Hence, they will be the focus of the remaining experiments.
5.2 Integration for State-free Priors
After showing motivating application for state-free priors, argue the importance of our proposed mixture-based integration scheme for priors in offline reinforcement. learning algorithms. Two common existing integration schemes for be- havioral priors rely on regularizing the policy with a KL-term (irumala et al. 2020 Pertsch et al. 2020), or on a based transformation of the action space (Singh al. 2021). Both methods strongly build on the assumption that the prior is
KL-vegularisation
Flow-based
Mixture
meta-world (downstream)
Test Return
300+
100
only conditioned the current state 8t, or subset thereof.
In the case of flo-bbsed action space warping, the prior is ef- fectively integrated in the environment dynamics: as result, if the prior is conditioned on previous actions af or states s t-1 the environment loses its Markov property, and station- ary policies are no longer guaranteed to be optimal. On the other hand, penalizing the KL-divergence between the policy and non-Markovian prior encourages the stationary policy to
01
1
Steps
x x10
Figure 5: Performance of different integration
schemes: our mixture-based integration com- pares favorably against existing methods (flow- based and KT-regularriaaion).

---

Published in Transactions on Machine Learning Research (08/2022)
SAC
SAC-PolyRL
SFP (ours)
SAC
SAC-PolyRL
SFP (ours)
Figure 6: A qualitative comparison of sampled exploration trajectories in robotic reaching task and in empty 2D maze. Our method achieves directed behavior while covering most of the state space. SAC fails to cover the full state space, while SAC-PolyRL fails to reach distant consistently.
match the distribution of potentially non-stationary prior, which in general an ill-posed lask. Instead, our bexture-based integration only suffers from biased learning objectives in the case of non-Markovian action prior (sce Appendix A) Empirically, we found this to be mild limitation, sensibly better performance compared to the two baselines in downstream learning, reported in Figure where experimental settings are the same as those described in Section 5.1. and details can be found in
integration achieves
5
Appendix K.3
5.3 Correlation and State Coverage
In this experiment, we show how a one-step stale-independent state-free prior ((ae(a)-1) can generate corre- lated and directed behavior, which leads to more complete coverage of the state space during exploration. To this end, sample 20 random trajectories of 500 steps cach with method and two relevant basclines in the room and reach enviromments.
As shown in Figure G the state-free prior produces directed behavior which covers most of the state space. As expected, uniform sampling (which approximates SAC's exploration) fails to reach the boundaries the environment; SAC-PolyRI. capable of producing correlated and directed behaviors, but only after careful design and tuning. This qualitative assossment consistent with the quantitative evaluation Table which reports state space coverage and radius of gyration squared (Amin et al. 2021 (see Appendix K.1 for details).
5.4 Transfer Learning
Our main results are obtained by comparing our method against several basclines in downstream learning tasks with a vectorial state space, as presented in Figure 7 We report learning curves averaged across downstream tasks, as well as on training tasks for reference. Results for each task are disentangled Appendix
Table State coverage metrics for our method and baselines. SFP's trajectories are locally directed and cover the state space well in both environments.
U3
% Coverage
SAC
0.006=t0.001
0.137+0.01
reach
SAC-PolyRL
0.025-10.001
0.272+0.01
SFP (ours)
0.053+0.000
0.357=00.02
SAC
0.005++0.001
0.333+0.02
room
SAC-PolyRL
0.026-L0.002
0.850+0.04
SEP (ours)
0.05440.008
0.963+1.04
10

---

Published in Transactions on Machine Learning Research (08/2022)
SAC
SAC+BC
SAC-PolyRL
PARROT-state
SPP (ours)
(training)
Text Return
400
meta-world (downstream)
point-man maze (downstream)
800
200
300
Test Return
Test Return
100
Steps
X10
2004
point-maze (training)
Teat Retur
200
600
100+
100
04
Steps
Steps
X10
Steps
x10
Figure 7: Accelerating downstream RL. While other methods are mostly competitive on training tasks (reach room), SFP is better suited for accelerating RL in unseen lasks. Results are averaged over 10 seeds per task.
As expected, we observe that the performance of behavioral priors depends on the similarity of between the task at hand and expert demonstrations. This the case for PARROT-state, which instantly solves the training lasks (reach and room), as its behavioral prior already represents a strong policy. lasks which are significantly different from the training tasks, PARROT-state is however unable to guide the policy, as it receives out-of-distribution states, resulting in low average performance. On the other hand, SFP largely more capable of transferring knowledge to unseen tasks, while rapidly catching up with PARROT-state in the training tasks.
Other baselines are in general less effective across the benchmarks: Vanilla SAC only makes progress on easier tasks, which can be achieved even with weak exploration.
SAC-PolyRL able to produce good explorative trajectorics through its hand-crafted policy and improves SAC, but fails to explore after its initial phase and does not solve harder tasks. As previously reported by Singh et al. (2021). , initializing SAC through behavioral cloning (SAC+BC) help guide exploration without constraining the policy, but it fails to generalize across tasks and is regularly outperformed by stronger methods.
5.5 Additional Applications
We finally provide proofs of concept for possible applications of SFP. Per-task results are available in Ap- pendix L
Visual RL State-free priors allow the state space of the downstream task be defined arbitrarily. Hence, they also allow transfer to the visual RL setting, which avoids reliance low-dimensional vectorized state space and is purely based on RGB observations. To this end, compare SFP with Vanilla SAC and with PARROT in its original, visual setup, i.e. by conditioning its behavioral prior on images. We remark that,
SAC
PARROT
SFP (ours)
PARROT-state
SFP
0000
1000
100%
100%
Tost Return
300
(dovnatrean)
Tost Return
100
point-naze (dosnatrean)
Sparsity
750
Sparsity
80%
750
500
60%
500
100-
300
200
10%
250
20%
20%
2
0
0.1
1.0 Bias
10.0
0.1 1.0 10.0 Bias
Steps
X10
Steps
X1P
Figure 8: Performance on downstream learning in
Figure Mean performance gym-mujoco envi- ronments for different sparsity thresholds and bias
visual settings, averaged per suite.
11

---

Published in Transactions on Machine Learning Research (08/2022)
in this case, PARROT needs access to additional data (i.e., RGB observations) compared to SFP. We report the results in Figure In general, we observe a performance drop for all methods compared to non-visual settings, which we hypothesize is due to the inherent challenge of learning in high-dimensional state spaces [Laskin et al.) 2020 Nonctheless, our findings remain generally valid in visual settings, as SFP retains most of its exploration capability and, despite failing to make progress on a majority of visual lasks, consistently outperforms the baselines. Per-task results are reported in Appendix L
Biased Downstream Learning A final potential application for stee priors is that of guiding ex- ploration when downstream learning is corrupted by biased observations. This scenario could model form of sim-to-real transfer, in which training trajectories were cheaply collected in simulation, but the physical testbed was miscalibrated, resulting in biased observations. We provide experimental results in this setup based on sym-mujoce environments (Ant-v2, HalfCheetah-v2, Swimmer-v2, Hopper-v2, Walker2d-v2). Training trajectorics are collected by a SAC agent that was fully trained on dense rewards. In downstream learning, the reward is instead sparse (i.e. only positive when the displacement of the CoM from its initial position surpasses a sparsity threshold ž), and significant exploration is required. Moreover, the observations in downstream learning are corrupted by fixed additive bias + b. Figure compares test performance of downstream agents when guided by behavioral prior (PARROT-state) or by state-free prior (SFP), after 500k steps and averaged across environments. We observe that behavioral priors are capable of extract- ing a near-optimal policy from the training data, which transfers perfectly to the sparsified environments if the observations are uncorrupted However, as bias increases, their performance sharply drops. On the other hand, state-free free priors are less effective in unbiased conditions, but their performance is generally retained as bias incrcases. More details on this experiment are presented in Appendix J.3 and an ablation introducing transformation on the action space is reported in Appendix E
6 Discussion and Conclusion
Limitations While state-free priors are able to generalize to a broader set of tasks with respect to be- havioral priors, they are data-driven, and as such will only reconstruct behavior that appears the offline dataset of trajectories they are trained on. For instance, when training on demonstrations for reach, state- free priors will not encode any grasping strategy. While such behavior can still be recovered by the policy ", there is no incentive to do so. As a result, successfully transferring to tasks that require additional strategies, such as grasping (c.g., pick-place) remains an open challenge. Secondly, as shown in Figure 7 behavioral priors remain superior when the downstream state distribution is not distant from the state distribution observed in the training task. As state-free priors do not model state-action relationships, they can only provide weaker guidance in this privileged case. This is in particular true when priors are trained on expert trajectories for complex manipulation tasks, as shown in Appendix [ For this reason, a strategy to combine benefits from behavioral and state-free priors would represent an interesting direction for the future.
Conclusion In this paper we proposed a method for improving exploration efficiency in off-policy reinforce- ment learning. In particular, we introduced state-free priors, family of Markviakov,n,, state-independent and blowd action priors, and proposed principled manner for integration into an off- off-policy reinforce- ment learning framework. While leveraging a relatively simple idea, state-free priors represent a powerful method for exploration. As shown by significant empirical evidence, they can massively accelerate learning across a diverse set of tasks, while solely requiring a modest amount of offline, weakly informative trajecto- ries.
Broader Impact Statement
Our main contribution revolves on accelerating and enabling reinforeement learning in environments with a strong exploration component. As a consequence, we believe that concerns with respect to our method are for the most part aligned with general RL research. For instance, improved sample efficiency could on one hand accelerate the process of automation, which might have a negative impact on socictal equality, and on the other hand democratize access to powerful RL methods by lowering the amount of required resources. Due to the general nature of our method, we believe that we do not introduce fundamentally new risks.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Acknowledgments
We are grateful to Núria Armengol Urpí, Emre Aksan and others for providing constructive feedback on this work. We also acknowledge the anonymous reviewers for their important contributions towards improving the manuscript.
References
Joshua Achiam. Spinning up in deep reinforcement learning. 2018. URL https://githinn htt.gggttuu...ooopeeee cww cpeeopanai spinningup
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: discovery for accelerating offline reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
offline primitive
Susan Amin, Maziar Gomrokchi, Hossein Aboutalebi, Harsh Satija, and Doina Precup. Locally persistent exploration in continuous control tasks with sparse rewards. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 275-285, 2021.
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep netwoos... In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The Option-Critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-bascd exploration and intrinsic motivation. In Advances in Newral Information Processing Systems, pp. 1471 1479, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47, 2013.
Miroslav Bogdanovic and Ludovic Righetti. Learning to explore in motion and interaction tasks. In Proceed- ings 2019 IEFE/RSJ International Conference on Intelligent Robots and Systems (TROS), pp. 2686 26922. JEEE, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distilla- tion. In 7th International Conference Learning Represenlations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Sammy Christen, Stefan Stevšié, and Otmar Hilliges. Guided deep reinforcement learning of control policies for dexterous human-robot interaction. Tn 2019 International Conference on Robotics and Automation (ICRA), pp. 2161 2167, 2019.
Sammy Christen, Lukas Jendele, Emre Aksan, and Otmar Hilliges. Learning functionally decomposod hierarchies for continuuu control tasks with path planning. IEEE Robotics and Automation Letters, 6(2): 3623 3630, 2021.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Will Dabney, Georg Ostrovski, and André Barreto. Temporally-cxtender e-greedy exploration. In 9th International Conference on Learning Represenlations, ICLR 2021, Virtual Event, Austria, 2021.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. Advances in Newral Information Processing Systems, 09 2000.
Thomas G Dicttcrich. Hicrarchical reinforcement learning with the maxq value function dccomposition. Journal of artificial intelligence research, 13:227 303, 2000.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Densily estimation using real NVP. In 5th Inter-
national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580 586, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you ncod: Learning skills without reward function. In 7th International Conference on Learning Represenlations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017.
Alexandre Galashov Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Tn- formation asymmetry in kl-regularized RL. In 7th International Conference on Learning Representations, TCLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. In Advances in Neural Information Processing Systems, 2021.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. International Conference on Machine Learning (ICML), 2018
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI Conference on Artificial Intelligence, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Represendations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015.
George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforcement learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAP07, pp. 895-900, 2007.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020.
Andrew Levy, George Dimitri Konidaris, Robert Platt Jr., and Kate Saenko. Learning multi-level hierarchies with hindsight. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimouus control with deep reinforcement learning. In Yoshua. Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan,
Puerto Rico, May 2-4, 2016.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbecl. A simple neural attentive meta-learner. In 6th International Conference on Learning Regresentations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient Hierarchical Reinforcement Learning. In Advances in Newral Information Processing Systems, pp. 3303-3313, 2018.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6292 6299.. TEFE, 2018.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In Advances in Newral Information Processing Systems, volume 32, 2019.
OpenAI, Iige Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in newral information processing systems, 29:4026-4034 2016.
Tan Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. arXiv preprint Xr:: 03335, 2018.
Simone Parisi, Victoria Dean, Deepak Pathak, and Abhinav Gupta. Interesting object, curious agent: Learning task-agnostic exploration. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Proceedings of the 1997 Conference on Advances in Neural Information Processing Systems 10, NIPS '97, pp. 1043-1049, 1998.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning compos- able hierarchical control with mulúplicative compositional policies. In Advances in Neural Information Processing Systems, volume 32, 2019.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J. Lim. Demonstrration-guided reinforcement learning with learned skills, 2021.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761, 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupla, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Kate Rakelly, Aurick Zhou, Chelsca Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta- reinforeement learning via probabilistic context variables. In Proceedings of the 36th International Con- ference on Machine Learning, Proceedings of Machine Learning Research, 2019.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Sasha Salter, Kristian Hartikainen, Walter Goodwin, and Ingmar Posner. Priors, hicrarchy, and information asymmetry for skill transfer in reinforcement learning. CoRR, 2022.
Stefan Schaal. Learning from demonstration. Advances in Neural Information Processing Systems, 1997.
Lukas Schäfer, Filippos Christianos, Josiah Hanna, and Stefano V Albrecht. Decoupling exploration and exploitation in reinforcement learning. ar preprint arXiu2217.089666 2021.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, on learning how to learn: the meta-mela-... hook. PhD thesis, Technische Universität München, 1987.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep condi- tional generative models. Advances in neural information processing systems, 28:3483 3491, 2015.
Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Daisuke Okanohara. Conditional density estimation via least-squares density ratio estimation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 781-788, 2010.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artif. Intell,, 112:181-211, 1999.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Picter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems (NIPS), volume 30, pp. 1-18, 2017.
Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascamu, Yee Whye Tch, and Nicolas Hoess. Exploiting hicrarchy for learning and transfer in kl-regularized rl arXie preprint orXiv:1903.077 074488 2019.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever, Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojeiech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, and Nicolas Heess. Behavior priors for efficient reinforcement learning, 2020.
Emanuel Todorov, Tom Frez, and Yuval Tassa. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026 5033, 2012.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hicrarchical Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Leurning, pp. 3540-3549, 2017.
N. Vulin, S. Christen, S. Stevšić, and O. Hilliges. Improved learning of robot manipulation tasks via tactile intrinsic motivation. TEEE Robolics and Aulomation Lellers, 6(2):2194-220) 2021.
Jane X. Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning reinforcement learn. In CogSci, 2017.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), pp. 1094 1100, 2020.
16

---

Published in Transactions on Machine Learning Research (08/2022)
A Objective Derivation
The purpose of this section is that of describing how learning objectives for the policy т, the Q-estimator Q" and the mixing function A can be derived from a max-cntropy objective; while in practice each function is respectively parameterized by Φ, and w, in order to ease notation we will only explicitly mention these parameters when introducing empirical loss estimates. The objective is defined
Σ TMO 1=
+ ( +(H((()))))
where the expectation computed over trajectories generated by the mixture म = (1 Ae) Ae क is a fixed action prior and X A(s:) is the mixing weight. The optimization of Equation g can be performed by modeling the action prior a(.) as Markovian distribution over the action space A: note that non- Markovian action priors do not necessarily satisly Equation TO and therefore can introduce bias in Q-value targets (Equation 12 The steps of these derivations follow those reported in Haarnoja et al. (2018), as our method designed to fit within the proposed framework. First, let us formally introduce the Q-function for the mixture #:
(²(s,a)= = E
(r'R(se, + Σ (1))( = ao =
(9)
We can then formulate the Bellman Equation and explicitly unravel the entropy term:
(8)
E s'ws())),)) a'ma(11ss')
R(s,a) + + + aH(T((1)))
B(s,a)+7(Q0F) + E
| log(((()))))))
s'~T(a)s a'wt(1ls')
d'mx(1le)
(10)
R(s,a) + +7(00 (s',a") log(T(d"(s')))
E s'1a(-)),a) ä' (-1s" a' ('(s')
The right-hand side of Equation 10 can be estimated via Monte Carlo sampling and used as target for Q-estimates, which can then be trained by minimizing a standard MSE loss. In practice, as is done for SAC, two separate parameterized Q-function estimators (Q5,)==1. are used to prevent overestimating Q-values. Additionally, we also adopt target networks (Q$ (0eegen )i=1,2 updated via Polyak averaging. As a result, when sampling batch B from a replay buffer, the empirical estimates for Q-losses are:
JQ..
1 B
Σ
(s, De(5',
(11)
where the target for the Q-value is computed as
(e rr. dr =r + dd] a0( min Q
Qu ($ (5, a') log logg
with ă' =t(ss'),, d' vo(-1s)).
(12)
111
and r, d stand for the reward and done signal, respectively.
As the prior सेt is fixed, optimizing the mixture policy म = Am + (1 A)TT only involves the optimization of w and X. The policy N can be trained to minimize the KI-divergence with the soft-max of the Qufnction,, or alternatively to maximize the value function (Haarnoja et al. 2018). Our method relies on the second option and trains both π and A to maximize the value function V (*(s), which can formulated as follows:
V*(s) =
E (~7(118) anT()18)
Q*((s â) log(T(als))
(13)
17

---

Published in Transactions on Machine Learning Research (08/2022)
Interestingly, the value function v can be decomposed
(s) = E
(0(
-a E
log(T(als))
aro((11))
ava((11e)
-
(=(s)))((a1s))
log(T(als))
amm(11))
Vbea (f(s + -))(((())) - a awa())s)
log(T(als))
(14)
A ( (*{),a)((a)) + (1 (1- +11- (1 aEA
o (*(s
log(n(als))
E
ama(-1s))
A ( (1 a)a(()
E (07(3,0)
log(T(als))
amo((1s)
where À As)) is the mixing weight computed on the current state. When differentiating this objective with respect to the parameters & of the policy network n, the first term does not contribute to gradients, resulting in
max V7 = max 6
(1=))) a) - a log(T(
(15)
Computing the expectation over actions can then be circumvented by using the reparameterization trick, which enables expressing actions s)(E,s), where & ~ N is sampled from a standard Gaussian. In practice, considering the two Q-networks, the policy network can be optimized by minimizing the empirical loss estimated over batch B:
1 Σ (1 (1- (s))( min Q7, (s)))(s)) a log(T((((s))) with a TT(-s))
Jx. =
(16)
B
1112
(B)EB
which can be minimized via standard first-order optimization techniques.
The last learning objective to recover is that for the parameterized mixing function Au. Once again, we obtain this by maximizing the value function (s). Starting from Equation 14 we can now drop the final term, which does not depend on w:
max (s) max (s)
E
Q (s, a)
(1 As))
@0(3.0)
6 w
max (s)
E
Q 28,s (8,0) a)
+ E
(5(5,
- As(ss
E
(3())a)
(~
amwa(-11s
am)
(17)
max (s)
E
Q (s, (5 a)
As)ss)
(8(3.0)
(m)(()
anta(-11))
max (s)
E
Q5(8, a)
(7())a)
(-)))))6) a
In practice, the loss estimate for training the mixing network An can be computed over a batch B as
Ja_
(=)
min Q (s, i=,2 112
min Q (s,
with ā ~ a(), (-1)).
(18)
B
B
18

---

Published in Transactions on Machine Learning Research (08/2022)
meta-world (downstream)
meta-world (downstream)
300-
Test Return
Test Return
400
200
NVP Flow
78% data quant.
90% data quant.
25% data quant.
data
500% rand. data
77%% rand. data
<VAE
LSCDE
200
100
Deterministic
SFF
25%
0
2
3
Steps
x10
Figure 11: Performance downstream meta-world tasks when training on fewer offine demonstrations, when they are corrupted by uniform noise.
Figure 10: Performance on downstream meta- tasks when modeling one-step state-free prior through various generative models.
The gradient of this objective with respect to the mixing weight = A(s) is intuitively the difference in Q- values between actions drawn from the prior and policy. As training procoods, and the policy æ improves, the gradient turns negative and encourages lower mixing weights, gradually abandoning the prior ä's guidance.
B Model Ablation
We now set out to provide empirical backing for important design choice. Our policy mixing approach grants freedom in choosing generative models capable of describing complex distributions, as computing distance metrics to the prior is not required. We compare the performance of Real NVP Flows with a Con- ditional VAF (Shhn et al. 2015), a non-parametric conditional Least Squares Density Estimator (Suiyaama et al. 2010), and an MLP modeling a deterministic prior. shown in Figure I0 we found flow-based modeling to be both competitive and practical.
C Data Ablation
Figure 11 includes data quantity ablations (with 75%, 50% and 25% of training data), and data quality
ablations (by substituting 25%, 50% and 75% of the training data with uniformly sampled actions). In the latter case, we observe a sharp drop in final performance (averaged over downstream meta-world tasks), while we find SFP to remain viable in low-data regimes.
D Training Task Ablation
The effectiveness of learned prior depends the nature of both training, and downstream tasks. In the context of this paper, focus on simple training lasks, which can be solved easily by existing RT algorithms (e.g. reach from meta-world). In this section, however, we instead evaluate several prior designs when training on expert trajectories from more complex and informative task (i.e. pick-place). We set out to verify whether (a) state-free priors can fail to exploit high-quality expert training trajectories due their lack of expressiveness, and (b) how state-free priors compare to behavioral priors on unseen tasks whose state distribution still matches the expert state distribution (i.e. push).
Our experimental comparison includes SFP (T(aelae-1)), PARROT-state, that involve priors conditioned on the current state (#)))s))) or on the current state and previous action (T(a,)8,, (1))). Figure| I2 gathers the performance of all methods in the training task (pick-place, left), averaged over all downstream meta-world tasks (middle), and on a downstream task which is unseen, but remains in-distribution the training task (push, right).
well two variants of SFP
We observe that only state-conditioned approaches are able make progress on the training task, 7 (aa (a.1)) lacks sufficient. expressiveness for guiding exploration, since its state-free prior cannot encode
19

---

Published in Transactions on Machine Learning Research (08/2022)
meta-world (training)
meta-Rorld (domnstream)
push
Test Return
Test Return
Tost Return
400
400
200
200
H(a 62-1)
PARROT-state
0
2 Steps
2
x10*
Steps
x10
Steps
x 10
When trained task involving manipulation, state-conditioned priors can succeed in complex tasks (the training task pick-place on the left, and an unseen in-distribution task push on the right), while a( at-1) fails. However, when averaging returns over all downstream tasks (middle), a(a partially close the gap.
Figure 12:
closed-loop interaction with objects. Similarly, when evaluating on an unseen in-distribution task requir- ing manipulation (push), we observe that behavioral priors are, as expected, able to learn the task, while it (a (a-1) fails to find a solution. However, when performance averaged over all downstream tasks, we find that the robustness to domain shifts compensates the lack of expressiveness, and म ( (a:-))'s performance is roughly comparable with other methods.
Within state-conditional approaches, note that the two methods adopting integration scheme dynamically manage the frequency to which the prior is sampled, and can ignore the prior when it suggests suboptimal solutions. As a consequence, their performance on downstream tasks is, on average, improved. PARROT-state, on the other hand, relics on a hard integration of the prior, and is the most sample efficient method for in-distribution tasks, but suffers largely when dealing with out-of out-of-distribute distriibibttoon downstream tasks.
Finally, we observe that priors conditioned on action-state pairs ( at-1) are underperforming with respect to purely behavioral priors a(ar)a [se] in this setting. Considering the results reported in Section 5.1 which show inverted pattern, note that conditioning the prior action-state pairs allows it to extract conditional dependencies on both the state space and the action space. Such priors are not robust to distribution shifts in the state space, but may also learn to focus more on their action inputs. As a consequence, we hypothesize that their average performance generally falls between that of purely state-free, and purely behavioral priors.
E
Ablation on Biased Downstream Learning
The experiment presented in Scction 5.5 introduces potential application of stato-frce priors in case the downstream task's state space corrupted by bias. We study the setting in which the state space of training and downstream tasks matches, but the action space is disrupted through transforma- tion. For simplicity, we consider a permutation of the action vector, and in particular an inversion (i.c. actionnaction[:-1]]] which occurs before the action is executed in the environment. This Iransformation
Test Return
1000
20%
40%
60% Sparsity %
80%
100%
Figure 13: SFP (red) and PARROT-state (orange) performance aggregated mujoco tasks. Dotted bars represent performance on the unmodified training task, and undotted bars represent performance on the training task, when introducing disruptive transformation of the action space.
20

---

Published in Transactions on Machine Learning Research (08/2022)
meta-world (downstream)
Actions PSD
Mixing Weight
Test Return
300
Data SFP
40
SAC
200
0.5
40
100
40-1
0.0
0
0.0[Hz)
0.2
Steps
x10
Figure 15: PSD of trajectories sampled from the offiie dataset, from SFP's prior and from uniform random actor.
Figure Evolution of mixing weight À and Lest performance during training, averaged over meta-world tasks.
is chosen because, contrary to adding noise to the state space, it would not affect the performance of a naive RL algorithm (e.g. SAC), while potentially impacting the effectiveness of learned priors.
In Figure 13 we report final returns for SFP and PARROT-state averaged over the five muj mujoco environments
used in Section 5.5 We evaluate different degrees of reward sparsity (as explained Section 5.5 and for each one we compare the performance of SFP and PARROT-state on the training task, as well as on a corrupted version of the training task. We note that the added transformation results in comparable drop in performance for both methods. In the case of SFP, this be partially traced back to the fact that the prior may be conditioned on out-of-distributee state, this transformation disrupts the actions suggested by the prior, the trajectorics it was trained on cannot be reproduced in the environment.
actions. Most importantly, for both SFP and PARROT-
F Study on Learned Mixing Weights
In Figure we plot the mixing weight during the course of training against returns, averaged over the downstream meta-world tasks. As formally described in Section 4 the mixing weight generally decreases returns increase; in particular, observe À - O for solved tasks and À 4-1 for unsolved tasks. We found
this trend to be consistent for most states visited. Per-task plots are reported in Appendix L We also note that the initial trajectory is influenced by the initialization discussed in Appendix K.4
G Study on Temporal Correlation
In order to better quantify the concept of temporally correlated exploration, we plot the PSD (averaged over action dimensions) of action sequences sampled from the training set, from our learned prior, as well as those obtained by sampling from uniform distribution on room. In Figure we note prominent low frequencies for both the training data and SFP trajectories.
H Additional Baselines
In this section we report results for an additional bascline, namely SPIRL (Pertsch et al. 2020). SPIRL is a hierarchical method that combines skill learning with a high-level behavioral prior, which is integrated in a SAC agent via KL-regularization. We evaluate this method on the meta-world suite in the same settings reported in Section 5.4 namely by training all priors on reaching trajectorics for random goals, and then reporting performance on the training task, as well as averaged over all downstream tasks. We also report reference.
the performance of SFP and PARROT-state from Section 5.4
21

---

Published in Transactions on Machine Learning Research (08/2022)
meta-world (training)
meta-world (downstream)
Test Return
Test Return
300
400
SFP
200-
PARROT- state
200
SPIRL
Steps
x10
Steps
x10
Comparison of SPIRI with SFP, as well with another method based on behavioral priors (PARROT- state), reported on the training task (left) and averaged over downstream tasks (right). The setting the same as for Figurel
Figure 16:
Figure 16 highlights that, similarly to PARROT-state, SPIRL can instantly solve the training task, but suffers when transferred to unseen tasks with out-of-distribution states. We note that the lower performance in the trained task can be traced back the hicrarchical nature of SPIRL: its low level skill dccoder is trained on reaching trajectories, which do not include static behavior. As consequence, the agent has no access to skill that keeps the gripper still, while reach requires the agent to hold the gripper in its goal position for several steps. Finally, we note that, while this skill-based approach needs further assumption on the training data to ensure learning all necessary skills, the soft integration of the behavioral prior could in principle allow the agent to mitigate issues related to out-of-listribttion states.
This experiments relies on the official implementation (Pertsch al. 2020), and adopts the hyperparameter set provided for Frankakitchen experiments. Further tuning of the target divergence parameter did not lead to signifieant improvements in performance.
Additional Related Works
Meta-RL and Task Inference The idea of quickly adapting to unseen tasks is fundamental concept in meta-reinforcement learning (Schmidhuber 1987 Wang et al.) 2017). While SFP has a similar goal, it crucially relies on training a policy from scratch for downstream learning and is not designed for zero-shot adaptation. line of research in meta-RL, referred to as contert- (Wang et al. 2017 Mishra et al. 2018 Rakelly et al.] 2019), performs task inference order to identify the current task and quickly extrapolate how maximize returns. A task representation can in practice be extracted from recent experiences. Interestingly, state-free priors in principle handle the same type of input, i.e. sequences of recent action- state pairs. However, instead of producing explicit representation of the current task, state-free priors directly model an action distribution.
Learning from Demonstrations SFP is aligned with existing methods that rely demonstrations
to accelerate RL complex tasks (aair et al. 2018 Rajeswaran et al. 2018 Christen et al. 2019). In principle, the cost of acquiring few expert trajectorics can be small compared to the significant engineering effort required in their absence (OpenAl et. al.) 2019 In most cases, such trajectories are required to be near-optimal and collected in the same environment (Schaal 1997) or from the same distribution tasks (Singh al. 2021). In our case, expert trajectories can be weakly informative, unlabeled and collected on a significantly different task, as we focus on reconstructing correlated exploration trajectorics instead of exploitative behavior.
22

---

Published in Transactions on Machine Learning Research (08/2022)
J Environment Details
J.1 Robot Manipulation
We rely on the meta-world suite Yu et al. 2020) for robot manipulation experiments. It consists of a simulated DoF' Sawyer arm, implemented in the MuJoCo physics engine (Todorov et al. 2012). By default, states are represented as 36-dimensional vectors across all tasks. The state contains the 3D location and aperture of the gripper, the 3D location and quaternion of one object (e.g. door or window), measured for the current and previous time step. Goals are represented as the desired 3D location of the end effector object, according to the task. Actions 4-dimensional vectors containing 3D movement and ID control over the aperture of the effector.
We additionally render 64x64 RGB images as observations for the visual setup in Section 5.5 using the camera angle corner (as can be seen in Figure 3). Instead of the originally proposed dense reward, we adopt binary sparse reward which non-zero only upon task completion. For more details, we refer to the original implementation Yu et al. 2020
J.2
Maze Navigation
Our point-maze enviromments are adapted from Pitis et al. (220 The state consists of the 2D position of the agent, which can be actuated via a velocity-controlle through 2D actions. The goal space matches the state spaces in dimensions and representation. The reward signal is when the Euclidean distance to the goal is lesser than 1.2 units, clse it 0. We experiment with two diffcrent layouts (renderings can be found in Figure]
room is a large, 29x29 square room. The starting position is at the center, and the goal is initialized randomly in one the corners at each reset. Trajectories for training the priors are obtained from this environment. Only for Figure the size was increased 81x81, to allow and better visualize long trajectories.
corridor is larger u-shaped corridor with three parts of lengths 60, and 60 respectively, assembled at 90º clockwise rotations. The structure of the corridor can therefore be contained in a 60 x 3 rectangle. Starting and goal position are fixed and located at opposite ends of the corridor.
maze a maze presenting intersections and dead ends. Starting and goal positions are fixed, and the optimal policy requires 66 steps.
For further details, we refer directly to the published codebase (URL in Footnote
J.3 Continuous Control
Due to their prevalence as a benchmark, we use gym-mujoco environments to evaluate SFP and baselines the presence of biased observations. We evaluate methods across 5 popular environments, namely Ant-v2, HalfCheetah-v2. Hopper-v2, Swimmer-v2 and Walker2d-v2 (see Figure 17 Additionally, we parameterize each environment with two variables, namely the bias and the sparsity parameter Observations in downstream learning are corrupted as s + s b, and rewards are sparsified as 1,2a where is the L2 distance between the initial and current position of the CoM. In order to aggregate data across environments, we empirically find an auvironment-specifn maximum sparsity threshold Imaz for which SFP fails to reach meaningful rewards. For cach environment, we then evaluate all methods on a set of 6 sparsity coefficients scaled linearly from to Æmaz: that are referred to as percentages in Figure and 23 The maximum sparsity thresholds Zmar 12 for HalfCheetah-v2, 2 for Ant-v2, Hopper-v2 and Swimmer-v? v2 and 1 for Walker-v2.
23

---

Published in Transactions on Machine Learning Research (08/2022)
Swimmer-v2
HallChestah-v2
Hopper-e2
Ant-v2
Walker2d-v2
Figure 17: Overview of the different gym-mujoco environments presented in Section 5.5
K
Implementation Details
K.1
Metrics
All metrics reported in this paper (e.g. cumulative returns per episode) are averaged over 10 random seeds (excluding gym-mujoco experiments and visual RL experiments, which are respectively limited to 5 and 2 seeds for computational reasons); mean learning curves are smoothed over 5 steps and shaded areas represent standard deviation across seeds.
We rely on two metrics for measuring the quality exploration trajectories in Section 5.3
%Coverage divides the reachable state space in n cubic buckets (n = 1000 for reach and n = 100 for room) and reports the ratio between the number of buckets visited by a set of 20 trajectories of length 500 and the total number of buckets.
Radius of Gyration Squared measures the spread in visited states, averaged over all trajectories, and adapted from Amin et al. 2021). Given a set T trajectories, the metric can be computed as:
(T) = ön Σ TET T
1
Σ (s, SET
where a trajectory + is modeled as a sequence of states (st) de(,,) measures the Euclidean distance and F = + E.sr.. s. We additionally normalize the metric by 8, which measures the diagonal of the box containing reachable states.
K.2 Data Collection
Training tasks for point-mant and meta-world are relatively simple, and allow collecting demonstrations through scripted policy. Since these environments do not include obstacles and allow direct control over the agent position (in 2D for room and 3D for reach), scripted policies simply receive the 2D/3D positions of the agent and of its goal, and output a distance vector. This vector then corrupted with isotropic Gaussian noise and scaled to fit within action limits. simple implementation of scripted policies is available part of our codebase (sce URL in Footnote D. For each training environment (reach room), collect 4000 trajectories of 500 steps each. Goals for the scripted policy are sampled uniformly from the reachable state space. On goal achievement, a new goal is sampled, once again uniformly. In the context of gym-mujoce experiments, 1000 trajectories of 500 steps each are instead collected by fully-trained SAC [Harnoja et al.] (2018) agents. The same expert datasets are then used for training state-free free-feee priors, behavioral priors or behavioral cloning.
K.3 Baselines
SAC We build upon the implementation provided by SpinningUp (Achiam| 2018), which is reported to be roughly on-par with the best results achieved on zym-mujoco (Brockman et al. 2016). For simplicity, we do not use automatic entropy tuning and keep a constant during learning. Nonetheless, our method could easily tune dynamically at the expense of increased complexity, as is described in Haarnoja et. al. (2018).
24

---

Published in Transactions on Machine Learning Research (08/2022)
Table 2: Hyperparameters for SAC.
Hyperparameter
Value
EPOCHS
125
STEPS PER EPOCH
4e3
STEPS OF INITIAL EXPLORATION
le4
STEPS BEFORE TRAINING
le3
ENVIRONMENT STEPS PER ITERATION
50
0.99
POLYAK AVERAGING RATE
0.995
INVERSE OF REWARD SCALE (a)
0.2 FOR META-WORLD AND GYN-XUJDCO, 0.02 FOR POINT-MAZE
BATCH SIZE
100
OPTIMIZER
ADAM
B1
0.9
B2
0.999
LEARNING RATE
0.001
HIDDEN UNITS
256
HIDDEN LAYERS
HINDSIGHT REPLAY RATIO
REPLAY SIZE
5e5 FOR VECTOR-RASED RL, 2c5 FOR IMAGE-RASED RT.
Moreover, we introduce n-step returns HHesell et al. 2018) with n = 10. We experimented with importance sampling for off-policy correction, but, similarly to what is reported by Hessel et al. (2018), we observed no
empirical benefit. All remaining hyperparameters are reported in Table 2 We anticipate that all baselines also rely n-step returns.
SAC+BC Behavioral cloning (BC) is performed on the entire dataset D for 10 epochs, by maximizing the log-likelihood of the Gaussian policy with respect to expert actions. The optimizer and batch size used for BC are the same as for downstream learning.
SAC-PolyRL SAC-PolyRL (Amin et al. 2021 replaces the initial uniform exploration phase of SAC with trajectories collected by a hand-crafted policy. While SAC's hyperparameters are unvaried, SAC-PolyRL specific parameters are tuned from those reported in various settings in the original paper. We use 0 = 0.35, o² = 0.017 and B = 0.01.
PARROT An official implementation for PARROT (Singh et al. 2021 writing. We reproduce the training routine and downstream application and adopt all hyperparameters reported in the original paper. Generative models size of 400 samples and Adam (Kingma & Ba 2015) as optimizer, with a learning rate of 0.0001, , 81 = 0.9.
not available at the time of
trained until convergence (100 epochs) using batch
B2 0.999 and a weight decay penalty of le 6.
In addition to the image-based version of PARROT used in Section 5.5 we introduce a version with a vectorized state space, dubbed PARROT-state, since the method is originally only applicable in visual settings. The only modification consists in replacing the image encoder with a 3-layer MIP with 256 neurons per layer and ReLU activations. In this setting, the input the encoder is therefore vector-based (non-visual).
The baseline method introduced in Section 5.2 replaces the entropy term in SAC with a KL-regularizatin term encouraging the policy to math the learned prior, as done in Pertschh et al. (2020). Existing hyperparameters are inherited from SAC, and the prior modcled as a Gaussian in order compute KL-terms in closed form. The inverse of reward scale a was tuned in the range (2e-4,20 2], and set to -3
KI-regularization
25

---

Published in Transactions on Machine Learning Research (08/2022)
K.4 SFP
Prior We model all families of state-Tree priors with conditional Real NVP Flows (Ardizoone el al. 2019
sharing the same architecture across all experiments. The invertible transformation fe is a composition of 6 coupling layers, each followed by a batch-normalization layer (Dinh et al. 2017]. For cach layer, a 3-layer MLP with 128 hidden units per layer used to preprocess the conditioning input. Scale and transition networks are also implemented as shared 3-layer ML.P with 128 hidden units, whose output split two provide scale and shift coefficients. All MLPs use ReLU activations. We train our state-free priors following the same protocol used with behavioral priors (100 epochs, batch size of 400 samples and Adam (Kingma & Ba 2015 with a learning rate of 0.0001, B1 = 0.9, 32 0.999 and weight decay penalty of le 1e-6) 6).
Downstream Learning Our method does not introduce additional hyperparameters with respect to SAC, excluding those involved with the design and optimization of the neural network parameterizing the mixing function Aw- This implemented as a 2-layer ReLU network with 128 units per layer and a sigmoid for output activation. We report that scaling the gradients for w by fixed coefficient was found beneficial for stabler training of the mixing network (c le for meta-world, point-maze and le 7 for gym-mujoco). As exploration is mainly driven by the prior, SFP also relies on lower a parameters, namely = 0.01 for meta-world, point and a 0.005 for gym-mujoco.
Finally, we introduce two implementation choices motivated by the prior knowledge that, during carly stages of training, exploration should mainly be driven by samples from the action prior æ. First, while SAC samples action uniformly for the first 10000 steps to encourage exploration, SFP directly samples actions from its prior instead. Second, we initialize the bias parameter of the final layer of the mixing network A. to o o= (Ao) in order to target mixing weights around Åo 0.95 during carly training.
All remaining hyperparameterss are shared with SAC (see Table 2
Test-time Policy At test time, instead of sampling an action from the mixture between prior मे and policy 2, the agent executes an action corresponding to the mean of the policy distribution. We note that this is consistent to what is done in SAC Haarnoja et al. (2018).
L Detailed Results
This section reports individual results were aggregated in previously occurring figures. In particular, Figure and Figure 19 report performance for each meta-world and point-maze task, respectively. Figure 20 reports the evolution of average mixing weights as training progresses for all meta-world tasks (an aggregation can be found in Figure 14h. Figures 21 and 22 also report performance for the two suites, but focus on visual RL settings. Figure 23 disentangles the results from Figure g across the five mujoco-gym
environments.
For completeness, we also include plots for meta-world tasks that remain out of reach of all tested methods, namely veg-insert-siee pick-place and push. In general, these tasks require both deep exploration and precise grasping and control. While SFP remains capable of deep exploration, precise manipulation constitutes a challenge, as expert demonstrations do not contain grasping behavior (see Section 6 In visual settings, due to increased complexity, observed drop in performance on reach, button-press and door-open across all methods.
26

---

Published in Transactions on Machine Learning Research (08/2022)
SAC-PolyRL
PARROT-state
SEP (ours)
button- prese
doer peor-open
draper draper-close
draver- opea
sineert-side
Test Retara
100
Test Retara
Test Retara
Test Retarn
Test Retarn
300
200
Steps pick-ppace
x100
Stepe
x1IP
Steps
X10
Steps
x10
Steps
posh
reach
Vindov-clll :loce
windox-opan
Tet Returr
Test Ratura
Test Retura
Tost Roturn
Return
100-
200
310
Steps
100
Stops
X10
Steps
110
Steps
X10
Steps
x100
Figure 18: Learning curves for each downstream meta-world task.
SAC
SAC+BC
SAC-PolyRL
PARROT-atate
SPP (ours))
Tiot Return
Test Return
Tost Roturn
400
200
Steps
Steps X10
Steps
Figure 19: Learning curves for each downstream point-maze -mae task.
Test Iecuurn
hutton-presa
Welght
Test Recura
door-apen
Weight
Test Recurn
drever-cloee
kilxing Welght
Recurn
draper-open
Mfxing Weight
Recurn
pag-inaert-a11
Aibing Weight
Stepe x100
Steps
Stepe
X10F
Steps X110
Scepe X10F
Test Retor
pick-place
Mixlog Woight
Tust Return
pueh
1.0
Test Return
reach
1.0
Weight
Retur
window-close
1..0
Mixing Weight
800
windox-cpen
Moding Woight
9..
Mixing
30
0,
Moding
Staps X10
Steps
X100
Scaps
X10
Steps X1I
Scaps X10
Figure 20: Evolution of mixing weight X and test performance for each downstream meta-world task.
SAC
PARROT
SPP (ours)
buttan-preaa
dncy-open
draser-open
pag- ina-rn-aattaeee
Tes Returm
Tost Return
Toot Return
Tost Return
Test Return
200-
2004
2004
310
2004
300
230
Steps pick-plase
10
Stepe
x10*
Steps
x10
Stops
X10
Steps unndo-open
x1
posh
window-s1ese
Test Returm
Tost Return
Test Return
Test Return
Test Return
100-
400
2004
310
2004
30
250-
Steps
168
Stope
x10
Steps
11
Stops
x100
Steps
x10
Figure 21: Learning curves for each downstream meta-world task visual settings.
27

---

Published in Transactions on Machine Learning Research (08/2022)
SAC
PARROT corridor
(ours)
Tost Return
Tes Return
Test Raturn
100
100
300
200
30
Steps
X10
Steps
X10
Steps
x100
Figure Learning curves for each downstream point-maze task visual settings.
Ant-v2 PARROT-state
HalfChoctah-v2
Hopper-v2
Swimmer-v2 PARROT-stat
Walker2d-v2
PARROT-state
PARROT-state
PARROT-state
1000
1000
2000
1000
2000
100%
10054
1007
1006
12050
750
8054
750
750
750
730
6035
500
808
500
Sparsity
600
500
Spacsity
008
500
Sparsity
6035
300
40%
4050
250
un3
406
250
250
SFP
SFP
SFP
SF P
SFP
1020
1200
10035
100%
10005
100%
10036
38
88
8035
809
5035
805
80%
Spersity
Sparsity
Sparsity
Sparsity
503
620
500
4036
250
40%
400
40%
259
40%
250
4036
350
2035
205
2035
2/5
207
01
LO
10.0
0.1
10.0
011
1.0 10.0
0.1 1.0
1.0
10.0
Bias
Biss
Bias
Bius
Blas
Figure 23: Mean performance on each gym-mujoco environment for different sparsity thresholds and bias norms.
28

---

Published in Transactions on Machine Learning Research (08/2022)
meta-world (training)
meta-world (downstream)
100
1(
Test Return
Test Return
400
R(adloL-))
200-
200
m(ad/s))
01
0
3
2
Steps
x x10
Steps
x10
Figure 24: Comparison downstream performance action priors with different conditioning variables from Subsection 5.3
aggregated over downstream tasks via IQM.
meta-world (training)
meta-world (downstream)
100-
Test Return
400
Test Return
300-
#(adlof:))
300
200
2000
100
100
Figure 25: Comparison of final downstream performance of action priors with different conditioning variables from Subsection 5.3 aggregated over downstream tasks via IQM.
M Alternative Aggregation and Visualization
In this Section, we present an alternative aggregation scheme for key figures in the main paper. In particular, we compute the TQM with 95% stratified basic boostrap confidence interals as suggested Agarwal et al.)
(2021b. We further provide bar plots representing the final performance for each figure. In particular, Figures
26 and 28 use IQM aggregation for the data used for Figures 7 and respectively. Figures 25
and 29 report bar plots represent the final IQM over task performances in Figures and respectively.
SAC
SAC-BC
SAC-PolyRL
PARROT-state
SFP (ours)
neta aeta-vorld (treining)
Tex Return
meta-world (downstream
point-man (downstream)
200
300
Test Retur
Test Return
400
Steps
XII
900-
point-maze (training)
Teat Return
100-
100
3
Steps
x10F
Steps
X109
Steps
x10
Figure 26: Comparison SFP with several baselines from Subsection 5.4 Results are aggregated across tasks via IQM, instead of mean.
29

---

Published in Transactions on Machine Learning Research (08/2022)
SAC
SAC+BC
SAC-PolyRL
PARROT-state
SFP (ours)
peta-world (training)
Teat Retum
mO
meta-world (downstream)
point (downstream)
2004
Test Return
Test Return
300
point-maze (training)
200
Text Rotmm
Figure 27: Comparison of SFP with several baselines from Subsection 5.4 Final returns are aggregated across tasks
via IQM, instead of mean.
SAC
PARROT
SFP (ours)
SAC
PARROT
SFP (ours)
meta-vorld (dounstrean)
point-nazz (dovnstrean)
neta-world (dounstrean)
point (dounstrean)
Test Return
Test Roturn
Test Return
100
Test Return
400-
900-
100
50
200
Steps
x10
Steps
X10
Figure 29: Final performance on downstream learning in visual settings from Subsection aggregated via IQM.
Figure Performance on downstream learning vi-
sual settings from Subsection 5.5]
aggregated via IQM.
30


Reviews:
[{"review_id": "XNzhGA3hgU", "reviewer": "Reviewer_7Mab", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Simple, easy-to-implement approach\n- Can be useful in a number of settings where there is a state-space mismatch between the training tasks and the downstream tasks \n\n### Weaknesses\n- As is often the case in this line of work, the learned priors are not very helpful when the downstream tasks require substantially novel behavior. For example, since the priors in this work are mainly trained on reaching tasks, they cannot help pick and place objects in downstream tasks (as acknowledged by the authors, and as seen in Figure 15 in the appendix). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a technique for incorporating behavioral priors in reinforcement learning, where a behavioral prior is a model over actions learned using some previously collected (near-expert) data. In contrast to prior works which typically learn a state-conditioned prior, this paper learns a temporal prior, wherein the action selected by the prior at any given time step depends only on the prior action(s) taken in that environment. This requires the prior dataset to contain temporally directed behavior, but the state space for the prior and downstream tasks can be different. The authors combine their learned temporal prior with an existing off-policy RL algorithm (SAC) in a straightforward way: the exploration policy in SAC becomes a mixture of two policies: the learned temporal prior, and the policy learned by SAC. The mixing weight for this mixture policy is itself learned via the maximum entropy objective in SAC. The authors show results on a suite of simulated manipulation and navigation tasks, and find that temporal priors hold significant advantage when state distribution is different for training tasks and downstream tasks. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- The authors talk a lot about their approach being “task-agnostic”, but I don’t think any kind of data-driven exploration method is truly task agnostic - you need some kind of shared structure between the tasks in the offline dataset (training task) and the new tasks you are trying to learn (downstream task). There is more nuanced discussion about this in Section 4.1, so I believe some of it should be noted in the introduction as well (i.e. avoid overpromising to readers). \n- Related work. Paper is missing an important citation in the “temporally extended exploration” section of the paper. Bogdanovic et al [1] do something very similar, except they condition on the history of past states, as opposed to the last action. For the kind of trajectories considered in this paper (a robot end effector reaching a specific point), I accept the practical behavior of these learned priors to be similar (but the prior work requires shared state spaces). \n- Visual RL results. While the proposed technique does better than the baseline in the visual RL setting, it is important to note that TempoRL fails to make progress on a majority visual RL tasks, as shown in FIgure 17 in the Appendix (but is still better than the baseline which does not work on any visual tasks in this setting). This should be noted in the main paper, and some discussion should be provided on why this is the case. \nI would encourage the authors to open-source their code for this project, as it would help accelerate research in this area. \n\n[1] Miroslav Bogdanovic and Ludovic Righetti. ​​Learning to Explore in Motion and Interaction Tasks. IEEE International Workshop on Intelligent Robots and Systems (IROS), 2019. https://arxiv.org/abs/1908.03731\n", "broader_impact_concerns": "I do not have any broader impacts concern for this work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YUJtyzPuk2", "reviewer": "Reviewer_J9Aw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe use of behavior priors can help to solve one of the notorious problems of (deep) RL: sample efficiency. The core problem the paper tackles -- learning more generalizable behavior priors -- is important for wider applicability of behavior priors and thus impactful in the community.\n\nThe paper is well-written and easy to follow. It briefly summarizes the relevant background on behavior priors in RL and then clearly lays out the proposed idea(s). The figures help to communicate the ideas throughout the paper. The experimental evaluations highlight the scenarios in which the proposed priors lead to superior generalization and give a good intuition *why* they lead to faster learning downstream with qualitative examples.\n\nI like the \"additional applications\" section (5.4) since the authors show some less intuitive but relevant use cases for the proposed prior design -- transferring priors from state to visual observations and transferring in the presence of systematic corruptions in training or downstream task observations.\n\nI also found the second contribution of the paper interesting: apart from the state-free prior design, the paper proposes an alternative to the usual KL-regularization approach for using priors to guide downstream learning: they use directly learn the policy as a weighted mixture of the prior and a learned policy, vaguely similar to a residual RL formulation with a fixed \"base policy\" (the learned behavior prior) and a residual policy that is trained on the downstream task, just that instead of adding the two (as I believe is commonly done in residual RL), they mix them probabilistically and sample from one or the other during downstream learning based on a learned advantage estimate. This way of using the learned behavior prior is novel to my knowledge and can be impactful even independent of the particular state-free prior design proposed in the paper.\n\n\n## Weaknesses\n\nWhile I like the general direction of the paper and the presentation of the overall idea, there are a few weaknesses that should be fixed before I can recommend acceptance.\n\n(A) **More detailed discussion of prior work on information asymmetry**: The concept that withholding certain information / inputs from learned behavior priors will lead to better generalization (at the cost of lower prior expressivity) has been well-discussed in prior work (eg [1-3]). Tirumala et al [2] explicitly mention on page 8 of their paper the option to condition behavior priors only on prior actions instead of states. Although all of these papers focus their experimental evaluations on different information asymmetries than this paper (eg training prior on only proprioceptive state information), the fact that information asymmetry is not a novel concept introduced in this paper should be adequately discussed in the related work. I believe the main novelty of this paper with regards to the prior design is the actual experimental evaluation of the state-free priors which was not performed in prior works. The writing should reflect this.\n\n(B) **Cleanly distinguish between two contributions in experimental evaluation**: The paper makes two contributions: the state-free behavior prior design and the novel approach to using the prior to accelerate downstream learning. However, the paper focusses all the attention in the experimental evaluation on the first point and conflates the contribution of the second point. Instead, both contributions should be cleanly investigated separately. Eg the paper could first establish that state-free priors lead to superior transfer (with the existing experiments) and then ablate the choice of prior regularization technique, comparing to the two most common approaches: (1) softly regularizing the policy towards the prior with a KL constraint, (2) warping the action space like in PARROT. Specifically the first evaluation is missing from the paper now, but is vital to show the benefits of the proposed policy mixing scheme.\n\n(C) **Show performance decrease for downstream tasks with in-distribution states**: It is intuitive that a state-free prior will generalize better to OOD states in the downstream task. But it is equally intuitive that the learned prior will be less expressive than a state-conditioned prior, since it is essentially blind to the agent's current state. It would be important to quantify this trade-off by showing the performance of the proposed prior design in scenarios where the downstream task has states that are in-distribution to the training data & compare to state-conditioned priors. The paper already has such evaluations, but only on the same tasks as used for collecting the training data. In contrast, prior works that use behavior priors for accelerating RL have shown transfer to *unseen* tasks with *seen* state distributions. It seems important to show evaluations of the state-free vs state-conditioned priors in this scenario -- eg evaluations on the commonly used FrankaKitchen environment could be added to show the performance decline one would need to accept for in-distribution state scenarios (this could serve to motivate future research into automatically figuring out the best prior conditioning).\n\n(D) **Naming of \"temporal priors\" not ideal**: The authors call their proposed state-free prior design \"temporal priors\" and their method accordingly TempoRL, presumably since temporal correlations between actions is all a state-free prior can learn to model. However, prior works have investigated conditioning on prior temporal information (like past states) too, eg [3]. The *temporal* aspect of the proposed priors is not the novel part, but the *state-free* part. Thus, I suggest the authors consider renaming their approach to focus on the state-free aspect of the learned priors rather than the temporal modeling aspect to clearly distinguish it from other temporal prior designs.\n\n\n## Minor points and suggestions\n\nThere were a few minor points, which I don't see as critical to the acceptance of the paper, but could improve the overall quality of the paper:\n\n- It could be nice to give an intuitive example for a scenario in which the structure of the observations would not be known while learning the prior (and thus a state-free prior is the best modeling decision) -- ie it is clear that state-free priors reduce the required assumptions on the data, but the importance of this could be made clearer with an example.\n\n- Regarding the sentence in section 3.1: \"... we do not require high quality trajectories to be collected on a task that is similar to the one at hand.\" This has two parts: (1) high quality, (2) on a related task to the target task -- it would be good to make clearer which of these is not required (presumably (2), while (1) is still required to learn a reasonable prior).\n\n- It would be interesting to show the development of the learned mixing coefficient lambda over time / some visualizations of situations where it is high vs low to get better insight into what it learns.\n\n- The characterization in the conclusion that state-conditioned priors are superior when \"the data collection policy can solve the downstream task\" is too narrow -- they should work well when the downstream state spaces are in distribution of the training state space, even if new tasks need to be learned.\n\n- It could be interesting to apply the proposed priors to the CALVIN environment: CALVIN provides large unstructured datasets that can be used for training behavior priors and crucially features four different environment layouts. One could expect that state-conditioned priors would have trouble generalizing between the environments, while the proposed state-free priors could generalize much better.\n\n## Questions\n- I am surprised to see in Fig 4 that p(a | s, a_t-1) is doing better than p(a | s) --> it should suffer from the same domain shift -- do you have any intuition why it is doing better? --> It would also be really interesting to see this comparison in an environment where state-based transfer should work (see point (C) above) to see whether conditioning on past actions too is generally useful in these scenarios too.\n\n- How hard is it to accurately learn the advantage used to update the lambda parameter? I have experimented with advantage weighting for prior regularization myself before (although in the context of KL-regularized RL) and found it very hard to accurately learn to estimate the advantage, leading to meaningless regularization weights. Did you find any particularly sensitive parameter in that optimization?\n\n\n\n[1] Galashov et al., Information asymmetry in kl-regularized RL, 2019\n\n[2] Tirumala et al., Behavior priors for efficient reinforcement learning, 2020\n\n[3] Salter et al., Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning, 2020\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper explores the transfer of behavior priors, learned on an offline dataset of diverse experience, to accelerate reinforcement learning on a new downstream task. It specifically investigates scenarios in which the behavior priors are solely conditioned on past actions p(a_t | a_t-1) instead of the environment state p(a_t | s_t), as is commonly done in prior work. In experimental evaluations on transfer problems, including Metaworld, the paper shows that non-action conditioned priors transfer better to unseen state spaces and allow transfer from state-based pre-training data to visual RL downstream tasks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See \"weaknesses\" section above, particularly points A-D seem critical for acceptance to me.", "broader_impact_concerns": "The provided broader impact statement seems sufficient to me.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lxSRbqU6zs", "reviewer": "Reviewer_SWNm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The idea of removing the state-dependency in expert-provided trajectories is, as far as I'm aware a novel approach, which I'm surprised works as well as it does. This alone makes the work interesting in my opinion. \n- Significant performance gap over baselines on provided tasks in the transfer (downstream) setting. \n- High quality presentation/polish. Visually, the paper looks good, and the figures are easy to follow. \n- Thorough appendix with details, analysis, etc. Code is provided & reproducibility feels high.\n\n**Weaknesses:**\n\nExperiments: The experiments part of the paper is very weak in my opinion. \n- Presentation of results: While the results are visually pleasing, how the authors have chosen to present the results is at best confusing, and at worse deceptive. In Figures (4, 6, 8, 9) multiple tasks are aggregated into a single figure. This is particularly bad as the fact that this aggregation has occurred is not stated and there is not description of how it occurred (i.e mean? median?). There are shaded areas surrounding learning curves but there is no description of what they represent. See [1] for some possible discussion/suggestions on result aggregation. In general, there is a lot of information that is unavailable without combing through the appendix, such as the number of demonstrations used. \n- Baselines/Environments: The baselines overall felt lacking. The authors make extensive reference to both [2,3] and yet only [2] is included as a baseline. Notably while [2] shows some negative examples for transfer, [3] focuses on transfer/downstream tasks, so it feels like a critical baseline. Given the performance of SAC-PolyRL, other exploration methods could also be useful as a strong baseline. \n \n  In terms of environments, compared to prior work [2,3], the domains examined seem relatively simplistic. For example, on the training task, the performance is strong across all baselines, and in particular, PARROT seems to learn in <20k timesteps, suggesting these tasks may be too easy. Examining Figure 15, many tasks are solved almost immediately, or not at all. \n\n  Part of the issue being that I did not expect any of the baselines to perform well. While I'm not super familiar with what is state-of-the-art on MetaWorld, [4] for example has some good results, including on \"insert peg side\" where all the baselines used in this paper fail (as well as TempoRL). Some baselines that can learn well on MetaWorld would allow for a better understanding of how much benefit, in terms of learning speedup, TempoRL provides.\n\nWriting: There are a few places where the writing comes across as vague and omits necessary details to follow the logic the authors are presenting. \n- \"The main challenge introduced by temporal priors stems from their non-Markovianity, which renders existing integrations schemes unsuitable Singh et al. (2021); Tirumala et al. (2020).\" This statement confused me because (1) these prior methods are not explained in the paper, so I'm not sure why they are necessarily not amenable to a non-Markovian prior and (2) it's unclear to me TempoRL deals with this problem either. As far as I can tell, the objectives Equation (3-6) ignore the non-Markovian nature of the prior. \n- \"the entropy term is computed on the policy alone, as it is not desirable to incentivize exploration of states according to the prior's entropy.\" I'm not sure why this is true. Additionally, since this change affects the derivation of SAC (as now the entropy is no longer over the current policy), I'm not sure why it's correct either. \n\nNovelty: While I feel the authors do enough to make the work interesting, it is worth mentioning that the novelty isn't too high. The proposed prior is a small modification of behavioral priors and the integration into SAC is very similar to existing work in the learning from demonstrations field, particular Q-filter [5]. \n\nTypos:\n- Missing period: \"between the policy and temporal prior Crucially\" (page 2).\n- Should be \\citep: \"if they are rather different from those demonstrated Parisi et al. (2021)\" (page 1).\n\nReferences:\n- [1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" 2021.\n- [2] Singh, Avi, et al. \"Parrot: Data-Driven Behavioral Priors for Reinforcement Learning.\" 2020.\n- [3] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. \"Accelerating Reinforcement Learning with Learned Skill Priors.\" 2021.\n- [4] Kumar, Aviral, Abhishek Gupta, and Sergey Levine. \"Discor: Corrective feedback in reinforcement learning via distribution correction.\" 2020.\n- [5] Nair, Ashvin, et al. \"Overcoming exploration in reinforcement learning with demonstrations.\" 2018.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce temporal priors, an extension of behavioral priors, which use provided expert trajectories to accelerate learning by guiding the policy towards relevant data/increasing coverage of the state space. Temporal priors are state-independent, and instead conditioned on the sequence of prior actions.\n\nThey motivate this difference by suggesting that when considering task transfer (i.e. expert trajectories are provided for task A but we are interested in task B), state-dependent priors are more likely to overfit to the provided behavior, limiting their effectiveness on new tasks. \n\nMain contributions:\n- Introduction of action-conditioned behavioral priors (temporal priors). \n- Provide a way to integrate temporal priors into the formulation of SAC. \n- Demonstrate the effectiveness of temporal priors to accelerate learning, particularly in the case of task transfer on the meta-world and point-maze benchmarks. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- (High importance). The authors need to improve the presentation of the results. I would suggest replacing Figures (4, 6, 8, 9) with better aggregation or a different presentation of the results such as bar graphs (as done in Atari), tables, presenting a subset of learning curves, or taking in consideration some of the ideas/recommendations in [1]. The authors also need to clearly label if/what aggregation is occurring and details such as what the shaded area represents and the amount of expert data.\n- (Mid importance). I would like to see [3] included as a baseline and ideally some methods which don't use behavioral priors but perform well on MetaWorld. \n- (Mid importance). Additional justifications/clarity should be added to the places I noted. I would also like to see a bit more details in the body of the paper on the derivation of the objectives. \n- (Mid-low importance). The paper would be improved with some additional ablations, particularly varying the amount of expert data used and changing the training task used for transfer, as [2] noted this can substantially affect performance. \n- (Low importance). The biased downstream learning experiments seems a bit overfit to making PARROT perform poorly. I would be interested in seeing it repeated with bias/noise on the actions rather than the states.\n\nReferences:\n- [1] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" 2021.\n- [2] Singh, Avi, et al. \"Parrot: Data-Driven Behavioral Priors for Reinforcement Learning.\" 2020.\n- [3] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. \"Accelerating Reinforcement Learning with Learned Skill Priors.\" 2021.", "broader_impact_concerns": "I have no concerns rearding the broader impact statement made by the authors.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel and surprisingly simple approach to distill offline data from an environment into "priors" that can usefully guide exploration even in new environments and new tasks. The key idea is simply to omit states like prior methods have done, and instead model action-only sequences. Through evaluation that has become significantly more robust and thorough over the course of the rebuttal discussions, they show that this does indeed perform well, particularly for transfer to new environments/new state spaces, where prior methods would be operating out of domain and hence perform poorly. The new experiments added during the rebuttal phase more clearly circumscribe where this method is useful, pointing out limitations. 

A second interesting idea is also proposed for probabilistically combining policy prescriptions from the prior policy and a new task-specific policy trained with SAC, and is now evaluated to show gains over alternative approaches for influencing task policies with priors.

Connections to prior work have also been more clearly stated now. I thank the reviewers and authors for a very productive review and rebuttal phase.

As the paper stands, it is nearly ready for publication. Most comments from reviewers have already been addressed in the paper. As reviewer J9Ab suggests, I recommend that the authors change the name of the method (and correspondingly, the title) to something more transparent and indicative of the key idea: "state-free", or "action-only" behavioral priors. I recommend acceptance for publication after this minor revision. If the authors so please, they may acknowledge the anonymous reviewers for their contributions towards improving the manuscript. No other changes necessary.
Decision: Accept (poster)