TempoRL: Temporal Priors for Exploration in Off-Policy Reinforcement Learning
Authors: Marco Bagatella
Sammy Joe Christen
Otmar Hilliges
Venue: ICLR 2022 Submitted (2022)
Keywords: deep reinforcement learning
exploration
prior

Abstract: Effective exploration is a crucial challenge in deep reinforcement learning. Behavioral priors have been shown to tackle this problem successfully, at the expense of reduced generality and restricted transferability. We thus propose temporal priors as a non-Markovian generalization of behavioral priors for guiding exploration in reinforcement learning. Critically, we focus on state-independent temporal priors, which exploit the idea of temporal consistency and are generally applicable and capable of transferring across a wide range of tasks. We show how dynamically sampling actions from a probabilistic mixture of policy and temporal prior can accelerate off-policy reinforcement learning in unseen downstream tasks. We provide empirical evidence that our approach improves upon strong baselines in long-horizon continuous control tasks under sparse reward settings.

Under review as a conference paper at ICLR 2022
TEMPORL: TEMPORAL PRIORS FOR EXPLORATION IN OFF-PYLICY REINFORCEMENT LEARNING
A Anonymous authors Paper under double-blind review
ABSTRACT
Effective exploration is crucial challenge in deep reinforcement learning. Behav- ioral priors have been shown to tackle this problem successfully, at the expense of reduced generality and restricted transferability. We thus propose temporal priors as a non-Markovian generalization of behavioral priors for guiding exploration in reinforcement learning. Critically, we focus on state- indeeeenndde temporal pri- ors, which exploit the idea of temporal consistency and are generally applicable and capable of transferring across wide range of tasks. We show how dynami- cally sampling actions from a probabilistic mixture of policy and temporal prior can accelerate off- off-policy reinforcement learning in unseen downstream tasks. We provide empirical evidence that our approach improves upon strong baselines in long-horizon continuous control tasks under sparse reward settings.
INTRODUCTION
Exploration is fundamental issue in reinforcement learning (RL): in order for agent maxi- mize its reward signal, it needs to adequately cover its state space and observe the outcome of its actions. This becomes increasingly harder when dealing with large, continuous state and action spaces, which includes many real world applications. There exists a large and fruitful body of re-
search on exploration (Bellemare al., 2016; Osband et 2016 Tang al. 2017 Osband al. 2018 Azizzzaeeneshlli et al.) 2018 Burda et al. 2018 Dabney et al. 2021 Ecoffet et al. 202
however most general purppose algorithms remain based on e-greedy exploration (Mnih et al. 2015)
or entropy-regularizee Gaussian policies (Harrojj et al. 2018). In the absence of an informative re- ward signal, both methods rely on uniformly sampling actions from the action space, independently of the history of the agent. Unfortunately, in sparse reward settings, achieving positive returns by uncorrelated exploration becomes exponentially less likely as the horizon length increases (Dabney) al. 2021). Moreover, within this setting, the lack of temporal correlation result in undesirable behaviors during exploration, such as reversing recent actions.
A promising approach to achieve efficient exploration is that of using behavioral prior to guide the policy (Pertsch et al. 2220); Tirumala et al., [2200] Singh et al. 2021). Typically, this is learned from expert trajectories
state-conditional action distribution. Behavioral priors are able to foster directed and correlated exploration (Singh et al. (021). by assuming strong similarity between the agent and expert tasks. However, an agent should ideally be able to produce efficient explorative behaviors even in unseen environments and unrelated tasks.
To overcome the shortcomings of behavioral priors, we propose temporal priors, powerful non- Markovian generalization which is capable of guiding exploration in challenging settings. In partic- ular, we put our attention on the family of indeeinnee temporal priors, which enable accel- erating reinforcement learning and transferring knowledge to unseen tasks by focusing temporal correlation between actions.
In our method, which we dub T'MMporal Priors for exploration in Off-policy Reinforcement Learn- ing (TempoRL), we introduce a temporal action prior (ag]st, Hy), where H, = (si, a4)) 1 is the history of the agent. In particular, we find that the class of state-independent temporal priors ā (a))(a))) -1) is sufficient for capturing desirable properties for exploration, such as directness and temporal correlation, and advantageous in situations where the prior carries knowledge about the current state. Temporal priors can be trained offline from few task-agnostic expert trajectories (see Figure T. Furthermore, we propose principled manner of integrating priors into the Soft Actor

---

Under review as a conference paper at ICLR 2022
Temppers Prise Rie
Hatory
Tadeaggootee
Mixtreo
aeogle
Action
Domalnn
Gap
Stuto
Actor
Dosessieoors Knviranment
Figure 1: TempoRL: A temporal prior is trained on task-agnostic trajectories. Actions are then sampled from a dynamic mixture between a state-independent temporal prior and the policy in downstream learning of more complex tasks. Our method works with both vector-hased and image-based state inputs.
Critic framework (Haarnoja et al. 2018) without breaking the Markovian assumption of the learning rule. In downstream learning more complex tasks, our method samples actions from dynamic mixture between the policy and the temporal prior. Moreover, the policy is regularized by directly maximizing the likelihood of sampling from the prior instead the policy's 's entropy. Our approach is general and can be applied for arbitrarily complex priors.
In our experiments, we first compare the adequacy behavioral priors and different families of temporal priors for accelerating downstream RL. We then focus inte-inndepndeet temporal priors and verify their capability to produce correlated and directed behavior. We provide empirical evidence that method can accelerate learning in long-horizon control tasks with sparse rewards. We demonstrate the effectiveness of our approach by comparing against state-of-the-art baselines.
Our contributions can be organized as follows:
1. We propose learnable non-Markovian action priors conditioned on the history of the agent. We show that sampling from these prior produces directed and correlated trajectories.
2. We introduce novel manner of integrating temporal priors into the Soft Actor Critic frame- work (Haarnoja et al. 2018)_
3. We show how state-independent temporal priors can be learned from few expert trajecto- ries simple tasks and used to improve exploration efficiency in new tasks, despite the presence massive domain gaps (c.g. from a simple reaching task to opening a window) and across entirely different settings (e.g., from non-visual to visual RL).
After discussing our method's novelty and related literature Section 2 we introduce our setting Section The method described in Section while empirical evidence of its effectiveness reported in Section [ Finally, Section [ contains a brief closing discussion of our work. We make our code available for research purposes
RELATED WORK
Demporally-Extentee Exploration Several works have attempted directly address the inability
of traditional methods, such as e-greedy or uniform action sampling (Lillicrap et al. [2016] Haarnoja
al., (2188), produce correlated trajectories. An interesting study (Dabney et al. [202]) highlights this issue and shows how repeating random actions for multiple steps is sufficient to significantly accelerate Rainbow (Hessel et al., 2018) on Atari (Bellemare et al.. 2013). Similarly, AAmin et al.)
(2021) propose non-learned policy inspired by the theory of freely-rotating chains in polymer physics to collect initial explorative trajectories continuous control tasks. Both methods pinpoint fundamental issue, but rely scripted policies which are hand-crafted for particular family of
'We borrow this term from domain adaptation literature hint the different nature of the environment used for collecting expert trajectories and of the environment the RL. agent deployed in. Informally, consider the domain gap hetween two environments to large policy trained struggles to solve the second environment, or vice versa, even after additional training. https: tes. google.....oooooooooooooooo..g e. wttp//////////eeee/eeeeeeeeeeeee
environment
2

---

Under review as a conference paper at ICLR 2022
environments. On the other hand, our method is learned from task lask-agnostic trajectories and does not require engineering an explorer, which can be unfeasible for complex tasks.
Hierarchical Reinforcement Learning Another approach to tackle exploration-hard tasks is to rely on hierarchical decomposition of the agent into different levels of temporal and functional
abstraction (Parr & Russell 1998; Dietterich 2000, Sutton al., 1999 Dayan & Hinton 2000).
For instance, the task can be decomposed into bigh level planning and often referred to as skills (Konidaris & Barto 2007 Eysenbach et al. 2018 or options (Sutton
set of low-level policies,
et al. 1999 |acon et al., (017). This approach effectively reduces the planning horizon and allows efficient sol sing of complex tasks from scratch (Bacon et el. 2017] Vezhnevets et al., 2017 Nachum
al. 2018 Levy al.) 2019] Christen al. 2021). Low-level policies can be trained without
supervision to achieve correlated and directed behaviors (Eysenbach et al. 2018), however, the issue of temporal correlation is merely relocated in the hierarchy, as the high-level planner is not encouraged to produce correlated sequences of skills. Incidentally, our method is not designed to achieve temporal abstraction, but can be interpreted in hierarchical framework (Schäfer et al. 2021] in which a high-level criterion (the mixing function) governs a probabilistic choice between an explorer (temporal prior) and an exploiter (policy).
Behavioral Priors Behavioral priors are generally represented by state-conditional action distri- butions modeling strategies for the current state of the environment. Such priors can be learned jointly with the policy in the context of KL-regularized RL (Tirumala et al.) 2019 2020), which in some cases restricts the information available to the prior (Galashov et al. 2019). second ap-
proach consists in learning behavioral priors from expert policies on related tasks. This is the case
for several works (Peng et al. 2019; Pertsch et al. [2200] 2021 Ajay et al., 202I which adopt a Gaussian behavioral prior in latent skill-space. In particular, Pertsch et al. (2020) report that prior is crucial to guiding a high-level actor in an HRL framework. An important contribution to the field is made by PARROT (Singh et al. 2021), which focuses visual setup and introduces a fowebased transformation of the action space to allow arbitrarily complex prior distributions. We extend this idea to prior action distributions that are not only conditioned on the current state part thereof, but rather on the history of the agent, and are therefore non-Markovian. Moreover, we propose novel and more flexible way of integrating the prior distribution into the learning algo- rithm. Most importantly, we overcome the reliance of the last two methods on a tight domain gap between tasks for training the prior and the agent.
BACKGROUND
3.1 SETTING
Reinforcement learning (RL) is the problem that an agent faces when learning to interact with a dynamic environment. Albeit with a slightly different definition, we formalize the environment as goal-conditioned Markov Decision Process (gc-MDP) (Nasiriany et al. 2019), that consists of 6-tuple (S, A, G, R. 7). where S is the state space, A is the action space. g C S is the goal space, R S x G + R is scalar reward function, T : S x A - II() probabilistic transition function that maps state-action pairs to distributions over S and, finally, y is a discount factor. Assuming goals to be drawn from distribution pç. the objective of an RI. agent can then be expressed over a time horizon T as finding a probabilistic policy x² = arg max, Egwpe Ei= >*R(st. ())))))) with St ~ T(s=-1, 1)) and a:-1 ~ x (St-1: g). In order to simplify notation, from this point on, we will implicitly include the goal into the state at each time step: 8: 4 (st.9).
We focus on long-horizon control problems with continuous state and action spaces and sparse re- wards, i.e. non-zero only after task completion. Although our method can be generally applied to stochastic oof-policy RL methods, we build upon Soft Actor Critic (Haarnoja et al. 2018)) with Hind- sight Experience Replay (Andrychowicz et al. 2017), due to their wide adoption in these settings.
Finally, in contrast with several behavioral prior approaches (Galashov et al. 2019 Pertsch et al., 2020 Singh et al. 2021). we adopt a more general and challenging setting. First, we do not assume that prior information on the structure of the state space is available. Second, while we also assume access to a collection of expert trajectories D = {(s). ag. sf. .i..... ..... s'y. a }/- we do not require high quality trajectories collected the exact environment and task. From this point on, refer
3

---

Under review as a conference paper at ICLR 2022
Behavioral Prior
Temporal Priors (ours)
B2
tt (te
t T(a St, He) b)
т T(aelaj -
(at at-1) d)
a)
Figure 2: Graphical models representing different priors: from left right, a behavioral prior, general temporal prior, a state-independent temporal prior and single-step state-independent temporal prior. Solid arrows represent the environment's s transition function T, while dashed arrow indicate conditional modeling.
to the environment used for collecting data as the training enviroument, and to the environment in which the RL agent deployed as the downstream environment.
3.2 BEHAVIORAL PRIORS
A behavioral prior ((lss) (Pertsch et al. 2020 Singh al. 2021 is ctate-conditional probability distribution over the action space (ct. Figure 2). Behavioral priors can be trained to assign high probability useful actions with respect the current state, and hence be used accelerate RL. A behavioral prior can only guide the policy effectively as long as prior information on the structure of the state space available (Galashov al.] 2019). or the prior has been trained on data collected on closely related task and environment (Pertsch et al. 2020 Singh et al. 2021).
In our settings, the structure of observations is unknown and expert trajectories may be collected on unrelated tasks. This means that the distribution of training states might not match the distribution of states produced by downstream environments: behavioral priors will then be evaluated on out-of- distribution samples and their performance will degrade drastically, as shown in Section 5.1
METHOD
Our method relies on the integration of learned temporal action prior into off-policy RL algo- rithm. Thus, we first define and discuss the class of temporal priors of interest, and later describe how they can be integrated into existing off-policy algorithms.
4.1 TEMPORAL PRIORS
Within the settings outlined in Section it is still possible to extract and transfer knowledge from the expert dataset D to the agent, despite a significant domain gap. Namely, we can model the temporal correlation of the expert trajectories and use it to speed up downstream learning.
In order to recover this information, we thus propose to learn a temporal prior. A temporal prior is a non-Markovian action prior, representing probability distribution over the action space that is conditioned on the current state alone, but also on the past history of the agent: # #(ae|St, H,) (see Figure Zp), where H, (Sy, ags .... 5 0t-1). Temporal priors are powerful generalization of behavioral priors, which directly address temporal correlation.
In the challenging settings we describe Section 3.1, conditioning on the current state not just insufficient, but might also be counterproductive, as a state-conditie state-conditional action prior would receive out-of-distribution samples as inputs. Hence, we focus on the class of state-independent temporal priors, which drop their dependence on the environment's state. This is indeed viable strategy in a hard-exploration setting, when no prior information is available on the state space and no reward observed: in this case, no information can be extracted from the state in any case. The most general definition of state-independent temporal prior is a probability density 7 (aagla))) ( (see Figure 2 Our empirical evidence suggests that its simplest form, i.c., ( (a-1) (see Figure 21)
surprisingly competitive with variants conditioned on multiple past actions (cf. Section 5.1 and therefore sufficient to capture complex temporal relations.

---

Under review as a conference paper at ICLR 2022
Independently from the conditioning variables, temporal priors can conveniently be modeled as parametric or non-paaraan parametric conditional generative models and learned through empirical risk min- imization. For the purpose of this paper, we choose to the conditional variant of the Real Non Value Preserving Flow (Dinh et al. 2017, Ardizzone et al. 2019), which has been successfully ap- plied in similar settings before (Singh et al. (02T) and is well suited for Fuclidean action spaces. Our integration in the SAC framework allows arbitrarily complex prior distributions, which Real NVP Flows are in principle able to capture.
In the context of NVP Flows, training samples are actions a E A. paired with conditioning variables (s, H), thus the learned mapping is analytically compute the likelihood of a single training pair (a, (s, H)) and maximize its expected value through standard gradient-based optimization techniques. An empirical justification of this choice is found in Appendix F.4 while implementation details are reported in Appendix [ For complete introduction to Real NVP Flows, we refer the reader to Doninn et al. (2017).
fo(2: (s. H)), with 2 ~ .. Since fo is invertible, we
One final concern regards the nature of the data for training the temporal prior. The main require- ments for the training data are two: (1) the environment which the data is collected needs to share the same action space of the downstream environment and (2) the training trajectories should display the desired qualities of correlation and directness. In general, we adopt task-agnostic expert trajec- tories generated by achieving simple random goals. Such simple trajectories can be learned from scratch using standard RI. or, as we do in practice, produced by a scripted policy. We remark that, in contrast with existing approaches (Pertsch et al. 2020; (Singh et al.] (202)) this framework poses very weak requirements on the similarity between the environments used for data collection and the target environments. As we show in Section 5.3 this allows our method to bridge the gap between fundamentally different environments and settings, such as transferring from simple reaching task with access to the true state of the system to door-closing task in a visual RL setting.
4.2 SOFT ACTOR CRITIC WITH TEMPORL
The main challenge introduced by temporal pri- ors stems from their non-Markovianity, which renders existing integrations of priors in RL unsuitable. Existing methods for accelerating RL through behavioral priors can only handle state-conditional distributions, which are mod- elled as Gaussians in most cases. For this rea- son, we introduce a novel method for the inte- gration of an action prior in an off-policy RL framework, Our method is suitable for both be- havioral and temporal priors, independently of their conditioning variables. The key strategy
Algorithm SAC with TempoRL.
1: Train temporal prior (aagss, H₂)
2: Initialize history Ho = 0
3: Initialize policy and Q-parameters 0,
4: for each iteration do
5: for each environment step do
6:
A A(H(TT(((::))
7:
ae ~ (1 (1-x) (c) + A,a((alst, Hi)
8:
8141 ((et, at)
9:
D = D U (st, at, r(s,, 01),
10:
HeI1 = H, U (8t,
revolves around sampling actions from a mix-
11:
end for
ture between the policy and prior distribution, 12:
for each gradient step do
dynamically weighted through uncertainty es-
13: Update 8,
timation. We demonstrate it into the Soft Actor Critic framework.
integration
14: end for
15:
end for
SAC's objective is designed to pursue large rewards while maximizing the entropy of its policy. When prior knowledge on the structure of the environment or task is available, simply sampling actions from maximal entropy policy may not be optimal. On the other hand, blindly sampling from a behavioral or temporal prior ग prevents exploitation of reward signals as well as any behavior which is not encoded in the prior. Ideally, it is desirable control the degree which actions sampled from the prior. We propose to achieve this in a natural way by sampling actions from a mixture between the policy π and the prior म:
a₂ ~ (1 - (-)x( + (- H+) with < X 1
(I)
where the mixing parameter X is computed dynamically at each step.
In principle, the current policy π should be centered upon actions that maximize returns, while R's sole purpose is to suggest suitable actions according to behavioral or temporal knowledge. When observing state 8t, the agent should then sample directly from its policy æ in case path from such
5

---

Under review as a conference paper at ICLR 2022
state the goal is known, and sample from ā in case the state is unknown and further exploration needed. For this reason, the mixing weight ) should ideally estimate the probability failing to reach the goal while only sampling from the policy
We therefore propose to compute the mixing weight directly as a function of the policy's entropy (+(()))))), which intuitively quantifies the agent's s confidence in its plans. Since (++((+)x)) cannot always be computed in closed form, we can estimate it via Monte Carlo sampling. While the number of samples can control the variance of the estimator, we simply use the current policy sample:
H(((-1ss))
E - log (a)lss)) x log (acls)) with a, x((1s)).
(2)
di Puat
The mixing weight can then be computed at each step as:
d = (HH(T(11s.) A log π (a:l=:)) with ag t (-(18)),
where A(-) is a monotonically increasing mixing function bounded to the range [0, 1].
We further incorporate this novel action sampling scheme by reformulating the objective directly encourage sampling from the prior म:
Σ
7' ( ((ss, + as (H(((T(())))))
arg
THOIT
Through straightforward derivations (see Appendix A). one retrieve modified objective for training the policy π and a Q-function estimator Q" in the Soft Actor Critic framework. Given a distribution D of observed states and actions, the loss functions can be defined as:
Jn = E ani
Q (5(s, + ((oooo))o(a)))) with a TT(-1s),
(5)
JQ=
E
(6)
(e.0)~
where the target for the Q-value is computed as
Ve a) = R( + ( (all,o)))aa
with T(-ls), ma(-ls)). (7)
We note that minimizing Equations 5 6nd bias with respect to the objective in Equation 4 for non-linear mixing functions due to Monte Carlo estimation of entropy. The two objectives can be empirically estimated and minimized through standard procedures, reported in Haarnoja al. 2018) and in Appendix A
Algorithm I summarizes (in blue) the modifications be applied in order integrate prior into the SAC framework. Namely, actions are sampled from a mixture (line 7) weighted according to the output of mixing function (line 6). Finally, the history of the agent needs to be initialized (line 1) and updated at each step (line 10). Update rules for 0 and á are modified and computed from Equations and
We finally note that the modified learning objective remains aligned with the original formulation. As consequence, while our method is superior on more complex tasks (cf. Section 5.3). leaving SAC's s objective unchanged can perform on an-par on some tasks, with reduced sensitivity to hyperpa- rameter tuning. We refer the reader to Appendix B for more details.
Mixing Function The output of the mixing function A should be constrained to the range [0, 1] and monotonically increasing with respect to its input. Intuitively, if the entropy of the policy increases (i.e. in the absence of strong reward signal), the mixing weight should also increase, sampling from the prior becomes more desirable. A natural choice is to simply apply sigmoid function after scaling and translating the entropy via parameters 8e, 3r, These parameters are estimated empirically and kept fixed across all experiments. The resulting function for computing the mixing weights is:
A A(H(T(-(s.)) (3. 3z).
(8)

---

Under review as a conference paper at ICLR 2022
zeta-vorid
point-Taze
Torniig Enceromen
Thining Eoveeouree
Dressenoo Rentootoent
xeach (cundon geais)
reach
Golt-clote
V10SOW ogen
Gout-spen
Posernaze Crandon geels)
rootrmaze
1-9200
Figure
Overview environments used our experimental validation.
5 EXPERIMENTS
We evaluate our method in a series of experiments to empirically validate our contributions. First, in Section 5.1 we compare the effectiveness of behavioral and temporal priors justify our choice of state-independent conditioning. In Section 5.2 we verify that sampling from our temporal prior produces correlated and state-covering behavior, without the need hand-craft an exploration pol- icy. Next, we show how our method can improve exploration efficiency in unseen long- morizoo tasks by comparing against various baselines in state-based RL (Section 5.3). Furthermore, we demon- strate that our prior enables transfer to different settings, i.e., from non-visual to visual state space, whilst retaining the aforementioned benefits exploration. An ablation of the generative model provided in Appendix F.4
Baselines
We now present the baselines. Implementation details are provided in Appendix D.3
SAC: vanilla Soft Actor Critic (Haarnoja et al. 2018)
SAC+AR(n): SAC with non-learning based prior that repeats action times to enforce more directed behavior. We choose n for our experiments.
SAC+BC: SAC with warm-started policy through behavior cloning.
SAC-PolyRL: SAC with locally self-avoiding walks (Amin et al. 2021
PARROT-state: flow-based behavioral prior enforced through transformation of the ac- tion space (Singh al. 2021
We benchmark state-based variant in non-visual settings.
Environments We evaluate our method on two types of domains, namely robotic manipulation and maze navigation. Specifically, we make use subset of robot manipulation tasks from the publicly available meta-words suite Yu et al. (020)) and adapt the point coint-maze ze implementa- tion from Pitis et al. (2020). More details can be found Appendix while visual examples the environments are provided in Figure 3
Our temporal priors, as well as prior-based baselines, are learned in relatively simple training en- vironments that make very weak assumptions about the environment and task structure. To this end, we train priors for robot manipulation in a reaching task (reach) and for maze navigation in empty environment (room-maze), where the task is completed upon reaching goal that is sampled uniformly from the whole environment space.
wider range of downstream environments and tasks. For analyzing how learned priors can improve exploration efficiency, we deploy an RL agent on both the environment used for training the priors and set of unseen, more difficult test tasks. The latter consist of manipulating objects, such as opening a window, or navigating in more complex mazes.
RL agents then trained and evaluated
5.1 CONDITIONING VARIABLES FOR BEHAVIORAL AND TEMPORAL PRIORS
The goal of this section is to empirically show how the effectiveness of an action prior depends on conditioning variables and on the domain gap between the training and the downstream environment. For this purpose, we train several variants flow-hased action priors on the robotic reaching task (zeach) and train TempoRL from scratch the same environment, well as a different one (window-open). In particular, we compare temporal priors conditioned on action sequences of different lengths (1, 2, 5, 10), temporal priors conditioned the previous state-action pair and behavioral prior (conditioned on the state alone).
The results provided in Figure 4 We observe that both types of priors capable of guiding downstream RL as long as the downstream environment matches the training environment. How- ever, find that including the state in the conditioning variables can jeopardize the ability to trans-

---

Under review as a conference paper at ICLR 2022
reach
sindox-open
a(ailae-1)
600
600
7(vilof-3)
Tos Rotuun
100-
400-
a(ailois))
7(ai(s))
R(arles,ar-) (1-1)
Figure 4: Comparison of downstream performance of behavioral and temporal priors when conditioned on actions, states and combinations thereof.
fer knowledge to a different task, such as wi window-open. We hypothesize that this is due to a mismatch between the states used for training the prior and those returned by the downstream vironment. For this reason, behavioral priors (() fail on the unseen task. Even when the prior conditioned state-action pairs (a St; 0r-1), ) learns rely its state input to model the action distribution, and therefore fails to transfer. Furthermore, we observe that state-conditional priors suffer when training in the same environment that was used for data collection, which we argue is due to the limited variance in sampled trajectories at training time (see Appendix D.4).
On the other hand, state-independen temporal priors prove to be a capable alternative across both settings and are able to transfer knowledge to unseen tasks. While conditioning on longer action sequences can improve performance, we note that wel-aaccioonnniiin sufficient for capturing complex temporal dependencies within our settings. Hence, they will be the focus of the remaining experiments.
models 7 (ag (4-1) are
SAC
SAC+AR(2)
SAC-PolyRL
TempoRL (ours)
Table 1: State coverage metrics for our method and baselines. TempoRL's trajectories locally directed and cover state space well in both en- vironments.
U2
% Coverage
SAC
0.005-10.001
0.1373:001
SAC+AR(2)
0.010++0.002
0.199++0.02
Figure 5: A qualitative comparison of sampled exploration trajectories in 2D room. Our method achieves directed hehavior while covering most of the state space. SAC and SAC+AR(2) fail cover full state space, while SAC- PolyRL fails to reach distant areas consistently.
reach
SAC-PolyRL
0.025:10.001
0.272:10.01
TempoRL (ours)
0,053--0.009
0.357:10.2
SAC
0.005+-0.001
0.333++0.2
SAC+AR(2)
0.008:10.001
0.493:: 0.08
maze
SAC-PolyRL
0.026:1:0.00
0,830
TempoRL (ours)
0.054:0.008
0.963-10.04
5.2 CORRELATION AND STATE COVERAGE
In this experiment, we show how a one-step Mate-independen temporal prior (a (a--) produces correlated and directed behavior, which leads to more complete coverage the state space during exploration. To this end, we sample 20 random trajectories of 500 steps each with our method and several baselines in the room-maze and reach environment.
As shown in Figure 5 our temporal prior produces directed behavior which covers most of the state space. As expected, uniform sampling (SAC) and action repeat (SAC+AR(2)) fail to reach the boundaries of the environment. SAC-PolyRL. capable of producing correlated and directed behaviors, but only after careful hyperparameter tuning. Equivalent behavior can be observed in the robot manipulation environment (ef. Appendix F.7p. This qualitative assessment is verified quantitatively in the evaluation presented in Table [ We report state space coverage and radius of gyration squared (Amin al.] (021) (see Appendix D.] for details). Evidently, our method outperforms the presented baselines on both metrics.
5.3 TRANSFER LEARNING
Our main results are obtained by comparing our method against several baselines downstream learning tasks with vectorized state space, presented in Figure 6 As expected, observe that the performance of behavioral priors depends on the similarity the downstream task with the expert dataset. This the case for PARROT-state, which solves reach and room-no easily, as
8

---

Under review as a conference paper at ICLR 2022
reach
door-close
windoe-close
500-
313
decr-open
vindou- open
SAC
U-IAZZ
SAC+AR(2)
SAC4BC
SAC-PolyRL
PARROT--dale
30-
TempoRL (ours)
Accelerating downstream RI. While other methods are mostly competitive on training tasks (reach and room room-maze), TempoRL is able to accelerate RL for unseen tasks (door-close) door-open, windox-clcse, window-open, u-maze).
Figure 6:
its behavioral prior already represents a strong policy. PARROT-state also offers good performance in door-clon ose, as the act of reaching for the goal, which is the final position of the door, is sufficient to swing closed. On tasks which are significantly different from the training task, namely window-open, window-close and u-maze, PARROT-state is unable to guide the policy, as it receives out-of-distribution states (cfr. Appendix (C. On the other hand, TempoRL is capable of transferring unseen tasks, while rapidly catching up with PARROT-state in the training tasks.
Other baselines are in general less effective across the benchmarks: Vanilla SAC only makes progress in the reaching task, due to the presence of easily reachable goals that be achieved even with weak exploration. Enabling action repeat (SAC+AR(2)) can effectively speed up explo- ration, but only on tasks that are slightly out of reach when the heuristic is not enabled. SAC-PolyRL. is able to produce good explorative trajectories through its hand-crafted policy, but its performance strongly dependent on the task. As previously reported by |iighh al. (2021). initializing SAC through behavioral cloning (SAC+BC) can help guide exploration, but it fails to generalize across tasks and is regularly outperformed by stronger methods.
Visual RL We finally demonstrate the benefits of temporal priors in more complex settings. In particular, state-independent tempo- ral priors allow the state space of the downstream environment to be defined arbitrarily. Hence, they also allow transfer to the visual RL setting, which avoids reliance on low-dimensional vectorized state space and is purely based on RGB observations. To this end, we compare TempoRL with Vanilla SAC in 7 with PAR- ROT in its original, visual setup, i.c., by conditioning its behavioral prior on images. We report that our findings hold visual settings: temporal priors are capable of generalizing to unseen tasks, even when transferred to fundamentally different state spaces. More re- sults are presented in Appendix F8
door-close
8010
The: Iaeum
400-
200-
Steps
Figure 7: Transfer of a one-step state-independent temporal prior to Visual RL. TempoRL (red) compared with SAC (brown) and PARROT (orange).
6 CONCLUSION
In this paper, we propose a method for improved exploration efficiency in off-policy reinforcement learning. In particular, we introduce Mar--aaaov,,,, flow-based temporal prior and show how it can be integrated into an off-policy reinforcement learning framework. Crucially, we argue that slate-conditioned priors struggle with transferring knowledge across domain gaps and provide em- pirical evidence on how a intepeeedeenn temporal prior can accelerate learning in unseen long- horizon control tasks with sparse rewards. As our method shows promising results, there are exciting directions for future work. State-independent temporal priors demonstrated their usefulness in un- seen tasks, to which behavioral priors often cannot extrapolate. On the other hand, condicoondi prior can directly transfer knowledge in the absence of domain gap. Since both families of meth- ods remain strong in complementary settings, we hope to explore the direction of a flexible prior, capable of both general and task-specific exploration.
9

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
Our main contribution revolves on accelerating and enabling reinforcement learning in environments with a strong exploration component. As a consequence, we believe that concerns with respect to our method are for the most part aligned with general RL research. For instance, improved sample efficiency could on one hand accelerate the process of automation, which might have a negative impact on societal equality, and on the other hand democratize access to powerful RL methods by lowering the amount of required resources. Due to the general nature of our method, we believe that we do not introduce fundamentally new risks.
8 REPRODUCIBILITY STATEMENT
We aim to achieve full reproducibility in our experiments. In practice, we extensively describe implementation details in Appendix D and, most importantly, we make our codebase public for research purposes (see URL in Footnote 2).
REFERENCES
Joshua Achiam. Spinning up in deep reinforcement learning. 2018. URL https:
com/
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: offline prim- itive discovery for accelerating offline reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Susan Amin, Maziar Gomrokchi, Hossein Aboutalebi, Harsh Satija, and Doina Precup. Locally persistent exploration in continuous control tasks with sparse rewards. In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 275-285, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight Experience Re- play. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe. Analyzing inverse problems with invertible neural networks. In 7th International Conference on Learning Representations, ICIR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Kamyar Azizzadeneshelli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q g-networks. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9. IEEE, 2018.
Pierre-Lur Bacon, Jean Harb, and Doina Precup. The Option-Critic architecture, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Sammy Christen, Lukas Jendele, Emre Aksan, and Otmar Hilliges. Learning functionally decom- posed hicrarchies for continuous control tasks with path planning. IEEE Robotics and Automation Letters, 2021.
10

---

Under review as a conference paper at ICLR 2022
Will Dabney, Georg Ostrovski, and André Barreto. Temporally-extender e-greedy exploration. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Peter Dayan and Geoffrey Hinton. Feudal reinforcement learning. Advances in Neural Information Processing Systems, 09 2000.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. Journal of artificial intelligence research. 13:227-303, 2000.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, 7, Toulon, France, April 24-26, 2017, Conference Truck Proceedings, 2017.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580-58, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: /802.0607, 2018.
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M. Czamecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry kl-regularized rl. In International Conference on Learning Representations, 2019.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint 1812v:1812.055
2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications, 2019.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI Conference on Artificial Intelligence, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
George Konidaris and Andrew Barto. Building portable options: Skill transfer in reinforcement learning. In Proceedings of the 20th International Joint Conference Artifical Intelligence, IJCAI'07, pp. 895-900, 2007.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning Multi-Level Hierarchies with Hindsight. In International Conference on Learning Representations, 2019.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature. 5 318(7540):5295533 2015.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient Hierarchical Reinforcement Learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018.
11

---

Under review as a conference paper at ICLR 2022
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In Advances in Neural Information Processing Systems, volume 32, 2019.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026-4034, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. arXiv preprint arXiv: urXiv:/806.033: 1806.03336 2018.
Ronald Parr and Stuart Russell. Reinforcement learning with hierarchies of machines. In Proceed- ings of the 1997 Conference on Advances in Neural Information Processing Systems 10, NIPS *97. pp. 1043-1049, 1998.
Xue Bin Peng, Michael Chang, Grace Zhang, Picter Abbeel, and Sergey Levine. Mep: Learn- ing composable hierarchical control with multiplicative compositional policies. In Advances in Neural Information Processing Systems, volume 32, 2019.
Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J. Lim. Demonstration guided reinforcement learning with learned skills, 2021.
Silviu Pitis. Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761, 2020.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy meta-reeinforcement learning via probabilistic context variables. In Proceedings of the 36th Inter- national Conference Machine Learning, Proceedings of Machine Learning Research, 2019.
Lukas Schäfer, Filippos Christianos, Josiah Hanna, and Stefano V Albrecht. Decoupling exploration and exploitation in reinforcement learning. arXiv preprint arXiv:2107.0896. 2021.
Jürgen Schmidhuber. Evolutionary principles self-referential learning, or on learning how to learn: the mela-mela-.. hook. PhD thesis, Technische Universität München, 1987.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Par- rot: Data-driven behavioral priors for reinforcement learning. International Conference on Learning Representations, 2021.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28: 3483-3491, 2015.
Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Daisuke Okanohara. Conditional density estimation via least-squares density ratio estimation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statis- tics, pp. 781-788, 2010.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A Frame- work for Temporal Abstraction in Reinforcement Learning. Artif. Intell., 1 112:181-211, 1999.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. In 31st Conference on Neural Information Processing Systems (NIPS), volume 30, pp. 1-18, 2017.
Dhruva Tirumala, Hyeonwoo Noh, Alexandre Galashov, Leonard Hasenclever, Arun Ahuja, Greg Wayne, Razvan Pascanu, Yee Whye Teh, and Nicolas Heess. Exploiting hierarchy for learning and transfer in kl-regularized rl. arXiv preprint /rrxiv/903.007
2019.
Dhruva Tirumala, Alexandre Galashov, Hyeonwoo Noh, Leonard Hasenclever. Razvan Pascanu, Jonathan Schwarz, Guillaume Desjardins, Wojeiech Marian Czarnecki, Arun Ahuja, Yee Whye Teh, and Nicolas Heess. Behavior priors for efficient reinforcement learning, 2020.
12

---

Under review as a conference paper at ICLR 2022
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 0026-5033, 2012.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning, pp. 3540-3549, 2017.
Jane X. Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning to reinforcement learn. In CogSci, 2017.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100, 2020.
13

---

Under review as a conference paper at ICLR 2022
A OBJECTIVE DERIVATION
The purpose of this section is that of describing how learning objectives for the policy and Q- estimators can be derived from the TempoRL objective:
Σ o (R(sn.a)) as (H(()(()))))))
20
E THOT
The steps of these derivations follow those reported in Haarnoja (2018), as our method is designed to fit into their framework. First, let us formally introduce the Q-function for a generic policy a:
Qa (s) = E 7 R(s + Σ (')()(((,,))))) 30 = 3,00 =
(10)
We can then formulate the Bellman Equation and explicitly unravel the entropy term:
(s)
R(s,a) + +(F aa(H((((,
s'muT s'muTrat
(11)
E s'm-T aatn
a(s,a)+ + ad E am
log (ag<(("|(\1))
The right-hani side of Equation II can be estimated via Monte Carlo sampling and used as target, as shown in Equation 7 Q-estimates can be trained by minimizing standard MSE error, resulting the Q-loss in Equation
In practice is done for SAC, two separate parameterized Q-function estimators (Qo,)i=1.2 are used to prevent overestimating Q-values; the policy is also parameterized as тæ. Additionally, we also adopt target networks (Q (i updated via Polyak averaging. As result, when sampling a batch B from a replay buffer, the empirical estimates for the Q-losses are:
Jo.: =
Σ
(12)
B
(s,0,r,8 deen
where the target for the Q-value is computed as
(, (().))))rr((((d))))) min Q oonge, (8', ( (o'ls')
with a' =o(-ss"). (13)
and d stand for the reward and done signal, respectively.
On the other hand, the policy ma can be trained to minimize the KL.-divergence with the soft-max of the Q-function (Haarnoja al., 2019), alternatively to maximize the value function (Haarnojal ctt al. 2018). Our method relies on the second option and trains æ to maximize its value function V*(s), which can be formulated follows:
(*(()= E (0((s,))
(14)
Optimizing over a distribution of states s ~ D results the biased objective Equation 6 Com- puting the expectation over actions then be circumvented by using the reparameterization trick,
14

---

Under review as a conference paper at ICLR 2022
reach
door-closa
windoe-close
303
decr-opor open
vindou-cpor open
U-ZAZZ
TunpoRL-ssupplllee
TempoRL
304
Figure 8: Comparison between the full method and its simplified version. The simplified version does not modify SAC's update rules.
which enables expressing actions as us(E, s), where & ~ N is sampled from a standard Gaussian. In practice, since the entropy term can once again be estimated by Monte Carlo sampling, this finally translates into the empirical loss estimate:
1
Σ
min (s, aA( log log
(15)
B
(B)EB
which can be minimized via standard first-order optimization techniques.
B SIMPLIFIED ALGORITHM
note that a policy क maximizing Objective 4 is not necessarily a maximizer for the original objective in SAC Haarnoja et al. (2018), which is:
Σ t=
R((s),))) (H(((((ss)))
(16)
However, as note in Section 4 due to monotonicity of the mixing function A(-), the regulariza- tion term the TempoRL. objective and the regularization term in SAC are both maximized by a max-entropy policy л² (AA). This observation suggests that modifying the learning objective might not be essential to our method. We thus set out to validate this hypothesis empirically, by removing all modifications the objective or the learning rules of SAC. The resulting algorithm, which we refer to as TempoRL-simplifed was validated on the same settings of Section 5.3 in Figure 8 In general, we find that the simplified version mostly on par with the full method in terms of downstream performance on the easier tasks (e.g. door-cll (oor-close), but it is not able to learn the more more robust
complex tasks (e.g. door-opent Interestingly, find the simplified version with respect to changes in hyperparameters, and we would suggest it as practical alternative to the full method on easier tasks.
C ENVIRONMENT DETAILS
C.1 ROBOT MANIPULATION
We use the meta- -wrlld suite Yu et al. 2020) for robot manipulation experiments. It consists of a simulated DoF Sawyer arm, implemented in the MuloCo physics engine (Todorov et al.) 2012). By default, states represented 36-dimensional vectors across all tasks. The state contains the 3D location and aperture of the gripper, the 3D location and quaternion of one object (e.g. door window), measured for the current and previous time step. Goals represented as the desired
15

---

Under review as a conference paper at ICLR 2022
3D location of the end effector or object, according the task. Actions are 4-dimensional vectors containing 3D movement and a 1D control over the aperture of the effector.
We additionally render 64x64 RGB images observations for the visual setup in Section 5.3 using the camera angle corner (as can be seen in Figure 3). Instead of the originally proposed dense reward, we adopt sparse reward which is only non-zero upon task completion. For more details, we refer to the original implementation Yu et al. 2020
C.2 MAZE NAVIGATION
Our point maze environments were adapted from Pitis et al. (2020). The state consists of the 2D position of the agent, which can be actuated via velocity-controller through 2D actions. The goal space matches the state spaces in dimensions and representation. The reward signal is when the Euclidean distance to the goal is lesser than 1.2 units, else it is O. We experiment with two different layouts (renderings can be found in Figure 3
room-maze ze a large, 29 29 square room. The starting position at the center and the goal is initialized randomly in of the 4 corners each reset. Trajectories for training the priors are obtained from this environment. Only for Figure [ the size was increased to 81 x81, to allow and better visualize long trajectories.
u-maze is a larger u-shaped corridor with three parts of lengths 50, 3 and 50 respectively, assembled at 908 clockwise rotations. The structure of the maze can therefore be contained in 50 x 3 rectangle. Starting and goal position are fixed and located at opposite ends of the maze.
For further details, we refer directly to our published codebase (URL in Footnote 2
D IMPLEMENTATION DETAILS
D.1
METRICS
All metrics reported in this paper (e.g. cumulative returns per episode) averaged over 7 random seeds (or 9 for Figure 6): mean and standard deviation are reported and plots are smoothed over steps.
We rely on two metrics for measuring the quality of exploration trajectories in Section 5.2
% Coverage divides the reachable state space in n cubic buckets (n = 1000 for reach and 100 for room-maze) and reports the ratio between the number of buckets visited by a set of 20 trajectories of length 500 and the total number of buckets.
Radius of Gyration Squared measures the spread in visited states, averaged over all tra- jectories, and is adapted from Amin al. (2021). Given set T of n trajectories, the metric can be computed as:
(T) on
Σ
Σ FET
Σ (s,
where a trajectory 7 modeled as sequence of states (+.)/ ((, measures Fuclidean distance and [ E.er We additionally normalize the metric by which measures the diagonal of the box containing reachable states.
D.2 DATA COLLECTION
Although the training tasks are relatively simple and a data collection policy can in principle be trained from scratch, for simplicity we fill the expert dataset D with the trajectories of scripted policy. Since training enviromments do not include obstacles and allow direct control over the agent position(in 2D for room-maze and in 3D for reach), scripted policies simply receive the 2D/3D positions of the agent and of its goal, and output a distance vector. This vector then corrupted with isotropic Gaussian noise and scaled fit within action limits. A simple implementation of scripted
16

---

Under review as a conference paper at ICLR 2022
policies is available part of our codebase (see URL in Footnote 2 For each training environment (reach or room-maze), we collect 4000 trajectories of 500 steps each. Goals for the scripted policy sampled uniformly from the reachable state space. On goal achievement, new goal is sampled, once again uniformly. The same expert datasets are then used for training temporal priors, behavioral priors behavioral cloning.
D.3 BASELINES
SAC We build upon the implementation provided by SpinningUp (Achiam 2018), which is re- ported to be roughly on-par with the best results achieved on MuJoCo Gym (Brockman et al.) 2016). For simplicity, we do not use automatic entropy tuning and keep constant during learning. Nonetheless, our method could easily tune a dynamically at the expense of increased complexity, as is described in |Harnoja et al. (2018). Moreover, we also introduce HER (Andrychowicz et al.) 2017) as well as n-step returns (Hessel et al. 2018). We experimented with importance sampling for off-policy correction, but, similarly to what is reported by Hessel et al. (2018), we observed no empirical benefit. By default, we sum rewards over n 10 steps before clipping the range to [0, 1]. All remaining hyperparameters HER and n-step returns as just described.
reported in Table
?
We anticipate that all baselines also rely
Table 2: Hyperparameters for SAC.
Hyperparameter
Value
Epochs
125
Steps Per Epoch
4e3
Steps of Initial Exploration
1e4
Steps Before Training
1e3
Environment Steps per Iteration
50
7
0.99
Polyak Averaging Rate
0.995
Inverse of Reward Scale (a)
0.2 for meta-world, 0.02 for point-maze
Batch Size
100
Optimizer
Adam
81
0.9
32
0.999
Learning Rate
0.001
Hidden Units
256
Hidden Layers
2
Hindsight Replay Ratio
4
Replay Size
5e5 for vector-based RL, 2c5 for image-based RL
This baseline shares all hyperparameters with SAC. The only difference lies in the fact that the policy is only sampled from every nith step, while the previous action is repeated for the remaining steps. We use 2 in our experiments.
SAC+AR(n)
SAC+BC Behavioral cloning (BC) is performed on the entire dataset D for 10 epochs, by maxi- mizing the log-likelihood of the Gaussian policy with respect to expert actions. The optimizer and batch size used for BC are the same for downstream learning.
SAC-PoLyRL SAC-PolyRL AAmin et al. 2021) replaces the initial uniform exploration phase of SAC with trajectories collected by a hand-crafted policy. While SAC's hyperparameters are un- varied, SAC-PolyRL specific parameters are tuned from those reported in various settings in the original paper. We use 0 = 0.35, o² = 0.017 and 3 = 0.01.
PARROT An official implementation for PARROT (Singh et al.) 2021 is not available at the time of writing. We reproduce their training routine and downstream application and adopt all hyperparameters reported in the original paper. Generative models are trained until convergence (100 epochs) using a batch size of 400 samples and Adam (Kingma Ba 2017) as an optimizer, with learning rate of 0.0001, 31 0.9, B2 0.999 and weight decay penalty of le 6.
17

---

Under review as a conference paper at ICLR 2022
In addition to the image-based version of PARROT used in Section 5.3 a vectorized state space, dubbed PARROT-state since the method is originally only applicable in visual settings. The only modification consists in replacing the image encoder with a 3-layer MLP with 256 neurons per layer and ReL.U activations. In this setting, the input to the encoder is therefore bastrr-hased (non-visual).
we introduce a version with
D.4 TEMPORL
Prior We model all families of temporal priors with conditional Real NVP Flows (Ardizzone et al.) 2019), sharing the same architecture across all experiments. The invertible transformation fe is composition of coupling layers, each followed by a batch-normalization layer (Dinh et al. 2017 For each layer, a 3-layer MLP with 128 hidden units per layer is used to preprocess the conditioning input. Scale and transition networks are also implemented as a shared 3-layer MLP with 128 hidden units, whose output is split in two to provide scale and shift coefficients. All MLPs use ReLU activations.
We train our temporal priors following the same protocol used with behavioral priors (100 epochs, batch size of 400 samples and Adam (Kingma & Ba 2017). with a learning rate of 0.0001, 31 = 0.9, 82 0.999 and weight decay penalty of le- 6).
Downstream Learning Our method introduces pair of hyperparameters governing the mixing function A(-), which we tune on w window open and apply to all other tasks and environment: 3. = 0.7, 3, O.75] where A is the action space. Due to the modifications to the objective, we also tune a to 0.2 for point- maze and 1.0 for meta-world. While SAC samples action uniformly for the first 10000 steps encourage exploration. TempoRL has access to temporal prior f and therefore, during this initial phase, we sample directly from it in met a wowlldd. To overcome stability issues, TempoRL still samples initial actions uniformly and adopt lower learning rate of 0.0001 in point taant-maaee Among these changes, those that could also be applied to the baselines were tested and found to be beneficial to them. All remaining hyperparameters are shared with SAC (see Table 2).
Test-time Policy At test time, instead of sampling an action from the mixture between prior म and policy π, the agent executes an action corresponding to the mean of the policy distribution. We note that this is consistent to what is done in SAC Haarnoja et al. (2018).
[A] while all remaining hyperparameters are shared with SAC. Similarly to the full method, it also samples initial actions from the prior.
Simplified Method The simplified version uses 3e 0.65 and B, 40
State-conditional Prior
When using state-dependent priors म (ar)s)) and ग (ar)sst 4;-1) to accel- erate downstream learning in Section 5.1 trajectories at training time immediately near-optimal and often distributed over an excessively small support, which hurts test-time performance. To miti- gate this issue, when using state-dependent priors in framework, multiply the mixing weight A by scheduling coefficient e, which we linearly increase from 0 to 1 over the duration of training. Intuitively, this leads more diverse initial trajectories.
E LIMITATIONS
Section 3 empirically argues in favor of our method's ability to produce correlated trajectories and accelerate downstream learning in unseen tasks. While state-independent temporal priors are able to generalize across larger domain gaps with respect to baselines, they can only be as general as the data used for training. In other words, temporal priors are data-driven, and as such will only reconstruct behavior that appears in the offline dataset of trajectories they are trained on. For in- stance, when training on zeach with random goals, temporal priors will not encode any grasping strategy. While such behavior still be recovered by the policy æ, there is no incentive do As a result, successfully transferring to tasks that require additional strategies such as grasping (e.g. pi pick-s cccpplee ace) remains an open challenge, as we show in Appendix R.
18

---

Under review as a conference paper at ICLR 2022
outtoo p praxx
drawerrel
pissh
MT10 (mean)
SAC
dravez-open
pick-place
2004
SAC1
SAC+-I
SAC-PolyRL
PARROT-state
Storpe
Thmpolt, (omrs)
Singe
20ge
Figure 9: Left: performance on additional downstream tasks from MT10 (Yu et al., 2020). Right: mean performance over 9 out of 10 tasks in MT10 Shaded area represents variation
does not allow hindsight relabeling). seeds. We note that training and evaluation protocol presented in Section 5.3 and does not rely on single, multi-task policy.
More generally, temporal priors can still suffer from domain gaps, especially when they state- dependent and there is a strong correlation between the current state and the expert action during training. In this case, the prior model's output might rely on its state input, and therefore suffer from nut-of-distribution samples during downstream learning. State-independent temporal priors alleviate this issue. On the other hand, they might provide weaker guidance very similar to the training task since they are agnostic to the state. As shown in Figure behavioral priors are in practice superior when the downstream environment can be solved by the policy used for collecting expert trajectories. For this reason, strategy to combine benefits from behavioral and temporal priors would represent an interesting direction for the future.
tasks that
One final concern regards entropy quantification. As we discuss in Section mixing weights A should represent the confidence of the agent in its plans. For simplicity, we estimate this through the policy's 's entropy in the SAC framework. While we empirically found this choice to be effective, in certain settings (e.g. when multiple actions lead large rewards), the policy's entropy might remain high even when the agent is capable of solving its environment. This might lead to overly sampling from the temporal prior, which is undesirable in this situation. However, in practice, this is a rare occurrence when dealing with morizooi sparse reward tasks, in which strong policies also need to be precise. We did not find this issue to be significant the context of experimental validation.
F ADDITIONAL RESULTS
F.1 ADDITIONAL TASKS FROM META-WORLE
Our selection of meta wontt--wrrll d tasks is meant to keep the evaluation in Section 5 connciee. We select the (arguably) casiest task for training (reach) and all tasks on two representative objects (door and window) as additional downstream tasks. We now report and discuss performance for the remaining tasks in MT10, adopting the settings used in Section
Additional results remain consistent with our previous experiments. In particular, TempoRL per- forms well on two tasks that are out of reach for all baselines. All methods fail in environments that require grasping or precise manipulation. We note that this is expected result of training state-independent temporal prior on a task that does not require actuating the gripper. Please note that pick-place could not be included in this comparison, it not possible to extract its reward function for hindsight relabeling.
F.2 ADDITIONAL TASKS FROM POINT-MAZE
We found our results in Section 5.3 to remain valid for different structures. In Figure 10 visualize an additional task (a comples mereleex-mae maaze of size 28 x 28) and we report results for TempoRL. and all baselines.
19

---

Under review as a conference paper at ICLR 2022
complex-maze
000
SAC
Tre Ibeturr
SAC+AR(2)
-00-
SAC+BC
PARReTivaate
Teeppolil (murs)
Figure 10: Performance additional structure (complex-maze), Settings identical to those
presented Section 5.3
F.3 ADDITIONAL TASKS FROM GYM MUJOCO
In this section, we apply our method a substantially different setting, represented by swimmer-v2 environment from the gym-Mujoco suite Brockman al. (2016). While tensive evaluation on tasks from this suite outside of the scope of this paper, we believe that this represents example of the potential general applicability of our method.
The agent in swi mmer-v2 is a snake-like robot that can be controlled by actuating its two joints. This is in contrast with environments from meta-world and point-maze, in which the agent's position can be directly controlled. Moreover, instead of extracting expert trajectories through scripted policy, trajectories are then used to train behavioral and temporal priors, as described in Section 5.3 We then train our method, as well all baselines on two downstream task. In swimmer ceemmee--sarssee dense rewards are replaced with a binary reward signal that is only positive when the agent is farther than sparsity factor X₂ 1.5 from initial position. sw swimmer-spel memee-apare-eeeee ropparseenissy also adopts this sparse reward scheme and, additionally, corrupts observation by constant additive noise on the state 8t = st te with 0.5. This can interpreted as bad calibration of sensors or faulty readings from real-world robot. Results for this experiments are reported in Figure I1
policy trained from scratch with DDPG (Lillicrap et al. 2019). These
sparse-swimmes
swinmer--sarre-noti
600
SAC+)R(()
SAC+BC
SAC-PolyRL
PARROT-stale
(ours)
Figure 11: Performance on tasks derived from swinmer-v2. Temporal priors are capable guiding explo- ration even when states are corrupted by noise.
We observe that, as expected, swimmer-sparre PARROT very effective, since training and downstream environments are identical. TempoRL. is also able to learn successful policy, albeit at slower rate, while the remaining baselines fail to observe any meaningful reward. However, when noise is introduced in the observation space, TempoRL still retains good performance, while PARROT fails to transfer to this unseen task.
F.4 MODEL ABLATION
We finally set out to provide empirical backing for an important design choice. Our policy mixing approach grants freedom in choosing generative models capable of describing complex distributions, as computing distance metrics to the prior is not required. We compare the performance of Real NVP Flows with Conditional VAE (Sohn et al. 2015), non-parametric conditional Least Squares Density Estimator (Sugiyama et al. 2010), and a MLP modeling a deterministic prior. As shown in Figure T2
observe that Flow modeling is consistently more powerful.
20

---

Under review as a conference paper at ICLR 2022
reach
xindow-open
600
600
NYP Flow
100-
400-
eVAE
ISCDE
Deterministic
Figure 12: Performance downstream tasks when modeling one-step itte--ineepeneett temporal prior through various generative models.
F.5 LEARNING ALGORITHM ABLATION
As described in Appendix D.3 HER (Andrychowicz et al. 2017) and n-step returns are the two main algorithmic modifications that we introduce on top of the SAC framework (Haarnoja et al. 2018). As our method significantly benefits from these two components, we also apply them to all baselines ensure fair comparison across our experiments.
In this section, we provide a concise discussion on the importance of these two modifications in the form of an ablation study. We compare the performance of TempoRL on reach and window-open when i) disabling HER and ii) changing the number of steps for reward computa- tion from 10 5, and In Figure 13 we can observe that n-step rewards are a crucial component to performance in long-horizon tasks, as they can effectively accelerate the propagation of value to states that are distant from the goal. On the other hand, we find HER's s contribution to be marginal, and counterproductive in certain settings (e.g. in reach).
F.6 STUDY OF SAC+AR(N)
SAC+AR(n) be strong baseline, when compared most elaborate methods. However, the value of n is crucial to final performance and it requires extensive tuning or domain knowledge. In this section, we present brief study of its performance on the meta-worid tasks. We observe that, within the experimental settings outlined in Section [.3 larger values of n can help in some tasks (e.g. door-close). However, they are not sufficient solve the harder tasks and can be detrimental when precise control is required, as in reach.
reach
window-open
600
000
10sttep
5-dep returns w/ HER
400
1-tep returns w/ HER
S-tep returns w/o HRR
1-dtep returns w/o HRR
Steps
Stapoo
Figure 13: Performance downstream tasks when disabling enabling HER, with different horizons for n-step returns.
21

---

Under review as a conference paper at ICLR 2022
Teach
doer-close
stndoe-cl cloae
34
deor-epen
vindow-cpan cpen
SAC
SAC+ ARCOA
Figure 14: Performance on SAC+AR(n) for different values of n.
F.7 CORRELATION AND STATE COVERAGE
SAC
SAC+AR(2)
SAC-PolyRL
TempoRL (ours)
Figure 15: A qualitative comparison of sampled exploration trajectories in robot manipulation environment (reach). Our method achieves directed behavior while covering most of the state space, outperforming SAC- PolyRL. On the other hand, uniform sampling (SAC) and action-repeat (SAC+AR(2)) fail the full state space.
F.8 VISUAL RL
We report the performance on an additional unseen task www.nddwwoppee in the same settings in Section 5.3 Similarly, our method outperforms both PARROT and SAC (see Figure 16). which are able leam the task.
window-open
600
200-
0-
Starp
Figure 16: Comparison of methods window-open with image-based states: (brown) PARROT (orange).
TempoRL (red), SAC
G ADDITIONAL RELATED WORKS
Meta-Ra RLL and Task Inference The idea of quickly adapting to unseen tasks is a fundamental con-
cept in meta-reenfor learning (Schmidhuber 1987 Wang et al. 2017). While TempoRI. also
22

---

Under review as a conference paper at ICLR 2022
aims at tackling unseen tasks, it crucially relies training policy from scratch for downstream learning and is not designed for zero-shot adaptation. A line research in meta-RL, referred to context-based (Wang et al. 2017: Mishra al. 2018 Rakelly et al. 2019), performs task infer- ence order to identity the current task and quickly extrapolate how to maximize returns. A task representation in practice be extracted from recent experiences.
Interestingly, temporal priors can in principle handle the same type of input, that is sequences of recent action-state pairs. However, instead of producing explicit representation of the current task, temporal priors directly model an action distribution. Nonetheless, while in the context of this paper we only train temporal priors on single tasks, they could in principle be trained on multiple tasks, and learn to perform implicit task inference on the currett trajectory in order to produce promising actions.
23


Reviews:
[{"review_id": "fw0vEgZG682", "reviewer": "Reviewer_j8Xm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and clear. The main contributions are novel, interesting, and well motivated. The experimental results seem informative and in support of the claims made throughout the paper, but the low choice of seeds leaves room for doubt in some cases.\n\nThe decision to only use 3 seeds greatly weakens the significance of these results, and is my only major concern with this work. For instance, the low seed count is likely why in the \"reach\" plot in Figure 4, the 2 and 5 action results appear lower than the 1 and 10 action. Similarly, it is very difficult to conclude anything from the TempoRL-simplified vs TempoRL comparison as a result of the low seed count. With so few seeds, a single bad run can drastically alter the mean. This issue of significance doesn't seem as bad in the case of Figure 6 which use 5 seeds and also shows some consistent patterns across the different domains so I don't think this necessarily invalidates all the empirical results.\n\nSimilarly, with so few seeds, e.g., 3 or even 5, the empirical standard deviation can be very noisy which makes it a very poor representation of the expected variance in the results. With so few seeds, plotting the min, max, and median or mean would provide a much more representative visualization of the learning behavior of each method. Alternatively, with few seeds, it isn't unreasonable to simply plot each learning curve with the addition of curve representing the mean/median (see [1, Figure 1] for an example of what I mean).\n\nOverall, I think the main idea is good, and the work is interesting. If it weren't for the aforementioned issues, this would be a strong paper. That being said, given not all results are equally affected, I don't think it fair to reject. I feel like the paper would still be borderline if you were to remove the problematic results which is my recommendation for now.\n\n# Questions:\n\n- p. 6, Why do we need a new objective, i.e., eq. (4)? What is the motivation for wanting policies that are less certain to reach the goal?\n\n- p. 6, \"We finally note that the modified learning objective remains aligned with the original formulation.\", it's not quite clear to me what aligned means here. Is this a formal statement or an observation based on empirical evidence? If it's the latter, then what is the evidence of their alignment?\n\n- Appendix A, eq. (9) and (10) seem to differ by a missing $\\Lambda (\\mathcal{H}(\\pi(\\cdot|s_t)))$ term. What is the motivation for dropping this term?\n\n- Appendix D.1, cumulative returns per episode?\n\n- Appendix D.2, What were the scripted policies that generated data for the priors in \"reach\" and \"room-maze\"?\n\n\n# Minor comments and nitpicks:\n\n- p. 3, \"the issue of temporal correlation is merely relocated in the hierarchy, as the high-level planner is not encouraged to produce correlated sequences of skills\", I'm not sure this is a fair assessment. Although it is true that the high-level planner is not encouraged to produce correlated sequences of skills, the skills themselves will generate correlated sequences of actions. The issue of temporal correlation isn't relocated so much as implemented at the skills/options level and  abstracted away at the high-level. Doing so doesn't change the fact that when the high-level planner executes just a single skill, it will be producing a correlated sequences of primitive actions, by construction, regardless of any encouragement to do so.\n\n- p. 3, definition of goal-conditioned MDP, the cited work doesn't seem to use the same definition, most notably it doesn't include a discount factor in its definition and the distribution over goals is part of the definition. They don't need to match but I would make it clear in the wording.\n\n\n[1] Fujimoto S, Meger D, Precup D. Off-policy deep reinforcement learning without exploration. InInternational Conference on Machine Learning 2019 May 24 (pp. 2052-2062). PMLR.\n\nPost-Rebuttal\n=========\n\nThe increase in number of seeds and the corrections of the error in the one of the figures gives me significantly more faith in the empirical results. I've increased my scores to reflect this.", "summary_of_the_paper": "The authors propose a novel method for encouraging exploration based on the idea of behavior priors. The proposed prior, called temporal prior, considers the history of the agent when determining the prior probability of an action. The authors then derive an variant of soft-actor critic (SAC) adapted to make use of these temporal priors. This is done by defining a behavior policy that is a mixture of the priors and the agent's learned policy based on the certainty of the agent's policy, as defined by its entropy in that state. Additionally, the authors propose using a custom loss meant to encourage uncertainty in the learned policy and therefore encourage actions sampled from the temporal priors. This approach is shown to encourage state coverage, to be capable of accelerating learning in unseen and different tasks.", "summary_of_the_review": "The paper is well written and clear. The main contributions are novel, interesting, and well motivated. The experimental results seem informative and in support of the claims made throughout the paper, but the low choice of seeds leaves room for doubt in some cases.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U_sv1vcjz9q", "reviewer": "Reviewer_xn1S", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is novel, simple and shows good empirical results. They use a good choice for the baselines and show that their methods outperforms them on environment from the meta-world suite.\n\nOn the weaknesses side, I've found the description of SAC + TempRL quite unclear and not exactly correct:\n\nFirst, given the objective that takes $\\Lambda$ in the formulation (eq 4), the soft-policy target (soft-max of the Q-functions in SAC) may differ, but is not described. \n\nThen, the objective $J_\\pi$ looks like the target of the value function $V_\\psi$ in SAC, if the expectation was on the action instead of the states. In sac, the objective for the policy $J_\\pi$ should be the KL divergence between the policy and the soft-max of the Q-function. Only the formulation of $J_\\theta(Q) looks correct. \n\nThe parameters $\\psi$ and $\\theta$ should appear in the objectives function to understand what 'update $\\psi$ and $\\theta$ means in Algo.1.\n\nAnd since, at the end, these objectives are not used but they simply used the original SAC, why not just say, after equation 3:\n\"To directly encourage sampling from the exploration prior, we use an entropy-regularized RL algorithm, for instance SAC\" ?\n\nRegarding the experiment, I was expecting more type of environment, to show that this method does not only work on the meta-world suite. For example, using Mujoco or even tasks with discrete actions.\n\n", "summary_of_the_paper": "This paper suggests a task-agnostic exploration priors that can be trained on policies that solve random tasks in a given environment (task-agnostic data). Given a new task, the induced exploration quickly finds how to solve it, compared to other methods that disposes of the same pre-training data. \n\nTemporal priors learns by imitation (of task-agnostic data) a policy conditioned by the whole history, rather than only the current states.\nHere, they observe that only using a condition on the previous action is already sufficient to outperforms other methods. \n\nTo use the prior, they use an $\\epsilon$-greedy approach where $\\epsilon$ ($\\lambda_t$ in the paper) depends on the entropy of the exploitation policy: the more the exploitation policy is random, the more the agents refers to the exploration policy (the temporal prior). \n\nIn order to favorise the utilisation of the exploration, they use SAC which enforces the entropy of the exploitation policy. \n", "summary_of_the_review": "+) novel and simple approach and nice results on meta-world.\n\n-) Unclear and useless SAC + TempRL description in section 4.2 and appendix A. Needs more environments for experiments.\n\nI am on the fence on this paper. Since the results are nice and the method is corrects (they finally used the original SAC), I would be more favorable for a weak accept, but I strongly suggest to remove the paragraph about the entropic regularisation in section 4.2 as well as the appendix A and re-write Algo1 to explicite that the used RL algorithm to learn the exploitation policy is SAC.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TDtmDogFWor", "reviewer": "Reviewer_wMYF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper tackles a difficult but prevalent problem in RL: solving control environments with long horizons and sparse rewards.\n2. The proposed learned priors showed good transfer properties on selected environments, some even with a wide gap.\n\nWeaknesses:\n1. The baselines being compared to are rather weak both in terms of their exploration capabilities, and that they were not designed with transfer tasks in mind. There seem to be many other baselines that could be considered in this space, such as intrinsic motivation based agents (e.g. Agent57), randomized value function / policy based exploration, or model-based exploration agents (e.g. Plan2Explore) etc.  \n2. There is danger of overfit to selected control environments when the paper's statements for contribution seem much broader. After reading the paper, I'm not sure why such temporal priors would necessarily generalize onto other long horizon sparse reward environments. Is there a piece of theory, an explanation or an illustration that could help convince us of the general usefulness of these temporal priors?", "summary_of_the_paper": "The paper proposed learned temporal action priors conditioned on past trajectories of the agent, and demonstrated using it in a soft actor critic agent, applied to long-horizon control problems.", "summary_of_the_review": "The paper presents a set of temporal priors for exploration, empirically showing good results for certain transfer tasks. I'm concerned with the generality of the approach beyond the selected environments and the lack of competitive baselines that the paper considered. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PrWf7ik1tgP", "reviewer": "Reviewer_txKn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the empirical results are impressive, but I feel that the potential weakness of the method is neither tested nor discussed.\n\n# Strength\n- S1. The empirical results of the proposed method are impressive.\n- S2. It is nice that the proposed method is agnostic of the state space of the downstream task.\n\n# Weakness\n\nThe detailed comment is given in the next section.\n\n- W1. The paper seems to claim that previous methods suffer from the domain gap while the proposed method does not. However, I can think of some situations that this method could be problematic.\n- W2. This is somehow related to W1. I feel that the paper tries to overly sell the proposed method.\n- W3. The objective function for the proposed SAC-like algorithm contains a mistake.\n\n# Detailed Comments\n\nAs for W1, it seems to me that the proposed method can suffer from the domain gap in some cases. For example, let us consider a maze \n(not a simple one like U-maze) with visual state information, i.e., the agent can observe the maze from bird's-eye view. Then, it is likely that without conditioning the prior by a state, the agent might simply hit the wall and continue to do so because of the state-agnostic prior. To claim that the proposed method does not suffer from the domain gap, would you add experiments in an environment like a general maze?\n\nIn addition, from Figure 5, I have an impression that the proposed method learns to simply repeats the same action, occasionally changing it. It seems that all downstream tasks are solvable with this simple strategy since the agent can occasionally hit the door or window knob, resulting in opening/closing it. Would you provide results of SAC+AR(n) with a higher n, like 5, 10, or so.\n\nAs for W2, for example, the paper says that \"... we propose a principled manner of integrating priors...\" What do you mean by \"principled\"? The proposed algorithm seems to use an ad-hoc way to adjust the ratio of a mixture policy. (Furthermore, the derivation of the proposed objective seems to contain a mistake.)\n\nAlso, the paper says \"We show how state-independent temporal priors can be learned from few expert trajectories on simple tasks and used to improve exploration efficiency in new tasks, despite the presence of massive domain gaps\". How do you measure the size of domain gaps? To me, a reaching task and a window opening task are not so different. In case of a reaching task, a robot needs to move its arm to the direction of a target, whereas in case of a window opening task, a robot similarly needs to move its arm to the direction of window opening while pushing the knob of the window. If a downstream task is grasping and stacking objects, I would say it is completely different, though.\n\nThere are many other claims which I feel is not well supported, and this is confusing to me.\n\nAs for W3, the maximization of the objective (Eq 4) cannot be achieved by the minimization of Eq 5. To see this, let us look at Eq 11 and Eq 13. If you take the expectation of Eq 13 (ignoring $\\min$) wrt the next state, reward, and next action, what you will get is $\\mathbb{E}[r + \\gamma (1-d) Q(s', a') + \\alpha \\Lambda (-\\log \\pi (a'|s'))]$, which is not equal to Eq 11 unless $\\Lambda$ is a linear function. If $\\Lambda$ is a linear function, the proposed objective is the same as SAC's objective. In other words, Eq 13 is NOT an unbiased estimate of Eq 11. The same argument can be made about Eq 4. I think the biasedness is OK as long as the paper clarifies this, and the algorithm still works. But it needs to be clarified.\n\n# Suggestions\n\nIn sum, I suggest the authors to improve the draft by removing some overly claimed sentences and adverbs, add some more experiments with downstream tasks which require completely different skills like grasping, and consider pointing out the biasedness of derived estimators.\n\n# Post-discussion\n\nAfter the author rebuttal, I am mostly satisfied with it and raised the score to accept. While I am still OK to accept the paper if the authors discuss the limitation of the method and correct some overselling wordings in the main text, I was convinced that my score should be based on what is in the current pdf not the paper's potential. Therefore, I lowered the score to weak accept. I strongly encourage the authors to resubmit the paper with discussion on the limitation of the method and correction of some overselling wordings in the main text.", "summary_of_the_paper": "This paper considers the exploration problem in RL and proposes a new type of the behavioral prior, one only conditioned by recent actions. The paper empirically demonstrates the effectiveness of this approach in sparse-reward environments. It also shows that this type of the behavioral prior can be learned in a simple non-visual task and transferred to a visual task. ", "summary_of_the_review": "The paper proposes an interesting idea, and current experimental results are impressive. However, I think the proposed method is not well-tested or discussed deeply. Accordingly, I am more inclined to the rejection unless my concerns are resolved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "L3zsToxxBXP", "reviewer": "Reviewer_AE8g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The general premise of the paper is that learning priors on actions conditioned on past actions allow the agent to have a reasonable directed behaviour even when the observation space of the data used to learn the priors is different from the observation space of the RL problem. \n\nFor the general RL problem, the premise of the paper is false --- one can design an offline dataset, or design a downstream task such that priors on actions conditioned on past actions from an offline dataset would hurt more than help. Similarly, one can design a problem for which priors conditioned on the state would outperform priors conditioned on just a sequence of actions.\n\nFor instance, imagine a grid-world in which the policy for the offline dataset is to go right with 97% probability, and the other directions with 1% probability, whereas the optimal policy on the downstream environment is to always go up. Action priors conditioned on past actions would only mislead the agent in such an enviroment. Similarly, an environment in which different parts of state space require very different action sequences for directed exploration would clearly benefit from priors conditioned on the state instead of past actions. \n\n\nAs a result, I find the primary claim \"However, an agent should ideally be able to produce efficient explorative behaviors even in unseen environments and unrelated tasks. Behavior priors don’t give us that, temporal priors must.\" unfounded. The priors can absolutely hurt when the downstream task is unrelated to the offline data, and a few empirical results are not sufficient to support the broad claim. The fact that a prior over actions only conditioned on the last action works only tells me that achieving directed exploration in the benchmarks the authors considered is not particularly challenging, and a simple prior enables directed exploration across the state space. ", "summary_of_the_paper": "The paper proposes a method to incorporate prior knowledge in an RL system to improve exploration. The authors propose learning a prior over actions conditioned on either the state, or just past actions and show that priors conditioned on just past actions generalize better than those conditioned on the full state when the observation space of offline data is different from the RL problem. \n", "summary_of_the_review": "The main claim of the paper is not supported by the evidence. It's not hard to imagine scenarios for which the main claim can be demonstratively false. In my opinion, the paper must clearly mention when the proposed method would help and hurt, and tone down the main claim.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GGw8DYZZiY", "reviewer": "Reviewer_66v7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed solution is benchmarked against a rather large set of related approaches and it is shown to perform very well, for tasks that differ significantly from the original tasks in particular. It also allows for the exploration of larger actions spaces without sacrificing the ability to achieve high rewards.\n\nIt is shown that conditioning the prior on the current state complicates transfer to tasks that differ too much from those of the source domains. What is clearly shown in the paper is that you often benefit from instead ignoring the state and only relying on the recent actions. Of particular interest is the fact that a prior only conditioned on the most recent action, rather than a longer sequence of actions, is as competitive as it is.\n\nUnlike behavioural priors, the proposed temporal do not assume that the state space has yet been explored, which in turn would assume that the novel tasks need to be similar to those the prior was trained on. Another benefit of the proposed priors is that they can be trained off-line in many different ways, such as by exploiting policies from earlier tasks or from demonstrations.\n\nHowever, the most important concern when it comes to the proposed approach is the inclusion of the mixing weights in (4). It is hard to see where this actually comes from and it would be good for the authors to sort this out. The addition would suggest the entropy component is suppressed for samples drawn from the policy and is instead dominated by those drawn from the prior, which should further encourage the policy to explore a wider space of actions, which might explain the good performance. Another concern is that since the samples are drawn from the mixture you no longer have an approximation of an entropy. It seems more like a cross-entropy between the mixture distribution and the policy distribution. \n\nThe addition of the monotonically increasing function with parameters that have to be experimentally set also seems like an arbitrary solution that needs a better explanation. The reasoning behind the function is quite unclear in the paper. If the mixing weights in the objective are seen as a kind of importance sampling, then the expectation would not be over a distribution that sums up to 1, which might motivate the need for the monotonic function. It is true though that without the inclusion of the mixing weights in the objective, the proposed solution becomes very similar to Soft Actor-Critic, with the main difference being the sampling.\n\nThe notation a_t ~ pi(s_t) in (2) and (3) is a bit confusing since it gives the impression that the policy could be deterministic. It would be better to include a do, as in pi(.|s_t). Further, since policy parameters are used in (5) and (6), it would be good to include also the parameters for the Q-function. By the way, the parameters for the Q-function are called both phi and psi in the paper. The paper does not mention how alpha is controlled over time, which can be understood from earlier literature but would be good to include also here.\n\nFinally, from the experiments, it is hard to deduce whether observed improvements are due to the mixed sampling or from the changed policy objective that includes the mixing weights. It would be good to test with and without the weights, assuming there is no good motivation why the weights need to be in the objective.\n", "summary_of_the_paper": "Temporal priors for exploration in off-policy reinforcement learning are proposed, priors for action sampling that are state-independent and depend only on the most recent past actions. In essence, an earlier trained temporal prior come in as a replacement, if a policy shows a great deal of uncertainty at a particular, possible yet unexplored, state. Exploration resorts to more generic task-agnostic trajectories instead of trusting the current policy. It is shown that sampling from such a mixture of policy and temporal priors can accelerate transfers to novel tasks that differ significantly from those originally trained for.", "summary_of_the_review": "Given the concerns raised in the core part of the paper that relates to the policy objective, the paper cannot be directly accepted, but not completely rejected either. There might be a good motivation behind the objective that the authors may be able to clarify, something that would lead to a minor modification of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DgCoTe-goFx", "reviewer": "Reviewer_eq2m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Strengths__:\n\n1. The paper is well-written and easy to follow.\n\n2. The authors provide an appropriate and concise overview of the most related prior work and its limitations.\n\n3. The proposed idea of capturing correlation between different actions is simple, intuitively useful, and general since it only requires the action space to match between the 'training' and 'downstream' environments.\n\n4. The authors provide experimental results analyzing different aspects of temporal priors, such as the effects of the conditioning variables and the type of employed generative model.\n\n__Main Concerns/Doubts__:\n\nMy major concerns with this work are the objective formulation and the experimental results. In order of relevance:\n\n1. Both hindsight experience replay (HER) and n-step returns are prior techniques that are orthogonal to the author's contribution. Both applicability and effectiveness of these techniques are very much environment dependant (e.g. HER requires access to the true reward function). Hence, it would be important if the authors could clarify whether these are also adopted by the other baselines. Moreover, it would be very relevant to include an ablation study to test the contribution of these ancillary methods and the effectiveness/sample-efficiency of TempoRL without them.\n\n2. The authors benchmark their algorithm on a very small, handpicked selection of adapted environments. For a more comprehensive and fair evaluation for robotics manipulation, I believe it would have been more appropriate to consider all of the environments in one of the pre-specified evaluation modes in meta-world (e.g., MT10). Thus, it would be important to either include additional experiments or, at least, provide a justification for the specific environment choices.\n\n3. It is unclear what is the effect of the proposed optimization objectives (Equations 4-7) on the sampling distribution policy. The authors propose to use a sampling policy derived from a mixture of $\\pi$ (the parametric downstream policy) and $\\bar{\\pi}$ (the temporal prior). Hence, to directly optimize w.r.t. the sampling distribution (as in SAC), both the policy's optimization objective and the targets for the Q-value would need to sample actions from the same mixture distribution. Yet, both of these objectives inconsistently consider sampling actions solely from $\\pi$.  If possible, It would be important to justify this specific choice over its natural alternative through formal analysis of these objectives.\n\n4. The baseline algorithms utilized have been proposed for different problem settings:\n - PolyRL (Amin et al. 2020) does not make use of prior trajectory data.\n - PARROT (Singh et al. 2020) assumes the prior trajectory data comes from a diverse set of multiple training environments from the same distribution as the downstream environment.\n  -> It would be appropriate to compare TempoRL to additional methods/algorithms that have been designed for reinforcement learning transfer given experience/demonstrations from a single environment, since this problem setting more closely matches the examined one. For instance, the authors could consider algorithms that learn diverse low-level skills in a training environment. E.g. the work from Florensa et al. 2017 and/or Haarnoja et al. 2018 (both of which share their code) could be adapted to use offline data by treating each trajectory as representing a different skill. \n -> This work asserts itself as removing the fundamental limitations of behavioral priors (Introduction, 3rd paragraph), without analyzing its own introduced limitations. Hence, for a more comprehensive comparison with behavioral prior algorithms, such as PARROT, it would be important to perform evaluation also on the behavioral priors problem setting. For instance, this could be done by adding an experiment treating all environments but one as training environments. This would assess the scalability of the proposed algorithm by evaluating how much useful information can be extrapolated from the correlation of consecutive actions in trajectories from multiple diverse training environments.\n\n5. The authors introduce a conspicuous amount of additional hyperparameters without providing details on how they were selected or an ablation study that provides information about the algorithm's robustness to their choices (e.g., $\\alpha$, $\\beta_t$, $\\beta_s$, hindsight replay ratio, n-step, ...).\n\n__Minor concerns__:\n\n1. Given the high standard deviations for some of the performance curves, I believe the authors should increase the number of utilized random seeds from the current 3/5 per experimental setting (Peter et al. 2018).\n\n2. It is unclear to me how the policy's entropy quantifies the agent's confidence to reach the goal, as stated in Section 4.2 (if multiple actions lead to the same outcome, wouldn't the entropy still be high regardless of the agent's confidence?).\n\n3. Figure 5 does not show the exploration results for Parrot.\n\n4. which is capable -> that is capable (Introduction, 3rd paragraph).\n\n__Further suggestions__:\n\n1. It would be interesting to compare the proposed SAC integration with utilizing the temporal prior as a reparameterized action space, as done in many hierarchical RL methods and PARROT.\n\n2. Since incorporating past states and actions provides information about the underlying task, a further connection could be drawn between the concept of temporal priors and implicitly performing task inference (e.g. as in Rakelly et al. 2019).\n\n\n__Post-rebuttal__\n\nWhile I appreciate the additional experiments performed by the authors, I believe they accentuate very severe issues w.r.t. the paper's empirical results. For instance, there is a large gap between the results in the original hand-picked meta-world tasks and the other tasks in MT-10. Moreover, there is no formal analysis to provide strong intuition about the algorithm's exploratory properties. This would be particularly important given some of the authors' algorithm design choices, which appear to be inconsistent with prior literature (not optimizing policy and Q-function w.r.t. exploratory policy).\n\nWhile I like the simple idea introduced in the paper, after reading the rebuttal and the other reviews, I do not think that TempoRL is currently shown to make a significant contribution. Thus, I think this paper is not yet ready for publication.\n\n\n__References__\n\nYu, Tianhe, et al. \"Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.\" Conference on Robot Learning. PMLR, 2020.\n\nAmin, Susan, et al. \"Locally Persistent Exploration in Continuous Control Tasks with Sparse Rewards.\" arXiv preprint arXiv:2012.13658 (2020).\n\nSingh, Avi, et al. \"Parrot: Data-driven behavioral priors for reinforcement learning.\" arXiv preprint arXiv:2011.10024 (2020).\n\nFlorensa, Carlos, Yan Duan, and Pieter Abbeel. \"Stochastic neural networks for hierarchical reinforcement learning.\" arXiv preprint arXiv:1704.03012 (2017).\n\nHaarnoja, Tuomas, et al. \"Latent space policies for hierarchical reinforcement learning.\" International Conference on Machine Learning. PMLR, 2018.\n\nHenderson, Peter, et al. \"Deep reinforcement learning that matters.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 32. No. 1. 2018.\n\nRakelly, Kate, et al. \"Efficient off-policy meta-reinforcement learning via probabilistic context variables.\" International conference on machine learning. PMLR, 2019.\n\n", "summary_of_the_paper": "This paper proposes the concept of temporal priors, extending previously proposed behavioral priors. Behavioral priors try to recover some prior (state conditioned) policy utilizing a set of trajectories obtained from multiple 'training' tasks, to be used to guide exploration in 'downstream' tasks. Instead, temporal priors take a more general non-Markovian approach by learning a prior policy that is potentially conditioned on the whole history of previous states and actions. This work proposes a new algorithm, TempoRL, that employs a generative flow policy to parameterize the temporal prior and a modified SAC agent that is incentivized to perform exploration with the temporal prior. In practice, the authors propose to limit the conditioning information of the temporal prior to, exclusively, the previous action in the trajectory, only learning the correlation between consecutive actions. This choice makes the resulting temporal prior less dependent on the diversity and the optimality of the trajectory data from the training tasks than traditional behavioral priors. Empirically, the proposed algorithm obtains promising results in learning temporal priors from a single training task in a few robotics environments from meta-world (Yu et al. 2020) and simple navigation environments with sparse rewards.", "summary_of_the_review": "While this paper introduces some simple and interesting ideas, without a more comprehensive experimental evaluation and a clearer explanation of the properties of the algorithm, it is not yet ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was close and also very polarizing with the reviewers. On the positive side, some reviewers found:
1. the results impressive
2. the proposed method to be novel, interesting, and produce good performance across several settings
3. the paper was well written

On the other hand, others found:
1. the motivation suspect
2. missing experiments to characterize the sensitivity to numerous hyper-parameters 
3. the baselines compared with weak and not representative
4. significant performance drop comparing the results in the original submission and the new ones added during discussion period
5. low number of seeds initially

In the end, multiple reviewers raised serious issues regarding the motivation for the approach and the quality and ultimately credibility of the results presented. One of the high-scoring reviewers agreed the paper was a bit misleading (limitations relegated to the appendix). Unfortunately, none of the high-scoring reviewers provided counters to this points.
Decision: Reject