Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
Authors: Michael R Zhang
Thomas Paine
Ofir Nachum
Cosmin Paduraru
George Tucker
ziyu wang
Mohammad Norouzi
Venue: ICLR 2021 Poster (2021)
Keywords: Off-policy policy evaluation
autoregressive models
offline reinforcement learning
policy optimization

Abstract: Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure. This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics. In this paper, we challenge this conditional independence assumption and propose a family of expressive autoregressive dynamics models that generate different dimensions of the next state and reward sequentially conditioned on previous dimensions. We demonstrate that autoregressive dynamics models indeed outperform standard feedforward models in log-likelihood on heldout transitions. Furthermore, we compare different model-based and model-free off-policy evaluation (OPE) methods on RL Unplugged, a suite of offline MuJoCo datasets, and find that autoregressive dynamics models consistently outperform all baselines, achieving a new state-of-the-art. Finally, we show that autoregressive dynamics models are useful for offline policy optimization by serving as a way to enrich the replay buffer through data augmentation and improving performance using model-based planning.


Published as a conference paper at ICLR 2021
AUTOREGRESSIVE DYNAMICS MODELS FOR OFFLINE POLICY EVALUATION AND OPTIMIZATION
Michael R. Zhang
Tom Le Paine
Ofir Nachum
Cosmin Paduraru²
George Tucker
Ziyu Wang
Mohammad Norouzi
University of Toronto mi michael@cs. twwonto.ooooo...oo edu,
DeepMind
Google Brain
mmorozzi@google..oooo
ABSTRACT
Standard dynamics models for continuous control make use feedforward compu- tation predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure. This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable deterministic transition dynamics. In this paper, we challenge this conditional inde- pendence assumption and propose a family of expressive autoregressive dynamics modely that generate different dimensions of the next state and reward sequentially conditioned on previous dimensions. We demonstrate that autoregressive dynamics models indeed outperform standard feedforward models in log- liglliiel on held- out transitions. Furthermore, we compare different model-based and model-free off- off-policy evaluation (OPE) methods on RL Unplugged, suite of offline MuJoCo datasets, and find that autoregressive dynamics models consistently outperform all baselines, achieving a new state-of-the-arr. Finally, we show that autoregressive dynamics models are useful for offline policy optimization by serving as way to enrich the replay buffer through data augmentation and improving performance using model model-based planning.
pryssic-based simulation environments entail
INTRODUCTION
Model-based Reinforcement Learning (RL) aims learn approximate model of the environment's dynamics from existing logged interactions to facilitate efficient policy evaluation and optimization. Early work Model-based RI. uses simple tabular (Sutton 1990, Moore and Atkeson 1993 Peng
and Williams 1993) and locally linear (Atkeson et al.) 1997) dynamics models, which often result in large degree of model bias (Deisenroth and Rasmussen [201T). Recent work adopts feedforward neural networks to model complex transition dynamics and improve generalization to unseen states and actions, achieving a high level of performance on standard RL benchmarks (Chua al. 2018 Wang et (201)). However, standard feedforward dynamics models assume that different dimensions of the next state and reward are conditionally independent given the current state and action, which may lead to a poor estimation of uncertainty and unclear effects on RI. applications.
In this work, we propose new family of autoregressive dynamics models and study their effectiveness for off-policy evaluation (OPE) and offline policy optimization on continuous control. Autoregressive dynamics models generate each dimension of the next state conditioned on previous dimensions of the next state, in addition to the current state and action (see Figure I. This means that to sample the next state from an autoregressive dynamics model, one needs n sequential steps, where n is the number of state dimensions, and one more step to generate the reward. By contrast. standard feedforward dynamics models take current state and action as input and predict the distribution of the next state and reward multivariate Gaussian with diagonal covariance structure (e.g., Chua et al. (2018); Janner et al. (2019)). This modeling choice assumes that different state dimensions are conditionally independent.
WWok done intern Google Brain.

---

Published as a conference paper at ICLR 2021
Autoregressive generative models have seen success in generating natural images (Parmarr al. 2018), text (Brown et al.] 2020). and speech (Oord et al. 2016), but they have not seen use in Model-based RL for continuous control.
We find that autoregressive dynamics models achieve higher log- bog-likelihood compared to their feed- forward counterparts heldout validation transitions of all DM continuous control tasks (Tassal et al. 2018) from the RL Unplugged dataset (Gulcehre et al. 2020). To determine the impact of improved transition dynamics models, we primarily focus OPE because allows us isolate contributions of the dynamics model in value estimation vs. the many other factors of variation in policy optimization and data collection. We find that autoregressive dynamics models consistently outperform existing Model-based and Model-free OPE baselines on continuous control in both ranking and value estimation metrics. We expect that our advances model-based OPE will improve offline policy selection for offline RI. (Paine et al., 2020). Finally, we show that our autoregressive dynamics models can help improve offline policy optimization by model predictive control, achieving new late-of-theeart chectah-run and fish-swim from RL Unplugged (Gulcehre al. 2020).
Key contributions of this paper include:
We propose autoregressive dynamics models to capture dependencies between state dimensions forward prediction. We show that autoregressive models improve log-likelihood over non- autoregressive models for continuous control tasks from the DM Control Suite (Tassa al.) 2018).
We apply autoregressive dynamics models to Off-Policy Evaluation (OPE), surpassing the per- formance of state-of-the art baselines in median absolute error, rank correlation, and normalized top-5 regret across 9 control tasks.
We show that autoregressive dynamics models more useful than feedforward models for offline policy optimization, serving as a way to enrich experience replay by data augmentation and improving performance via model-based planning.
2
PRELIMINARIES
Here we introduce relevant notation and discuss off-policy (offline) policy evaluation (OPE). We refer the reader to Lange al. (2012) and Levine et al. (2020) for background on offline RL, which is also known as batch RL the literature.
A finite-horizon Markov Decision Process (MDP) is defined by tuple M = A, do. r, 7), where is a set of states E S, A is a set of actions a A. T defines transition probability distributions p( (st, az), ), do defines the initial state distribution do p(so), r defines reward function r x AA + R, and y is a scalar discount factor. A policy π (a s) defines a conditional distribution over actions conditioned
states. A trajectory consists of sequence of states and actions (so, do, $1, (),.... 8H) of horizon length H. We use 8t, to denote the i-th dimension of the state at time step (and similarly for actions). In reinforcement learning, the objective is to maximize the expected sum of discounted rewards over the trajectory distribution induced by the policy:
H (rr((sss Entr(or
VT(T)) = R (r)
Σ 0=
The trajectory distribution is characterized by the initial state distribution, policy, and transition probability distribution:
H-1
(T( = do II t=
((aass)p(ss4)l at).
(2)
In offline RL, we are given access to a dataset transitions D {s}, aj, ri41, 11 and a set of initial states Sp. Offline RL is inherently a data-driven approach since the agent needs to optimize the same objective as in Eq. IT] but is not allowed additional interactions with the environment. Even though offline RL offers the promise of leveraging existing logged datasets, current offline RL algorithms (Fujimoto et al.) 2019 Agarwal et al.) 20200 Kumar et al.) 2019) are typically evaluated using online interaction, which limits their applicability in the real world.
2

---

Published as a conference paper at ICLR 2021
4r ... dg
$
(H. o²) for 51-1,1
8r.1 $1.0 a. dr
(H, of) for Spel.n
Stt.x [[x < one_bot
Tr+l
mean and variance for next state and reward
state and action
state, action, i- dim next state
mean variance for
1 dim next state
Standard Feedforward Dynamics Models
Proposed Autoregressive Dynamics Model
Figure I: Standard probabilistic dynamics models (e.g.. Chua et al. 2018)) use a neural network to predict the mean and standard deviation of different dimensions of the next state and reward simultaneously. By contrast, we the same neural network architectures with several additional inputs and predict the mean and standard deviation of each dimension the next state conditional on previous dimensions of the next state, As empirical results indicate, this small change makes a big difference in the expressive power of dynamics models. Note that reward prediction is not shown the right to reduce clutter, but it can be thought of as (n++ 1- 1)th state dimension.
The problem of off-policy (offline) policy evaluation (OPE) entails estimating (T(),, the value of a target policy based on a fixed dataset of transitions denoted DD without access to the environment's s dynamics. Some OPE methods assume that is generated from a known behavior (logging) policy and assume access to a in addition to D. In practice, the logged dataset D> may be the result of following some existing system that does not have a probabilistic form. Hence, in our work, we will assume no access to the original behavior policy μ for OPE. That said, for methods that require access to J4, we train a behavior cloning policy on D.
3 PROBABILISTIC DYNAMICS MODELS
Feedforward dynamics model. In the context of our paper, we use the term "model" to jointly refer
to the forward dynamics model (s |s,, a²) and reward model p, Pr (alss,,,)), a.). We use neural nets to parameterize both distributions since they are powerful function approximators that have been effective for model-based RL (Chua et al. 2018 Nagabandi et al. 2018 Janner et al. 2019).
Let 0 denote the parameters of fully connected network used to model p8(8141, Tt+1 Tt+1 expect joint modeling of the next state and reward to benefit from sharing intermediate network features. Similar prior work (Janner et al. 2019), our baseline feedforward model outputs the mean and log variance of all state dimensions and reward simultaneously, as follows:
8t, at). We
P't St, a.) -N a.), Diag(exp(((ss,.n))
where plst, 0c) E pa+1 denotes the mean for the concatenation of the next state and reward, I(s,, at) E 1= denotes the log variance, and Diag(u) an operator that creates a diagonal matrix with the main diagonal specified by the vector During training, seek to minimize the negative log likelihood of the parameters given observed transitions in the dataset D:
((0 D) - Σ (saar log Pe('s, ('' s,
(4)
While it is possible to place different weights on the loss for next state and reward prediction, we did not apply any special weighting and treated the reward as an additional state dimension in all of our experiments. This is straightforward to implement and does not require tuning an additional hyperparameter, which is challenging for OPE. Note that the input has |s| + |a] dimensions.
Autoregressive dynamics model. We now describe our autoregressive model. We seek to demon- strate the utility predicting state dimensions in an autoregressive way. Therefore, rather than using complex neural network architecture, where improvements in log logelikelihood and policy evaluation are confounded by architectural differences, we opt to make simple modifications to the feedforward model described above. This allows us isolate the source of performance improvements.
The autoregressive model we use is a fully connected model that predicts the mean and log variance of single state dimension. We augment the input space of the baseline with the previous predicted
3

---

Published as a conference paper at ICLR 2021
state dimensions and one-hot encoding indicate which dimension to predict. This is illustrated in Figure [I] The autoregressive model therefore has 3|s| + |a] input dimensions. Hence, the autoregressive model has small number of additional weights in the first fully connected layer, but as will be shown in our experiments, these extra parameters are not the reason for a performance gain.
At training time, the autoregressive model has similar computational cost to the fully connected model as we can mask ground truth states and use data parallelism to compute all state dimensions simultaneously. At inference, the autoregressive model requires additional forward passes, on the order of the number of state dimensions in given environment. We use the default ordering for the state dimensions in given environment, though it is interesting explore different orderings in future works. The negative log-likelihood for an autoregressive model takes the form of:
( ()
- D)
E (0,,,, (')ED
[oggo(('' 8 s') + Σ "2
log po(s 8, s, 8 $1...,,....................... (i..).....)) (5)
:=1
where we use chain rule to factorize the joint probability of p(s', r' 8,
The main advantage of the autoregressive model is that it makes no conditional independence assumption between next state dimensions. This class of models can therefore capture non-unimodal dependencies, e.g., between different joint angles of robot. Paduraru (2007) demonstrates this increased expressivity in the tabular setting, constructing an example on which a model assuming conditional independence fails. While the expressive power of autoregressive models have been shown in various generative models (Parmar et al. 2018 Oord et al. 2016). autoregressive dynamics models have not seen much use in Model-based RL for continuous control before this work.
Model-based OPE. Once dynamics model is trained from offline data, OPE can be performed in direct and primitive way. We let the policy and model interact- policy generates the next action, the model plays the role of the environment and gener- ates the next state and reward. Due to the stochasticity in the model and the policy, we estimate the return for policy with Monte-Cari sampling and monitor standard error. See Algorithm for pseudocode.
Algorithm 1 Model-based OPE
Require:
Number of rollouts n, discount factor 7. horizon length H. policy x, dy- namics model p, set of initial states So for i 1,22..... do
R3 + 0 sample initial state Sp ~ Sg
for = 0, 1. 2........ H - do
sample from policy: at ~ T(- 8t)
sample from the dynamics model:
4 RELATED WORK
8141, ~ p(:, 81, a.)
Our work follows long line of OPE research, which especially relevant to many practical domains such as medicine (Murphy et al.] 200T), recommendation systems et. | [01 1), and education (Mandel et al.] 2014) in order to avoid the costs and risks associated with online evaluation. There exists a large body of work on OPE, including methods based on
R, + R, + 7rr41
end for end for return Σm R
importance weighting (Precup 2000 Li et al., 2014) and Lagrangian duality (Nachum al. 2019 Yang et al. 2020 Uchara and Jiang 2019). The model-based approach that we focus on in this paper lies within the class of algorithms referred to as the direct method (Kostrikov and Nachum 2020
Dudik et al. 2211T: Voloshin et al., 2019), which approximate the value of a new policy by either explicitly or implicitly estimating the transition and reward functions of the environment. While model-based policy evaluation has been considered by previous works (Paduraru) [0077; Thomas and Brunskill 2016a Hanna al. 2017). it has largely been confined to simple domains with finite state and action spaces where function approximation is not necessary. By contrast, our work provides an extensive demonstration of model-based OPE in challenging continuous control benchmark domains. Previous instances of the use of function approximation for model-based OPE (Hallak et al. (015) impose strong assumptions on the probabilistic dynamics models, such as factorability of the MDP. Our results indicate that even scemingly benign assumptions about the independence of different state dimensions can have detrimental consequences for the effectiveness of a model-based OPE estimate.
While the use of model-based principles in OPE has been relatively rare, it has been more commonly used for policy optimization. The field of moodel-based RL has matured in recent years to yield impressive results for both online (Nagabandi et al.) 2018 Chua al. [018 Kurutach al. 2018 Janner et al. 2019) and offline (Matsushima et al. 2020 Kidambi et al. 2020; Yu et al. 2020 Argenson and
2020) policy optimization. Several of the techniques we employ, such
4

---

Published as a conference paper at ICLR 2021
Table 1:
Summary of the offfine datasets used. Dataset size indicates the number of (s, a, tuples.
caripole swingup
cheetah
finger turn hard
fish swim
humanoid run
walker stand
walker walk
manipulator insert ball
manipulator insert peg
State dim. Action dim.
17
12
24
67
24
24
44
44
2
5
21
6
5
5 1.5M
Dataset size
40K
300K
500K
200K
3M
200K
200K
1.5M
Table Negative log-likelihood on heldout validation sets different RL Unplugged tasks (lower is better). For both family of dynamics models, we train 48 models with different hyperparameters. We report the Top-1 NLL on the top and average of S models on the bottom. On all of the tasks autoregressive dynamics models significantly outperform feedforward models in terms of NLL for both Top-I and Top-5.
Dynamics model cartpole cheetah
finger
fish
humanoid
walker walker manipulator insert ball
manipulator insert peg
architecture
swingup
turn hard swim
stand
walk
Top-1
Feedforward
-6.81
-4.90
-5.58
-4,91
-3.42
-4.52
-3.84
-4.74
-4.34
Autoregressive
-7.21
-6.36
-6,14
-5.21
-4.18
-4.73
-4.17
-5.62
-5.73
Feedforward
-6.75
-4.85
-5.50
-4.90
-3.40
-4.49
-3.81
-4.64
-4.31
Autoregressive -7.14
-6.32
-5.94
-5.18
-4.15
-4.11
-4.15
-5.58
-5.29
as the normalization of the observation space, are borrowed from this previous literature (Nagabandi et al., 2018 Chua et al. 2018). Conversely, we present strong empirical evidence that the benefits of our introduced autoregressive generative models of state observations do earry over to model-based policy optimization, at least in the offline setting, and this is an interesting avenue for future work.
5 RESULTS
We conduct our experiments on the DeepMind control suite Tassa et al. 2018), a set of control tasks implemented in MuJoCo (Toooovv et al. 2012). We use the offline datasets from RL Unplugged (Gulcehre al. 2020), the details of which are provided in Table These environments capture a wide range of complexity, from 40K transitions in a 5-dimensional cartpole environment to 1.5 million transitions on complex manipulation tasks. We follow the evaluation protocol in the Deep OPE (Fu et al. 2021) benchmark and use policies generated by four different algorithms: behavioral cloning (Bain 1995). D4PG (Barth-Maron al. 2018), Critic Regularized Regression (Wang et al. 2020). and ABM (Siegel et al. 2019). With varied hyperparameters, these form a diverse set of policies of varying quality.
We perform thorough hyperparameter sweep in the experiments and use standard practice from generative modeling to improve the quality the models. We allocate 80% of the data for training and 20% of the data for model selection. We vary the depth and width of the neural networks (number of layers E (3, 4). layer size E {512, 1024)), add different amounts of noise to input states and actions, and consider two levels of weight decay for regularization (input noise E {0, 1e-6, 1e-7]. weight decay {0, le- le-6)). For the choice optimizer, we consider both Adam (Kingma and Ba 2014) and SGD with momentum and find Adam to be more effective at maximizing likelioiddood across all tasks in preliminary experiments. We thus use Adam in all of our experiments with two learning rates E (1e-3, 3e 4]. We decay the optimizer's 's learning rate linearly to zero throughout training, finding this choice to outperform a constant learning rate. Lastly, we find that longer training often improves log- lideeooll results. We use 500 epochs for training final models.
For each task we consider in total 48 hyperparameter combinations (listed above) for both models and pick the best model in each model family based on validation log-likelihood. This model then used for model-based OPE and policy optimization. Note that, in our experiments, 20% of the transitions are used only for validation, but we believe one can re-train the models with the best hyperparameter configuration on the full transition datasets to improve the results even further.
5.1 AUTOREGRESSIVE DYNAMICS MODELS OUTPERFORM FEEDFORWARD MODELS IN NLL
To evaluate the effectiveness of autoregressive dynamics models compared to feedforward coun- terparts, Table 2 reports negative log-likelihood (NLL) the heldout validation set for the best
5

---

Published as a conference paper at ICLR 2021
Cheetah run
Fish swim
Humanoid run
Manipulator insert peg
Negative log-likeee ee-likellhood
Feodforrard
Negative aiglllllhhodd
Roodforword Autoregressvvé
Negative log-likellnodd
Feadionward Auttregresiin
Negative lilkllihood
Feodf Aucuregressvve
Parameter count (x1000)
Parameter count (x1000)
Parameter count (X1000)
Parameter count (x1000)
Figure 2: Network parameter count vs. validation negative log-likelihood for autoregressive and feedforward
models, Autoregressive models often have a lower validation NLL irrespective of parameter count.
cartpole swingup
cheetah run
fish swim
walker walk
Correlation coeff.
Correlation coeff.
Correlation coeff.
Correlation coeff.
Autoregressive
Autoregressive
Autoregresddoe
Autoregressive
Negalive log- liglliikihhod
Negative log- log-likelihood
Negative log-likelihood
Negative log-likelihood
Validation negative log- likelihoodd OPE correlation coefficients different tasks. On 4 RL Unplugged tasks, we conduct extensive experiment in which 48 Autoregressive and Feedforward Dynamies models are used for OPE. For each dynamics model, we calculate the correlation coefficient between model- based value estimates and ground truth values at a discount factor of 0.995. We find that low validation NLL numbers generally correspond to accurate policy evaluation, while higher NLL numbers are less meaningful.
Figure 3:
performing models from our hyperparameter sweep. For each environment, report the NLL for the best-performing model (Top- 1) and the average NLL across the Top-5 models. The autoregressive model has lower NLL on all environments, indicating that it generalizes better to unseen data.
To study the impact of model size on NLL, Figure 2 ssows validation NLL as a function of parameter count. We find that small datasets large models hurt, but more importantly autoregressive models outperform feedforward models regardless of the parameter count regime, i.e., even small autoregressive models attain lower validation NLL compared big feedforward models. This indicates that autoregressive models have a better inductive bias in modeling the transition dynamics than feedforward models that make conditional independence assumption.
5.2 ARE DYNAMICS MODELS WITH LOWER NLL BETTER FOR MODEL- MODEL-BASED OPE?
We ultimately care not just about the log- likelioel numbers, but also whether or not the dynamics models are useful in policy evaluation and optimization. To study the relationship of NLL and OPE performance for model-barn based methods, we compute OPE estimates via Algorithm I and compute the Pearson correlation between the OPE estimates and the true discounted returns. This serves as measure of the effectiveness of the model for OPE. We repeat this for all 96 dynamics models we trained on a given environment and plot the correlation coefficients against validation NLL in Figure
Models with low NLL are generally more accurate in OPE. Lambert et al. (2020) have previously demonstrated that in Model-based RL, "training cost does not hold strong correlation to maxi- mization of episode reward." We use validation NLL instead, and our results on policy evaluation decouple the model from policy optimization, suggesting more nuanced picture: low validation NLL numbers generally correspond to accurate policy evaluation, while higher NLL numbers are generally less meaningful. In other words, if the dynamics model does not capture the transition dynamics accurately enough, then it is very hard to predict its performance on OPE. However, once the model starts to capture the dynamics faithfully, we conjecture that NLL starts to become reasonable metric for model selection. For instance, validation NLL does not seem to be a great metric for ranking feedforward models, whereas it is more reasonable for autoregressive models.

---

Published as a conference paper at ICLR 2021
cartpole swingup (r- 0.81)
cartpole swiingup 0.80)
caripole swinguup (r- 0.89)
cartpole (r= 0.953
FOE (distribution mntonnal
Foodfonwrrd Model
e Model
FQR
Return (df 0. 0.995)
Return (d 0.995)
Return (d 0.995)
Return (d 0.995)
fish
fish 0.51)
fish swim
fish (+= 0.99)
Messeeeeess Mogeel
Fecoforword Model
FOF (dietributtonnll
FOE (L.2
Retum (d-
(d 0.995)
(r- 0.04)
huma noid run = 0.72)
humanoid run 0.44)
humaneid run (r = 0.89)
FOR (distributionay
Focdforrennd Model
A eeessegess Mordel
FOE (12)
Return 0.995)
Return (d 0.995)
Return 0.995)
Return (d 0.995)
walker stand (r 0.97)
walker stand (r- 0.84)
walker stand (f 0.83)
walker stand (f= 0.94)
FOF Colstritigg sonernttuttion (araa
Foedforwaadd Modeal
Autoregreeet reeeegrresivee Medal
FQE (L2)
0.995)
(d 0.995)
0.995)
0.995)
Figure 4: Comparison of model-based OPE using autoregressive and feedforward dynamics models with state-of-the-art FQE methods based on L2 and distributional Bellman against ground truth returns with discount .995 on the x-axis. We report the Pearson correlation coefficient (r") the title. While feedforward models fall behind FQE on most tasks, autoregressive dynamics models are often superior. See Figure B.4 for additional scatter plots on the other environments.
We plot OPE estimates the y-axis
5.3 COMPARISON WITH OTHER OPE METHODS
We adopt recently proposed benchmark for OPE (uu al. 2021) and compare model-based approaches with state-of-the- OPE baselines therein. Figures and B. B.4ccomarre OPE estimates
from two Fitted-Q Evaluation (FQE) baselines (Le al. 2019 Kostrikov and Nachum 2020 Paine
et al. 2020 our feedforward models, and the autoregressive approach. Each plot reports the Pearson correlation between the OPE estimates and the returns. The autoregressive model consistently outperforms the feedforward model and FQE methods on most environments. We report ensembling results in the appendix, but compare single models for fairness in the rest of the paper.
We compute summary statistics for OPE methods in Table 3. Table A.1 and Table A.2 These tables report the Spearman's 's rank correlation, regret, and absolute error, respectively. These metrics capture different desirable properties of OPE methods (Fu et al. 2021); more details about how they are computed are the appendix. In three metrics, the autoregressive model achieves the best median performance across nine environments, whereas the baseline model is not good FQE. The only environment in which the autoregressive model has negative rank correlation manipulator insert ball. In addition, major advantage of our model-based approach needs to be trained once per environment-ww do not need to perform additional policy-specine optimization, whereas FQE needs to optimize a separate Q-function approximator per policy.
FQE is that the model only
7

---

Published as a conference paper at ICLR 2021
Cartpole swingup
Cheetah run
Finger
Fish
Humanoid
turn hard
swim
run
Rank Correlation btw.
OPE and ground truth
Importance Sampling
0.23 #0.11
0.01 #0.12
0.45 +0.08
0.17 #0.11
0.91 +0.02
Best DICE
0.16 #0.11
0.07 +0.12
0.22 s0.11
0.44 60.09
-0.10
Variational power method
0.01 #0.11
0.01 40.12
0.25 +0.11
0.56 +0.08
0.36 +0.09
Doubly Robust (IS, FQB)
0.55 =0 09
0.56
0.67 20.05
0.11 10.12
-0.03 1:.12
Foedforward Model
0.83 2.005
0.64 1.0.08
0.08 20.11
0.95 ::0 02
0.35 10010
FQE (distributional)
0.69 =0.07
0.67 -:0.000
0.94 10.01
0.59 10.10
0.74
FQE (1.2) Autoregressive Model
0.70 =0.07
0.56 20.08
0.83 #0.04
0.10 10.12
0.00 1:012
0.91
0.74 00.07
0.57 #0.09
0.96 1.96.10.01
0.90 +0.02
Walker stand
Walker walk
Manipulator insert ball
Manipulator insert peg
Median t
Rank Correlation buw.
OPE and ground truth
Importance Sampling
0.59 20.08
0.38 10.10
0.72 #0.05
0.25 #0.08
0.17
Best DICE
00.11 10.12
0.5810..0
0.19 10.11
0.35 10.10
-0.11
Variational power method
00.35 10.10
10.10
0.61 10.08
0.41 10.09
0.01
Doubly Robust (IS, FQE)
0.88 20.03
0,85 1:0.04
0.42 10.10
00.47 10.09
0.55
Feedforward Model
0.82 20.04
#0.05
0.06 10.10
0.56 20.08
0.64
FQE (distributional)
0.87 =0.02
10.03
0.63 20.08
-0.23 10.10
0.69
FQE (I.2) Autoregressive Model
0.96
0.94
10.02
0.70 =0.07
-0.48 10.08
0.70
0.96 0.002
0.0.00
0.33 $0.09
0.47 +0.09
0.90
Table 3: Spearman's 's rank correlation (p) coefficient (bootstrap mean + standard deviation) between different OPE metrics and ground truth values discount factor of 0.995. each column, rank correlation coefficients that not significantly different from (p > 0.05) are bold faced. Methods are ordered by median. Also see Table A.I and A.2 for Normalized Regret and Average Absolute Error results.
run
finger turn
cartpole swingup
Und saausscuunt Episodic Return
CRR-glanning dlaning Published SOTA
1909 u9de
D000
1000
12000
IIO 2000
2009 1000
1900
Steps
Steps
Steps
Steps
Figure 5: Model-based offline policy optimization results. With planning and data augmentation, we improve the performance over CRR exp (our baseline algorithm). When using autoregressive dynamics models (CRR- planning AR), outperform state-of-the-ar
Cheetah run and Fish swim. Previous SOTA results (Gulcehre al. 2020 Wang al. (0200) obtained using different offline RL algorithms: Cheetah CRR exp. Fish
swim CRR binary max, Finger turn hard CRR binary max, Cartpolc swingup BRAC [Wu al. 2019
5.4 AUTOREGRESSIVE DYNAMICS MODELS FOR OFFLINE POLICY OPTIMIZATION
Policy evaluation is integral part reinforcement learning. Improvement in policy evaluation can therefore be adapted for policy optimization. In this section, we explore two possibilities of using models to improve offline reinforcement learning. Regression (CRR) as a base offline reinforcement learning algorithm (Wang et al. 2020)
experiments, we use Critic Regularized
First, we utilize the model during test time for planning by using modified version of Model Predictive Path Integral (MPPI) (Williams al., 2015). Unlike MPPI, truncate the planning process after 10 steps of rollout and use the CRR critic to evaluate future discounted returns. We provide additional details in the appendix. Secondly, dataset to learn a better critic for CRR. More precisely, given st ~ D. and the current policy n, we generate additional data using the following process: af T(-1st), 8 Att p(: -s), ā,).
the model augment the transition
These two options are orthogonal and can be applied jointly. We implemented both techniques on top of the CRR exp variant (Wang et al. 2020) and show their combined effect in Figure 5 The
8

---

Published as a conference paper at ICLR 2021
figure shows that autoregressive dynamics models also outperform feedforward ones in the policy optimization context. Notably, in the case of cheetah run and fish swim, using autoregressive models for planning well data augmentation enables to outperform the previous ofte-of-theean on these offline datasets. Additionally, when using autoregressive dynamics models, both techniques improve performance. In the appendix, show this result well more ablations.
CONCLUSION
This paper shows the promise of autoregressive models in learning transition dynamics for continuous control, showing strong results for off-policy policy evaluation and offline policy optimization. Our contributions offline model-based policy optimization are orthogonal prior work that uses ensembles to lower the values when ensemble components disagree (Kidambi et al., 2020) Incorporating conservative value estimation into our method is an interesting avenue for future research. We use relatively primitive autoregressive neural architectures in this paper to enable fair comparison with existing feedforward dynamics models. That said, will be exciting to apply more sophisticated autoregressive neural network architectures with cross attention (Bahdanau et al. 2014) and self-attention /Vaswani al. 2017) to Model-based RL. for continuous control.
Acknowledgements We thank Jimmy Ba, William Chan, Rishabh Agarwal, Dale Schuurmans, and Silviu Pitis for fruitful discussions on our work. We are also grateful for the helpful comments from Lihong Li, Jenny Liu, Harris Chan, Keiran Paster, Sheng Jia, and Tingwu Wang earlier drafts.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. International Conference on Machine Learning, 2020.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. rrXiv:2008.05556 2020.
Christopher G Atkeson, Andrew W Moore, and Stefan Schaal. Locally weighted learning. In Lazy learning, pages 11-73. Springer, 1997.
Damitry Bahdanau. Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: arXiv:1409.00 1409.0473, 2014.
Michael Bain. A framework for behavioural cloning, In Machine Intelligence 15, pages 103-129, 1995.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. a 00610, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv:2005.1 101.2 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML--I), pages 472, 2011. 2011.
Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv:1103.4600. 2011.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang. Michael R. Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, Sergey Levine, and Thomas Paine. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2021.

---

Published as a conference paper at ICLR 2021
Scott Fujimoto, David Meger, and Doina Precup. Off- Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062, 2019.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: Benchmarks for offline reinforcement learning. arXiv:2006. 1.888, 2020.
Assaf Hallak, François Schnitzler, Timothy Mann, and Shie Mannor. Off-policy model-based learning under unknown factored dynamics. In International Conference on Machine Learning, pages 711-719, 2015.
Josiah Hanna, Scott Nickum, and Peter Stone. Importance sampling policy evaluation with an estimated behavior policy. In International Conference Machine Learning, pages 2605-2613. PMLR, 2019.
Josiah P Hanna, Peter Stone, and Scott Niekum. Bootstrapping with models: Confidence intervals for off-policy evaluation. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based
policy optimization. In Advances in Neural Information Processing Systems, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL Model- based offline reinforcement learning. wwxiv:2005.0595/ 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
Ilya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation, 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11784-11794 2019.
Thanard Kurutach, Ignasi Clavera, Yan Duan. Aviv Tamar, and Pieter Abbeel. Model-Ensemble Frust-Region Policy Optimization. In International Conference on Learning Representations, 2018.
Nathan Lambert, Brandon Amos, Omry Yadan, and Roberto Calandra. Objective mismatch in model-based reinforcement learning. 20ti::0002004233,, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
Hoang M Le. Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. arXiv preprint 19X3:1993300878, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. TrXi::2005.01643, 2020.
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. In Proceedings of the fourth ACM interna- tional conference Web search and data mining, pages 297-306. ACM, 2011.
Lihong Li, Remi Munos, and Csaba Szepesvári. On minimax optimal offline policy evaluation. arXiv:1409.3653 2014.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off- off-policy estimation. In Advances in Neural Information Processing Systems, pages 5356-5366, 2018.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages 1077-1084, International Foundation for Autonomous Agents and Multiagent Systems, 2014.
10

---

Published as a conference paper at ICLR 2021
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model- offine optimization. arXiv:2006.036/1 2020.
Andrew W Moore and Christopher G Atkeson. Memory-basee reinforcement learning: Efficient computation with prioritized sweeping. In Advances in neural information processing systems, pages 263-270, 1993.
Susan Murphy, Mark van der Laan, James M Robins, and Conduct Problems Prevention Re- search Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association, 96(456):140014222 2001.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pages 2318-2328, 2019.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynam- ics for model--bsee deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 2559-7566. IEEE, 2018.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv:1609.088
2016.
Cosmin Paduraru. Planning with approximate and learned models of markov decision processes. MsC Thesis, University of Alberta, 2007.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv:2007.09055 2020.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. 100.0555,,, 2018.
Jing Peng and Ronald J Williams. Efficient learning and planning within the dyna framework. Adaptive behavior, 1(4):337 454, 1993.
Doina Precup. Eligibility traces for off- off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
Noah Siegel, Jost Tobias Springenberg. Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2019.
Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. 180I. 2018.
Thomas and E. Brunskill. Data-efficient oof-policy policy evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, pages 1149, 2016a.
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference Machine Learning, pages 2139-2148, 201
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
Masatoshi Uchara and Nan Jiang. Minimax weight and q q-function learning for off-policy evaluation. arXiv: 12809, 2019.
11

---

Published as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6000 2017.
Cameron Voloshin, Hoang M Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. 1.Xiv:9911.06844 2019.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang. Pieter Abbeel, and Jimmy Ba. Benchmarking model- based reinforcement learning. 1eXivv9907.000
2019.
Ziyu Wang, Alexander Novikov, Konrad Zoin, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv:2006. 2020.
Junfeng Wen, Bo Dai, Lihong Li, and Dale Schuurmans. Batch stationary distribution estimation. arXiv preprint arXiv:2003.0071 00722, 2020.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv: 149, 2015.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arX1v:1911.11311, 2019.
Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu,, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. arXiv:2005. 13239, 2020.
12

---

Published as a conference paper at ICLR 2021
A OFFLINE POLICY EVALUATION
We use the baseline results in Fu et al. 2021 For convenience, we replicate their description of the OPE baselines and metrics.
A.1 1
OPE METRICS
To evaluate the OPE algorithms, we compute three different metrics between the estimated returns and the ground truth returns:
1. Rank correlation This metric assesses how well estimated values rank policies. It equal to the correlation between the ranking (sorted order) by the OPE estimates and the ranking by the ground truth values.
2. Absolute Error:
This metric measures the deviations of the estimates from the ground truth and does directly access the usefulness for ranking.
3. Regret@k This metric measures how much worse the best policies identified by the esti- mates are than the best policy in the entire set. Regret @ k is the difference between the actual expected return the best policy in the entire set, and the actual value of the best policy in the top-k set.
A.2 OPE BASELINES
train neural network estimate the value of the evaluation policy Te by bootstrapping from Q T (x))). We tried two different implementations,
Fitted Q-Evaluation (FQE) As in ee al.
2019),
one from Kostrikov and Nachum 2020) and another from Paine al, (2020).
Importance Sampling (IS) We perform importance sampling with a learned behavior policy. We use the implementation from Kostrikov and Nachum (2020), which uses self-normalized (also known as weighted) step-wise importance sampling (Liu et al.) (0188] Nachum et al. (019). Since the behavior policy not known explicitly, we learn an estimate of
a max-likelihood objective over the
dataset D, as advocated by Hanna et al. (2019). In order to be able to compute log- bog-probabilities when the target policy is deterministic, we add artificial Gaussian noise with standard deviation 0.01 for all deterministic target policies.
Doubly-Robust (DR) We perform weighted doubly-robust policy evaluation based on Thomas and Brunskill 6b016b) and using the implementation of Kostrikov and Nachum (2020). Specifically, this method combines the IS technique above with value estimator for variance reduction. The value estimator is learned according to Kostrikov and Nachum (2020), using deep FQE with L2 loss function.
DICE This method uses saddle-point objective to estimate marginalized importance weights d di a)/d"M (s, a); these weights are then used to compute weighted average of reward over the offline dataset, and this serves implementation from Yang et al. (2020) corresponding to the algorithm BesIDICE.
estimate of the policy's value in the MDP. We use the
Variational Power Method (VPM) This method runs variational power iteration algorithm to estimate the importance weights d= (s, a) "d (s, a) without the knowledge of the behavior policy. It then estimates the target policy value using weighted average of rewards similar to the DICE method. Our implementation is based on the same network and hyperparameters for OPE setting as in Wen al. (022)). We further tune the hyperparameters including the regularization parameter X, learning rates ag and Og. and number of iterations on the Cartpole swingup task using ground-truie policy value, and then fix them for all other tasks.
A.3 ENSEMBLING
As in Chua et al. 2018); Janner et al. (2019), we can form an ensemble using our best persfrerformnn models. We generate rollouts using the procedure detailed JnJanner et al. (2019) forming an ensemble with 4 models. We see some improvement in policy evaluation results, as shown in Figure [A.] Ensembling could likely be further improved by forcing unique hyperparameter settings and seeds.
13

---

Published as a conference paper at ICLR 2021
Carppole swingup
Cheetah run
Finger
Fish
Humanoid
turn hard
swim
run
OPE vs. ground truth
Importance Sampling
0.73 10.16 6
0.40 20.21
0.64 10.05
0.12 10.05
0.31 1:009
Regret for
Best DICE
0.68 10.41
0.27 10.05
0.44 10.04
0.35 1: 24
0.84 10.22
Variational power method Doubly Robust (IS, FQE)
0.50 10.13
0.37 10.04
0.45 10.13
0.02
0.56 10.008
0.28 10.05
0.09 00.05
0,56 10.12
0,61 1:0.12
0.99 1:0 00
FQE (1.2)
0.06 20.04
0.17 #0.05
0.30 20.11
0.500s..0s
0.99 20000
Feedforward Model
0.02 =0.02
0.24 10.12
0.43 #0.04
0.00 +0.00
0.44 #0.022
FQF (distributional)
0.03 20.09
0.11 20
0.10 10.12
0.49 10:06
0.24 :0.10
Autoregressive Model
0.00 +0.02
0.01
00.02
0.63 #0.11
0.03
0.32
Walker stand
Walker walk
Manipulator insert ball
Manipulator insert peg
Median +
OPE vs. ground truth
Importance Sampling
0.54 10.11
0.54 10.23
0.83 10.05
0.22 10.03
0.54
5 for
Best DICE
0.24 00.07
0.55 10.06
0.44 :: 0.07
0.75 40.04
0.44
Variational power method Doubly Robust (IS, FQE)
0.41 =0.02
0.39 10.02
0.52 20.20
0.32
0.41
0.02 20.01
0.05 =0.07
0.30 =0.10 10
0.70
0.30
FQE (I.2)
0.04 :0.02
0.00 20.02
0.37 20.07
0.74 00.01
0.30
Feedforward Model
0.18 +0.10
0.03 +0.05
0.83 60000
0.74 40.02
0.24
FQE (distributional)
0.03 +0.03
0.01
0.50
0.73 40.00
0.11
Autoregressive Model
0.04 60.02
0.04 +0.02
0.85 t0.02
0.30 40.04
0.04
Table A.I: Normalized R (bootstrap mean + standard deviation) for OPE methods vs. ground truth values discount factor of 0.995. each column, normalized regret values that significantly different from the best (p > 0.05) bold faced. Methods ordered by median.
Cartpole swingup
Cheetah run
Finger turn hard
Fish swim
Humanoid
run
Absolute Error btw.
OPE and ground truth
Variational power method
37.53 23.50
61 61.89
24.25
46.22 23.93
31 22 20.99
35.29
#3.00
Importance Sampling
68.75 22.30
44.29
11.1
90.10 24.68
34.82 1.99
27.89
1.98
Best DICE
22.73 #1.65
23.35
#1.22
33.52 #3.48
59.48
31.42 #2.04
Feedforward Model
6.80 +0.85
13.64
20.59
35.99
#3.00
4.75 #0.23
30.12 +2.40
FQE (L2)
19.02 #1.34
48.26
11.7
27.91
#1.18
19.82 +1.07
56.28 +3.02
Doubly Robust (IS, FQE)
24.38 #2.51
40.27
22.05
25.26
22.46
20.28
11.00
53.64
13.68
FQE (distributional)
12.63 11.21
36.50
1.66
10.23
20.93
7.76
20.95
32.36
1:2.27
Autoregressive Model
5.32
20.54
4.64
20.46
22.93 1.72
4.31
0..22
20.95
1.61
Walker stand
Walker walk
Manipulator insert ball
Manipulator insert peg
Median +
Absolute Error how.
OPE and ground truth
Variational power method
96.76 +3.59
87.24
24.25
79.25 #0.19
21.95 11.17
46.22
Importance Sampling
66.50
67.24
22.70
29.93 11.10
12.78
44.29
Best DICE
27.58 #3.01
47.28
23.13
103.45
#3.21
22.75 #3.00
31.42
Feedforward Model
23.34 12.41
52.23
22.34
34.30
12.55
121.12 1155
30.12
FQE (L2)
6.51 10.71
18.34
20.95
36.32
1107
$1.12:2.5
27.91
Doubly Robust (IS, FQE)
26.82 12.86
24.63
1166
13.33
11.16
22.28 22.34
24.63
FQE (distributional)
21.19
27.57
11.64
9.75
12.66 11.39
12.66
Autoregressive Model
19.12
+1.23
5.14
20.49
17.13
9.71 20.70
9.71
Table A.2: Average absolute error between OPE metrics and ground truth values at discount factor of 0.995.
In each column, absolute values that not significantly different from the best (p >> 0.05) bold faced. Methods are ordered by median.
14

---

Published as a conference paper at ICLR 2021
humanoid 0.46)
humanoid 0.67)
humanoid (/= 0.89)
humanoid (/= 0.91)
Fendforwarg Model
Foedforward Ensemble
Auteregressi Model
Peoorspee Ensemble A 0 Ensemble
Return 0.9951
Retum (d - 0.9951
Retum (d 0.995)
Return (d 0.995)
walker stand 0.83)
walker stand (r= 0.81)
walker stand 0.94)
walker stand 0.96)
Feedforward Ensemble
Feedforward
Autorogressivv
Return (d 0.995)
Return (d 0.995)
Return (d 0.995)
Return (d 0.995)
Figure A.I: Estimates of returns using the top model versus estimates of returns using an ensemble of the top-4 models.
Algorithm Model Predictive Path Integral Planning
Require: state 8, policy æ, dynamics model critic Q, temperature A, and noise variance o².
for m I.... M do
for I,... N do
5g f- 8
R, +0
for r = 0, O.....
do
o ((1(5))
8n r 111 π(. 18
mi 4
Rn <- Rn + + 1
end for a" 1 (+1s")
Rn + Rn H (( ;
end for Re-define и such that x (-115)) = En Σn expl 13 end for sample final action ~ En Em expl Pem / 8(a return
exp(B.18)
NN r2I). (a depends r and not 8.)
exp(Rn// 13)
B ADDITIONAL DETAILS REGARDING POLICY OPTIMIZATION
To test dynamic models for policy optimization, implement the two methods discussed in Section 5.4 on top of CRR exp, one of the CRR variants (Wang et al., 2020). We use the RL Unplugged datasets (Gulcehre al. 2020) for all environments studied in this section. When using data augmentation, we adopt a 1-to-1 ratio between the original dataset and the augmented dataset.
To take advantage of the dynamies models test time, we use a variant of Model Predictive Path Integral (MPPI) for planning. To reduce the planning horizon, we truncate the model rollout using CRR critics. The details of the planning procedure summarized in Algorithm All hyperparameter the "cartpole swingup" task. The hyperparameters
tuning for the planning process is conducted used in the planning process are M 3, N = 16, H = 10, 8 0.1, and の² 0.01. To match the temperature used in the planning component, CRR. This change, however, does not impact the baseline CRR agent performance much. With the exception of 3 and the planning component, all hyperparameters kept the
choose 3 0.1 for the CWP component of
CRR exp.
15

---

Published as a conference paper at ICLR 2021
cheetah
fish swim
finger turn hard
cartpole swingup
Undiscounted Episodic Return
CRR-planning AR
200to Use Learner Steps
vie
208
2008 2uRUU
Uoe .W08 2,00
2000 Learner Steps
Learner Steps
Learner Steps
Figure B.2: Effects of the planning procedure. Here we compare using planning (CRR-planning AR) vs not ((CRR AR)) while using augmented data generated by the autoregressive model. Planning with autoregressive models helps in all environments tested.
cheetah
cheetah
CRR data Aug. AR
Undiscounted Episodic Return
Undiscounted Episodic Return
pla-niannnng CRR-planning AR (no data aug
Learner Steps
Learner Steps
Figure B.3: Effects of the data augmentation cheetah run. [LEFT] In the absence of planning, data augmentation significantly increase the performance of CRR agent. [RIGHT] With the planning procedure, data augmentation still effective albeit to lesser extent.
We compare the agents' performance with and without the planning procedure to test its effects. As shown in Figure B.2, planning using an autoregressive model significantly increases performance.
Data augmentation does not change the agents' performance cartpole swingup, fish swim, finger turn hard. It, however, boosts performance considerably on cheetah run. In Figure B.3] we show the effects of data augmentation cheetah run.
16

---

Published as a conference paper at ICLR 2021
cheetah (r- 0.35)
cheetah 0.74)
cheetah run (r= 0.55)
cheetah run (r= 0.88)
Model
bustonabtional)
Feedforward Model
Autoregrass 3
0.995)
Retum (d 0.995)
Return (d 0.995)
finger (r-
finger turn hard (r= 0.93)
finger hard 0.09)
finger hard (r- 0.42)
reedforward Model
FOE (distribution
Autorogressive
Return [d'= 0.995)
Return 0.995)
Return (d== 0.995)
Return (d 0.995)
0.95)
0.82)
(r= 0.99)
(distributionall
Feedforward Modell
Autoreg Autoregreszivee
2
0.995)
0.995)
0.995)
manipulator (r- 0.67)
manipulator 0.60)
manipulator (r- 0.04)
manipulator ball (r- 0.38)
buttonay
Mntoegressivg Model
FOR
FOR
Return (d' 0.995)
Retum (d' 0.995]
Return (d 0.995)
Retum (d 0.9955
manipulator peg (r-
manipulator peg (r 0.48)
manipulator peg (r- 0.76)
manipulator peg (r= 0.66)
FOE (distriibutooaa) sarr
Foodforwerd Model
(12)
hutoregressive
Return id 0.995)
Retur (d 0.995)
Retum (d 0.9955
Return 0.995)
Figure B.4: Comparison of model-based OPE using autoregressive and feedforward dynamics models with distributional Bellman error. We plot ground truth returns on
state-of-the-art FQE methods based x-axis against estimates of returns from various OPE methods on the y-axis. While feedforward models fall behind FQE most tasks, autoregressive dynamics models are often superior. The remaining environments are plotted in Figure 4
17


Reviews:
[{"review_id": "pbLhyelQya", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes extra conditioning in a dynamics model, wherein each dimension of the next state is generated on previous dimensions as well as previous state and action. This allows for a richer model (as a model without conditioning on previous dimensions is a special case).The paper claims that this additional conditioning adds a better inductive bias in certain tasks.\n\n\nThe new models fit data better (Deepmind suite, RL Unplugged data) in terms of log-likelihood. The authors study the impact of better models on OPE and on policy optimization. The paper is generally well written. The contribution seems straightforward.\n\n\nWhy is this relevant only in continuous control tasks? Is this inductive bias really a general pattern observed in multiple tasks? What happens when there is no structure a priori (e.g., independence holds): do you lose in terms of sample efficiency?\n\nShouldn't any dynamics model be able to enrich replay buffer? \n\nThe default ordering may be completely arbitrary, how is the new dynamics model able to cope with this in the experiments? In other words, it is unclear if P(s_i|s_{j<i},...) is appropriate at all without knowing the ordering.\n\nAlso since p(s|s_prev,a) = \\Prod_{i}p(s_i|s_prev,a,s_{j<i}) by definition, how is the feedforward model restrictive and making explicit conditional independence assumption? It seems that the feedforward model is not restrictive but too general and explicitly capturing this sequential dependency across states is useful.\n\n>> We ultimately care not about the log-likelihood numbers, but whether or not the dynamics models are faithful in policy evaluation and optimization.\n\nThe above needs clarification. What is the faithfulness property? Also, why would we not care about log-likelihood?", "extra_scores": {}, "extra_text": {}}, {"review_id": "adb2Dse6KU6", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThe authors consider the usage of autoregressive dynamics models for batch model-based RL, where state-variable/reward predictions are performed sequentially conditioned on previously-predicted variables. Extensive numerical results are provided in several continuous domains for both policy evaluation and optimization problems. The results showcase the effectiveness of autoregressive models and, in particular, their superiority over standard feed-forward models.\n\n#### Pros\n\n- The paper is very well-written and easy to follow. The experiments are described with sufficient details to understand the results\n- The usage of these autoregressive models for model-based RL is, to my knowledge, novel\n- The paper presents extensive experiments on several challenging domains. The results are convincing and significant. In particular, they show that autoregressive models are superior to feedforward ones\n\n#### Cons\n\n- The paper's sole contribution seems to be empirical since autoregressive models are (as acknowledged) not novel, though their application to this setting is.\n- While the empirical results are very convincing, I did not find much intuition on where this big improvement over feedforward models comes from (see detailed comments below).\n- The ordering of the state variables might be a limitation (again, see below).\n\n#### Detailed comments\n\n1. As mentioned above, I did not find much intuition on the better performances of autoregressive models vs feedforward ones. As I am not entirely familiar with the system dynamics of the considered domains, do you think that they possess any property which makes autoregressive models more suitable than feedforward ones (e.g., strong correlations between next-state variables)? Aren't the transition dynamics deterministic in most of the considered domains?\n\n2. Since the reward in most of the considered domains is (I suppose) a function of state, action, and next-state, could it be that one of the reasons behind the worse performance of feedforward models is that they try to predict the reward as a function of state-action only? Would their performance change if they explicitly modeled the reward as a function of s,a,s'?\n\n3. Related to the previous point, the autoregressive model naturally predicts a reward as a function of s,a,s' since r is considered as the (n+1)-th state component. But what if we re-ordered the state variables with r as the first component instead of the last one? Would the performance change?\n\n4. More generally, do you think that the ordering of the state variables might be a limitation? For instance, could there be an ordering of these variables that makes the model perform well and one that makes it perform poorly? While in, e.g., image/text generation problems where autoregressive models are applied we have a natural ordering between the variables involved (e.g., by space or time), here there seems to be no particular relationship between state variables with similar index. Maybe some additional experiments could help in clarifying whether this could be a limitation or not.\n\nSome minor comments/questions:\n\n- In Eq. 2, should the product be up to H-1?\n- Before Sec. 3, a citation for \"behavioral cloning\" could be added\n- Sec. 5.3: the FQE acronym was not introduced\n- Fig. 4: what is \"r\" above each plot?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-E2PeTM2At", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper studies offline policy evaluation (OPE) and optimization in the model-based setting. The main methodological contribution of the paper is using autoregressive models for the next state and reward prediction. The authors demonstrate that autoregressive models achieve higher likelihood compared to feedforward models on 9 environments from RL Unplugged [1] offline dataset. Given that model likelihood is only a proxy quality metric in OPE and control, they further demonstrate a positive correlation between likelihood and OPE estimates. The paper shows quantitatively that using autoregressive models results in more accurate OPE estimates than for feedforward models and model-free benchmarks. Finally, the authors apply autoregressive models for offline control and achieve higher returns than for feedforward models.\n\n\nStrengths\n- The paper is written clearly and generally easy to follow.\n- The proposed modification is simple, straightforward to implement, and demonstrates convincing results consistently on different environments. For example, the median rank correlation between OPE and ground truth is the best against 7 OPE baselines on 9 environments from RL Unplugged.\n- The experimental setup follows the standard practices (e.g. using a validation set for hyperparameter selection) and the details necessary for the reproduction of the results are provided (e.g. optimizers, learning rate schedules, number of epochs, architectures).\n\nWeaknesses\n- The authors claim that “standard feedforward dynamics models assume that different dimensions of the next state and reward are conditionally independent given the current state and action”. In other words, p(s’,r|s,a) is claimed to be equal to p(s’_1|s,a) … p(s’_n|s,a) p(r|s,a) when using a feedforward model. The statement is incorrect unless we use a linear function approximator as a model. However, this mistake does not affect much the quality of the paper.\n- Using autoregressive models does not address aspects that are specific to the offline setting. Providing results for the online setting will be helpful for understanding whether autoregressive models should be favored in general for model-based reinforcement learning.\n\n\nRecommendation\n\nThe reviewer votes for accepting the paper. The paper is well-written, the proposed extension is simple to implement and convincingly outperforms baselines on a variety of environments.\n\n\nNotes\n1. Appendix A.2 is identical to Section 5.1 of another submission [2].\n2. The abbreviation FQE is used throughout the paper but expanded only in the appendix.\n\n\nReferences\n\n[1] Gulcehre, Caglar, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal et al. \"Rl unplugged: Benchmarks for offline reinforcement learning.\" arXiv preprint arXiv:2006.13888 (2020).\n\n[2] Anonymous. Benchmarks for deep off-policy evaluation. In Submission to International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=kWSeGEeHvF8. Under review.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is about the use of autoregressive dynamics models in the context of offline model-based reinforcement learning.
After reading the authors' responses and the other reviews, the reviewers agree that this paper has several strengths (well written, easy to follow, the approach is novel and simple to implement, the empirical evaluation is well executed and the results are reproducible) and it deserves acceptance.
The authors need to update their manuscript by keeping into considerations all the suggestions provided by the reviewers (clarifications and additional empirical comparisons).
Decision: Accept (Poster)