Variational Latent Branching Model for Off-Policy Evaluation
Authors: Qitong Gao
Ge Gao
Min Chi
Miroslav Pajic
Venue: ICLR 2023 poster (2023)
Keywords: Model-Based Off-policy Evaluation
Reinforcement Learning
Variational Inference

Abstract: Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies. Model-based OPE methods face two key challenges. First, as offline trajectories are usually fixed, they tend to cover limited state and action space. Second, the performance of model-based methods can be sensitive to the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function of MDPs by formulating the environmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Specifically, VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, we also introduce the branching architecture to improve the model’s robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed to result in varied coverage of the state-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.

Published as a conference paper at ICLR 2023
VARIATIONAL LATENT BRANCHING MODEL FOR EVALUATION
Qitong Gao"
Ge Gao!
Min Chit
Miroslav Pajic
ABSTRACT
Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies.
Mldeel-based OPE methods face two key challenges. First, offline trajectories are usually fixed, they tend cover limited state and action space. Second, the performance of model-based methods can be sensitive the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function MDPs by formulating the environmental dynamics as compact latent space, from which the next states and rewards then sampled. Specifically. VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, also introduce the branching architecture to improve the model's s robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed result in varied coverage the aate-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.
INTRODUCTION
Off-policy evaluation (OPE) allows for evaluation of reinforcement learning (RL) policies without online interactions. It is applicable to many domains where on-policy data collection could be prevented due efficiency and safety concerns, e.g., healthcare (Gao et al. 2022c a Tang & Wiens]
2021), recommendation systems (Mehrotra et al. 2018 Li et 201 1), education (Mandel et al.
2014). social science (Segal et al. 2018) and optimal control (Silver al. 2016 Vinyals et a 2019
Gao al. 2020a; 2019 2020b). Recently, as reported in the deep OPE (DOPE) benchmark (Fu et al. 2020b), basel OPE methods, leveraging feed-forward (Fu et al. 2020b) and auto-regressive
(AR) (Zhang et al., 2020a) architectures, have shown promising results toward estimating the return of target policies, by fitting transition functions of MDPs. However, modell-ased OPE methods remain challenged as they can only be trained using offline trajectory data, which often offers limited coverage of state and action space. Thus, they may perform sub-optimally on tasks where parts the dynamics are not fully explored (Fu al. (020b). Moreover, different initialization of the model weights could lead to varied evaluation performance (Hanin & Rolnick [018 Rossi et al., 2019). reducing the robustness of downstream OPE estimations. Some approaches in RL policy optimization literature use latent models trained to capture a compact space from which the dynamics underlying MDPs are extrapolated; this allows learning expressive representations over the state-action space. However, such approaches usually require online data collections as the focus is on quickly navigating to the high-reward regions (Rybkin et al. (02T), as well as on improving coverage of the explored
state and action space (Zhang et al. 2019; Hafner et al. 2019 2020a) sample efficiency (Lee et al. 2020).
In this work, we propose the variational latent branching model (VLBM), aiming to learn a compact and disentangled latent representation space from offline trajectories, which can better capture the
gao, mi roslav. pa jijjj @duke.edu. North Carolina State University, USA. Emails: {ggao5, mch (eccu.uueee Code available httptttpps.. /ttpa://giihhpp www.gaooiitooggegeeoooo.ooo
Duke University, USA. Emails: (qi gginton... gao,

---

Published as a conference paper at ICLR 2023
dynamics underlying environments. VLBM enriches the architectures and optimization objectives for existing latent modeling frameworks, allowing them to learn from a fixed set of offtine trajectories. Specifically, VLBM considers learning variational (encoding) and generative (decoding) distributions, both represented by long short-term memories (LSTMss) with reparameterization (Kingma & Welling 2013), to encode the state-action pairs and enforce the transitions over the latent space, respectively. To train such models, we optimize over the evidence lower bound (ELBO) jointly with a recurrent state alignment (RSA) term defined over the LSTM states; this ensures that the information encoded into the latent space can be effectively teased out by the decoder. Then, we introduce the branching architecture that allows for multiple decoders to jointly infer from the latent space and reach a consensus, from which the next state and reward are generated. This is designed to mitigate the side effects of model-based methods where different weight initializations could lead to varied performance (Fu et al., 2020b] Hanin & Rolnick 2018
Rossi et al. 2019).
We focus on using the VLBM to facilitate OPE since it allows to better distinguish the improvements made upon learning dynamics underlying the MDP used for estimating policy returns, as opposed to RL training where performance can be affected by multiple factors, egg., techniques used for exploration and policy optimization. Moreover, model-based OPE methods is helpful for evaluating the safety and efficacy of RL-based controllers before deployments in the real world (Gao et al. 2022b), e.g., how surgical robot would react states that are critical to a successful procedure. The key contributions of this paper are summarized as follows: (i) to the best of our knowledge, the VLBM is the first method that leverages variational inference for OPE. It can be trained using offline trajectories and capture environment dynamics over latent space, as well as estimate returns of target (evaluation) policies accurately. (ii) The design of the RSA loss term and branching architecture can effectively smooth the information flow in the latent space shared by the encoder and decoder, increasing the expressiveness and robustness of the model. This empirically shown in experiments by comparing with ablation baselines. (161) Our method generally outperforms existing model-based and model-free OPE methods, for evaluating policies over various D4RL environments (Fu et al.) 2020a). Specifically, we follow guidelines provided by the DOPE benchmark (Fu et al. (2020b). which contains challenging OPE tasks where the training trajectories include varying levels of coverage of the state-action space, and target policies are designed toward resulting in state-action distributions different from the ones induced by behavioral policies.
2 VARIATIONAL LATENT BRANCHING MODEL
In this section, we first introduce the objective of OPE and the variational latent model (VLM) we consider. Then, we propose the recurrent state alignment (RSA) term as well as the branching architecture that constitute the variational latent branching model (VLBM).
2.1 OPE OBJECTIVE
We first introduce the MDP used to characterize the environment. Specifically, an MDP can be defined as a tuple M = (S. AA P. R, 7)) where S is the set of states, AA the set of actions, P P:Sk x A- + S the transition distribution usually captured by probabilities p(s; $1-1, ( R x A + R is the reward function, $p is the initial state sampled from the initial state distribution p(so), 7 E 1) is the discounting factor. Finally, the agent interacts with the MDP following some policy (als) which defines the probabilities of taking action at state Then, the goal of OPE can be formulated as follows. Given trajectories collected by a behavioral policy 3, d²
[[(80 Go. ro, $1)...., (ST-1. 1r- (r)) (0) [s00, ap, ro, 81 s.).... (()) {(),,.. la, ~ B(a, (sx) I es- timate the expected total return over the unknown state-action visitation distribution p" of the target (evaluation) policy π - i.e., for T being the horizon.
T 7 > ((st
Breassp R
2.2 VARIATIONAL. LATENT MODEL
We consider the VLM consisting of prior p(z) over the latent variables 2 Z C R with Z repre- senting the latent space and the dimension, along with a variational encoder go (2: 24 Ot-1, 9t)
We slightly abuse the notation p², to represent either the trajectories or state-action visitation distribution under the behavioral policy, depending on the context.
2

---

Published as a conference paper at ICLR 2023
and generative decoder Pe(z,, St, 11-1121-11111111 a1-1) parameterized by db and & respectively. Basics of variational inference are introduced in Appendix
Latent Prior p(=)). The prior specifies the distribution from which the latent variable of the initial stage, 20, is sampled. We configure p(zo) to follow a Gaussian with zero mean and identity covariance matrix, which a common choice under the variational inference framework (Kingma Welling 2013
Lee et al. 2020).
Variational Encoder for Inference 4y (2 8t). The encoder is used to approximate the intractable posterior, (( (t,,,,
So
S1
To
S2
P Ja,zz 102 where the de- nominator requires integrating over the unknown latent space. Specifically, the encoder can be decomposed into two parts, given that
Z2
ho
lt
ht
lty
qu (20:3 (0:11 d0:r-1)
Ao
a1
GT-11
[0p(2o(s0) II a (a
Inference Process
Generative Process
here, (=0/(8))) encodes the initial state 8o
Figure I: Architecture of variational latent model
in to the corresponding latent variable 2o. then, go 11 0r-1 1, St) enforces the transi- (VLM) we consider.
tion from ž o conditioned on 0:-1 and 8t. Both distributions are diagonal Gaussiang with means and diagonal of covariance matrices determined by multi-layered perceptron (MLP) (Bishop 2006) and long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) respectively. The weights for both neural networks are referred to as v in general.
Consequently, the inference process for 2, can be summarized =0 ~ qu (2a]88)),
by = [w - (t),,,) = (z(11)1)))
where f represents the LSTM layer and h the LSTM recurrent (hidden) state. Note that we use vi in superscripts to distinguish the variables involved in this inference process, against the generative process introduced below. Moreover, reparameterization can be used to sample 2/ and 2 such that gradients of sampling can be back pook-propagated, as introduced in (Kingma & Welling 2013 Overview of the inference and generative processes are illustrated in Fig. I
Generative Decoder for Sampling Po Po( 8t, Tt- 1 0z-1)- The decoder is used interact
with the target policies and acts as synthetic environment during policy evaluation, from which the expected returns can be estimated the mean return of simulated trajectories. The decoder can be represented by the multiplication of three diagonal Gaussian distributions, given that
T
Po(21:r So.r, Pu:r 1 T) II Po(Se t=u
II
pa (ze 20- (r (=1
(4)
with 0r xaa, (8)) each time step. Specifically, P=(2z/=4-1;0 0111) has its mean and covariance determined by an LSTM, enforcing the transition from - to in the latent space given action a--1. In what follows, Ps (st(at) and po(r(-1(z)) generate the current state 8, and reward rt-1 given 21, whose mean and covariance are determined by MLPs. As result, the generative process starts with sampling the initial latent variable from the latent prior, i.e., =0 ~ p(=a). Then, the initial state of (=(5))=)) and action ag <aaols)) are obtained from pa and target policy n, respectively; the rest of generative process can be summarized as
h (6 2 5/ ~ (s( p (r a ((rl)))
hg (h)), of 2~~ pop))
(5)
Assume that different dimensions of the states non-correlated with each other. Otherwise, the states he projected orthogonal basis, such that non-diagonal elements of the covariance matrix will be zeros.
3

---

Published as a conference paper at ICLR 2023
Inforence
mow Liadhhood
x(5
Recurnett State
Ae =(E)
Alignment (RSA)
Civergence
Generalive
Figure 2: (Left) Recurrent state alignment (RSA) applied over the recurrent hidden states between inference and generative process illustrated separately. (Right) Single-step forward pass of the varia- tional latent branching model (VLBM), the training objectives for each branch and final predictions.
where fa is the LSTM layer producing recurrent state he. Then, an MLP do is used to generate mapping between h. and h that will be used for recurrent state alignment (RSA) introduced below, to augment the information flow between the inference and generative process.
Furthermore, to train the elements the encoder (3) and decoder ( one can maximize the evidence lower bound (ELBO), lower bound of the joint log-likelihood p(50:T, ro. following
T
T Σ Po - (T.
LELBO Φ) =E.g.
log Po (8)l2))
a. (x
(6)
here, the first two terms represent the log-likelihood of reconstructing the states and rewards, and the last two terms regularize the approximated posterior. The proof can be found Appendix E
2.3 RECURRENT STATE ALIGNMENT
The latent model discussed above is somewhat reminiscent of the ones used in model-based RL policy training methods. e.g., recurrent state space model (RSSM) used in PlaNet (Hafner al.) 2019) and Dreamer (Hafner et al. 2020a b). as well as similar ones in Lee et al. (2020); Lu et al. (2022). Such methods rely on growing experience buffer for training, which collected online by the target policy that is being concurrently updated (with exploration noise added); however, OPE aims to extrapolate returns from a fixed set of offtine trajectories which may result in limited coverage of the state and action space. Consequently, directly applying VLM for OPE can lead to subpar performance empirically; see results in Sec. 3 Moreover, the encoder above plays a key role of capturing the temporal transitions between latent variables, i.e., Py: (2)) 4 st) from (2.. However, it is absent the generative process, as the decoder leverages a separate network to determine the latent transitions, i.e., P=(=z/ze-1, (t-1). Moreover, from the ELBO (6) above it can be seen that only the Kl.-divergence terms are used to regularize these two parts, which may not be sufficient for OPE limited offline trajectories are provided. As result, we introduce the RSA term as part of the training objective, to further regularize pw ( (2 2t- Qt-1, st) and graphical illustration of RSA can be found in Fig.
4--1,
11. A
Specifically, RSA is defined as the mean pairwise squared error between h." from the encoder and h. from the decoder
i.e.
M(M
CRSA h8
Σ
(G+b) (x))))) (4 LI A [kD))
k=j+1
here, we assume that both LSTM recurrent states have the same dimension T 2 E RM,, with b. [ referring to the j-th element of the recurrent state, and N the number of training trajectories.
Here, we choose the pairwise squared loss over the classic mean squared error (MSE), because MSE could be too strong to regularize h and i which support the inference and generative processes respectively and are not supposed to be exactly the same. In contrast, the pairwise loss (7) can
Rewards and actions are omitted for conciseness of the presentation.
4

---

Published as a conference paper at ICLR 2023
promote structural similarity between the LSTM recurrent states of the encoder and decoder, without strictly enforcing them to become the same. Note that this design choice has been justified in Sec. through an ablation study by comparing against models trained with MSE. In general, the pairwise loss has also been adopted in many domains for similar purposes, e-g. object detection (Gould et al., 2009; Rocco et al. 2018), ranking systems (Doughty al.] 2018: (Saquil et al. (021) and contrastive learning (Wang et al. 2021 Chen et al. (2020). Similarly, we apply the pairwise loss over h2 and h instead of directly over h and hy, as the mapping 96 (from equation 5 could serve as
regularization layer ensure optimality over CRSA without changing h hy significantly.
As result, the objective for training the VLM, following architectures specified in (3) and 5 can be formulated
max max LV 6) v1.
(CELBO(V: の) C CRSAA (hg b v (6))))))))
(8)
vi.0
with C > and C' E R being the constant balancing the scale of the ELBO and RSA terms.
2.4 BRANCHING FOR GENERATIVE DECODER
The performance of model-based methods can vary upon different design factors (Fu et al. 2020bb
Hanin & Rolnick 2018). Specifically, Rossi al. (2019) has found that the convergence speed and optimality of variational models are sensitive to the choice of weight initialization techniques. More- over, under the typical variational inference setup followed by the VLM above, the latent transitions reconstructed by the decoder, pa (z( 1t-1 a;-1), 1), are only trained through regularization losses in (6) and 7.. but are fully responsible for rolling out trajectories during evaluation. Consequently, in this sub-section we introduce the branching architecture for decoder, with the goal of minimizing the impact brought by random weight initialization of the networks, and allowing the decoder to best reconstruct the latent transitions Pa (=+/ Fe| 1;) as well as & and r:-1 's correctly. Specifically, the branching architecture leverages an ensemble of B E Z decoders to tease out information from the latent space formulated by the encoder, with final predictions sampled from a mixture of the Gaussian output distributions from (5). Note that the classic setup of ensembles is not considered, i.e., train and average over B VLMs end-to-end; because in this case B different latent space exist, each which is still associated with a single decoder, leaving the challenges above unresolved. This design choice is justified by ablations studies in Sec. by comparing VLBM against (classic) ensemble of VL.S.
Branching Architecture. Consider the generative process involving B branches of the decoders parameterized by (dI....,on.. The forward architecture over a single step is illustrated in Fig. Specifically, the procedure of sampling =;" and Sh for each b E [1, B] follows from (5). Recall that by definition Pas (5)12)²)) follows multivariate Gaussian with mean and diagonal of covariance matrix determined by the corresponding MLPs, i.e., p(s(")) ) 0b,1" (x(""())")) and Ediag (st²) 0b
MML (=8").
In what follows, the final outcome sf can be sampled following diagonal Gaussian with mean and variance determined by weighted averaging across all branches using weights Wp 's, i.e..
St (
2B) -N N Σ wh pl((")) Edsag Σ w E (9)
b
The objective below can be used to jointly update, wb's, 'ss ib and do 's ie.,
max LVLBM Φ1 ................................ ØB, W.,...,UB WB)
0.,0.00
max
Σ
loggooss ......,"")
2² - C. (h + C2 CELNOO(s
s.t.
W1 wp 0. Σ wb 1 and constants C1, C2 0.
(10)
Though the first term above already propagates through all w/ and de 's the third term and constraints over Wo's 's regularize Φb in each individual branch such that they are all trained toward maximizing
*For simplicity, the parts generating rewards omitted without lost of generality.
5

---

Published as a conference paper at ICLR 2023
the likelihood Pe. (5t (2("). Pseudo-code for training and evaluating the VLBM can be found in
Appendix C Further, in practice, one can define w/ = and 0 < c & c, IR, the constant ensuring denominator to be greater than zero, convert (TO)
of with Us E R the learnable variables
of
into unconstrained optimization and solve it using gradient descent. Lastly, note that complementary
latent modeling methods, e.g., latent overshooting from Hafner et al. (2019), could be adopted in (TO).
However, we keep the objective straightforward, so that the source of performance improvement's can be isolated.
3 EXPERIMENTS
To evaluate the VLBM, we follow the guidelines from the deep OPE (DOPE) benchmark (Fu et al. 2020b). Specifically, follow the D4RL branch in DOPE and use the Gym- Mujoco and Adroit suites the test base (Fu et al.) (020a).. Such environ- ments have long horizons and high- dimensional state and action space, which are usually challenging for model-based methods. The provided offline trajectories for training are collected using behavioral policies at varied scale, including limited ex- ploration, human telcoperation etc.. which can result in different levels of coverage over the state-action space. Also, the target (evaluation) policies are generated using online RL training, aiming to reduce the similarity between behavioral and target policies; it introduces another challenge that during evaluation the agent may visit states unseen from training trajectories.
VLBM
AR Ens.
FQE
DR
DICE
VPM
is
0.
84
1000
3458 1902
Rank
Regret@11
MAE J
VLBM
Baselines
Figure 3: Mean rank correlation, regret@ @ and MAE over all the 32 Gym-Mujoco and Adroit tasks, showing VLBM achieves state-of-the-ar performance overall.
Environmental and Training Setup. A total of environments are provided by Gym-Mujoco and Adroit suites (Fu et al. 20200b a). Moreover, each environment is provided with 5 (for Gym-Mujoco) or (for Adroit) training datasets collected using different behavioral policies, resulting in total of 32 sets of env-dataset task full list can be found in Appendix A DOPE also provides 1 target policies for each environment, whose performance are to be evaluated by the OPE methods. They in general result in varied scales of returns, as shown in the x-axes of Fig. 7 Moreover, we consider the decoder to have = 10 branches, i.e., {M Po10 } The dimension of latent space
set to be 16, i.e., Z E Z 16, Other implementation details can be found in Appendix A
Baselines and Evaluation Metrics. In addition to the five baselines reported from DOPE, i.e.,
importance sampling (IS) (Precup 2000), doubly robust (DR) (Thomas & Brunskill 2016), variational power method (VPM) (Wen et al., 2020), distribution correction estimation (DICE) Yang et al. 2020). and fitted Q-evaluation (FQE) (Le et al. 2019), the effectiveness of VLBM is also compared against the state-of-the model-baser OPE method leveraging the auto-regressive (AR) architecture (Zhang let al. 2020a). Specifically, for each task we train an ensemble of 10 AR models, for fair comparisons against VLBM which leverages the branching architecture; see Appendix A for details of the AR ensemble setup. Following the DOPE benchmark (Fu et al.) (020b), our evaluation metries includes rank correlation, regret @ 1, and mean absolute error (MAE). VLBM and all baselines are trained using 3 different random seeds over each task, leading to the results reported below.
Ablation. Four ablation baselines are also considered, i.e., VLM, VLM+RSA, VLM+RSA(MSE) and VLM+RSA Ensemble. Specifically, VLM refers to the model introduced in Sec. 2.2 trained toward maximizing only the ELBO, i.e., proposed in existing RL policy optimization literature (Lee) et al. 2020 Hafner et al. 2019 2020a by Lu et al. 2022);; details can be found in Sec. 1 below. The VLM+RSA ablation baseline follows the same model architecture as VLM, but trained to optimize over both ELBO and recurrent state alignment (RSA) introduced in (). i.e. branching is not used comparing to VLBM. The design of these two baselines can help analyze the effectiveness of the RSA
(6). Note that, arguably, VLM could be seen as the general-
ization of directly applying
From now on the dataset names abhreviated by their initials, egg., Ant-M-R refers Ant-Mediumm-Rplpg

---

Published as a conference paper at ICLR 2023
Art
Haltcheetan Welker2d
Hopper
Ant
Haitcheetsh Walker2d
Hooper
Ant
Hallcheetan
Walker2d
Hopper
VLEM
VLM
VLM+-SA (M5S))
VIM
FOE
DR
Rank1
Rank1
Rank1
RankT
Regretig Regretig 1eotipl--eee Regretii Regret@i 4
NAE
MAE
MAE
MAET
VLBM
Ablation Baselines
Otner Baselines
Figure 4: Mean rank correlation, regret and MAE over all datasets, for each Mujoco environment.
Per
Hammar
Door
Relocate
VLBM
Ersemblle
FOE
DICE
VPM
15
Rank?
Rank1
Rank1
Rankt
Regretg1 Regretgh gll
MAE
MAEL
VIDM
Beseliee
Figure 5: Mean rank correlation, regret @ and MAE over all datasets, for each Adroit environment.
loss term and branching architecture introduced in Sec. 2.3 and 2.4 Moreover, VLM+RSA(MSE)) uses mean squared error to replace the pairwise loss introduced in and the VLM+RSA Ensemble applies classic ensembles by averaging over B VLM+RSA models end-to-end, instead branching from decoder as in VLBM. These two ablation baselines can help justify the use of pairwise loss for RSA, and the benefit of using branching architecture over classic ensembles.
Results. Fig. 3 shows the mean overall performance attained by VLBM and baselines over all the 32 Gym- Mujoco and Adroit tasks. In general VLBM leads to significantly increased rank correlations and decreased regret@| I's over existing methods, with MAEs maintained the state-of-the level. Specifically, VLBM achieves tfate-of-the-ar performance in 31, 29, and 15 (out of 32) tasks in terms of rank correlation, regret@ and MAE, respectively. Performance for each task can be found in Tables M- at the end of Appendices. Note that results for IS. VMM, DICE, DR, and FQE are obtained directly from DOPE benchmark (Fu et al., 2020b), since the same experimental setup is considered. Fig. and 5 visualize the mean performance for each Gym-Mujoco and Adroit environment respectively, over all the associated datasets. It can be also observed that the model-based and FQE baselines generally perform better than the other baselines, which is consistent with findings from DOPE.
Dist. of Branching Weights
67.0%
50.0%
33.0%
13.0%
0.6
02
05
Figure 6: Distribution of all branching weights, wy 's over all VLBMMs trained on the tasks.
The fact that VLM+RSA outperforming the VLM ablation baseline, shown in Fig. 4 illustrates the need of the RSA loss term to smooth the flow of information between the encoder and decoder, in the latent space. Moreover, one can observe that VLM+RSA(MSE) sometimes performs worse than VLM, and significantly worse than VLM+RSA in general. Specifically, it has be found that, compared to VLM and VLM+RSA respectively, VLM+RSA(MSE) significantly worsen least two metrics in 7 and 12 (out of 20) Gym-Mujoco tasks; detailed performance over these tasks can be found in Tables a the end of Appendices. Such finding backs up the design choice using pairwise loss for RSA instead of MSE, as MSE could be overly strong regularize the LSTM recurrent states of the encoder and decoder, while pairwise loss only enforces structural similarities. Moreover, VLBM significantly improves rank correlations and regrets greatly compared to VLM+RSA. illustrating the importance of the branching architecture. In the paragraph below, we show empirically the benefits brought in by branching over classic ensembles.
7

---

Published as a conference paper at ICLR 2023
Highest
Halfcheetah-Me Walker2d-M-E
Ant-M-E
Hopper-M-E
Performncce
Figure 8: t-SNE visualization over the latent space, capturing encoded state-action visitations induced from all target policies. Each point is colored by the corresponding policy from which it is generated. Policies in the legend are sorted in the order of increasing performance.
Branching versus Classic Ensembles. Fig- shows that the VLM+RSA Ensemble does not improve performance over the VLM+RSA in general, and even leads to worse overall rank correlations and regrets in Walker2d and Hopper environments. This supports the rationale provided in Sec. 2.4 that each decoder still samples from different latent space exclusively, and averaging over the output distributions may not help reduce the disturbance brought in by the modeling artifacts under the variational inference framework, e.g., random weight initializations (Hanin & Rolnick] [2018] Rossi jtt al. 2019). contrast, the VLBM leverages the branching architecture, allowing all the branches to sample from the same latent space formulated by the encoder. Empirically, we find that the branching weights, wp 's in (). allows VLBM to kill branches that are not helpful toward reconstructing the trajectories accurately, to possibly overcome bad initializations etc. Over all the the 32 tasks consider, most of VLBMs only keep 1-3 branches (out of 10), Le.. Wp < 10 for all other branches. The distribution of all wy ss from VLBMs trained the 32 tasks, are shown in Fig. 6 can observe that most of the wi, 's are close to zero, while the others generally fall in the range of (0, 0.25] and [0.75,1 1).
AR ensembles also lead to compelling rank cor- relations and regrets, but attains much smaller margins in MAEs over other baselines in gen- eral; see Fig. 3 From Fig. 7 one can observe that it tends to significantly under-estimate most of the perforeffonnng policies. Scatter plots for the other tasks can be found in Appendix A which also show this trend. The reason could be that its model architecture and training objec- tives are designed directly learn the transitions of the MDP; thus, may produce biased predic- tions when the target policies lead to visitation of the states that are not substantially presented in training data, since such data are obtained us- ing behavioral policies that sub-optimal. In contrast, the VLBM can leverage RSA and branching against such situations, thus outperforming AR ensembles in most of the OPE tasks in terms of all metries we considered. Interestingly, Fig. also shows that latent models could sometimes over-estimate the returns. For example. in Hopper-M-E and Walker2d-M-FF VL.M tends over-estimate most policies. The VL.BM performs consistently well in Hopper-M-F, but is mildly affected by such an effect in Walker2d-M-FF though over fewer policies and smaller margins. It has been found that variational inference may fall short in approximating true distributions that are asymmetric, and produce biased estimations (Yao et al. 2018). So the hypothesis would be that the dynamics used to define certain environments may lead to asymmetry the true posterior p(2 2 at-1, s.), which could be hard to be captured by the latent modeling framework we consider. More comprehensive understanding of such behavior can be explored in future work. However, the VLBM still significantly outperforms VLM overall, and achieves top-performing rank correlations and regrets; such results illustrate the VIBM'ss improved robustness as a result of its architectural design and choices over training objectives.
AR Ensemble
4
VLM
M0
VLBM
TOP Halfcheetah--M. M-E Hopser-MiE
Ant-M-E
Walker2d-M-t
Figure 7: Correlation between the estimated (y- axis) and true returns (x-axis), across different model-based OPE methods and environments.
(-SNE Visualization of the Latent Space. Fig. 8 illustratee t-SNE visualization of the latent space by rolling out trajectories using all target policies respectively, followed by feeding the state-action pairs into the encoder of VLBM which maps them into the latent space. It shows the encoded state-action pairs induced from policies with similar performance are in general swirled and clustered together, illustrating that VL.MM can learn expressive and disentangled representations of its inputs.
8

---

Published as a conference paper at ICLR 2023
4 RELATED WORK
Latent Modeling in RL. Though variational inference has rarely been explored to facilitate model- based OPE methods so far, there exist several latent models designed for RL policy optimization that are related to our work, such as SLAC [Lee et al., 2020). SOLAR (Zhang et al. 2019). LatCo (Rybkin et al., (021), PlaNet (Hafner et al., 2019). Dreamer (Hafner et al.) 2020a b). Below we discuss the connections and distinctions between VLBM and the latent models leveraged by them, with a detailed overview of these methods provided in Appendix G Specifically, SLAC and SOLAR learn latent representations of the dynamies jointly with optimization of the target policies, using the latent information to improve sample efficiency. Similarly, LatCo performs trajectory optimization over the latent space to allow for temporarily bypassing dynamic constraints. As a result, latent models used in such methods are not designed toward rolling out trajectories independently, as opposed to the use of VLBM in this paper. PlaNet and Dreamer train the recurrent state space model (RSSM) using a growing experience dataset collected by the target policy that is being concurrently updated (with exploration noise added), which requires online data collection. In contrast, under the OPE setup, VLBM is trained over a fixed set of offline trajectories collected over unknown behavioral policies. Moreover, note that the VLM baseline is somewhat reminiscent of the RSSM and similar ones as Lee et al. (2020); [u et al. (2022). however, experiments above show that directly using VLM for OPE could lead to subpar performance. On the other hand, though MOPO Yu et al. 2020), LOMPO (Rafailov et al. 202T) and COMBO (Yu et al., 202T) can learn from offline data, they focus on quantifying the uncertainty of model's predictions toward next states and rewards, followed by incorporating them into policy optimization objectives to penalize for visiting regions where transitions are not fully captured; thus, such works are also orthogonal to the use case of OPE.
OPE. Classic OPE methods adopt IS to estimate expectations over the unknown visitation distribu- tion over the target policy, resulting in weighted IS, step-wise IS and weighted step-wise IS (Precup) 2000). IS can lead to estimations with low (or zero) bias, but with high variance (Kostrikov & Nachum 2020: Jiang & Li] (016), which sparks a long line of research to address this challenge. DR methods propose to reduce variance by coupling IS with a value function approximator (Jiang & Li 2016: Thomas & Brunskill 2016 Farajtabar et al.) 2018). However, the introduction of such approximations may increase bias, so the method proposed in Tang et al. (2019) attempts to balance the scale of bias and variance for DR. Unlike IS and DR methods that require the behavioral policies to be fully known, DICE family of estimators (Zhang et al. 2020c by Yang et al. 2021 [0200] Nachum let al. 2019 Dai al., 2020) and VPM (Wen et al. (2220) can be behavioral-agnostic they directly capture marginalized IS weights as the ratio between the propensity of the target policy to visit particular state-action pairs, relative to their likelihood of appearing in the logged data. There also exist FQE methods which extrapolate policy returns from approximated Q-functions (Hao et al. 202T Le et al. 2019) Kostrikov & Nachum, 2020). Existing model-based OPE methods are designed to directly fit MDP transitions using feed-forward (Fu et al.. 2020b) or auto-regressive (Zhang et al., 2020ab models, and has shown promising results over model-free methods as reported in a recent benchmark (Fu et al. 2020b). However, such model-based approaches could be sensitive to the initialization of weights (Hanin Rolnick 2018 Rossi et al.) 2019) and produce biased predictions, due to the limited coverage over state and action space provided by offline trajectories (Fu et al., 2020b). Instead, VLBM mitigates such effects by capturing the dynamics over the latent space, such that states and rewards are evolved from compact feature space over time. Moreover, RSA and the branching can lead to increased expressiveness and robustness, such that future states and rewards predicted accurately. There also exist OPE methods proposed toward specific applications (Chen)
let al. 2022 Saito et al., 2021 Gao et al. 2023 2022b).
CONCLUSION AND FUTURE WORK
We have developed the VLBM which can accurately capture the dynamics underlying environments from offline training data that provide limited coverage of the state and action space: this is achieved by using the RSA term to smooth out the information flow from the encoders to decoders in the latent space, as well as the branching architecture which improve VLBM's robustness against random initializations. We have followed evaluation guidelines provided by the DOPE benchmark, and experimental results have shown that the VLBM generally outperforms the state-of-the-art model- based OPE method using AR architectures, well other model-free methods. VLBM can also facilitate off- off-policy optimizations, which can be explored in future works. Specificallyy, VLBM can regulator) can
serve synthetic environment on which optimal controllers (e.g.. linear-quadren be deployed. On the other hand, similar to Dreamer and SLAC, policies can be updated jointly with training of VLBM, but without the need of online interactions with the environment during training.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work sponsored in part by the AFOSR under award number FA9550-19-1-0169, and by the NSF CNS-1652544, CNS-1837499 DUE-1726550, I1S-1651909 and DUE-2013502 awards, as well as the National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant 2NS-2112562.
REFERENCES
Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.
Minmin Chen, Can Xu, Vince Gatto, Devanshu Jain, Aviral Kumar, and Ed Chi. Off-policy actor- critic for recommender systems. In Proceedings of the 16th ACM Conference on Recommender Systems, pp. 338-349, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607. PMLR, 2020.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in Neural Information Processing Systems, 33:9398- 9911,,911111 2020.
Hazel Doughty, Dima Damen, and Walterio Mayol-Cuevas. Who's better? who's best? pairwise deep ranking for skill determination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6057-6066, 2018
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference Machine Learning, pp. 1447-1456. PMLR,
2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020a. 2020a.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen, Aviral Kumar, Cosmin Paduraru, et al. Benchmarks for deep off-policy evaluation. In International Conference on Learning Representations, 2020b.
Ge Gao. Qitong Gao, Xi Yang, Miroslav Pajic, and Min Chi. reinforcement Geaning-inffrmmer pat- tern mining framework for multivariate time series classification. In International Joint Conference on Artificial Intelligence (IJCAI), 2022a.
Ge Gao, Song Ju, Markel Sanz Ausin, and Min Chi. Hope: Human- off off-policy evaluation for e-learning and healthcare. In International Conference Autonomous Agents and Multiagent Systems (AAMAS), 2023.
Qitong Gao, Davood Hajinezhad, Yan Zhang, Yiannis Kantaros, and Michael M Zavlanos. Reduced variance deep reinforcement learning with temporal logic specifications. In Proceedings of the 10th ACM/IÉEE International Conference on Cyber-Physica, Systems, pp. 237-248, 2019.
Qitong Gao, Michael Naumann, Ilija Jovanov, Vuk Lesi, Karthik Kamaravelu, Warren M Grill, and Miroslav Pajic. Model-based design of closed loop deep brain stimulation controller using reinforcement learning. In 2020 ACM/IEEE 1 1th International Conference on Cyber-Physich Systems (ICCPS), pp. 108-118. IEEE, 2020a.
Qitong Gao, Miroslav Pajic, and Michael M Zavlanos. Deep imitative reinforcement learning for temporal logic robot motion planning with noisy semantic observations. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 8490-8499 8496. IEEE, 2020b.
Qitong Gao, Stephen L Schmidt, Karthik Kamaravelu, Dennis A Turner, Warren M Grill, and Miroslav Pajic. Offline policy evaluation for learning- beasedg-bbseed deep brain stimulation controllers. In 2022 ACM/EEE 13th International Conference on Cyber-Physice Systems (ICCPS), pp. 80-91. IEEE, 2022b.
10

---

Published as a conference paper at ICLR 2023
Qitong Gao, Dong Wang, Joshua D Amason, Siyang Yuan, Chenyang Tao, Ricardo Henao, Majda Hadziahmetovic, Lawrence Carin, and Miroslav Pajic. Gradient importance learning for incomplete observations. International Conference on Learning Representations, 2022c.
Stephen Gould, Tianshi Gao, and Daphne Koller. Region-based segmentation and object detection. Advances in Neural Information Processing Systems, 22, 2009.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on PMLR, 2019.
machine learning. pp.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference Learning Representations, 2020a.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference Learning Representations, 2020b.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018.
Botao Hao, Xiang Ji, Yaqi Duan, Hao Lu, Csaba Szepesvari, and Mengdi Wang. Bootstrapping fitted q-evaluation for off-policy inference. In International Conference on Machine Learning, pp- 4074-4084. PMLR, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning, In International Conference Machine Learning, pp. 652-661. PMLR, 2016.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.611 14, 2013.
Ilya Kostrikov and Ofir Nachum. Statistical bootstrapping for uncertainty estimation in off-policy evaluation. arXiv preprint arXiv:2007.13609 2020.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna- tional Conference on Machine Learning, pp. 3703-3712, PMLR, 2019.
Alex Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. Advances in Neural Information Processing Systems, 33, 2020.
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit- based news article recommendation algorithms. In Proceedings of the fourth ACM Interna- tional Conference on Web Search and Data Mining, pp. 297-306, 2011.
Cong Lu, Philip Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206. 244.77990..47799 2022.
Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications educational games. In AAMAS, volume 1077, 2014.
Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. Towards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. In Proceedings the 27th ACM International Conference on Information and Knowledge Management, pp. 2243-2251 2018.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnosti estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32:2318-233 2019.
11

---

Published as a conference paper at ICLR 2023
Doina Precup. Eligibility traces for off- oif--poliyy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154-1168.
PMLR. 2021.
Ignacio Rocco, Relja Arandjelović, and Josef Sivic. End-to-end weakly-supervisse semantic align- ment. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6917-6925, 2018.
Simone Rossi, Pietro Michiardi, and Maurizio Filippone. Good initializations of variational bayes for deep models. In International Conference on Machine Learning, pp- 5487-5497. PMLR, 2019.
Oleh Rybkin, Chuning Zhu, Anusha Nagabandi, Kostas Daniilidis, Igor Mordatch, and Sergey Levine. Model-based reinforcement learning via latent-space collocation. In International Conference on Machine Learning, pp. 9190-9201 PMLR, 2021.
Yuta Saito, Takuma Udagawa, Haruka Kiyohara, Kazuki Mogi, Yusuke Narita, and Kei Tateno. Evaluating the robustness of off-policy evaluation. In Fifteenth ACM Conference on Recommender Systems, pp. 1 123, 2021.
Yassir Saquil, Da Chen, Yuan He, Chuan Li, and Yong-Liang Yang. Multiple pairwise ranking networks for personalized video summarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1718-1727, 2021.
Avi Segal, Kobi Gal, Ece Kamar, Eric Horvitz, and Grant Miller. Optimizing interventions via offline policy evaluation: Studies in citizen science. In Shcroty-eccond AAAI Conference on Artificial Intelligence, 2018.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. Machine Learning for Healthcare Conference, pp. 2-35.
PMLR, 2021.
Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly robust bias reduction in infinite horizon off- estimation. In International Conference on Learning Representations, 2019.
Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. International Conference on Machine Learning, pp. 2139-2148. PMLR, 2016.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforeement learning. Nature, 575(7782):350-3555 2019.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supervud visual pre-training In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3024-3033, 2021.
Junfeng Wen, Bo Dai, Lihong Li. and Dale Schuurmans. Batch stationary distribution estimation. In International Conference on Machine Learning, pp. (0203-10213, PMLR, 2020.
Mengjiao Yang. Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 13:651,-6561, 2020.
Mengjiao Yang, Bo Dai, Ofir Nachum, George Tucker, and Dale Schuurmans. Offline policy selection under uncertainty. In Deep RL Workshop NeurIPS 2021, 2021.
12

---

Published as a conference paper at ICLR 2023
Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Yes, but did it work?: Evaluating variational inference. In International Conference on Machine Learning, pp. 5581-5590 PMLR, 2018.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129- 14142, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
Marvin Zhang, Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew Johnson, and Sergey Levine. Solar: Deep structured representations for model-based reinforcement learning. In International Conference on Machine Learning, pp. 7444-7453. PMLR, 2019.
Michael R Zhang, Thomas Paine, Ofir Nachum, Cosmin Paduraru, George Tucker, Mohammad Norouzi, et al. Autoregressive dynamics models for offline policy evaluation and optimization. In International Conference on Learning Representations, 2020a.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. International Conference on Learning Representations, 2020b.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. In International Conference Machine Learning, pp- 11194- 11203. PMLR, 2020c.
13

---

Published as a conference paper at ICLR 2023
Ant
Halfcheetah
Hopper
Walker2d
Pen
Door
Hammer
Relocate
Figure 9: The Gym-Mujoco and Adroit environments considered by the D4RL branch of DOPE.
A ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
Additional Results and Discussions. Rank correlations, regret @ and MAEs for all 32 tasks are documented in Tables I 6 below The mean and standard deviation (in subscripts) over 3 random seeds are reported. Note that in each column, performance of multiple methods may be highlighted in bold, meaning they achieve the best performance and not significantly outperform each other. The fact that VL.BM outperforms the ablation baselines in most cases suggests that the RSA loss term and branching architecture can effectively increase model expressiveness, and allow to learn the dynamics underlying the MDP more accurately and robustly from offline data that provide limited exploration coverage. Yet, smaller margins are attained between the VLBM and VLM+RSA in Hopper-M-E and Hopper-M. It is likely because Hopper has relatively lower dimensional state space compared to the other three environments, from which the underlying dynamics can be sufficiently captured by the VLM+RSA. Fig. and shows the correlation between estimated (y-axis) and true IT
returns (x-axis) for all the OPE tasks we consider. It can be found that for Halfcheetah-B -M-R, -M, most of the model-based methods cannot significantly distinguish the returns across target policies. The cause could be that the offline trajectories provided for this task are relatively more challenging, compared to the other OPE tasks. Such an effect appears to affect IS, VPM, DICF, DR and FQE at larger scale. It can be observed from the scatter plots reported in the DOPE benchmark (Fu et al.) 2020b) that these methods could hardly tell the scale returns across different target policies; as the dots almost form horizontal line in each plot. However, the estimated returns from VLBM and IS still preserve the rank, which leads to high rank correlations and low regrets.
Implementation Details and Hyper-parameter. The model-based methods are evaluated by directly interacting with each target policy for 50 episodes, and the mean of discounted total returns (y = 0.995) over all episodes is used as estimated performance for the policy. We choose the neural network architectures
follows. For the components involving LSTMs, which include qu (222222 ( 4t) St) and 2e(2122111 a1-1), their architecture include one LSTM layer with 64 nodes, followed by dense layer with 64 nodes. All other components do not have LSTT layers involved, so they are constituted by a neural network with dense layers, with 128 and 64 nodes respectively. The output layers that determine the mean and diagonal covariance of diagonal Gaussian distributions use linear and softplus activations, respectively. The ones that determine the mean of Bernoulli distributions (e.g., for capturing early termination of episodes) are configured to use sigmoid activations. VLBM and the two ablation baselines. VLM and VLM+RSA, are trained using offline trajectories provided by DOPE, with mar iter in Alg. [ set to 1,000 and minibatch size set to 64. Adam optimizer is used to perform gradient descent. To determine the learning rate, perform grid search among (0.003, 0.001, 0.0007, 0.0005, 0.0003, 0.0001, 0.00005). Exponential decay is applied to the learning rate, which decays the learning rate by 0.997 every iteration. To train VLBM, we set the constants from equation 10 following C₁ C2, and perform grid search among
'Some VPM entries absent since they were not reported in Fu 2020h the code is open-sourced.
14

---

Published as a conference paper at ICLR 2023
{5, 1, 0.1.1 0.01, 0.005, 0.001, 0.0001). To train VLM+RSA, the constant C' from equation is determined by grid search among the same set of parameters above. L2-regularization with decay of 0.001 and batch normalization are applied to all hidden layers. Consider that some of the environments (e.g., Ant, Hopper, Walker2d, Pen) may terminate an episode, before timeout, the state meets specific conditions; details for VLBM to capture such early termination behavior is introduced in Appendix D
The DOPE Benchmark. The deep OPE (DOPE) benchmark (Fu al., 2020b) provides stan- dardized training and evaluation procedure for OPE works to follow, which facilitates fair and comprehensive comparisons among various OPE methods. Specifically, it utilizes existing environ- ments and training trajectories provided by D4R1 and RLUnplugged which are two benchmark suites for offline RL training, and additionally provide target policies for OPE methods to evaluate. In the D4RL branch, the training trajectories originally collected from various sources including random exploration, human teleoperation, and RL-trained policies with limited exploration; thus, can provide varied levels of coverage over the state-action space. Moreover, the target policies are trained using online RL algorithms, which can in general lead to different state-action visitations than in the training trajectories. We leverage the D4RL branch as our test base, since the OPE tasks it provides are considered challenging, i.e., the limited coverage introduced by training data, as well as the discrepancy between the behavioral and target policies. Graphical illustrations of the Gym-Mujoco and Adroit environments considered are shown in Fig. g Details on the environments and datasets used are shown in Tables 7 and X from the perspectives
state and action dimensions, if episodes can be terminated before timeout, if controls are performed over continuous space, and the size of the offline trajectories used for training. In contrast, in the RLUnplugged branch, the training trajectories are always collected using online RL training, which can result in adequate coverage over the sate-action space. The target policies are trained by applying offline RL over the training trajectories, so that behavioral and target policies can lead to similar distributions. As discussed in DOPE (Fu et al. (020b), such tasks are suitable for studies where ideal data are needed, such as complexity comparisons.
visitation
Evaluation Metries. Following from (Fu et al. 2020b), we consider rank correlation, regret@ I and mean absolute error (MAE) as the evaluation metries. Specifically, rank correlation measures the strength and direction of monotonic association between the rank of OPF-estimmted returns and true returns over all target policies. It is is captured by Spearsman's correlation coefficient between the ordinal rankings between estimated and true returns. Regret@1 is captured by the difference between the return of the policy corresponding to the highest return as estimated by OPE and the return of the policy that actually produces the highest true return. In other words, regret @ I evaluates how worse the policy resulting in the highest esttmmatd return would perform than the actual best policy. The two metrics above evaluate how useful OPE would be to facilitate important applications such as policy selection. Finally, we also consider MAE which is commonly used in estimation/regression tasks. Mathematical definitions of these metries can be found in (uu et al. 2020b).
Implementation of AR Ensembles. For fair comparisons with VLBM, in experiments we train an ensemble of the sate-of-the-arr model-based OPE method, auto-regressive (AR) models (Zhang) et al. 2020a), one of the baselines. Specifically, we train an ensemble of 10 AR models to learn (8++1, r2/0 St, ax) following the auto-regressive manner, with each individual model following the design introduced in (Zhang et al. 2020a), i...
S+1 ~ p(
18,, 01 8
(1)
(-1))
(11)
(0)) representing the element located at the j-th dimension of the state variable, and D the with 8:41
dimension of state space. The reward is treated as an additional dimension of the states, i.e.
re ((r ae, s1+1:..... .(D)). However, in the original literature (Zhang et al. (2020a) it does not introduce in details regarding which specific ensemble architecture is used (e.g., overall averaging or weighted averaging). As a result, we choose the same weighted averaging procedure as used in VLBM branching, to sort out the influence of different ensemble architectures and facilitate fair comparisons. Specifically, a total of 10 AR models, parameterized by (0)....,010 010}. along with 10
(1)
15

---

Published as a conference paper at ICLR 2023
weight variables {wf. ....... wig wff = 1}. are trained. Similar to weighted averaging architecture
used in VLBM, i.e., equation
9 the mean and variance of the prediction (
8411 captured by normal
distribution N (u, o²), follow
10
10 b41h o² =
E,_(((u0 (w,)² oā. (5111)
w po,
(12)
where me: (s and 0, ($f41) are the mean and variance produced from each individual AR model in the ensemble.
Training Resources. Training of the proposed method, and baselines, facilitated by Nvidia Quadro RTX 6000, NVIDIA RIX A5000, and NVIDIA TITAN XP GPUs.
License.
The use of DOPE and D4RL (Fu al. 2020a) follow the Apache License 2.0.
16

---

Published as a conference paper at ICLR 2023
AR Ensemble
202
1315
4607
354
723
1011
303
-100
105
533
1976
3008
-221
-33
354
142
128
197
168
1316
-472
515
1631
263
0.00
VLM
24
722
-105
125
033
1972
3003
-021
-31
354
140
378
538
207
750
1316
529
3033
200
VLBM
1970
354
1611
-10
311
1092
2003
-22
-33
142
-228
207
1319
-472
3051
2052
Halfcheetah-E
Hopper-E
Ant-E
Walker2d-E
AR Ensemble
1309
797
537
5677
237
727
210
-315
237
223
-035
1252
8000
VLM
201
517
237
220
202
237
VLBM
702
537
257
225
25
Halfcheetah-Me
Hopper-M-E
Ant-M-E
Walker2d-M-E
AR Ensemble
600
315
2577
3310
2140
100
1319
1730
000000
VLM
660
155
25/4
106
658
219)
11
1222 2002
-359
11.0
357
1254
2100
1652
2330
1220
VLBM
1000
850
1453
20d
-015
11
1322
2002
-355
1119
357
707
1201
Halfcheetah-te M
Hopper-M
Ant-M
Walker2d-M
Figure 10: Scatter plots between OPE-estimated (y-axis) and true (x-axis) returns over all 20 Gym- Mujoco tasks that are considered. Part 1.
17

---

Published as a conference paper at ICLR 2023
AR Ensemble
2130
1027
1372
644
293
324
559.6
0000
3043
VLM
1427
1110
142-
054.5
000
-128
660
1437
2100
-224
253
1809
1927
00
324
149
1312
45.1
299.
1943
9883
2190
1927
1372
053.
VLBM
044
202
324
235.6
1413
Halfcheetah-Me
Hopper-M-R
Ant-M-R
Walker2d-M-R
AR Ensemble
1425
379
113-
675
-122
300
650
-425
2100
-40
2200
-13
3310
209
675
1002
VLM
850
-42
357-
2091
000
122.205
-40
1130
VLBM
660
209
2005 160
1425
2300
-459
-40
379
159
-200
307
713
3330
-50
230
535
1092
Halfcheetah-R
Hopper-R
Ant-R
Walker2d-R
Figure 11: Scatter plots between OPE-estimated (y-axis) and true (x-axis) returns over all 20 Gym- Mujoco tasks that are considered. Part 2.
18

---

Published as a conference paper at ICLR 2023
B MORE t-SNE VISUALIZATIONS
Performonce
Highest
Halfcheetah-M-EE
Walker2d-M-E
Ant-M-E
Hopper-M-E
10 Performonce
Figure 12: 1-SNE visualization over the latent space captured by VL.,, illustrating encoded state- action visitations induced from all target policies. Each point is colored by the corresponding policy from which it is generated. Policies in the legend are sorted in the order of increasing performance.
Lowest Performance
Highesst Perforsance
Halfcheetah-M-E
Walker2d-M-E
Ant-M-E
Hopper-M-E
Figure 13: 1-SNE visualization over the latent space captured by VLM+RSA(MSE), illustrating encoded state-action visitations induced from all target policies. Each point is colored by the corresponding policy from which it is generated. Policies in the legend are sorted in the order of increasing performance.
Figures 12 and 13 above visualize the latent space captured by two ablation baselines, VLM and VLM+RSA (MSH). respectively. can observed that comparing to the latent space captured by VLM are not disentangled well compared to VL.BM (shown in Figure 8). as the state-action pairs induced by policies with different levels of performance are generally cluster together without explicit boundaries. Such a finding illustrated the importance of the use of RSA loss (7) empirically, as can effectively regularize (22 41-1, a,-1, &t) and allows the encoder to map the MDP states to expressive and compact latent space from which the decoder accurately. Moreover, Figure 13 shows that the latent representations of the state-action pairs captured by VIM+RSA(MSE distributed almost uniformly over the latent space. This justifies the rationale provided in Sec. 2.3 where MSE is too strong to regularize the hidden states of the encoder and decoder, and also consistent with the results reported in Figure that MSE+RSA(MSE) performs worse than VLM in general.
reconstruct states and rewards
19

---

Published as a conference paper at ICLR 2023
C ALGORITHMS FOR TRAINING AND EVALUATING VLBM
Algorithm Train VLBM.
Input:
Model weights th 01, Un, Wt, ....... wn, offline trajectories p², and learning rate
Begin:
1: Initialize v. .1...., ØB, W1...... wB
2: for iter in 1 mar mar_iter do
3: Sample trajectory [(so, do, ro, 81),-... 10-, p
4: =0 440 (2g1s))
5: 20 ~ p(zo)) for all b E [1, 1
6: Run forward pass of VLBM following 3 5 and 9 for 1 T, and collect all variables
needed to evaluate LVLBM as specified
7:
3 4 aV CVLBM
8:
for b in B do
9:
Φ - 6b + a LVLBM
10:
whb w/b a wa LVLBM
11:
end for
12:
end for
Algorithm 2 Evaluate VLBM.
Input:
Trained model weights t), . Un, WI...., w
Begin:
1: Initialize the list that stores the accumulated returns over all episodes R =
2: for epi in 1 max_ey epi do
3: Initialize the variable r = 0 that tracks the accumulated return for the current episode
4: Initialize latent states from the prior, i.e., =0" p(zo) for all b E [1. B]
5: Initialize LSTM hidden states h 0 for all b [1,
6: Sample de ~ (Po 1=; for [1, B] and generate initial MDP state of following
7: for in 1 ' do
8: Determine the action following the target policy म. i.e. a ~ π (a 15
9: for b in 1: B do
r following 10: Update h. Rr. 26 to
end for
12:
Generate the next state & following as well as the reward r
Pø(r=-1l27 ....,. 20 N E. wh u(r (()r-1): Eding En wB Eding (1e
13:
Update < + 1- 1-1 with being the discounting factor
14: end for
15:
Append into R
16: end for
17:
Average over all elements in R, which serves the estimated return
20

---

Published as a conference paper at ICLR 2023
D EARLY TERMINATION OF ENVIRONMENTS
Given that some Gym-Mujoco environments, including Ant, Hopper, Walker2d and Pen, may terminate an episode before reaching the maximum steps, if the state violates specific constraints. Below we introduce how VLM and VLBM can be enriched to capture such early termination behaviors.
VLM For VLM, we introduce an additional component de Po(de =2") to the generative pro- cess equation 5 where de is Bernoulli variable determining episode should be terminated its t-th step. Specifically, po(de jaf") follows Bernoulli distribution, with mean determined by an MLP with sigmoid activation applied to the output layer. As result, the generative process now follows
h = (0 0t-)).
i = gg()) ~ pppo)
of (s)22)))
r po(ro-ila"), de (d:l)==))
(a.)s)))
(13)
Moreover, we add in a new term to VLM's training objective, in order to update the component introduced above during training, i.e..
LVLM
pearly
(v,0) = LVLM(U, (v, Φ) + Σ log Ps (delzz),
(14)
with LV (V, Φ) being the original objective of VLM, as presented in equation
VLBM For VLBM, the termination of an episode is determined following, i.e.,
df ~ Poldeel
2 = Bernoulli(p= = Σ wb (a((d")))
(15)
where ua(df") ML 0b) (2()) is the mean of dehh produced from the b-th branch of the decoder, and MLP is the corresponding MLP that maps 2hhh to Ha(d?)). To update the components involved in Dbpe the procedure above, we introduce
term to the VLBM's objective, i.e.,
Cearly, (v, $1...., ΦB, W1. ww)
(16)
-LyL.M oi...., on, wT." wn) + Σ
log lo "
(17)
with LVLBM being the original objective of VLBM, as presented in equation
T0
21

---

Published as a conference paper at ICLR 2023
E BOUND DERIVATION
We now derive the evidence lower bound (ELBO) for the joint log-likelihood distribution, i.e,
log (o
(18)
=log
(o(0o.T,210 21:T, Yo:TYYoTT-i)d
(19)
FITTZ
Po 21:7, (oTT-1)
= -log
Z1:rE qu qu (20:T) So.r (0rr-1 Qu (20:r//0::: ao.r-
(20)
gg [logp(2)) p(2)) log ( 21,T, No:r-11200 log qv (Zuer borr, ao:--))
(21)
-Bgo
=
logp(20) + log (s 20) + Σ t=1
log Polp:s 2e, r 1e- 121,
-o 80)
1=1 log (2
(22)
=B -Bao
log (0) - log (20ls0) + log (ogrz(50120) + Σ 1=1 log bog.e
log 2o Lo (2 94)
(23)
=Bqo
log Po + E 1=1 log log
([( En, KL (2)2,,,,, 01-1 (t), san.l//.......ll/l////./.a..aaa
(24)
Note that the transition from equation 20 to equation 21 follows Jensen's inequality.
22

---

Published as a conference paper at ICLR 2023
F BASICS OF VARIATIONAL INFERENCE
Classic variational auto-encoders (VAEs) are designed to generate synthetic data that share similar characteristics than the ones used for training (Kingma Welling 2013). Specifically, VAEs learn an approximated posterior gy. (2(zr and generative model po(x|z), over the prior p(z), with a being the data and the latent variable, It's true posterior po (2(22) (z) is intractable, i.e.
P=(2|1)= P=(())2)p(=))
(25)
(x(
since the marginal likelihood in the denominator, Pa (2) = Jg (2(2)2)p(2)dz, requires integration over the unknown latent space. For the same reason, VAES cannot be trained directly maximize the marginal log-likelihoode max log (x)). To resolve this, one could maximize a lower bound of Po(x), i.e,
KT(Go Eg llg Pa (x/2))
(26)
which is the evidence lower bound (ELBO).
During training. is required to sample from (=(a)) and pa (x|²) constantly, The reparameterization technique is introduced in (Kingma & Welling 2013). to ensure that the gradients can flow through such sampling process during back-propagation. For example, both distributions (1w(2/z)) and p)(x]z)) follow diagonal Gaussians, with mean and diagonal covariance determined by MLPs, i.e.,
Reparameterization.
2 ~ (2(2))) = NN 1/ MLP (x),
= = DE (L))())
MLP
(I)
(27)
x :~ a~pp(El)) N NN(1= (m= of
MLP (z),
Σ S
(28)
here, 1/ W.LPP MLP DE . MLP WLPP are the MLPs that generate the means and covariances. The sampling processes above can be captured by reparameterization, i.e.,
2 = 1P (x) + MLP (x).e,
1/5 (x))
(29)
MLP Op (=) or L
(30)
with (0, 1). Consequently, the gradients over and Φ can be calculated following the chain rule, and used for back- pack-propagation during training. We direct readers to (Kingma & Welling 2013 for comprehensive review of reparameterization.
23

---

Published as a conference paper at ICLR 2023
G ADDITIONAL RELATED WORKS
Overview of latent-model based RL methods. SLAC, latent representations are used to improve the sample efficiency of model-free RL training algorithms, by jointly modeling and learning dynamics and controls over the latent space. Similarly, SOLAR improves data efficiency for multi-task RL by first learning high-level latent representations of the environment, which can be shared across different tasks. Then, local dynamics models are inferred from the abstraction, with controls solved by -near-quadratic regulators. PlaNet and Dreamer further improve the architecture and training objectives of latent models, allowing them to look ahead multiple steps and plan for longer horizon. There also exist LatCo which directly performs trajectory optimization over the latent space, allowing the agent to temporarily bypass dynamical constraints and quickly navigate to the high-reward regions in early training stage. To summarize, methods above leverage latent representations to gain sufficient exploration coverage and quickly navigate high-reward regions, improving sample efficiency for policy optimization. Note that they mostly require online interactions with the environment to formulate growing experience replay buffer for policy learning, which have different goals than OPE which requires learning from a fixed set of offline trajectories.
24

---

Published as a conference paper at ICLR 2023
Ant -B
Ant
Ant
Rank Corr.
-M-E
-M
-M-R
-R
IS
.1.41
.1.35
.7.32
.7.39
.26.34
VPM
- -42.38
.288288
2.31
.26.29
.24.31
DICE
-13..7
- 33,44
36
.24.39
-.21.35
DR
28
35,35
.66 26
.5,32
.01.33
FQE
13.32
.7.35
.65. 25
.57. 28
.04.33
AR Ensemble
40
24.25
.56011
.4.16
A8.17
VLM
.28
.39.16
.37.03
.37.19
.36,07
VLM+RSA (MSE)
.33.11
.29
.35.22
.30.42
.17.14
VLM+RSA
40.03
.53.19
.42.22
.53.19
.4..11
VLM+RSA Ens.
62.16
76,02
.65.07
.62.18
0..50
VLBM
.7.011
81.05
.65.0a
.59.14
.78.24
Halfchectah
Halfcheetah -M-F
Halfcheetah -M
Halfcheetah -M-R
Halfcheetah -R
Rank Corr.
-F
IS
.01.55
.06377
.00
.99
-224.38
VPM
.18 35
.7.29
-.07.36
.2.33
DICE
-.4..00
08.35
.26.u7
-15.41
-.70.2
.77.17
62.27
.32.22
.32.37
-002.38
FQE
.78.15
.2.27
.3.17
.26.37
-.11.41
AR Ensemble
.65.11
.65
.60.0a
.59
.60
VLM
.7.19
.45
.33.1
.64.
.4.0u
VLM+RSA (MSE)
54.31
.49.B uB
.6,08
.47.11
.13,u7
VLM+RSA
80.17
.4
.65,21
.61.ua
.51.08
VLM+RSA Ens.
71.
66.0s
.64.o2
.60
.45.77
VLBM
88,01
.74
.81 13
.64
.60
Walker2d
Walker2d -M-E
Walker2d -M
Walker2d -M-R
Walker2d
Rank Corr.
EE
R
IS
22.37
24
.2.335
.65 24
.05 38
VPM
.17 32
.49.r .37
.44,21
.52
.42..4
DICE
.37.22
-34..34
.12.38
.5.23
-99.36
DR
26.34
19.33
.02.
.37.39
.16.29
FQE
.3.33
.2.32
-.09.86
.19.36
.21.31
AR Ensemble
54.11
.25.33
.55.14
.38.17
.3.29
VLM
.57.13
.16.13
.18,30
.39
.4.
VLM+RSA (MSE)
.7.228
.20.u5
.09.18
.10.11
.3.199
VLM+RSA
56.11
.57
.46
.4.14
.59
VLM+RSA Ens.
62,
.57
.43
-.14.0
.3..4
VLBM
.70.3
.55.17
.66
.60.o7
.72
Hopper
Hopper -M-E
Hopper
Hopper -M-R
Hopper R
Rank Corr:
IS
37.27 27
.35g 26
-55. 28
.66.0 o3
23.34
VPM
.21.32
.13.37
.16.as
.46.20
DICE
.08.32
.08 14
.19.33
.7.28
-.13 -.13.99
DR
.4
-08.30
.31.44
.05.17
-.19.38
FQE
.33.30
.0L.ss
.229
.45.1
-.11.96
AR Ensemble
23 30
.14.29
.53
.28.
.26.
VLM
.00
.22.11
.3.08
.46.2
.36
VLM+RSA (MSE)
- -188
.05.0
.5.200
.43
.58.14
VLM+RSA
.15
.26.10
.51.11
.53.06
.55.18
VLM+RSA Ens.
.09 21
.13.12
-.01.s
.66.07
.63. 16
VLBM
28.16
.32.10
.70.03
.75.07
.77.04
Table 1: Rank correlation between estimated and ground- -ruth returns for all Gym- tasks. Results are obtained by averaging over 3 random seeds used for training, with standard deviations shown in subscripts.
25

---

Published as a conference paper at ICLR 2023
Door human
Door cloned
Door
Pen
Pen cloned
Pen
Rank Corr.
expert
human
expert
IS
-12..35
.6622
76.17
.28
.71.08
.45.31
VPM
-.29.36
65.23 23
.08.33
DICE
.002 20
.18.31
.06632
.17333 33
.077 26
-53..0
DR
.1.18
.60
76.13
-336,29
.9.25 25
.52,
FQE
.07.09
.55.27
.89.09
-31..21
.06
-.01.33
AR Ens.
58.06
.52.13
.61 .07
.3.07
.4,os
.60 .99
VLBM
80,14
.78 18
.93.00 03,03
.4.17
.8,,07
.58. 15
Hammer human
Hammer cloned
Hammer expert
Relocate human
Relocate cloned
Relocate
Rank Corr.
expert
.39.07
.58.27
64.24
-23.07
-.22.18
.52. 23
VPM
-.77.22
39.31
.9.31
DICE
11
.35,8
-42.31
-3.3
.22.16
-.27 31
DR
-04
-.70.
.4.31
.65 19
.10.16
-.40 21
FQE
.14.10
-.15,33
.29.34 .4
.62.11
.15.17
-.57 .28
AR Ens.
.44.22
.40,20
.53.11
.42233
.30.
.54 23
VLBM
.34.14
.58.18
.70.20
.68.17
.80.04
.58.17
Table Rank correlation between estimated and ground tround-truth returns for all Adroit tasks. Results are obtained by averaging over random seeds used for training, with standard deviations shown in subscripts.
26

---

Published as a conference paper at ICLR 2023
Ant -E
Ant
Ant
Ant
Ant
Regret@1
-M-E
-M
-M-R
-R
IS
A7.32
.46.18
.61.18
.16.28
.56
VPM
88.22
.32.24
.,21
.72.43
.15.20 14.24
DICE
.62.15
.60.16
.4.1
.64.13
.0.299
DR
A3,22
.37.13
.12.18
.05.09
.28.15
FQE
43,2
.36.14
.12118
.05
.28 15
AR Ensemble
.1,09
.17.20
.050
.3..20 20
.03 02
VLM
38,
.07.00 .22
.20.25
.08 02
.14
VLM+RSA (MSE)
.50.
.26.21
.28,4
.48,33
.43.44
VLM+RSA
.1
.13.12
.14.16
.17.24
.07
VLM+RSA Ens.
.1.0s
.05o.
.03.02
.03.02
.2.37
VLBM
.05p.
.05o.
.05p.
.11.09
0.o.
Halfcheetah -E
Halfeheetah -M-E
Halfcheetah -M
Halfcheetah -M-R
Halfcheetah -R
Regret@) @1
IS
11,,08
.73 42
.05 .05
.13,10
.31 11
VPM
.14,09
.80.34
.33.19
.2,00
.12.07
DICE
.3.410
.38,37
.82.29
.30,07
.81.30
DR
.11 .o8
.1.07
.3.115
.33.18
31
FQE
.12.07
.14.77
.38.13
36
.3.
AR Ensemble
.02.o3
.11.07
.13.
.07 .05
.04 or
VLM
.11.04
.12.06
.25.01
.04.o3
.23p.
VLM+RSA (MSE)
.09 .08
.2,099
.20
.09 08
.27 .05
VLM+RSA
08
.17.05
.09.12
.02.03
.230.
VLM+RSA Ens.
.1.05
.19
.07
.02
.69
VLBM
14.04
.09.02
0.o.
.07.0g
.15.07
Walker2d
Walker2d -M-E
Walker2d -M
Walker2d -M-R
Walker2d R
Regret@ @
-E
IS
43
.13,77
.70,39
.02.05
.74.3
VPM
.09.
24.42
.08.n6
.46.11
.8820
DICE
.5.36
.78.27
.27.43
.18.12
.3.33
DR
.06 07
.30
.25099
.68.23
.15
FQE
.06 06
.22.14
.31
.4.20
.15
AR Ensemble
.1.111
.17.19
.16.15
.1416
.66 02
VLM
.10.05
.51,25
.0.39
.33.8
.08 o7
VLM+RSA (MSE)
49.16
.39.00
.43.3
.860.
.31.29
VLM+RSA
.10.07
.11.22
.18.15
.3.37
.08 05.04
VLM+RSA Ens.
.1..44
.14.16
.02.02
.86.
.58.20
VLBM
.05
.05
.03 04
.1.16
.06 06
Hopper -E
Hopper -M-E
Hopper -M
Hopper -M-R
Hopper
Regret@ @1
-R
IS
.06 o3
.00
.38 28
.80.
.05 05
VPM
.13
.10,14
.26.10
DICE
.20
.1.08
.18.19
.16.13
.30
DR
.3.35
.3439
.32.22
.3424
.41
FQE
A1
.42.
.3232
.18
.3622
AR Ensemble
.07 .05
.23111
.14.00
.06.02
.12.11
VLM
.76
.35.22
.22
.14
.07 02
VLM+RSA (MSE)
.4.31
.51.
.33.39
.26
.06 04
VLM+RSA
.62.
.18
.13
.25
.33.3
VLM+RSA Ens.
31.18
.51o.
.47.36
.03.02
.06
VLBM
.10.03
.10
.11.11
.040.
.03 04
Table 3: Regret@1 I for all Gum-oujoco tasks. Results are obtained by averaging over 3 random seeds used for training, with standard deviations shown subscripts.
27

---

Published as a conference paper at ICLR 2023
Door
Door cloned
Door
Pen human
Pen cloned
Pen
Regret@1
human
expert
expert
IS
.45.00
02.07 .07
.01 04
.1715
.1400 .99
33.10
VPM
.69.24
.1.33
.0..03
.28.12
.36
.25.33
DICE
.10.27
.65.45
.3.227
.0.09
.12.0s
.33.20
DR
.05 09
.11
.05.0
.09 08
.13 .6
.05.0
FQE
.05 08
.11.06
.03.o3
.07.05
.12.07
.11.14
AR Ens.
.08 10
.4,31
.10
.09 .09.08
.1.05
.08.0n
VLBM
.03 04
.03 .4
00..03
.2.07
.08 06
.09.02
Hammer human
Hammer cloned
Hammer expert
Relocate human
Relocate cloned
Relocate expert
Regret @1
.19.00
.03.15
.01 .04
.63.11 41
.63.
.18.14
VPM
.18
.72,30
.04
.77.18
.11
.76.23
DICE
.04.os
.67.48
.24.44
.97.11
.96
.97.07
DR
.46
.78
.09
.17.15
.18.27
.980ss 00.8
FQE
.46
.36.3 39
.05.00
.17.14
.29.12
1.00
AR Ens.
.08 06
.05.05
0.o.
.26,33
.63.35
.26,33
VLBM
.08o.
0.o.
.01
01
.08
.02.02
.07.07
Table Regret@ for Adroit tasks. Results are obtained by averaging over random seeds used for training, with standard deviations shown in subscripts.
28

---

Published as a conference paper at ICLR 2023
Ant -E
Ant
Ant
Ant
Ant
MAE
-M-E
-M
-M-R
-R
IS
605
604102
594104
603
606103
VPM
607 10s
604106
570
612105
570ge
DICE
158
170
495p0
583110
530g2
DR
584114
32666
345e6
42172
404106
FQE
123
319
34564
41079
398111
AR Ensemble
551g1
629
57435
6421
57561
VLM
33115
31520
31031
4866
6632
VLM+RSA (MSE)
34313
3244
3063
46321
6618
VLM+RSA
3517
31423
30525
4483
6654
VLM+RSA Ens.
24220
31237
34580
4646
66720
VLBM
2024
269s5
33143
2652
59811
Halfcheetah
Halfcheetah -M-E
Halfcheetah -M
Halfcheetah -M-R
Halfcheetah
MAE
-E
IS
1404
1400
1217
1409154 54
1405
VPM
945164
1427111
1374153
1384 48
1411 154
DICE
944161
1078132
1382130
1440158
1446156
DR
1025gB
1015103
1222131
1001129
949126
FQE
103195
1014101
1211130
1003
938125
AR Ensemble
1226222
48024
55364
84664
153716
VLM
520242
52649
624g3
147827
14901
VLM+RSA (MSE)
159
42649
689g9
1432
1489o
VLM+RSA
414155
446%0
622
147320
1492s
VLM+RSA Ens.
253
773139
1306113
1468
152522
VLBM
20122
45630
51750
1281170
14952
Walker2d -E
Walker2d
Walker2d
Walker2d
Walker2d
MAE
-M-E
M
-M-R
R
IS
40562
436G2
42860
427e0
430G1
VPM
367es
42561
42660
42464
44058
DICE
43760
32260
273s1
374s1
41957
DR
519
21746
36874
29654
34774
FQE
453142
23342
35079
31373
35473
AR Ensemble
530102
408
4446
327
383
VLM
53830
38012
25017
160G6
452g
VLM+RSA (MSE)
52130
34020
36119
236
44315
VLM+RSA
52241
358s6
253g
1257
326161
VLM+RSA Ens.
538g
38623
201
16811
44124
VLBM
51724
28872
244a3
15628
26222
Hopper
Hopper -M-E
Hopper -M
Hopper -M-R
Hopper
MAE
-E
RR
S
10629
36047
40548
438
41245
VPM
44243
43344
43844
DICE
259:1
26640
215
3982
12216
DR
42699
23477
307
298
28950
FQE
28276
25228
28373
2957
261
AR Ensemble
36916
29211
393
47734
45434
VLM
14831
13619
210,2
138g
382
VLM+RSA (MSE)
24640
18610
232.9
12412
41515
VLM+RSA
270g
14015
11728
11716
41220
VLM+RSA Ens.
25323
14942
23362
11517
30647
VLBM
266g
140,
12647
12421
38527
Table 5: MAE between estimated and ground-truth returns for all Gym-Mujoco tasks. Results are obtained by averaging over 3 random seeds used for training, with standard deviations shown in subscripts.
29

---

Published as a conference paper at ICLR 2023
Door
Door
Door
Pen
Pen
Pen
MAE
human
cloned
expert
human
cloned
expert
IS
870173
891188
648122
3926128
1707128
4547222
VPM
862
1040
879182
1569215
2324
2325138
DICE
1108
69779
856134
4193244
1454219
2963279
DR
379G5
42473
1353218
2846200
1323gs
2013561
FQE
38960
43881
1343g1
2872170
1232105
1057281
AR Ens.
734s
8267
223616
216112
1981106
1803226
VLBM
710152
9331
600s4
1637286
669270
1002262
Hammer human
Hammer cloned
Hammer expert
Relocate human
Relocate cloned
Relocate
MAE
expert
73521118
74031126
3052608
638217
632215
2731
VPM
7105
7459
7312
806
586135
620214
DICE
5677936
4169839
3963758
4526474
1347485
1095221
DR
5768751
6101 679
3485590
606116
412121
1193350
FQE
6000
5415558
2950728
593118
439125
1351393
AR Ens.
689727
7240
3057g
8237
662g
21384
VLBM
6184479
7267
2682146
62425
388183
2021
Table MAE between estimated and ground-truth returns for Adroit tasks. Results are obtained by averaging over 3 random seeds used for training.
30

---

Published as a conference paper at ICLR 2023
State Dim.
Action Dim.
Early Term.
Continuous Ctrl.
Dataset
Datasct Size
random
999,427
medium- 301,698
Ant
27
8
Yes
Yes
replay
medium 999,175
medium- expert
1,998,158
expert 999,036
random 999,000
medium- 201,798
Halfcheetah
17
6
No
Yes
replay
medium 999,000
medium- expert
1,998,000
expert 999,000
random 999,999
medium- 401,598
Hopper
Yes
Yes
replay
medium 999,998
medium- 1,998,966
expert
expert 999,061
random 999,999
medium- 301,698
Walker2d
17
6
Yes
Yes
replay
medium 999,322
medium-
expert 1,998,318
expert
999,000
Table 7: Summary of the Gym-Mujoco environments and datasets used to train VLBM and baselines.
State Dim.
Action Dim.
Early Term.
Continuous Cirl.
Dataset
Dataset Size
human
4,975
Pen
45
24
Yes
Yes
cloned
496,264
expert
494,248
human 6,704
Door
28
No
Yes
cloned 995,642
expert 995,000
human 11,285
Hammer
46
26
No
Yes cloned
996,394
expert
995,000
human 9,917
Relocate
39
30
No
Yes
cloned
996.242
expert
995,000
Table 8: Summary of the Adroit environments and datasets used to train VLBM and baselines.
31


Reviews:
[{"review_id": "m3QJGBg5i-", "reviewer": "Reviewer_QLEv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This papar focuses on the model-based off-policy evaluation(OPE) method, where there exist both limited coverage of state-action space and sensitive initialization during training the transition model. To address these, this paper presents the variational latent branching model(VLBM) to obtain as much information from the limited offline data and improve its robustness, through variational inference framwork with the Recurrent State Alignment and Branching Architecture of decoders respectively.", "summary_of_the_review": "This work proposed a variant of VAE in which both encoder and decoder are LSTM to model the trajectory of a RL agent and use this for OPE. I think that this is less related to the problem of OPE than other model based RL problems, and wish to see more results on those applications.", "strength_and_weaknesses": "Strength:\n1. The proposed method is very detailed and sounds reasonable, and the empirical results and ablation studies are sufficient to verify its effectiveness;\n\nWeaknesses:\n1. technical novelty is limited: the proposed method is basically a straightforward extention to the VAE model in the sense that the latent variable is extended to a sequence of latent variables modeled, where both encoder and decoder is implemented with LSTMs.\n\n2. The main concern of this paper is how to train a better transition model, but why this is the core issue of OPE needs further explanation. I think that  it's better to evaluate the presented method across more model-based RL applications rather than OPE.\n\n3. typos: In formula (4), the left side doesn't contain a_{t}, but the right side contain a_{t}. \n\n4. In Fig.6, one can observe that only two weights are relatively large - what this means?  how does this affect performance?\n\n5. it is claimed that \"From Fig. 7, one can observe that it tends to significantly under-estimate\", why? it seems that there has over-estimate and under-estimate in Fig.7.\n\n6. in eq.(3), why a diagnal Gaussian is chosen to model the latent transition model?  It could be over-simplified in some complex enironment.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "iD_l76ey9K", "reviewer": "Reviewer_MDLU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work outlines a model-based approach for off-policy evaluation. It revolves around learning (from a static, offline dataset) a recurrent variational latent model of the environment. They outline two \"tricks\" to improve the approach. One is termed \"recurrent state alignment\", an additional loss is used to encourage the hidden state representations in the decoder and the encoder to be similar. Secondly, they use multiple decoders and when using the model for OPE average over the predictions of multiple decoders. They test their approach on some off-policy benchmarks on control tasks and compare with other OPE baselines as well as try ablations.\n", "summary_of_the_review": "Interesting paper providing some tricks to make variational autoencoders as models work better on offline datasets. It would be of wider interest if it also considered policy optimization.", "strength_and_weaknesses": "Strengths:\n- The paper is well-written.\n\n- Off-policy evaluation is an important topic for practical use of RL and of interest to the community.\n\n- Source code is provided.\n\n- The empirical results are compelling.\n\nWeaknesses:\n- The environments used (control tasks from D4RL) are typically more used for off policy optimization rather than evaluation. It would be helpful to test the approach on other kinds of OPE tasks such as recommender systems / advertising e.g. [1, 2].\n\n- It would be helpful to discuss (and ideally report experiments) on using this approach for policy optimization rather than just evaluation.\n\nMinor:\n- Spelling mistake in section heading 2.\n\n[1] Chen, Minmin, et al. \"Off-Policy Actor-critic for Recommender Systems.\" Proceedings of the 16th ACM Conference on Recommender Systems. 2022.\n\n[2] Saito, Y., Udagawa, T., Kiyohara, H., Mogi, K., Narita, Y., & Tateno, K. (2021, September). Evaluating the robustness of off-policy evaluation. In Fifteenth ACM Conference on Recommender Systems (pp. 114-123).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XoABsglbtx", "reviewer": "Reviewer_k3gq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to tackle the Off-Policy Evaluation problem in a model-based fashion, i.e., contrary to importance sampling-based methods, a model is trained to directly generate future trajectories based on past information. The proposed model is a Recurrent Variational Auto-encoder, where both the encoder and decoder are embodied by LSTMs. The main contribution, besides leveraging variational inference for model-based OPE, is the addition of two features: Recurrent State Alignment (RSA) and branching architecture for the decoder.\nRSA adds a term in the loss so that the hidden LSTM states of the encoder and the decoder stay close to each other. The authors claim that this term augments the information flow and helps the decoder generate more realistic trajectories.\nThe branching architecture is a kind of ensemble of decoders, where each branch is assigned a weight (summing to 1 over all branches) that is trained jointly with the parameters of the model. The authors claim that it helps alleviate the effect of initialization.\nThe authors present results in the DOPE benchmark against model-based and model-free baselines and achieve superior performance overall.", "summary_of_the_review": "This paper proposes an Off-Policy Evaluation problem in a model-based fashion using a generative model. This is in contrast to policy correction methods like IPS, and the Doubly Robust method. The paper is well-written and easy to understand. \n\nHowever, the claims and motivation for adding RSA and the branching architecture lack justifications, with some ambiguity in writing at places. \n", "strength_and_weaknesses": "Strengths:\n- The results on the DOPE benchmark are compelling\n- The ablation study is well conducted\n- Writing and presentation are pleasant and help get to the point quickly.\n\nWeaknesses:\n- The claims and motivation for adding RSA and the branching architecture lack theoretical or empirical justifications, beyond the final results: would it be possible to characterize the behavior observed when MSE is used / RSA is not used, and explain why decoder initialization can lead to such largely decreased performance\n- The intuitive statements made by the authors are sometimes vague: would it be possible t clarify terms such as \"structure\", \"information flow\" and \"impact of initialization artifacts\" in this particular context?\n- The results might be poorly reliable, as they are given on only 3 random seeds.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5erQWy8rLdm", "reviewer": "Reviewer_grrh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the application of recurrent world model to the task of off-policy evaluation (OPE) in offline RL setting. As contributions, the paper identifies two empirical techniques to help improve performance in the offline ODE setting: (1) recurrent state alignment (RSA); (2) branching architecture. Finally, experiment results show improvement over baseline methods.", "summary_of_the_review": "A few questions regarding the paper.\n\n=== **Motivation** ===\n\nMotivation. Since evaluated envs are Markovian, why is a Markovian world model not enough for offline OPE? I think providing a more clear and detailed investigation into this will greatly boost the motivation for using recurrent models for offline OPE in general. It can just be the case that in practice, using recurrent model is much more superior empirically, but conceptually, it remains unclear why it is the case. Is this because of statistical efficiency? Is this because of these envs are in fact non-Markovian and recurrent model is in fact being useful in a general POMDP setting? \n\nOverall, the paper has shown good empirical gains over prior methods. But the \"why\" is not clear. Thus far, I feel general readers cannot gain a better understanding as to why recurrent model is useful in OPE in general. If the paper is meant to just leverage a recurrent model training scheme, most part of which has existed in public, and show it delivers better performance, then the contributions are not novel nor insightful enough.\n\n=== **RSA** ===\n\nThe paper proposes that RSA is very important for practical benefits, and has shown evidence through ablation. However, what's not clear to me is why RSA is important specifically in the case of OPE? What is the implication of the RSA loss function that makes the OPE more accurate? \n\nFor example, if RSA benefits recurrent world model in general, what makes the loss function novel in the OPE case? Is this because in addition to the usual training stability benefits provided by RSA in the dreamer setting, extra benefits are present in the OPE case. Or maybe for the offline OPE case, it is even more important to carry out RSA than the dreamer case because now the data is offline.\n\nA clear explanation and motivation for the above question will help elucidate the novelty of the paper's contribution. If RSA does not provide benefits beyond how it helps dreamer, it is not clear if RSA is a novel contribution of the current paper.\n\n=== **Markovian world model** ===\n\nComparison to Markovian world model. In Fig 3 the paper compares VLBM with auto-regressive world model, obtained from prior work. An experiment to address the conceptual question in Q1 would be to compare with a Markovian world model, and maybe borrow results from prior work if there is any.\n\n=== **Why VLBM is better than AR** ===\n\nGiven VLBM and AR are both recurrent world models, a natural question is why one is better than another? It would be nice to have visualizations like in Fig 8 for the AR model too, to see whether it is because AR model fails to distinguish policies with different performance in the representation space.\n", "strength_and_weaknesses": "=== Strengths ===\n\nThe paper seems to propose an interesting application of learning recurrent world model for the purpose of offline ODE. Intriguingly, applying such world model technique does not work optimally out-of-the-box, and as the authors suggest in the paper, would require additional techniques to work better. The paper identifies such important elements as the RSA loss and proposes the novel branching architecture, which can be useful for future investigation in the model-based RL for OPE in general. Experiment results also seem solid.\n\n=== Weakness ===\n\nI think a major weakness of the paper lies in its conceptual contribution. One central question is: since all the tasks being evaluated are Mujoco continuous controls or Adroit tasks, arguably the environments are almost fully Markovian, then what is the principal reason why using a recurrent model works better than just using a Markovian world model? This is a key motivational question not addressed enough by the paper. I will expand more in the following.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster