Offline Policy Optimization with Variance Regularization
Authors: Riashat Islam
Samarth Sinha
Homanga Bharadhwaj
Samin Yeasar Arnob
Zhuoran Yang
Zhaoran Wang
Animesh Garg
Lihong Li
Doina Precup
Venue:  (2021)
Keywords: reinforcement learning
offline batch RL
off-policy
policy optimization
variance regularization

Abstract: Learning policies from fixed offline datasets is a key challenge to scale up reinforcement learning (RL) algorithms towards practical applications. This is often because off-policy RL algorithms suffer from distributional shift, due to mismatch between dataset and the target policy, leading to high variance and over-estimation of value functions. In this work, we propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regularization can be used to augment  any existing offline policy optimization algorithms. We  show that the regularizer leads to a lower bound to the offline policy optimization objective, which can help avoid over-estimation errors, and explains the benefits of our approach  across a range of continuous control domains when compared to existing algorithms. 

Under review as a conference paper at ICLR 2021
OFFLINE POLICY OPTIMIZATION WITH VARIANCE REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning policies from fixed offline datasets is a key challenge to scale up rein- forcement learning (RL) algorithms towards practical applications. This is often because off-policy RI. algorithms suffer from distributional shift, due to mismatch between dataset and the target policy, leading to high variance and over-estimation of value functions. In this work, we propose variance regularization for offline RL algorithms, using stationary distribution corrections. We show that by using Fenchel duality, we can avoid double sampling issues for computing the gradient of the variance regularizer. The proposed algorithm for offline variance regular- ization (OVR) be used to augment any existing offline policy optimization algorithms. We show that the regularizer leads to a lower bound to the offline policy optimization objective, which help avoid over-estimation errors, and explains the benefits of our approach across a range of continuous control domains when compared existing algorithms.
INTRODUCTION
Offline batch reinforcement learning (RL) algoithms are key towards scaling up RL for real world applications, such as robotics (Levine et al., 2016) and medical problems. This is because offline RL provides the appealing ability for agents learn from fixed datasets, similar to supervised learning, avoiding continual interaction with the environment, which could be problematic for safety and feasibility reasons. However, sigmificant mismatch between the fixed collected data and the policy that the agent is considering can lead to high variance of value function estimates, a problem encountered by most off-policy RI. algorithms (Precup et al., 2000). complementary problem that the value function can become overly optimistic areas of state space that outside the visited batch, leading the agent in data regions where its behavior is poor Fujimoto et al. (2019). Recently there has been some progress in offline RL (Kumar et al., 2019; Wu et al., 2019b; Fujimoto et al., 2019), trying to tackle both of these problems.
In this work, we study the problem of offline policy optimization with variance minimization. To avoid overly optimistic value function estimates, we propose to learn value functions under variance constraints, leading
pessimistic estimation, which can significantly help offline RL algorithms, especially under large distribution mismatch. We propose framework for variance minimization in offline RL, such that the obtained estimates can be used regularize the value function and enable more stable learning under different off-policy distributions.
We develop a novel approach for variance regularized offline actor-critic algorithms, which we call offline Variance Regularizen riieer (OVR). The key idea of OVR is to constrain the policy improvement step via variance regularized value function estimates. Our algorithmic framework avoids the double sampling issue that arises when computing gradients of variance estimates, by instead considering the variance of stationary distribution corrections with per-step rewards, and using the Fenchel transformation (Boyd & Vandenberghe, 2004) to formulate a minimax optimization objective. This allows minimizing variance constraints by instead optimizing dual variables, resulting in simply augmented reward objective for variance regularized value functions.
We show that even with variance constraints, we can ensure policy improvement guarantees, where the regularized value function leads to lower bound the true value function, which mitigates the usual overestimation problems in batch RL The use of Fenchel duality in computing the variance allows us avoid double sampling, which has been a major bottleneck in scaling up variance-constrained

---

Under review as a conference paper at ICLR 2021
actor-critic algorithms in prior work A. & Ghavamzadeh (2016); A. & Fu (2018). Practically, our algorithm is easy to implement, since it simply involves augmenting the rewards with the dual variables only, such that the regularized value function can be implemented offline policy optimization algorithms. We evaluate our algorithm on existing offline benchmark tasks based on continuous control domains. Our empirical results demonstrate that the proposed variance regularization approach is particularly useful when the batch dataset is gathered at random, or when it is very different from the data distributions encountered during training.
top of any existing
2 PRELIMINARIES AND BACKGROUND
We consider an infinite horizon MDP as (S, P.1) where is the set of states, A is the set of actions, P is the transition dynamics and y is the discount factor. The goal of reinforcement learning is to maximize the expected return [TT = Eg-dg [=-dd[V""s)]] where (*()) is the value
function (s) = Ei rr((s,, at) 80 s), and / is the initial state distribution. Considering parameterized policies Te (a|s), the goal is maximize the returns by following the policy gradient (Sutton et al., 1999), based on the performance metric defined as:
(I) where (s, a) is the state-action value function, since (*(s) -a (aals)("(s a). The policy optimization objective can be equivalently written in terms of the normalized discounted occu- pancy measure under the current policy тØ, where (s a) is the state-action occupancy mea- sure, such that the normalized state-action visitation distribution under policy æ is defined as (s. = (1 - 7) Ei=p >P((s; = 8, 8, a B.a ~ (ss))) The equality in equation 1 holds and can be equivalently written based on the linear programming (L.P) formulation in RL (see (Puterman, 1994; Nachum & Dai, 2020) for more details). In this work, we consider the off-policy learning problem under a fixed dataset DD which contains 8, a, r, tuples under known behaviour policy p(als). Under the off oof-po setting, importance sampling (Precup et al., 2000) is often used to reweight the trajectory under the behaviour data collecting policy, such as to get unbiased estimates of the expected returns. At each time step, the importance sampling correction z(a,ls)) used to compute the expected return under the entire trajectory as a(arlee)
[ = E =pppp.ao
QT ao) = (sap))) (assa)r((ss)) a)
J(x)= (1 7) (s.addd,, 7'rss,( (II-, x(ar ).))) Recent works (Fujimoto et al., 2019) have demonstrated that instead of importance sampling corrections, maximizing value func- tions directly for deterministic or reparameterized policy gradients (Lillicrap al., 2016; Fujimoto et al., 2018) allows learning under fixed datasets, by addressing the over-estimation problem, by maximizing the objectives of the form max, EEWD [Q (s, no(s)
u[ar]=:
3 VARIANCE REGULARIZATION VIA DUALITY IN OFFLINE POLICY OPTIMIZATION
In this section, we first present our approach based on variance of stationary distribution corrections, compared to importance weighting of episodic returns in section 3.1. We then present derivation of our approach based on Fenchel duality on the variance, to avoid the double sampling issue, leading to variance regularized offline optimization objective in section 3.2. Finally, we present our algorithm in I, where the proposed regularizer can be used in any existing offline RL algorithm.
3.1 VARIANCE OF REWARDS WITH STATIONARY DISTRIBUTION CORRECTIONS
In this work, we consider the variance of rewards under occupancy measures in offline policy optimization. Let us denote the returns as D = Σi= 7'r(s,, a,), such that the value function is V E,, [ The 1-step importance sampling ratio is Pt p(aj(s))*
a(ag]s.). and the T-steps ratio can
be denoted P1.T II. Pt. Considering per-decision importance sampling (PDIS) (Precup et al., 2000), the returns can be similarly written as D= = Ei= 7'rePu:e- The variance of episodic returns, which we denote by (T()), with off-policy importance sampling corrections can be written
Vp = (Ps-8aa-p(((1))) P-(aaa) ( (DD a) - (())] "
2

---

Under review as a conference paper at ICLR 2021
Instead of importance sampling, several recent works have instead proposed for marginalized impor- tance sampling with stationary state-action distribution corrections (Liu et al., 2018; Nachum et al., 2019a; Zhang et al., 2020; Uchara & Jiang, 2019), which can lead to lower variance estimators the cost of introducing bias. Denoting the stationary distribution ratios as w(s, a) (s,s)) the returns du(e,o)
can be written as W (s, a) w(s, ())(s
a). The variance of marginalized IS is:
(W" (s. - J(()) 21
VD(T) = ((s,a)=d
=B(ss))dd W (* (x,,)) (s.a) WT (s, a)]
(2)
Our key contribution is first consider the variance of marginalized IS VD(==) itself a as risk constraints, in the offline batch optimization setting. We show that constraining the offline policy opti- mization objective with variance of marginalized IS, and using the Fenchel-Legendre transformation on (T) can help avoid the well-known double sampling issue in variance risk constrained RL (for more details on how to compute the gradient of the variance term, see appendix B). We emphasize
that the variance here is solely based on returns with occupancy measures, and we do not consider the variance due to the inherent stochasticity of the MDP dynamics.
3.2 VARIANCE REGULARIZED OFFLINE MAX-RETURE ORJECTIVE
We consider the variance regularized off off-policy max return objective with stationary distribution corrections wx/D (which we denote for short for clarity) in the offline fixed dataset 'D setting:
max (TT) = R.ND Q (s, mo(s)) \V No)
We where A > O allows for the trade-off between offline policy optimization and variance regularization (or equivalently variance risk minimization). The max-return objective under QTT a) has been considered in prior works in offline policy optimization (Fujimoto et al., 2019; Kumar et al., 2019). We show that this form of regularizer encourages variance minimization in offline policy optimization. especially when there is large data distribution mismatch between the fixed dataset D and induced data distribution under policy mg.
3.3 VARIANCE REGULARIZATION VIA FENCHEL DUALITY
At first, equation 3 seems to be difficult to optimize, especially for minimizing the variance regular- ization w.r.t 0. This is because finding the gradient of V(w, ma) would lead to the double sampling issue since it contains the squared of the expectation term. The key contribution of OVR is to the Fenchel duality trick on the second term the variance expression equation 2, for regularizing policy optimization objective with variance of marginalized importance sampling. Applying Fenchel duality, x² = max, (2xy - y²), to the second term of variance expression, we can transform the variance minimization problem into equivalent maximization problem, by introducing the dual variables v(s, a). We have the Fenchel conjugate of the variance term as
1-7"(s, ( a) v(s, () a) + E(sa))dd ( ((s.)))(s.0)2))
1
(ww, Te) = max
(4)
max E(s,a)=dp - + (s)(ss)))(()))))) a) ((s,
Regularizing the policy optimization objective with variance under the Fenchel transformation, therefore have the overall max-m min optimization objective, explicitly written as:
max min J [[TT, v) EyDD (00 (s, (T [[
1 y² +v. ww..wr (s. (5)
2
3.4 AUGMENTED REWARD OBJECTIVE WITH VARIANCE REGULARIZATION
In this section, explain the key steps that leads to the policy improvement step being augmented variance regularized reward objective. The variance minimization step involves estimating the stationary distribution ration (Nachum et al., 2019a), and then simply computing the closed form solution for the dual variables. Fixing dual variables v. to update Tp. note that this leads to a standard maximum return objective in the dual form, which can be equivalently solved in the primal form,
3

---

Under review as a conference paper at ICLR 2021
using augmented rewards. This is because write the above above in the dual form as
J (TT, U, w. = ((sa))do(ss)) (w(s, r(s, - x(- 2" + M.w. r w r² (s, a)
= B(s,a)dd((sa)) ((s.u) u) (r- (r- (5. + [5)
=B(sa))d ((ss a) F(s, a) + - a) where we denote the augmented rewards as F(s,a) = [r- -. The policy improvement step can either be achieved by directly solving equation 6 or by considering the primal form of the objective with respect to Qa (s, mo) as in (Fujimoto al., 2019; Kumar et al., 2019). However, solving equation 6 directly can be troublesome, since the policy gradient step dry(s.a)
(6)
(7)
involves findinding the gradient w.r.t ((s a) do(s,a) too, where the distribution ratio depends on da, a). This means that the gradient w.r.t & would require finding the gradient w.r.t to the normalized discounted occupancy measure, ie, Vadag (+). Instead, is therefore easier to consider the augmented reward objective, using i(s, a) as in equation 7 in any existing offline policy optimization algorithm, where we have the variance regularized value function QT (s, o).
Note that as highlighted in (Sobel, 1982), the variance of returns follows a Bellman-like equation. Following this, (Bisi et al., 2019) also pointed
Bellman-lik, like solution for variance w.r.t occupancy measures. Considering variance of the form in equation 2, and the Bellman-like equation for variance, we can write the variance recursively as a Bellman equation:
V a) = (r(s.a) a) - ((T)) + -B, of (())))))))) V a
(8)
Since in our objective, we augment the policy improvement step with the variance regularization term, can write the augmented value function (x a) Q (s, a) AVg a). This suggests can modify existing policy optimization algorithms with augmented rewards on value function.
Remark: Applying Fenchel transformation to the variance regularized objective, however, first glance, seems to make the augmented rewards dependent on the policy itself, since F(s, a) depends on the dual variables v(s, a) well. This can make the rewards non-stationary, thereby the policy maximization step cannot be solved directly via the maximum return objective. However, as we discuss next, the dual variables for minimizing the variance term has a closed form solution v/s, a). and thereby does not lead to any non- non-stationarity in the rewards, due to the alternating minimization and maximization steps.
Variance Minimization Step: Fixing the policy no, the dual variables v can be obtained using closed form solution given by u(s.a) = w(s a) r(s, a). Note that directly optimizing for the target policies using batch data, however, requires a fixed point estimate of the stationary distribution corrections, which can be achieved using existing algorithms (Nachum et al., 2019a; Liu et al., 2018). Solving the optimization objective additionally requires estimating the state-action distribution ratio, (s,a)
w(s, a) (s.aa Recently, several works have proposed estimating the stationary distribution ratio, mostly for the off-policy evaluation case in infinite horizon setting (Zhang et al., 2020; Uchara & Jiang, 2019). We include a detailed discussion of this in appendix E.4.
Our proposed variance regularization approach with returns under stationary distribution conrections for offline optimization be built top of any existing batch off off-policy optimization algorithms. We summarize our contributions in Algorithm I. Implementing our algorithm requires estimating the stte-actioon distribution ratio, followed by the closed form estimate of the dual variable v. The augmented stationary reward with the dual variables can then be used to compute the regularized value function (s,, a). The policy improvement step involves maximizing the variance regularized value function, e.g with BCQ (Fujimoto et al., 2019).
Algorithm: :
4 THEORETICAL ANALYSIS
In this section, we provide theoretical analysis of offline policy optimization algorithms in terms of policy improvement guarantees under fixed dataset D. Following then, demonstrate that using the variance regularizer leads to a lower bound for our policy optimization objective, which leads to a pessimistic exploitation approach for offline algorithms.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Offline Variance Regularizer zer
Initialize critic Qs, policy Te, network we and regularization weighting X; learning rate 77 for ( to 7 do
Estimate distribution ratio (s, a) using any existing DICE algorithm
Estimate the dual variable v(ss a) = (s, a) F(ss,a
Calculate augmented rewards f(s, a) using equation 7
Policy improvement step using any offline policy optimization algorithm with augmented rewards f(s,a) 02 = 01-1 + [0,
end for
4.1 VARIANCE MARGINALIZED IMPORTANCE SAMPLING AND IMPORTANCE SAMPLING
We first show in lemma 1 that the variance of rewards under stationary distribution corrections can similarly be upper bounded based on the variance of importance sampling corrections. We emphasize that in the off-policy setting under distribution corrections, the variance is due to the estimation of the density ratio compared to the importance sampling corrections.
Lemma 1. The following inequality holds between the variance of per-step rewards under stationary distribution corrections, denoted by Vo(T) and the variance of episodic returns with importance sampling corrections VT
VD(TT)
Vp(T) <
7)
The proof for this and discussions on the variance of episodic returns compared to per-step rewards under occupancy measures is provided in the appendix B.
4.2 POLICY IMPROVEMENT BOUND UNDER VARIANCE REGULARIZATION
In this section, we establish performance improvement guarantees (Kakade & Langford, 2002) for variance regularized value function for policy optimization. Let us first recall that the performance improvement can be written in terms of the total variation Drv divergence between state distributions (Touati et al., 2020) (for more discussions
the performance bounds, see appendix C)
Lemma 2. For all policies x' and n, we have the performance improvement bound based on the total variation of the state-action distributions de'' and dx
J(r') 2 (x(T') Drv(drrrll
(10)
where _ max; [Ea= (AT a)]l. and (T(') = J(T) + Es-dt [A" (s,a).. For detailed proof and discussions, see appendix C. Instead of considering the divergence between state visitation distributions, consider having access to both state-action samples generated from the environment. To avoid importance sampling corrections on state-action visitation distributions, where we have an upper bound following from (Nguyen et al., 2010): Drv (s))(d (s)lld=(s)) & Div(d (s, a) (s, Following Pinsker's inequality, have: J(T') 2 J (B)d
further considers the bound on the objective based
(T a)) -c -c"B
DDL (s, (11)
Furthermore, we can exploit the relation between KL. total variation (TV) and variance through the variational representation of divergence measures. Recall that the total divergence between P and Q distributions is given by: Drv (p. q) = E. (p(x)pp((2))))) -q(z)) We can use the variational representation of the divergence measure. Denoting ds a) = (s, a). we have
Drv (18x) sup E (((.,)) (f,, a)] S(,aa"= o f(s,, a)]]
(12)
where Ø² is the convex conjugate of & and f is the dual function class based on the variational representation of the divergence. Similar relations with the variational representations of f-divergences have also been considered in (Nachum et al., 2019b; Touati et al., 2020). We can finally obtain a bound for the policy improvement following this relation, in terms of the per-step variance:
Theorem 1. For all policies and T', and the corresponding state-action visitation distributions dut and dn, we can obtain the performance improvement bound in terms of the variance of rewards under state-action occupancy measures.
J(7') - ((T] 2 (s) (AT a)] - Var [(s,a)
(13)
5

---

Under review as a conference paper at ICLR 2021
where f(s. 0.) is the dual function class from the variational representation of variance.
Proof. For detailed proof, see appendix C.I.
4.3 LOWER BOUND OBJECTIVE WITH VARIANCE REGULARIZATION
In this section, we show that augmenting the policy optimization objective with a variance regularizer leads to a lower bound to the original optimization objectiven J(T)). Following from (Metelli et al., 2018), we first note that the variance of marginalized importance weighting with distribution corrections can be written in terms of the a ox-Remm Renyi divergence. Let and q be two probability
When a = 1, this leads measures, such that the Renyi divergence is Fo = 1 log Σx ((x) p(1))
to the well-known KL divergence Fi (pllg) = Frt. (pllg)
Let denote the state-action occupancy measures under and dataset DD dz and dp- The variance of state-action distribution ratios
(w/)://dddda))//((( a)]. When a = 2 for the Renyi
divergence, we have:
(s)(s.)))d [We/D(s a)] F2d dd)
(14)
Following from (Metelli et al., 2018), and extending results from importance sampling to marginal- ized importance sampling wa/D. we provide the following result that bounds the variance of the approximated density ratio 2=/D p in terms of the Renyi divergence:
Lemma 3. Assuming that the rewards of the MDP are bounded by finite constant, ||r|lso S Rmar
Given random variable samples (s, a) ~ di(ss, a) from dataset DD for any N > 0, the variance of marginalized importance weighting be upper bounded
Var (sa))r))))))) @x//D(s a)] I
1 N F2
(15)
See appendix D.1 I for more details. Following this, our goal is to derive a lower bound objective to our off-policy optimization problem. Concentration inequalities has previously been studied for both off-policy evaluation (Thomas et al., 201 and optimization (Thomas et al., 2015b). In our case, we can adapt the concentration bound derived from Cantelli's 's ineqaulity and derive the following result based on variance of marginalized importance sampling. Under state-action distribution corrections, we have the following lower bound to the off-policy policy optimization objective with stationary state-action distribution corrections
Given state-action occupancy measures da and dp. and assuming bounded reward functions, for any 8 S and N > O, we have with probability at least 8 that
Theorem 2.
J(T) 2 B(s.a)dd((sa) W a) (aa(ss,)))r(..s.)))) r(s,a) m)] / Var (a.)d)))d (Dx, r(s,a)) (16)
Equation 16 shows the lower bound policy optimization objective under risk-sensitive variance constraints. The key to our derivation in equation 16 of theorem 2 shows that given off-policy batch data collected with behaviour policy μu(ass), we are indeed optimizing lower bound the policy optimization objective, which is regularized with variance term to minimize the variance in batch off-poll off-policy learning.
EXPERIMENTAL RESULTS ON BENCHMARK OFFLINE CONTROL TASKS
Experimental Setup:
We demonstrate the significance of variance regularizer on range of continuous control domains (Todorov al., 2012) based on fixed offline datasets from (Fu et al., 2020), which is standard benchmark for offline algorichms. To demonstrate the significance of our variance regularizer OVR, we mainly use on top of the BCQ algorithm and compare it with other existing baselines, using the benchmark D4RL (Fu et al., 2020) offline datasets for different tasks and off- off-policy distributions. Experimental results are given table
Performance on Optimal and Medium Quality Datasets We first evaluate the performance of OVR when the dataset consists of optimal and mediocre logging policy data. We collected the dataset using fully (expert) or partially (medium) trained SAC policy. We build our algorithm OVR top of BCQ. denoted by BCQ + VAR. Note that the OVR algorithm can be agnostic to the behaviour policy too for computing the distribution ratio (Nachum al., 2019a) and the variance. We observe that even

---

Under review as a conference paper at ICLR 2021
Domain
Task Name
BCQ+OVR
BCQ
BEAR
BRAC-p
aDICE
SAC-off
halfcheetah-rammee hopper-random
0,00
0.00
25.1
24.1
-0.3
30,5
9.51
9.65
11,4
11
0.9
11.3
walker-random
5.16
0.48
7.3
-0.2
0.5
4.1
baccheetah-meddiue
35.6
34,9
41.7
43.8
-2.2
-4.3
hopper-medium
71.24
57.76
52.1
32.7
1.2
0.9
walker-medium
33.90
27.13
59.1
77.5
0.3
0.8
halfchcetah-expee
100.02
97.99
Gym
hopper-expert
108.41
98.36
walker-expert
71.77
72.93
halfcheetah-
macei---ddmmmiieeeee expert
59.52
54.12
53.4
44.2
-0.8
1.8
hopper-medium medium expert
44.68
37.20
96.3
1.9
1.1
-0.1
34.53
29.00
40.1
76.9
0.4
1.6
halfcheetah-mixed
29.95
29.91
hopper-mixed
16.36
10.88
walker-mixed
14.74
10.23
kitchen-come complete
4.48
3.38
15
FrankaKitchen
kitchen-partial kitchen-mixed
25.65
19.11
13.1
0
30.59
23.55
47.2
2.5
2.5
Table 1: The results on D4RL tasks compare BCQ (Fujimoto et al., 2019) with and without OVR, bootstrapping actor critic with policy (BRAC-p) (Wu al.,
reduction (BEAR) (Kumar et al., 2019), echavior-regulatee offline SAC (SAC-off) (Haarnoja et al., 2018). The results presented are the normalized returns on the task as per Fu et al. (2020) (see Table in the unnormalized scores on each task). We see that in most tasks we are able to significant gains using OVR. Our algorithm be applied to any policy optimization baseline algorithm that trains the policy by maximizing the expected rewards. Unlike BCQ, BEAR (Kumar al., 2019) does have the same objective, they train the policy using MMD objective.
2019a), AlgeaDICE (aDICE) (Nachum al., 2019b)
al. (2020) for
though performance marginally improved with OVR under expert settings, since the demonstrations optimal itself, we can achieve significant improvements under medium dataset regime. This is because OVR plays a more important role when there is larger variance due distribution mismatch between the data logging and target policy distributions. Experimental results shown in first two columns figure 1.
Performance Random and Mixed Datasets We then evaluate the performance random datasets, i.c, the worst-case setup when the data logging policy is random policy, as shown in the last columns of figure 1. As expected, baselines such as BCQ (Fujimoto et al., 2019) can work poorly under random dataset setting. When we collect data using a mixture of random and mediocre policy, denoted by mixed, the performance is again improved for OVR on top of BCQ, especially for the Hopper and Walker control domains. We provide additional experimental results and ablation studies in appendix E.I.
observe improvements all, and even existing
RELATED WORKS
We now discuss related works in offline RL, for evaluation and opimization, and its relations to variance and risk sensitive algorithms. We include discussions of related works in appendix A. In off-policy evaluation, per-step importance sampling (Precup et al., 2000; 2001) have previously been used for off-policy evaluation function estimators. However, this leads high variance estimators, and recent works proposed using marginalized importance sampling, for estimating stationary state-action distribution ratios (Liu et al., 2018; Nachum al., 2019a; Zhang et al., 2019), to reduce variance but with additional bias. In this work, we build on the variance of marginalized IS, develop variance risk sensitive offline policy optimization algorithm. This is in contrast to prior works variance constrained online actor-critic (A. & Ghavanzadeh, 2016; Chow et al., 2017; Castro et al., 2012) and relates to constrained policy optimization methods (Achiam et al., 2017; Tessler et al., 2019).
For offline policy optimization, several works have recently addressed the overestimation problem in batch RL (Fujimoto et al., 2019; Kumar et al., 2019; Wu et al., 2019b), including the very recently proposed Conservative Q-Learning (CQL) algorithm (Kumar et al., 2020). Our work is done in parallel CQL, due which do not include value function which is guaranteed to lower-boun the true value function. This helps prevent value over-estimation for distr---ittiittionn (OOD) actions, which is important issue in offline RI. We
baseline in experiments. CQL learss
7

---

Under review as a conference paper at ICLR 2021
1100
-1.0
12000
4001
4250
2001
0000
4150-
$100
5m0
0000
800
4050
B20
300
1000
Dhude Miyu 2M1
(a) Cheetah Expert
(b) Cheetah Medium
(c) Cheetah Random
(d) Cheetah Mixed
700
8920-
1200
500
5000
2003
B00
200 VAA
1000
1500-
B03
200
200-
3C0
203-
BDQ
0.0
01.6
1.2
0.0
0.S
3.0
0.0
0.2
0.M
0.8
0.2
11.4
(e) Hopper Expeett
(1) Hopper Medium
(g) Hopper Random
Hopper
0909
2000
360
28G-
1200
5900
1000-
2003
200-0
BCD BCD+
200
0.0
0.2
0.5
0.5
Craduent Stege 150
Crutler
Gradket Stepe
( Walker Expert
(i) Walker Medium
(k) Walker Random
Walker Mixed
Figure 1: Evaluation of the proposed approach and the baseline BCQ (Fujimoto al., 2019) on suite of three OpenAl Gym environments. Details about the type of offline dataset used for training, namely random, medium, mixed, and expert are included in Appendix. Results are averaged over random seeds (Henderson et al., 2018). We evaluate the agent using standard procedures, as in Kumar ct al. (2019); Fujimoto et al. (2019)
note that our approach is orthogonal to CQL in that CQL introduces a regularizer on the state action value function (*, a) based on the Bellman error (the first two terms in equation 2 of CQL), while we introduce a variance regularizer on the stationary state distribution d (s). Since the value of a policy can be expressed in two ways either through Q (s, a) or occupancy measures d, (s). both CQL and our paper are essentially motivated by the same objective of optimizing a lower bound on J(8), but through different regularizers. Our work can also be considered similar to AlgaeDICE
(Nachum et al., 2019b), since we introduce a variance regularizer based on the distribution corrections, instead of minimizing the f-divergence between stationary distributions in AlgaeDICE. Both our work and AlgaeDICE considers the dual form of the policy optimization objective in the batch setting, where similar to the Fenchel duality trick on variance term, AlgaeDICE instead uses the variational form, followed by the change of variables tricks, inspired from (Nachum et al., 201 19a) to handle their divergence measure.
7 DISCUSSION AND CONCLUSION
We proposed a new framework for offline policy optimization with variance regularization called OVR, tackle high variance issues due distribution mismatch in offline policy optimization. Our work provides a practically feasible variance constrained actor-critic algorithm that avoids double sampling issues in prior variance risk sensitive algorithms (Castro et al., 2012: A. & Ghavamzadeh, 2016). The presented variance regularizer leads to lower bound to the true offline optimization objective, thus leading to pessimistic value function estimates, avoiding both high variance and overestimation problems in offline RL. Experimentally, we evaluate the significance of OVR on standard benchmark offline datasets, with different data logging off-policy distributions, and show that OVR plays a more significant role when there is large variance due to distribution mismatch. While we only provide a variance related risk sensitive approach for offline RL, for future work, would be interesting other risk sensitive approaches (Chow & Ghavamzadeh, 2014; Chow et al., 2017) and examine its significance in batch RL. We hope our proposed variance regularization framework would provide new opportunities for developing practically robust risk sensitive offline algorithms.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Prashanth L.. A. and Michael C. Fu. Risk-sensitive reinforcement learning: constrained optimization
viewpoint. CoRR, abs/1810.09126, 2018.
Prashanth L.. A. and Mohammad Ghavamzadeh. Variance-connstrainedd actor-critic algorithms for discounted and average reward mdps. Mach. Learn., 105(3):367-417, 2016. doi: 10. 10.1007/ $10994-016-5569-3 URL https: /tap:://do.... /tt.//ii...// ottp:///...1110/000000-011-----0
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017. pp. 22-31, 2017. URL http://ppattt httt:://procceedingg. mlr mww.press. v70/ 1.0/achtis aaaachiaam aml 7a html.
press/
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In NeurlPS Deep Reinforcement Learning Workshop. 2019. URL https: //arxiv. owg///g////... /bs/ 1977 04543. Contributed Talk at NeurlPS 2019 DRL Workshop.
Eitan Altman and hangyyiuudssiiippppppiznddddddas Asingleutility. Constrained markov decision processes, 1999.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation, In In Proceedings of the Twelfth International Conference on Machine Learning, pp- 30-37. Morgan Kaufmann, 1995.
Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk-averse trust region optimization for reward-volatiliy reduction. CoRR, abs/1912.03193, 2019. URL attt:://axiis org/ abs abbs/192.03393. 031933
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, USA, 2004. ISBN 0521833787.
Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 July 1, 2012, 2012. URL http: http///clll..//m/ cc/ pttp1///apppr///888 pdf.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for cvar optimization in mdps. In Advances in Neural Information Processing Systems 27: Annual Confer- ence on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3509-3517, 2014. URL http: http:://apppss.ttte 5246-algorithan core---r---vva------eoe inn.tiioiiniiiiiiiiiiiii-----
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. J. Mach. Learn. Res., 18:167:111166 16:16111166::51,2011 URL http: jmir org/ sappers/v8// 11111156660 636 htmll
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: convergent reinforcement learning with nonlinear function approximation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm. Sweden, July 10-15, 2018, pp. 1133-1142, 2018. URL http: http://poooeeedn....n.e mlr p press/ v80/dai18c. html.
/
Dongsheng Ding, Xiaohan Wei. Zhuoran Yang. Zhaoran Wang, and Mihailo R. Jovanovic. Provably efficient safe exploration via primal-dual policy optimization. CoRR, abs/2003.00534, 2020. URL https://eet //aa:v////xvvva orgg 2tt.a.///20..... 20t.///0333.0.
Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, 0bb/2004.00219,, 2020. URL https: /arxiv. org/ arg/abs// 2004.000. 07219.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüssann Stockholm, Sweden, July 10-15, 2018, pp. 1582-1591, 2018.

---

Under review as a conference paper at ICLR 2021
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 2052 2062, 2019. URL http:// proceedi hgs ml mlr: malr. press/v97 fu immool 9a. html.
Javier García, Fern, and o Ferúandez. A comprchensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1477-1480, 2015. URL http://:mlr... org/ papers/v16 6/ garcial5aaaaaaaa htmml.
lan J. Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1406 160,,,,,,,, 2014. URL http: /arxivv org/ abs 1466.2661.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 1856-1865, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI- 18), New Orleans. Louisiana, USA, February 2-7, 2018, pp. 3207-3214, 2018. URL https: // //www..... aaai. org/ oes/ index. php. AAA//AAAI///// RAAIIAAAAAN AAAIIAAAAAB..... wwtp/papereeeeee hwtppppppppepeeeeaeeaeeeeppeeese 16669.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances Neural Information Processing Systems 29: Anmual Conference Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4565-4573. 2016.
Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), University of New South Wales, Sydney, Australia, July 8-12, 2002, pp. 267-274, 2002.
llya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off off-policy distribution 2019. URL http: http/v.xvvvvoooo/oo org/ abs /19s//19111.. 05032.
matching, CoRR,
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 20X::2000.044799 2020.
Sascha Lange. Thomas Gabel, and Martin A, Riedmiller. Batch reinforcement learning. In Reinforce- ment Learning, 2012.
Hoang Minh Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 3703-3712, 2019. URL htttt::/pproceednngg/// mlr. press press/v97. 1el9a.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. J. Mach. Learn. Res., 17:39:1-39:400 2016. URL http:t////mlle /mtt::://mlr.orgg papers/v17/1s 522. html.
Lihong Li, Rémi Munos, and Csaba Szepesvári. Toward minimax off-policy value estima- tion. In Proceedings of the Eighteenth International Conference Artificial Intelligence and Statistics, AISTATS 2015, San Diego California, USA, May 9- 2015, 2015. URL httt:://procett ngs ml mwr.prres/vee /tt/////////bb//////////// html.
10

---

Under review as a conference paper at ICLR 2021
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference Learning Representations. ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL htLp:tt:// 02971.
1509.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou, Breaking the curse of horizon: Infinite- horizon off- off- oollcy estimation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, 3-8 December 2018, Montréal, Canada, pp. 5361-5371, 2018.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018. NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pp. 5447-5459, 2018.
Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality. CoRR, abs/2001.01 2020. URL http:// org/ abs/2001. 01866.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostie estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp.
2019a.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. CoRR, abs/1912.02074, 2019b. URL http://arxiv. org/abs/ / 912. 02074.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Information Theory. 56(11):
5847-5861, 2010. doi: 10.1 10.1109777220102268887 URL https: http://///ooooooor//////////oe// 1109/ TIT. 2010 2068870.
Theodore J. Perkins and Andrew G. Barto. Lyapunov design for safe reinforcement learning. J. Mach. Learn. Res., (oulll)803-8322 March 2003. ISSN 1532-4435.
Doina Precup, Richard S. Sutton, and Satinder Singh. Eligibility traces for off off-policy policy evaluation. In Proceedings of the Seventeenth International Conference Machine Learning (ICMI. 2000), Stanford University, Stanford, CA, USA, June 29 -July 2, 2000, pp. 759-766, 2000.
Doina Precup, Richard S. Sutton, and Sanjoy Dasgupta. Off Off-policy temporal difference learning with function approximation. In Carla E. Brodley and Andrea Pohoreckyj Danyluk (eds.), Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 July 1, 2001, pp. 417-424. Morgan Kaufmann, 2001.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, pp. 1889-1897, 2015.
Samarth Sinha, Jiaming Song, Animesh Garg, and Stefano Ermon. Experience replay with likelihood- free importance weights. arXiv preprint arXiv:2006. 102000066111... 13169, 2020.
Matthew J. Sobel. The variance of discounted markov decision processes. Journal of Applied
Probability, 19(4)794-802, 1982. doi: 10.2307/3213832.
James C. Spall. Multivariate stochastic approximation using a simultancous perturbation gradient approximation. IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 371):332341, 1992.
11

---

Under review as a conference paper at ICLR 2021
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Informa- fion Processing Systems 12, INIPS Conference, Denver, Colorado, USA. November 29 December
4, 1999]. pp- 1057-1063, 1999.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. J. Mach. Learn. Res., 16:1731-1755, 2015a. URL http:
//dl. acm. org http/iiaaiiiooo./ http://ctaatoooo..caaaa288880000 cfm2id-2885805
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 814-823, 2015b. URL http: http::/procceedigge mlr. press/ / v27 http///wmiiaaaaaaaaammm
htmii.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL https: http::s:///opnreeiwwweeeeeee net wwweee//ooonnnnnnntteeeeee
Philip S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off- off-policy evaluation. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pp. 3000-3006, 2015a. URL http: / /www. aaai org/
index. pwppppppppppppppppppppppppwwwawp BAA.AAAAAAAAAA AAA.AAAAAAAAAAAAAA.A/AAAAAAAAAAT wwt.a//////apee/eiieeeee /1042..
Philip S. Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2380-2388. JMLR.org. 2015b. URL http: //ppoceedding.... milr.iinnniimmi.....mmmmmm.mmmam press / v37 ttw.astooaasssa/ 1 15. html.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ahmed Touati, Amy Zhang, Joelle Pineau, and Pascal Vincent. Stable policy optimization via off-policy divergence regularization. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of the Thirty-Sixth Conference on Uncertainty in Artificial Intelligence, UAI 2020, virtual online, August 3-6, 2020, pp. 543. AUAI Press, 2020. URL http:/ /wwp///wwommmmmm/ aua www.ooorgg/aoo/ooooooo proceedings egeeediing///e 543 main wpppp.ppce
Masatoshi Uehara and Nan Jiang. Minimax weight and f-function learning for off-policy evaluation. CoRR, abs/1910. 12809, 2019. URL http: http://xxvvvv///// http://wwwwwooo///////////o 910 12809.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularrzre offline reinforcement learning. CoRR, abs/1911. 1361, 2019a. URL.ttpp///arr2v.orr/////////11
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularizeee offinn reinforcceeeeiieeeeeemmemmm learning.
CoRR, abs/191 11361, 2019b. URL UtLLLttt///aarx...rr///////11111
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL https://p///s.c...t...tt opw.revien.wweecceeeee. net./ / forum?id
HKx.cnVEWB.
Shangtong Zhang, Wendelin Bochmer, and Shimon Whiteson. Generalized off-policy actor- critic. In Advances in Neural Information Processing Systems 32: Annual Conference on
Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Van-
couver, BC, Canada, pp. 1999-2009, 2019. URL http::/papprs/ wwwps.ccc/eeec ccppperr//ppppp//pccpccccce..eee 4474-generalized off coaf-poliy-acco wwwtor-criitte
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX: ADDITIONAL DISCUSSIONS
A.1 EXTENDED RELATED WORK
Other related works: Several other prior works have previously considered the batch RL setting (Lange et al., 2012) for off-policy evaluation, counterfactual risk minimization (Swaminathan & Joachims, 20;5;;)), learning value based methods such as DQN (Agarwal et al., 2019), and others (Kumar et al., 2019; Wu et al., 2019b). Recently, batch off-policy optimization has also been introduced to reduce the exploitation error (Fujimoto et al., 2019) and for regularizing with arbitrary behaviour policies (Wu et al., 2019b). However, due to the per-step importance sampling corrections on episodic returns (Precup et al., 2000), off-policy batch RL methods is challenging. In this work, we instead consider marginalized importance sampling corrections and correct for the stationary state- action distributions (Nachum et al., 2019a; Uehara & Jiang, 2019; Zhang et al., 2020). Additionally, under the framework of Constrained MDPs (Altman & Asingleutility, 1999), risk-sensitive and constrained actor-critic algorithms have been proposed previously (Chow et al., 2017: Chow & Ghavamzadeh, 2014; Achiam et al., 2017). However, these works come with their own demerits, as they mostly require minimizing the risk (ie, variance) term, where finding the gradient of the variance term often leads double sampling issue (Baird, 1995). We avoid this by instead using Fenchel duality (Boyd & Vandenberghe, 2004), inspired from recent works (Nachum & Dai, 2020; Dai et al., 2018) and cast risk constrained actor-critic as a max-min optimization problem. Our work is closely related to (Bisi et al., 2019), which also consider per-step variance of returns, w.r.t state occupancy measures in the on-policy setting, while instead consider the batch off- off-policy optimization setting with per-step rewards w.r.t stationary distribution corrections.
Constrained optimization has previously been studied in in reinforcement learning for batch policy learning (Le et al., 2019), and optimization (Achiam et al., 2017), mostly under the framework of constrained MDPs (Altman & Asingleutility, 1999). In such frameworks, the cumulative return objective is augmented with a set of constraints, for safe exploration (García et al., 2015; Perkins & Barto, 2003; Ding et al., 2020), or reduce risk measures (Chow al., 2017; A. & Fu, 2018; Castro et al., 2012). Batch learning algorithms (Lange et al., 2012) have been considered previously for counterfactual risk minimization and generalization (Swaminathan & Joachims, 201 50:b)) and policy evaluation (Thomas et al., 2015a; Li et al., 2015), although little has been done for constrained offline policy based optimization. This raises the question of how can we learn policies RI. from fixed offline data, similar to supervised or unsupervised learning.
A.2 WHAT MAKES OFFLINE OFF-POLICY OPTIMIZATION DIFFICULT?
Offline RL optimization algorithms often suffer from distribution mismatch issues, since the under- lying data distribution in the batch data may be quite different from the induced distribution under target policies. Recent works (Fujimoto et al., 2019; Kumar et al., 2019; Agarwal et al., 2019; Kumar et (al.2)220) have tried to address this, by avoiding overestimation of Q-values, which leads to the extraplation eгror when bootstrapping value function estimates. This leads to offline RL agents generalizing poorly for unseen regions of the dataset. Additionally, due the distribution mismatch, value function estimates can also have large variance, due to which existing online off-policy algo- rithms (Haarnoja et al., 2018; Lillicrap et al., 2016 6; Fujimoto al., 2018) may fail without online interactions with the environment. In this work, we address the later problem to minimize variance of value function estimates through variance related risk constraints.
B APPENDIX : PER-STEP VERSUS EPISODIC VARIANCE OF RETURNS
Following from (Castro al., 2012; A. & Ghavamzadeh, 2016), let denote the returns with importance sampling corrections in the off-policy learning setting as
T
T (at 8t) su s. 4. - pla: st)
Dd (s(ssa)=
at)
II =
(17)
t=0
From this definition in equation 17, the action-value function, with off- policy trajectory wisee im- portance correction is Q²((sa)) a) ((.a))d,, (D*(s,)), and similarly the value function can
(D")
be defined as V (s) (s)dd (D*(s)) (s)). For the trajectory-wise importance corrections, we can
13

---

Under review as a conference paper at ICLR 2021
define the variance of the returns, similar to (A. & Fu, 2018) as:
(T( = E (ssa))dd,,ssa)
(s.a) (s. a) ((s.a)=d,, (s.a) DD as)
(18)
where note that as in (Sobel, 1982), equation 18 also follows a Bellman like equation, although due to lack of monotonocitiy required for dynamic programming (DP), such measures cannot be directly optimized by standard DP algorithms (A. & Fu, 2018).
In contrast, if we consider the variance of returns with stationary distribution corrections (Nachum et al., 2019a; Liu et al., 2018), rather than the product of importance sampling ratios, the variance term involves weighting the rewards with the distribution ratio wa/u- Typically, the distribution ratio is approximated using a separate function class (Uchara & Jiang, 2019), such that the variance can be written
(* a) Ws/DD(s a) r(ss a) s, a T(- |s) (s, a) dD(s,a)
(19)
where we denote D as the data distribution in the fixed dataset, collected by either a known or unknown behaviour policy. The variance of returns under occupancy measures is therefore given by
(T( ((..)))d W (s, (s, a)
((s.a)
(s,
(20)
where note that the variance expression in equation 20 depends on the square of the per-step rewards with distribution correction ratios. We denote this as the dual form of the variance of returns, in contrast to the primal form of the variance of expected returns (Sobel, 1982).
Note that even though the variance term under episodic per-step importance sampling corrections in equation 18 is equivalent to the variance with stationary distribution corrections in equation 20, following from (Bisi et al., 2019), considering per-step corrections, we will show that the variance with distribution corrections indeed upper bounds the variance of importance sampling corrections. This is an important relationship, since constraining the policy improvement step under variance constraints with occupancy measures therefore allows us to obtain lower bound to the offline optimization objective, similar to (Kumar et al., 2020).
B. PROOF OF LEMMA 1 VARIANCE INEQUALITY
Following from (Bisi et al., 2019), we show that the variance of per-step rewards under occupancy measures, denoted by VD(TT) upper bounds the variance of episodic returns Vp((T).
VD(
VT(T)) < (1
(21)
Proof. Proof of Lemma following from (Bisi et al., 2019) is as follows. Denoting the returns, as above, but for the on-policy case with trajectories under n, (T a) Ei 7'r(st, ar), and
denoting the return objective (TT) E Bxsppppar->(()))))... returns can be written as
D a) the variance of episodic
(s J(T)
(1 - )
VP(T)= = (Ma)
(22)
= (E(sa))dd
(8){ + (1 71-
J(T)
(1 (1-)) E (s.a)
(T)
(
D (s.
(23)
J(x)2
-B(sa)-da((..))
(24)
(1- 71
Similarly, denoting returns under occupancy measures as (T a) = a)(ss))(ss, a). and the returns under occupancy measures, equivalently written as ((T (s.a)) a)) based on the primal and dual forms of the objective (Uchara & Jiang, 2019; Nachum & Dai, 2020). we can equivalently write the variance as:
(T( = ((s,a)dd((aa) ((x))))((x)))
(25)
-((.a) ((., a(
+ J(T)² 25 217
(26)
-(aaa) (s.a)
((),a)2)
- [(T]2
(27)
14

---

Under review as a conference paper at ICLR 2021
Following from equation 22 and 25, therefore have the following inequality:
oc
(1 7) Bos
D (s(s $ (1 - 2) somppan
)r((s,,
(28)
1=
(1 -7)
,r(s), )
(29)
(((aa))d, ((s. a)p]
(30)
where the first line follows from
inequality. This concludes the proof.
We can further extend lemma 1, for off- off-policy returns under stationary distribution corrections (ie, marginalized importance sampling) compared importance sampling. Recall that denote the variance under stationary distribution corrections as:
VD (T) ((s.a)nd((sa)) ( a) (r(p))))))(()))))))))))))))))))) r(s
(31)
((aa))do(ea)) [M (8, a)² r(s, r(s.a)² J
(32)
where J(T) = ((ssa))do(saa)) a) 4 Wa/D(ss a) rss We denote the episodic returns with importance
sampling corrections D= 2 The variance, denoted earlier is given by
J(x)²
[D* (1 71-
VT( = (s.a)
(33)
We therefore have the following inequality
(1 7)² 2 "E
DJ (s,
S (1 - 2 Ess
(E 1=0
T
T r ax)
x (a. st)
uD ae (8))
1=0
T
T T(ae(s))
(1 Bso
-pann
Σ (r((s,,
II
JDD (at(s))
(34)
= dd WT/D(ss a)² as)
(35)
which shows that lemma also holds for off-policy returns with stationary distribution corrections.
B.2 DOUBLE SAMPLING FOR COMPUTING GRADIENTS OF VARIANCE
The gradient of the variance term often leads the double sampling issue, thereby making it impractical to use. This issue has also been pointed out by several other works (A. & Ghavanzadeh, 2016; Castro et al., 2012; Chow et al., 2017), since the variance involves the squared of the objective function itself. Recall that we have:
(0) = (s,a))) Ww//p(s a) r((s,a)]
(Eraa)ddo( wa/p(ss (esy)(s)).(ssa))) (36)
The gradient of the variance term is therefore
V VD) = Volce,a)vdo Wx (x//D(s a) r(s,
r(s,, } Ve E(s,a)=dD We/D(s, (t)pp(..)-rss.al]] r(s.a)}
- E Ess.a)ndp (
(37)
where equation 37 requires multiple samples compute the expectations in the second term. To see why this is true, let us denote
((0) = Edp (*.a) wx/d(ss a) ((s, ()
where we have IS(,, T the returns in short form. The variance of the returns with the stationary state-action distribution corrections can therefore be written as
VD (0) = (dp(...)
IS(w, }
(dn(( [S(,, TT)
(38)
(8)
15

---

Under review as a conference paper at ICLR 2021
We derive the gradient of each of the terms in (a) and (b) in equation 38 below. First, find the gradient of the variance term w.r.t 0:
V (s..)()..)) IS(W.
= V dp(s, (W.. mo)²= = Σ
mp)²
8,
3,0
Σ do(s, a) 2. IS(W, To) 1S( mo) V log To(a
3,
(39)
2
Σ 8,0
d To V log
= (d,(a))) [S(w, no)² Va log (a s)
Equation 39 interestingly shows that the variance of the returns w.r.t nee has form similar to the policy gradient term, except the critic estimate in this case is given by the importance corrected returns, since (w,, To) / (s, a) r(s, a)]- We further find the gradient of term (b) from equation 38.
Finding the gradient of this second term w.r.t 0 is therefore:
VgEdn (s.a)
[S( ma)]
(e)
(() (s.()
WT/D (VglogT((a s) (7 a)}
(40)
Overall, the expression for the gradient of the variance term is therefore
VoVD(0) 2 F.t((s.)) [S( m0)2 V log Ta(a |s]
2- J(0) Edv(s,a) Wx/D= : (Vo log To(a 8) (+,
(41)
The variance gradient in equation 41 is difficult estimate in practice, since it involves both the gradient of the objective and the objective J(Ø) itself. This is known to have the double sampling issue (Baird, 1995) which requires separate independent rollouts. Previously, (Castro et al., 2012) tackled the variance of the gradient term using simultaneous perturbation stochastic approximation (SPSA) (Spall, 1992), where we can keep running estimates of both the return and the variance term, and use a two time scale algorithm for computing the gradient of the variance regularizer with per-step importance sampling corrections.
B.3 ALTERNATIVE DERIVATION VARIANCE REGULARIZATION VIA FENCHEL DUALITY
In the derivation of our algorithm, we applied the Fenchel duality trick to the second term of the variance expression 25, An alternative way derive the proposed algorithm would be to see what happens if we apply the Fenchel duality trick to both terms of the variance expression. This might be useful since equation 41 requires evaluating both the gradient terms and the actual objective (0). due to the analytical expression of the form V (o. [()), hence suffering from double sampling issue. In general, the Fenchel duality is given by:
max(2ay - y²)
(42)
y and applying Fenchel duality to both the terms, since they both involve squared terms, we get:
Edp(s.)) IS(w, x0) = (dd(()))
max {2 IS(W, y(s, a) - a(s.a)²}
(43)
Ed [SS mo) uis, a)) EsS((a) [v(s.a)8]]
= max y
Similarly, applying Fenchel duality to the second (b) term we have
(dn(s.) IS(W, mo) = max 2 (dd(s.)) [IS(w, To) v(s, a)) - v2
(44)
Overall, we therefore have the variance term, after applying Fenchel duality as follows, leading to an overall objective in the form max, max,, (f), which we
our variance regularizer
(0) = 2 max Edo(t.a) IS(W, mo) v(s,a)) Ees [u(s)a)3
"
max
2 (s
IS(w, TO) v((s a) - D²
(45)
16

---

Under review as a conference paper at ICLR 2021
Using the variance of stationary distribution correction returns regularizer, can find the gradient of the variance term w.r.t as follows, where the gradient terms dependent on the dual variables y and are O.
IS(w, To) ()),s,)) 0 - 2 VoBdo IS(W T8) v(s,a) a) + VoVb(0) 2 (sLdd
[is(w, (() log To(a s) -2- 2-Eap(s.a) [iS(w, ( [s]]
2-
IS(w, mO) Vo log mo(a 8) {u((s - - -(s,)))
= 2 (e.(a
(46)
Note that from equation 46, the two terms in the gradient is almost equivalent, and the difference comes only from the difference between the two dual variables y(s, a) and v(s, a). Note that our variance term also requires separately maximizing the dual variables, both of which has the following closed form updates
V VD) = 2 (slEd [Ss Te) v (ss.a) ai] + V =0
(47)
Solving which exactly, leads to the closed form solution v(s, a) = (e [IS(ws ma) we can also solve exactly using closed form solution for the dual variables y/ such that
Similarly,
V VD))(0) 2 V,, (d.daas) [S, TO) y(s, a) 2 VyBBdo(s.)) y(( a)² =
(48)
Solving which exactly also leads to the closed form solution, such that y(s,a) = 1/2 IS(,, me) dr
(.(x,a) rr(saa, Note that the exact solutions for the two dual variables are similar to each other, where vss, a) is the expectation the returns with stationary distribution corrections, whereas y(s, a) only the return from a single rollout.
C APPENDIX : MONOTONIC PERFORMANCE IMPROVEMENT GUARANTEES UNDER VARIANCE REGULARIZATION
We provide theoretical analysis and performance improvements bounds for our proposed variance constrained policy optimization approach. Following from (Kakade & Langford, 2002; Schulman et al., 2015; Achiam et al., 2017), extend existing performance improvement guarantees based on the stationary state-action distributions instead of only considering the divergence between the current policy and old policy. We show that existing conservative updates in algorithms (Schulman et al., 2015) can be considered for both state visitation distributions and the action distributions, as similarly pointed by (Achiam al., 2017). We can then adapt this for the variance constraints instead of the divergence constraints. According to the performance difference lemma (Kakade & Langford, 2002), have that, for all policies ग and n':
J(TT) J(x) = Bewddr aovr' (s, a)]
(49)
which implies that when we maximize 49, it will lead to an improved policy n' with policy improve- ment guarantees over the previous policy æ. We can write the advantage function with variance augmented value functions as:
イg (x, VX (s) = Evep [r(3,0) A(r(s,)) J(x))² (V) V2
However, equation 49 is often difficult to maximize directly, since additionally requires samples from n' and der, and often a surrogate objective is instead proposed by (Kakade & Langford, 2002). Following (Schulman et al., 2015), we can therefore obtain a bound for the performance difference based on the variance regularized advantage function:
J(m')= 2 J(T)) Egnd (s).)>< AT a))
(50)
where we have the augmented rewards for the advantage function, and by following Fenchel duality for the variance, can avoid policy dependent reward functions. Otherwise, we have the augmented rewards for value functions F(s, a) r(s. a) X(r(s, a) { This however suggests that
the performance difference does not hold without proper assumptions (Bisi et al., 2019). We can
therefore obtain monotonic improvement guarantee by considering the KI. divergence between
17

---

Under review as a conference paper at ICLR 2021
policies:
J(T)) + Bo-de (s,
(51)
which ignores the changes the state distribution dy due to the improved policy n'. (Schulman
et al., 2015) optimizes the surrogate objectives Lx(T)) while ensuring that the new policy n' stays
close to the current policy æ, by imposing a KL. constraint (Eynd, [D (T'( (- ()( [)] & 8). The
performance difference bound, based on the constraint between and n' as TRPO (Schulman et al., 2015) is given by:
Lemma 4. The performance difference lemma in (Schulman et al., 2015), where a = DiV = max, Drv(T, n
4e7
J(r') 2 (x(
(1 (1- (DiV (a l
(52)
where= e max,,a (s, a)). which is usually denoted with where
The performance improvement bxound in (Schulman et al., 2015) can further be written in terms of the KL divergence by following the relationship between total divergence (TV) and KL, which follows from Pinsker's inequality, Drv Drv(plla) < (pllp),)) to get the following improvement bound
4ey
(( 2 C. (TT') (1 1 7) Dxt. (T'|[TT)
(53)
We have performance difference bound in terms of the state distribution shift d, and d This justifies that C, (œ') is a sensible lower bound to J (T²) as long as there a total variation distance between dy' and dz which ensures that the policies n' and π stay close to each other. Finally, following from (Achiam et al., 2017), we obtain the following lower bound, which satisfies policy improvement guarantees :
2nc -- ----Eaadd Bowd, (Div(r'() []][ |s)]]
J(x') 2 Lx
(54)
T-7
Equation 53 and 54 assumes that there is no state distribution shift between n' and æ. However, if we explicitly assume state distribution changes, d, and dz due to n' and respectively, then have the following performance improvement bound:
Lemma 5. For all policies x' and n, we have the performance improvement bound based on the total variation of the state-action distributions de and dx
J(x') 2 L. ( 6. Drv(ddr11lss
(55)
where en = max, Baaw'' (1,>) (s)
which can be further written in terms of the surrogate objective Lx(xx)
(xx) as
((T') 2 J(T) + Es-de [A (A= [s, E "Dv (rv (ld))
= L. (TT) Drv((drldd)
(56)
C.I PROOF OF THEOREM 1: POLICY IMPROVEMENT BOUND WITH VARIANCE REGULARIZATION
Proof. We provide derivation for theorem 1. Recall that for all policies V' and ग, and corresponding state visitation distributions d and d,, we can obtain the performance improvement bound in terms of the variance of state-action distribution corrections
J(m') ((x) Band, A (s, Var awa [(s,a)
(57)
where [(8, a) is the dual function class, for the divergence between dyr(s, (8, a) and di a) Following from Pinsker's 's incquality, the performance difference lemma written in terms of the state visitation distributions can be given by
J(x') L=(m') D'v
2 (TT + E Lendd [A (A - -'DDi((drrldd))
2 ((r) ) + Egndd AA (s, a)] - - VDk (d))
(58)
18

---

Under review as a conference paper at ICLR 2021
Following from (Schulman all, 2015), alternately write this follows, where further apply the variational form of TV
J(a) 2 J(x Bonderawt (TT(s
-C. - (rv((drlll=))
2 $1 [( maxx (5, a)] Esnd (s( a)]}
=() + BsnddrawT
(s
-C. -Bsddr
[ (Fanddr (s) Bgndd am [ "
J(z) + Byydyyann amot!
A (s)
C. max Bende
ane aaw [Bandy [f all) 2}
= J(t) + Exwdy awo aous
1 (s)
C. max (Eandd (s,, - Bxndy
(m)) + Bawd,
A (s)
C. max Varsad, [
(59)
Therefore the policy improvement bound depends on maximizing the variational representation f(s, a) of the f-divergence to guaranetee improvements from J(T) to [+r'. This therefore leads to the stated result in theorem 1.
D APPENDIX LOWER BOUND OBJECTIVE WITH VARIANCE REGULARIZATION
D.1 PROOF OF LEMMA 3
Recalling lemma 3 which states that, the proof of this follows from (Metelli et al., 2018). We extend this for marginalized importance weighting, and include here for completeness. Note that compared to importance weighting, which leads to an unbiased estimator as in (Metelli et al., 2018), correcting for the state-action occupancy measures leads to a biased estimator, due the approximation Ws />/D- However, for our analysis, we only require to show a lower bound objective, and therefore do not provide any bias variance analysis as in off-policy evaluation.
Var (a)().)))
(60)
Proof. Assuming that state action samples are drawn i.i.d from the dataset D, we can write: d= 01 . -r(s1, a1) a1)
1
Var (da))
S (a) (t./D(s.))
N
[
r(s1,a)) ' (drlor,aa)) do(si,
N ((e
1 B
(()))))) a1) 1 -
dz (51, (p)
$
(61)
N
D.2 PROOF OF THEOREM 2:
First let us recall the stated theorem 2. By constraining the off-policy optimization problem with variance constraints, we have the following lower bound to the optimization objective with stationary state-action distribution corrections
(s, a)
d= do
J ( 2 ((s.a))d(((.a))
dp(8, (s,
Var (s.a)
(s,a))
(62)
Proof. The proof for the lower bound objective can be obtained as follows. We first define a relationship between the variance and the a-divergence with = 2, also similarly noted in (Metelli et al., 2018). Given we have batch samples D, and denoting the state-action distribution correction write from lemma 3:
with ( a),
doe)) [+ [ 1 121
(63)
where the per-step estimator with state-action distribution corrections given by wn/p(s, a) -rss a). Here, the reward function r(s, a) is bounded function, and for any N > the variance of the
19

---

Under review as a conference paper at ICLR 2021
per-step reward estimator with distribution corrections be upper bounded by the denei-divergence (a = 2). Finally, following from (Metelli et al., 2018) and using Cantelli's inequality, we have with probability least 1 5 where 0 < 1:
Pr WD/DD - J(r) 2 A S
(64)
1
(o we (5. (s.)))((sa))
and by using 6 (x.a)mdp(ssa)) //D////
we get that with probability at least 6, we have:
E (ssa) Way r(s.a)) V
δ
( (ssa))
Var Va
r(s,a)
(65)
where we can further replace the variance term with a 2 for the Renyi divergence conclude the proof for the above theorem. We can further write the lower bound for for a- a-Renyi divergence, following the relation between variance and Renyi-divergence for 2 as
dx (8, (TT (s.a) a)] 2 ((s.a)-d ds a)
8)d2 (dd)
r(s a)] ||rl|
8N This hints the similarity between our proposed variance regularized objective with that of other related works including AlgaeDICE (Nachum et al., 2019b) which uses a f-divergence D D2 lkde) between stationary distributions.
E APPENDIX ADDITIONAL EXPERIMENTAL RESULTS
E.
EXPERIMENTAL ABLATION STUDIES
In this section, we present additional results using state-action experience replay weightings on existing offline algorithms, and analysing the significance of our variance regularizer on likelihood corrected offline algorithms. Denoting w(s, a) for the importance weighting of state-action occupancy measures based samples in the experience replay buffer, to account for state-action distribution ratios.
modify existing offline algorithms
The ablation experimental results using the Hopper control benchmark summarized in figure 2. The same base BCQ algorithm is used with modified objective for BCQ (Fujimoto et al., 2019) where the results for applying off-policy importance weights denoted as "BCQ+IW.". We employ the same technique to obtain wss, a) for both the baseline and for adding variance regularization described. The results suggest that adding the proposed per-step variance regularization scheme significantly outperforms just importance weighting the expected rewards for off-policy policy learning.
2003
3200
2000
8800
1000
Sppodd
BCO-1w RC0-WAR
BOQ+VAR
203
#1 Coadick Suiee D0
Crudeet Sugo (NI
#f Goaddea Stepx
(a) Hopper Expert ablation
(b) Hopper Medium ablation
(c) Hupper Random ablation
(d) Hopper Mixed ablation
Figure 2: Ablation performed on Hopper. The mean and standard deviation are reported over 5 random seeds. the corresponding ones in Fig of the main paper.
The offline datasets for these experiments
E.2 EXPERIMENTAL RESULTS IN CORRUPTED NOISE SETTINGS
We additionally consider setting where the batch data is collected from noisy environment, i.e, in settings with corrupted rewards, r + r + e, where e ~ / (0, 1). Experimental results are presented in figures 1, 3. From our results, we note that using OVR top of BCQ (Fujimoto et al., 2019), we can achieve significantly better performance with variance minimization, especially when the agent is given sub-optimal demonstrations. We denote it medium (when the dataset was collected by a half trained SAC policy) or a mixed behaviour logging setting (when the data logging policy is a mixture of random and SAC policy). This is also useful for practical scalability, since often data collection is
20

---

Under review as a conference paper at ICLR 2021
Domain
Task Name
BCQ+OVR
BCQ
BEAR
BRAC-p
aDICE
SAC-off
pen-human
64.12
56.58
-1
8.1
-3.3
6.3
hammer-human
1.05
0.75
0.3
0.3
0.3
0.5
door-human
0.00
0.00
-0.3
-0.3
0
3.9
relocate-human
-0.13
-0.08
-0.3
-0.3
-0.1
0
pen-cloned
40.84
41.09
26.5
1.6
-2.9
23.5
hammer-cloned
0.78
0.35
0.3
0.3
0.3
0,2
Adroit
door-cloned
0.03
0.03
-0.1
-0.1
0
0
relocate-cloned
-0.22
-0.26
-0.3
-0.3
-0.3
-0.2
pen-expert 99.32
89.42
105.9
-3.5
-3.5
6.1
hammer-expert
119.32
108.38
127.3
0.3
0.3
25.2
door-expert
100.39
101.33
103.4
-0.3
7.5
relocate-expert 31.31
23.55
98.6
-0.3
-0.1
-0,3
Table The results on D4RL tasks compare BCQ (Fujimoto et 2019) with without OVR, bootstrapping emrrr reduction (BEAR) (Kumar et al., 2019), behavior-regulariveed actor critic with policy (BRAC-p) (?), AlgeaDICE, (aDICF) (Nachum et al., 201 and offline SAC (SAC-off) (Haarnoja et al., 2018). The results presented are the normalized returns the task per Fu et al. (2020) (see Table in Fu et al. (2020) for the unnormalized scores each task).
expensive from an expert policy. We add noise to the dataset, to examine the significance of OVR under noisy corrupted dataset setting.
2300
200
310
1903
RD0
300
2002
B00+NWR
270
0.4
Crachee Mhem 190
afd Siees IVD
(a) Hopper Export w/ Noise
(b) Hopper Medium w/ Noise
(c) Hopper Random w/ Noise
(d) Hopper Mixed w/ Noise
2000
1000
noo-
1000
1300
B00
900 VAI
1000
200
D00
800
BDQ B0Q
0.0
0.4
0.0
0.2
0.6
0.0
0.6
02
(e) Walker Expert
Medium
(g) Walker
Mixed Noise
Figure Evaluation the proposed approach and the baseline BCQ on suite three OpenAl Gym environments. We consider setting rewards that are corrupted by Gaussian noise. Results for uncorrupted version are in Fig. 1. Experiment results are averaged over random seeds
E.3 EXPERIMENTAL RESULTS ON SAFETY BENCHMARK TASKS
Safety Benchmarks for Variance as Risk We additionally consider safety benchmarks for control tasks, to analyse the significance of variance regularizer as risk constraint in offline policy optimization algorithms. Our results are summarized table 3.
E.4 DISCUSSIONS ON OFFLINE PFLICY OPTIMIZATION WITH STATE-ACTION DISTRIBUTION RATIOS
In this section, we include several alternatives by which we can compute the stationary state-action distribution ratio, borrowing from recent works (Uehara & Jiang, 2019; Nachum et al., 2019a).
Off-Policy Optimization with Minimax Weight Learning (MWL) We discuss other possible ways of optimizing the batch off- off-policy optimization objective while also estimating the state-action density ratio. Following from (Uchara & Jiang, 2019) we further modify the off off-policy optimization part of the objective (() in L0, , X) min-max objective, consisting of weight learning wx/D
21

---

Under review as a conference paper at ICLR 2021
Table 3: Results the Safety-Gym envi- ronments Ray et al..
PointGoall
PointGoal2
Reward
Cost
Reward
Cost
We report the mean and S.D. of episodic returns and costs five random seeds and I million timesteps.
BCQ
43.1 = 0.3
137.0 + 3.6
32.7 0.7
468.2 = 9.1
BCQ+OVR
44.2 + 0.3 127.1 + 4.0
33.2 + 0.7
453.9 + 7.3
PointButton1
PointButton2
The goal of the agent is maximize the episodic return, while minimizing the cost in- curred.
Reward
Cost
Reward
Cost
BCQ
30.9 + 2.2
330.8 + 8.3
18.1 + 1.1
321.6 + 4.1
BCQ+OVR
30.7 + 2.3
321.5 + 6.8
19.6 = 1.0 305.7 = 6.1
and optimizing the resulting objective J(8, w). We further propose an overall policy optimization objective, where single objective can be used for estimating the distribution ratio, evaluating the critic and optimizing the resulting objective. We can write the off aff-policy optimization objective with its equivalent starting state formulation, such that we have:
(s.a) D a) r(s, a)] = (1 7) a0 Q an)
(66)
Furthermore, following Bellman equation, expect to have E[r(s, a)] B|Q (s, a) - (s= (dd((,a) (T.//D(s,a)- (s, a) Qs"
(o Q² (2 ao) (67)
We can therefore write the overall objective as:
J [w, Q) = Edd(s.)) Wnn/D(B,,a)
-00 (s',a')) a')}
- -(1- 7) sp s
(1000(1)))))) -000
( on)
(68)
This is similar to the MWL objective in (Uchara & Jiang, 2019) except we instead consider the bias reduced estimator, such that accurate estimates of Q or w will lead to reduced bias of the value function estimation. Furthermore, note that in the first part of the objective [mT,, w. Q) we can further use entropy regularization for smoothing the objective, since instead of Q= a') in the target, we can replace it with a log-sum-exp and considering the conjugate of the entropy regularization term, similar to SBEED (Dai al., 2018). This would therefore give the first part of the objective as an overall min-max optimization problem:
J [[,, mp) = (s.0) [wr//((s a) fr(s, a) + + Tog (s) - (s..a))
+ (1 (1-2) (1- a0 (so, on)
(69)
such that from our overall constrained optimization objective for maximizing 0 we have turned it into a min-max objective, for estimating the density ratios, estimating the value function and maximizing
the policies
wHyD: Q' n argmin argmax J(TT, w, Q)²
(70)
w.Q "
where the fixed point solution for the density ratio can be solved by minimizing the objective
/ = argmin [(Wa/D, Q)² (s.a) [Tw(s, a) ("'(s' a')
(1 (1-7)Bs(s.a)) Q (so. ( an)
(71)
DualDICE: :
In contrast to MWL (Uchara & Jiang, 2019), DualDICE (Nachum et al., 2019a)
introduces dual variables through the change of variables trick, and minimizes the Bellman residual of the dual variables v(s, a) to estimate the ratio, such that
v*(8, a) v0 (s, a) = wz/D(s, a)
(72)
the solution which be achieved by optimizing the following objective
min C(v) = - Edt (v - (Bl-BBuv( a)² - (1 - (
1
((50 an)
(73)
Minimizing Divergence for Density Ratio Estimation : The distribution ratio can be estimated using objective similar GANs (Goodfellow et al., 2014; Ho & Ermon, 2016), as also similarly
22

---

Under review as a conference paper at ICLR 2021
proposed in (Kostrikov et al., 2019).
max ((h) (s,))) log a)) + B(s,))=d= log(1 h(s,a)))
(74)
where h is the discriminator class, discriminating between samples from dp and dz. The optimal discriminator satisfies:
log h" (s, a) log(1 (s,, = log dD(s,a)
(75)
(s,
The optimal solution of the discriminator is therefore equivalent to minimizing the divergence between d and dp. since the KL divergence is given by
- -Kk (d (s.a) log dp(s,a)
(76)
du (s)
Additionally, using the Donsker-Varadhal representation, we can further write the KL divergence term as:
DKL (dxk(dd1(dl (d=l(d)) = min log (s.a))dd exp a) - (s.a)dd, (a(ss a)]
(77)
such that now, instead of the discriminator class h, we learn the function class æ, the optimal solution to which is equivalent to the distribution ratio plus constant
dz ds
x² (s. a) log dp(s.a)
(78)
However, note that both the GANs like objective in equation 74 or the DV representation of the KL divergence in equation 77 requires access samples from both de and do. In our problem setting however, we only have access to batch samples dp. To change the dependency on having access to both the samples, we can use the change of variables trick, such that z(s, a) v(s, a) B" a). to write the DV representation of the KL divergence as:
(d= dp) min log E (s.o)~do
exp v(s, a) Bd v(s - B(naaad, -B a)
(79)
where the second expectation can be written expectation over initial states, following from DualDICE, such that we have
-Dxu (d=lldp) min log ((aa))do -B -L-)))
(((o an)
(80)
By minimizing the above objective w.r.t u, which requires only samples from the fixed batch data dp and the starting state distribution. The solution the optimal density ratio is therefore given by
(= a) x" as) = (s) 13Th v" a) = llog ds a)
log "oggw"(s.. (s,
(81)
Empirical Likelihood Ratio We can follow Sinha et al. (2020) to compute the state-action likelihood ratio, where they use binary a classifier to classify samples between an on-policy and off-policy distribution. The proposed classifier, ø, is trained on the following objective, and takes as input the state-action tuples (s, a) to return probability score that the state-action distribution is from the target policy. The objective for ø can be formulated as
Lets max -E, EE,a- [log(g (oog(((s,) a))] + Pa--(llo(dde (s))]
(82)
where s, a ~ D are samples from the behaviour policy, and s, IT (s) are samples from the target policy. The density ratio estimates for given s, D are simply w(s, a) ( a)) like in Sinha et al. (2020). We then use these w(s, a) for density ratio corrections for the target policy equantion ??
23


Reviews:
[{"review_id": "nxe8zwmpHJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\nThis work is based on a recent work of [Bisi et al., 2019], where a per-step reward formulation is presented with some outstanding unresolved problems, e.g., the double-sampling issue and the policy-dependent reward issue. This paper proposes to use the Fenchel duality to solve the double sampling problem and extend it to the behavior-agnostic off-policy setting by leveraging the density ratio estimation technique.\n\nMajor concerns:\n\n1 \nThe derivations in several major equations are WRONG. The objective in Eq 5 is NOT the same as the objective in Eq 6. \\E_{s, a ~ d_D}[\\omega(s, a)r(s, a)] in Eq 6 is NOT equal to \\E_{s~D}[Q^\\pi(s, \\pi(s)] in Eq 5. \n\n2. The motivation and empirical demonstration of the variance regularization are unclear.\nFirst, the definition of the variance doesn’t make sense to me. The variance in mean-variance optimization is a long-established term, which refers to the var of the return (either one-step or cumulative). So it is not clear why V_P makes sense without further motivation or reference. Moreover, the variance term defined in Eq. 2 is very weird. It is neither the variance of the return nor the so-termed “variance of the marginalized IS” (since it involves the reward r). By definition, the variance V_D is the variance of d_\\pi(s, a)/d_\\mu(s, a)r(s, a). This expression involves both \\pi, \\mu, and r, and its randomness comes from the randomness of (s, a). It is unclear why minimizing this variance is useful.\n\nSecond, the empirical results are not convincing. As pointed out by Eq. 3, the overall objective is to achieve a trade-off E[Q] and V_D(\\pi) through \\lambda. The empirical results, however, do not show this trade-off. Then it becomes unclear where the empirical improvement comes from. I would like to see how changing \\lambda influences V_D(\\pi). \n\n3 Several key deductions, which hold in on-policy cases, may NOT be true in this paper’s off-policy setting. For example, Eq . 8. lacks proof. In Bisi’s setting, this holds only for on-policy cases. I don’t think it still holds for off-policy settings. The author needs to prove it.\nThe MDP setting is unclear. The authors consider an infinite horizon MDP, so what does T mean? The proof of Lemma 1 also seems problematic. First, without a clear definition of T, there is no way to check the proof of Lemma1. Is T a random variable? Second, Eq 33 is wrong. Eq 33 is the same as Eq 24, but the definition of D^\\pi is different, so how can they be the same?\nMoreover, It is hard follow the inequality in Eq 34. It looks wrong to me. Thee reviewer strongly suggest that the author write it step by step to make it clear? And also, it would be great to show how the products of IS in Eq 34 reduce to the density ratio in Eq 35.\n\n4. Theorem 2, which is the paper's major theoretical contribution,  is obvious and trivial. By definition, the first term of RHS of Eq 16 is exactly J(\\pi). So what theorem 2 says is that J(\\pi) \\geq J(\\pi) \\sqrt{c * variance of sth}}. This is fairly obvious and does not bring in any insight.\n\n5. The entire Appendix B.2 is wrong, where the 3rd equality (aka, line 2 of Eq. 39) does NOT necessarily hold. The term d_\\pi(\\theta)’s gradient is not computed at all, and therefore, any results afterward are not correct.\n\n6. There are some missing references as well. Using Fenchel Duality to solve the double-sampling issue in mean-variance optimization using variance as regularization has been solved by previous literature, e.g., Xie et al., (2018) and Zhang et al., (2020). The author should acknowledge this. Also, I encourage the authors to compare with them. Especially I think the authors may want to compare with Zhang et al., (2020). Algorithm 1 is very similar to the offline MVPI in Zhang et al., (2020). There is only a slight difference in computing the augmented reward.\n\nXie, T., Liu, B., Xu, Y., Ghavamzadeh, M., Chow, Y., Lyu, D., & Yoon, D. (2018). A block coordinate ascent algorithm for mean-variance optimization. In Advances in Neural Information Processing Systems (pp. 1065-1075).\nZhang, S., Liu, B., & Whiteson, S. (2020). Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning. arXiv preprint arXiv:2004.10888.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1AzapEGTDLa", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\n\nThis paper proposes a novel algorithm for offline policy optimization. The main idea is to prevent overestimation bias by regularizing against the variance of the importance weighted value estimate. There are two key modifications: (1) using an importance weight from the stationary distribution and (2) using Fenchel duality to introduce a min-max problem to avoid double sampling when estimating the gradient of the variance regularization term. The theory section motivates the use of variance regularization and the experiments show improvements over BCQ when adding the proposed variance regularization algorithm. \n\n--------------------------------------------------------------------\n\n**Strengths:**\n\n1. The paper provides theoretical motivation for variance regularization. Theorem 2 demonstrates that using variance regularization can be seen as optimizing a lower bound on the true value of a policy. \n2. The algorithm provides a novel way to implement variance regularization. By introducing the $ \\nu $ variables, the paper proposes a way to get around double sampling issues in estimating the gradient of the regularizer. \n\n--------------------------------------------------------------------\n\n**Weaknesses:**\n\n1. There are several technical steps that I am not convinced by. I am not sure if these are mistakes or just statements that require more explanation: \n   1. In the algorithm, $ \\nu $ is defined by $ \\tilde r $ and $ \\tilde r $ is defined by $ \\eta $ in equation (7). This definition seems somewhat circular and the paper never clearly explains why this should work. \n   2. The continual updating of $ \\tilde r $ seems to make the problem nonstationary for the policy improvement subroutine that depends on the current iterate of the distribution ratio $ \\omega_\\psi$. The argument made in a remark on page 4 does not seem to sufficiently resolve this issue since even though $ \\nu $ has a closed form, that closed form depends on the distribution ratio which depends on the current policy. \n   3. The variance term that shows up in Theorem 1 does not seem to be what is being estimated by $ V_D $ in the algorithm. In light of this, what is the point of Theorem 1 vis a vis the proposed algorithm?\n   4. Theorem 2 shows that using the square root of the variance as a regularizer provides a lower bound, but the algorithm uses the variance. What is the explanation for this mismatch?\n2. Notation is somewhat confusing and sloppy. For example, $J$ is defined in several different ways, first in equation (1) and then in equation (3). This makes the rest of the paper confusing since it is sometimes unclear which $J$ is being referred to. Another example is that $ V_D $ is defined as a function of $ \\pi $ in equation (2) and then a function of $ \\omega, \\pi $ in equation (3). Equation (4) just refers to $ V $ when I think it means $ V_D$. Also in equation (4) the LHS has no $s,a$ variables but the first two terms on the RHS are a function of $ s,a $. Where do these $s,a$ come from? Is there an expectation missing? In equation (5) it is not clear which $ \\omega $ is being referred to. Equation (6) overloads $J$ once more.  Equation (6) could also use a better explanation as to why this is the dual form of (5) rather than just asserting it. The definitions of $ \\epsilon_\\pi, L_\\pi $ are not included in Lemma 2, but are instead in the following paragraph. In equation (12) $ d_\\pi $ is denoted by $\\beta_{\\pi'}$ for no apparent reason. In equation (13), $f$ is supposedly a dual class of functions, is there a sup missing? In equation (15), ${\\hat \\omega_{\\pi /D}}$ is never explicitly defined. This sort of sloppiness pervades the paper and makes it often difficult to understand.\n3. The writing is unclear. One major issue is that lemmas are presented with no context or direction. It would be helpful to preface each section with an explanation of where the section is going before presenting technical lemmas. This is a problem for both Lemma 2 and Lemma 3 where it is unclear to the reader what the point of the lemma is until much later in the section. This is also a problem for Lemma 1 which has it's own subsection that does not seem to make any clear claim to connect to the thesis of the paper. The lemmas may be better suited in the appendix rather than the main text of the paper or just require more explanation. \n4. Empirical results are not very convincing. The plots presented in the experimental section seem to show a slight but not large or consistent advantage for the proposed method. Perhaps more worrying, the only type of result reported are the final performance of the algorithms. There is no empirical indication as to how the algorithm is working or whether the variance regulation is indeed having the desired effect of reducing overestimation. Moreover, there is no indication that the Fenchel duality tricks are doing anything useful empirically.\n5. Important ablations and baselines are missing. The authors do not include conservative Q learning (Kumar et al., 2020) as a baseline saying that it is too recent to compare. However, this paper came out in June and according to the ICLR reviewer guidelines we are only meant to consider work released in August or later as contemporaneous. So, CQL ought to be included as a baseline and I think generally outperforms the proposed methods. Additionally, while there is one ablation included in the appendix (adding importance weighting to BCQ), more ablations are needed. Specifically, since the main algorithmic innovation of the paper not just using variance regularization, but computing gradients using the Fenchel duality max-min procedure, there should be ablations showing whether this part of the algorithm is indeed necessary and useful. \n\n--------------------------------------------------------------------\n\n**Recommendation:**\n\nI recommend rejection and gave the paper a score of 4. While there may be a good idea in there, I do not think the paper is ready to publish. A more clear and careful exposition of the algorithm as well as more rigorous theory and experiments are needed.\n\n--------------------------------------------------------------------\n\n**Additional feedback:**\n\nTypos:\n\n- The first sentence of the intro has a space before the period. \n- The references on the top of page 2 seem to be incorrect. \"A.\" is the first initial of the first author, not their last name.\n- Throughout the paper there is usually a space preceding each colon. There should be no spaces before each colon.\n- After equation two \"a as\" should be \"as a \"\n- In the first sentence of section 3.4 \"leads\" should be \"lead\"\n- The paper sometimes refers to the proposed variance regularization as a \"framework\" (e.g. in the first sentence of the conclusion). It is not a framework, it is an algorithm.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0D35vqc5NS_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Disclaimer: this paper was assigned to me as an emergency review paper, so I might be missing something important.\n\n### Evaluation\nI recommend rejection. I think the paper presents an interesting idea to regularize policy updates by the variance of a return. Besides, it proves that the return-variance-regularized policy update leads to a monotonic policy improvement, which I think is novel. That being said, the paper seems to have several issues that decreased my evaluation. First, the clarity of the paper is really low: many typos, ambiguous notations, and confusing presentation of a new algorithm. Second, I could not understand some parts of the paper, especially a proof of the monotonic policy improvement theorem. Third, it is unclear why the proposed regularizer is suitable for offline RL.\n\n### Paper Summary\nOffline RL is (said to be) a key to enable RL applications to real world problems. The paper proposes a new algorithm, which regularizes policy updates by the variance of return, for offline RL. The paper proves a monotonic policy improvement when the return-variance-regularized policy update is used. Experiments show moderate performance gain by the regularization.\n\n### Strong Points\n1. Interesting idea to regularize policy updates by the variance of a return\n2. Moderate performance gain, especially when offline dataset is obtained by a suboptimal policy.\n\n### Weak Points\n1. The paper is not well-written. It contains typos, unclear sentences, and ambiguous notations.\n2. Some parts of the paper, like the proof of the monotonic policy improvement theorem, seem to contain mistakes. (I might be wrong, though.)\n3. The performance gain seems to be moderate, despite a high complexity of the computation of the proposed regularizer.\n\n### Comments to the Authors\n\nIf I am missing something, please feel free to let me know. As noted above, I could not spare much time to review the paper.\n\n1. The paper is not well-written. Please revise it again. (I don't point out all ambiguous notations and typos, but later I point out some serious ones.) Besides, please revise the references section. It refers to arxiv versions of papers that were accepted to conferences.\n2. In page 3, $\\omega_{\\pi / \\mathcal{D}}$ suddenly appears. What does it mean? Maybe $\\omega_{\\pi / \\mu}$?\n3. What does $s \\sim \\mathcal{D}$ mean? Does it mean $s \\sim d_\\mu$? Since the dataset $\\mathcal{D}$ contains states visited by $\\mu$, simply drawing states from $\\mathcal{D}$ will be different from $d_\\mu$.\n4. What is $d_{\\mathcal{D}}$ in, for example, Equation 4?\n5. The idea to use the Fenchel duality to avoid double-sampling problem seems to be not new (cf SBEED paper). While the paper mentions AlgaeDICE as an algorithm using a similar technique, it does not mention SBEED about the use of Fenchel duality. Why?\n6. In the beginning of Section 3.4, the min-max problem is being solved by repeating the inner minimization and outer maximization. As far as I remember (I might be wrong, though!), this way of solving a min-max problem might not find the exact solution. Isn't it a problem?\n7. In Equation 6, it seems that $Q^\\pi (s, a)$ is rewritten as $E_{(s, a) \\sim d_\\mathcal{D}} [ \\omega (s, a) r (s, a)]$. According to the notation of the paper, isn't $E_{(s, a) \\sim d_\\mathcal{D}} [ \\omega (s, a) r (s, a)]$ be $(1-\\gamma) E_{s_0 \\sim \\beta, a_t \\sim \\pi} [\\sum_{t=0}^\\infty \\gamma^t r(s_t, a_t)] \\neq Q^\\pi(s, a)$?\n8. Is Lemma 1 different from Lemma 1 of [Bisi et al, 2019](https://arxiv.org/pdf/1912.03193.pdf)? Also what is its meaning? Why is is useful to understand the variance regularizer?\n9. How is the beginning of Section 4.2 related to Theorem 1? Theorem 1 seems to be derived based on a different inequality in its proof.\n10. As far as I remember, the dual form of the total variation is $\\sup_{f \\in C_`1} E_{A \\sim P} f(A) - E_{B \\sim Q} f(B)$, where $C_1$ is the space of all continuous functions bounded by $1$. Therefore, we don't need $\\phi$ and can explicitly state the space of $f$ in Equation 12. Am I wrong or missing something? \n11. Why is there no sup over f in Theorem 1?\n12. As for Equation 59, how do you get the first line? In addition, do you need $E_{s \\sim d_\\pi}$? In the second line, the first $E_{s \\sim d_{\\pi'}, a \\sim \\pi}$ is sampling an action from $\\pi$. Isn't it $\\pi'$? In the fourth line, $d_{\\pi'}$ changed to $d_\\pi$. How is it possible?\n13. I don't fully understand what Section 4.3 does. In addition, what are random variables in Theorem 2?\n14. I don't understand what the final objective to be optimized is. In Algorithm 1, $J(\\theta, \\phi, \\psi, \\nu)$ appears. Is it the same as Equation 6?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are unanimous that the submission does not clear the bar for ICLR.
Decision: Reject