Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
Authors: Jiachen Li
Edwin Zhang
Ming Yin
Qinxun Bai
Yu-Xiang Wang
William Yang Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning algorithms
Deep Reinforcement Learning

Abstract: Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. By exploiting historical transitions, a policy is trained to maximize a learned value function while constrained by the behavior policy to avoid a significant distributional shift. In this paper, we propose our closed-form policy improvement operators. We make a novel observation that the behavior constraint naturally motivates the use of first-order Taylor approximation, leading to a linear approximation of the policy objective. Additionally, as practical datasets are usually collected by heterogeneous policies, we model the behavior policies as a Gaussian Mixture and overcome the induced optimization difficulties by leveraging the LogSumExp's lower bound and Jensen's Inequality, giving rise to a closed-form policy improvement operator. We instantiate an offline RL algorithm with our novel policy improvement operator and empirically demonstrate its effectiveness over state-of-the-art algorithms on the standard D4RL benchmark.

Under review as a conference paper at ICLR 2023
OFFLINE REINFORCEMENT LEARNING WITH CLOSED- FORM POLICY IMPROVEMENT OPERATORS
Anonymous authors
Paper under double-blind review
ABSTRACT
Behavior constrained policy optimization has been demonstrated to be successful paradigm for tackling Offline Reinforcement Learning. By exploiting historical transitions, policy is trained to maximize a learned value function while con- strained by the behavior policy to avoid a significant distributional shift. In this paper, we propose
closed-form policy improvement operators. We make novel observation that the behavior constraint naturally motivates the use of first-order Taylor approximation, leading to a linear approximation of the policy objective. Additionally, as practical datasets are usually collected by heterogeneous policies, model the behavior policies optimization difficulties by leveraging the LogSumExp's 's lower bound and Jensen's s Inequality, giving rise to a closed-form policy improvement operator. We instantiate offline RL algorithms with our novel policy improvement operators and empirically demonstrate their effectiveness over state-of-the-ee algorithms on the standard D4RL benchmark.
Gaussian Mixture and overcome the induced
INTRODUCTION
Deploying Reinforcement Learning (RL) (Sutton & Barto 2018) in the real world is hindered by its massive demand for online data. In domains such as roboties (Cabi et al. 2019) and autonomous driving (Sallab et al. 2017), rolling out a premature policy is prohibitively costly and unsafe. To address this issue, offline RL (a.k.a batch RL) (Levine et al., 2020 Lange et al. (012)) has been proposed to learn policy directly from historical data without environment interaction. However, learning competent policies from static dataset challenging. Prior studies have shown that learning a policy without constraining its deviation from the data-generating policies suffers from significant extrapolation errors, leading to training divergence (Fujimoto et al. 2019 Kumar et al. 2019).
Current literature has demonstrated two successful paradigms for managing the trade-off between pol- icy improvement and limiting the distributional shift from the behavior policies. Under the actor-critic 1999). behavior constrained policy optimization (BCPO) (Fujimoto ctt al.] 2019;; Kumar et al. 2019 Fujimoto & Gul 2021 Wu et al. 2019 Brandfonbrener et al. 2021
framework (Konda & Tiitsiklis
Ghasemipour al. 2021) explicitly regularizes the divergence between learned and behavior policies, while conservative methods (Kumar et al. 20206 Bai et al.) 2022 Yu et al., 22000) (021) penalize the value estimate for cut-o-diistriutton (OOD) actions to avoid overestimation error. However, most existing model-free offline RL algorithms still require learning off-policy value functions and target policy through stochastic gradient descent (SGD). Unlike supervised learning, off-policy learning with non-linear function approximators and temporal difference learning (Sutton & Barto 2018) notoriously unstable (Kumar et al. 2020a Mnih et al.) 2013 Henderson al.) Konda & Tsitsiklis 1999 Watkins & Dayan 1992) due to the existence of the deadly-triad (Sutton & Barto 2018 Van Hasselt et al. [ 2018)) The performance can exhibit significant variance even across different random seeds (Islam et al. (017). In offline settings, lcarning becomes even more problematic as environment interaction is restricted, thus preventing the learning from receiving corrective feed- back (Kumar al. 2020a). Consequently, training stability poses a major challenge in offline RL. Although some current approaches (Brandfonbrener et al. (002)) circumvent the requirement for learning off off-po value function, they still require learning a policy via SGD.
Can we mitigate the issue of learning instability by leveraging optimization techniques? In this paper, we approach this issue from the policy learning perspective, aiming design stable policy

---

Under review as a conference paper at ICLR 2023
improvement operator. We take closer look at the BCPO paradigm and make novel observa- tion that the requirement of limited distributional shift motivates the use of the first-order Taylor approximation (Callahan 2010), leading to linear approximation of the policy objective that is accurate in a sufficiently small neighborhood of the behavior action. Based on this crucial insight, we construct our policy improvement operators that return closed-form solutions by carefully designing a tractable behavior constraint. When modeling the behavior policies as a Single Gaussian, our policy improvement operator deterministically shifts the behavior policy towards a value improving direction derived by solving a Quadratically Constrained Linear Program (QCLP) in closed form. Therefore, our method only requires learning the underlying behavior policies of a given dataset with supervised learning, avoiding the training instability from policy improvement.
Furthermore, we note that practical datasets are likely to be collected by heterogeneous policies, which may give rise to multimodal behavior action distribution. In this scenario, Single Gaussian will fail to capture the entire picture of the underlying distribution, limiting the potential of policy improvement. While modeling the behavior as a Gaussian Mixture provides better expressiveness, it incurs extra optimization difficulties due to the non-concavity of its log-likelihood function. We tackle this issue by leveraging the LogSumExp's 's lower bound and Jensen's 's inequality, leading to a closed-foe form policy improvement (CFPI) operator compatible with a multimodal behavior policy. Empirically, we demonstrate the effectiveness of Gaussian Mixture over the conventional Single Gaussian when the underlying distribution comes from hetereogenous policies.
In this paper, we empirically demonstrate that our CFPI operators can instantiate successful offline RL algorithms in one-step or iterative fashion. Moreover, our methods can also be leveraged to improve policy learned by the other algorithms. In summary, our main contributions are threefold:
CFPI operators compatible with single mode and multimodal behavior policies.
An empirical demonstration of the benefit to model the behavior policy as Gaussian Mixture in model-free offline RL. To the best of our knowledge, we are the first to do this,
One-step and iterative instantiations of our algorithm, which outperform state-of-the-art (SOTA) algorithms on the standard D4RL benchmark (Fu et al. 2020
2 PRELIMINARIES
Reinforcement I Learning. RI aims to maximize returns in a Markov Decision Process (MDP) (Sutton & Barto 2018) M (S, A. T. with state space S, action space A, reward function R, transition function T. initial state distribution Po. and discount factor y E [o, 1). At each time step the agent starts from a state s; E S, selects an action o, T(-[st) from its policy æ, transitions to a new state St+1 ~ T(-|st, 0z), and receives reward := R(st, oz). The goal of an RL agent is to learn an optimal policy
that maximizes the expected discounted cumulative reward E,, Er y'r,] without access to the ground truth I and
T. We define the action value function associated with by
Q (s, = E, Er rrre ( = 8, 4g = a). The RL objective can then be reformulated as
arg max J (TT) == EBepo 1Q
(I)
In this paper, consider offfine RL settings, where assume restricted access to the MDP M. and a previously collected dataset D with N transition tuples {s} at. r policy that generates DD Ty, which may or may not be mixture of individual policies.
f=1- We denote the underlying
Behavior Constrained Policy Optimization. One of the critical challenges in offline RL is that the learned Q function tends to assign spuriously high values to OOD actions due to extrapolation error, which is well documented in previous literature (Fujimoto et al. 2019 Kumar et al. (019)). Behavior Constrained Policy Optimization (BCPO) methods (Fujimoto et al. [219 Kumar al.)
2019:Fujimon & Gu 2021 Wu et al., 2019 Brandfonbrener et al. 2021) explicitly constrain the action selection of the learned policy to stay close the behavior policy T3. resulting in a policy
improvement step that can be generally summarized by the optimization problem below:
max E.DD [Ea-((-s) IQ (s, a)] aD (T(- s), TT(- s))].
(2)
where (...)) is divergence function that calculates the divergence between two action distributions, hyper-parameter parameter controlling the strength of regularization. Consequently, the policy is optimized to maximize the Q-value while staying close to the behavior distribution.
and a is a hyper
2

---

Under review as a conference paper at ICLR 2023
Different algorithms may choose different D...-) (e.g.. ... KL Divergence (w et al. 2019 Jaques et al. 2019), MSE (Fujimoto Gu, 2021) 1 and MMD (Kumar et al. 2019)). However, to the best of our knowledge, all existing methods tackle this optimization via SGD. In this paper, take advantage of the regularization and solve the problem in closed form.
CLOSED-FORM POLICY IMPROVEMENT
In this section, we introduce our policy improvement operators that map the behavior policy to higher-valued policy, which is accomplished by solving linearly approximated BCPO. We show that modeling the behavior policy as a Single Gaussian transforms the approximated BCPO into a QCLP and thus can be solved in closed-form (Sec. 3.T. Given that practical datasets are usually collected by heterogeneous policies, we generalize the results by modeling the behavior policies as a Gaussian Mixture facilitate expressiveness and overcome the incurred optimization difficulties by leveraging the LogSumExp's lower bound (LB) and Jensen's Inequality (Sec. (.2). We close this section by presenting an offline RL paradigm that leverages our policy improvement operators (Sec. 3.3
3.1
APPROXIMATED BEHAVIOR CONSTRAINED OPTIMIZATION
We aim to design a learning- free policy improvement operator to avoid learning instability in offline settings. We observe that optimizing towards BCPO's policy objective (2) induces a policy that admits limited deviation from the behavior policy. Consequently, it will only query the Q-value within the neighborhood of the behavior action during training, which naturally motivates the employment of the first-order Taylor approximation to derive the following linear approximation of the Q function
(a az)² [Va ( al==a + Q(s,(aa)))
=
=a" (VQ((s.))l.c + const.
By Taylor's theorem (Callahan 2010), Q(s, a: ag) only provides an accurate linear approximation of (s,aa) in sufficiently small neighborhood of a... Therefore, the choice of ag is critical.
Recognizing (2) as a Lagrangian and with the linear approximation following surrogate problem of ( given any state
we propose to solve the 33)
max
E
[VQ((, (lo=as s.t. ( 8), x.(. |s) s 6. a
(4)
Note that is not necessary for D(...) to be (mathematically defined) divergence measure since any generic -). that can constrain the deviation of I'' action from Fy can considered.
Single Gaussian Behavior Policy. In general, (4) does not always have a closed-fom solution. We analyze special case where TH =N(u,, Σş) a Gaussian policy, and ((.,)) -) is a negative log-likelihowe function. In this scenario, a reasonable choice of should concentrate around μı limit distributional shift. Therefore. we set ag fıg and the optimization problem 4 becomes the following:
fr deterministic policy,
max 11 (V((s,)))
s.t. log TH (
We now show that 5 has a closed-form solution. Proposition 3.1. The optimization problem
5) has a closed-form solution that is given by
May (Ty(T) ps + /2log T Eg (V.Q(s, a)aa na
where 8 = log detddetti))pp log
VaQ al Eg as)
(6)
Proof sketch. (5 can be converted into the QCLP below that has closed-form solution given by (Boyd et al. 2004 A fulll proof is given in the Appendix
1 2 (p - - (n (48) s 5 log det (2mE.)
1
max I VaQ(ss, ())s a) s.t.
(7)
#
3

---

Under review as a conference paper at ICLR 2023
Although still have to tune tuning in for conventional BCPO methods, have transparent interpretation of r's effect on the action selection thanks to the tractability of (5). Due to the KKT conditions (Boyd al. 2004), (6) always returns action Mag with the following property
- (uglss)
1
log <a(ps)
log
( (s) T
(Molssss))
While setting T = will always return the mean of na, a large 7 might send Mog out of the support of TT, breaking the accuracy guarantee of the first-order Taylor approximation.
3.2 GAUSSIAN MIXTURE MORE EXPRESSIVE MODEL
Performing policy improvement with (6) enjoys favorable computational efficiency and avoids the potential instability caused by SGD. However, tractability relies on the Single Gaussian assumption of the behavior policy Te. In practice, the historical datasets are usually collected by heterogeneous policies with different levels of expertise. A Single Gaussian may fail to capture the whole picture of the underlying distribution, motivating the use of a Gaussian Mixture to represent n..
N
N
T
XAN E.),
A
=1
However, directly plugging the Gaussian Mixture Ty into 5 breaks its tractability, resulting in non-convex optimization below
max 1
4 [VoQ(s,)))
s.t. log \, det exp (- 1 (- (-)(-- (4)) 11)) 2 -
(10)
We confronted with two major challenges to solve the optimization problem (TO). First, it is unclear how to choose proper ag while we need to ensure that the solution lies within a small neighborhood of a,3 Second, the constraint of (10) does not admit convex form, posing non-trivial optimization difficulties. We leverage the lemma below to tackle the non-convexity of the constraint.
Lemma 3.1. log E
Ei=1 X; exp(x;) admits the following inequality:
1. (LogSumExp's 's LB)
log EN > cxp((x)) 2 max; fa, + log }
2. (Jensen's 's Inqquality)
N log ΣN X exp(x,) Σ A,a
Next, we show that applying each inequality in Lemma 3.1 to the constraint of (10) respectively resolves the intractability and leads to natural choices 0,3.
Proposition 3.2. By applying the first inequality Lemma 3.7 to the constraint of 10 we can derive an optimization problem that lower bounds (T))
max 1r QV. ((,,)))o==a a))o=ag
(11)
-1
-5 2-0
S.L. max (ou-Ja) " (m us) log det + log 21 log
and the closed-form solution to is given by
s.1. 8 -= min L
pise (T) = -arg max 1E!
log det Ei) - log di + log
PL(5)
(12)
2(6 + log ). log det (2T Ex (V. Q(s a),
where p=(0) u4
[v. a))a E, [V. a))aa
Proposition 3.3. By applying the second inequality of Lemma 3. to the constraint of 10 we can derive an optimization problem that lower bounds (10
N
max T [v. (2).)))))))))))) a=0==
Σ Ae
(-2
log del deddeeet222 (2+E.)
2 "2, (p Hii) -8 (13)
4

---

Under review as a conference paper at ICLR 2023
Ground Truth Iroquallyy LeggSmmappss..
Figure I: Apply Lemma 3.1 to Gaussian Mixture's log probability log F at different scenarios. (L) log T3 has multiple modes. LogSumExp's LB preserves multimodality. (M) log Ta reduces to Single Gaussian, Jensen's 's inequality becomes equality. (R) log xg similar uniform distribution.
and the closed-form solution
13 is given by
2 2log +
Ex A Dj Mu + E a
Hjenenn (T)
E ((s.)ap
[v.((s.a) FlVoQ(( allamp
(14)
where Σ
Σ ( Ā=
x Σ
Ms
ā = log 2
Σ A log deg
We defer the detailed proof of Proposition 3.2 and Proposition 3.3 as well how choose ag for each optimization problem to Appendix (A.2 and A.3 respectively.
Indeed, these two optimization problems have their own assets and liabilities. When ne exhibits an obvious multimodality as is shown in Fig. (L), the lower bound of log F constructed by Jensen's Inequality cannot capture different modes due its concavity, losing the advantage of modeling m as Gaussian Mixture. this case, the optimization problem (II) can serve as a reasonable surrogate problem of [T), LogSumExp's LB still preserves the multimodality of log nT..
When B is reduced to a Single Gaussian, the approximation with the Jensen's Inequality becomes equality is shown in Fig. (M). Thus Mjensen returned by (14) exactly solves the optimization problem (00). However, in this case, the tightness of LogSumExp's LB largely depends on the weights Ai=I...N. If each Gaussian component distributed and weighted identically, the lower bound will be log N lower than the actual value. Moreover, there also exists the scenario (Fig. (R)
when both and (33) can serve reasonable surrogates the original problem
10
Fortunately, we can combine the best of both worlds and design a policy improvement operator accounting for all the above scenarios. As both Proposition 3.2 and 3.3 have closed-form solutions, the operator returns policy that selects the higher-valued action from jise and [ijensen
(mg(T) arg max,, arg (s,,)), s.t. / E (/4se(T), Pjensen (T)}
(15)
3.3 ALGORITHM TEMPLATE
We have derived two CFPI operators that map the behavior policy to a higher-valued policy. When the behavior policy ky is Single Gaussian, I Q; returns a policy with action selected by (6). When Tg is a Gaussian Mixture, [MG Q: T) returns policy with action selected by (15). We note that our methods can also work with non-Gaussian πy. Appendix (eiixxDDppoooo provides the derivations for the corresponding CFPI operators when T is modeled as both a deterministic policy and VAE. Algorithm shows that our CFPI operators enable the design can yield one-step, multi-step and iterative methods, where & is general policy evaluation operator that returns value function Q. When setting T = 0, we obtain our one-step method. We defer the discussion on multi-step and iterative methods to the Appendix C
general offline RL. template that
While the design of our CFPI operators is motivated from the behavior constraint, highlight that they are compatible with general baseline policies nu besides Tg. Sec. 5.2 and Appendix G. show
that our CFPI operators can improve policies learned by IQL and COL (Kumar et al. 2020b
3.4 THEORETICAL GUARANTEES FOR CLOSED-FORM POLICY IMPROVEMENT
follows the approximate policy iteration (API) Perkins Precup/ 2002) by iterating over the policy evaluation (E step, Line 4) and policy improvement (I step. Line
At a high level, Algorithm |
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Offline RL with closed-form policy improvement operators (CFPI)
Input: Dataset D, baseline policy it. value function Q- HP
1: Warm start Qu = SARSA(Q-1: D) with the SARSA-style algorithm Sutton & Barto 2018
2: Get one-step policy #1 I(ft. Q0:T)
3: for = do
4: Policy evaluation: Q E m,, ()
5: Get policy: $441 Tm Q(:))
(conerete choices of I includes IMg and Isg)
6: end for
5). Therefore, to verify provides the improvement, we need to first show policy evaluation Qe is accurate. We employ the Fitted Q-Iteration (Sutton & Barto 2018) to perform policy evaluation, which is known to be statistically efficient (e.g. (Chen & Jiang 2019)) under the mild condition for
the function approximation class. Next, for the performance gap between J(a++1) - - we apply
the standard performance difference lemma (Kakade & Langford 2002: Kakade, 2003).
Theorem 3.1. [Safe Policy Improvement] Assume the state and action spaces are discrete Let #1 be the policy obtained after the CFPI update (Line of Algorithm [). Then with probability 6,
1
((+i) -(tt))
E Eanain
- -
2 E din Ba
C7,6 + CCFPI(S, = (D(s,a)
For multi-step T iterative update, we similarly have with probability 1-
-J Σ J(
10
where ((s,o) denotes number of samples at s, a, Cy.8 denotes the learning coefficient of SARSA and Ccrri(s, a) denotes the first-order approximation error from
B We defer detailed derivation and
the expression of Cr,6. (0) and CcrPi(s, a) in Appendix A.4 When ag, CCFPI a) O.
By Theorem 3.1 #ं is a Ç-safe improved policy. The ( safeness consists of two parts: CCFPI is caused by the first-order approximation, and the C,,5/ v VD(s, a) term is incurred by the SARSA update. Similarly, का is Dt= (()- improved policy.
4 RELATED WORK
Our methods belong and are motivated by the successful BCPO paradigm, which imposes constraints in (2) to prevent from selecting OOD actions. Algorithms from this paradigm may apply different divergence functions, e.g., KL-divergence (Wu et al. 2019 Jaques et al. 2019), MMD (Kumar et al., 2019) or the MSE (Fujimoto & Gul 2021). All these methods perform policy improvement via SGD. Instead, we perform CFPI by solving a linear approximation of (2). Another line of research enforces the behavior constraint via parameterization. BCQ (Fujimoto et al., 2019) learns a generative model as the behavior policy and Q function to select the action from set of perturbed behavior actions. Ghasemipour et al. 2202 further show that the perturbation model can be discarded.
The design of our CFPI operators is inspired by the SOTA online RL algorithm OAC (Ciosek et al. 2019). It treats the evaluation policy as the baseline na and obtains an optimistic exploration policy by solving similar optimization problem (7). We extend the result to accommodate multi-modal Ty and overcome the optimization difficulties by leveraging[Leme Lemma 3.1 In Appendix H we further draw connections with prior works that leveraged the Taylor expansion approach to RI..
Recently, one-step (Kostrikov et al. 2021 Brandfonbrener et al. 2021) algorithms have achieved great success. Instead of iteratively performing policy improvement and evaluation, these methods
Note here assume the discreteness only for the purpose of analysis. For the more general cases, please refer Appendix A.4 In Theorem 3.1 14, (s, also, ao) pa (sh 8, an a |so 8, ag a).

---

Under review as a conference paper at ICLR 2023
Table Comparison between our one-step policy and SOTA methods on the Gym-MuJoCo domain of D4RL. Our method uses the same т for all datasets except Hopper-M-E (detailed in Appendix (F.I). We report the mean and standard deviation of our method's performance across seeds. Each seed contains an individual training process and evaluates the policy for 100 episodes. We use Cheetah for HalfCheetah, M for Medium, E for Expert, and R for Replay. We bold the best results for each task.
SG-BC
MG-BC
OnestepRL
TD3+BC
CQL
IQL
(Ra, Qu)
Chestah-M-v2
40.6
40.6
42.6
46.9
55
48.3
44.0
47.4
52.1 + 0.3
Hopper.M
53.7
53.9
53.3
50.3
68.2
Walker2d-M-v2
71.9
70.0
74.0
79.8
85.6
83.7
72.5
78.3
88.5 1: 1.6
Cheetah-M-R-v2
34.9
33.0
36.6
42.4
44.6
44.5 0.4
Hopper-M-R-v2
12.1
21.2
82.7
91.5
71.0
60.9
95. 0
91.7
93.6 7.9
Walker2d-M-R-w2
22.9
22.8
66.6
71.6
$1.8
73.8
78.2 5.6
Cheetah-M-B-v22
46.6
51.7
86,8
95.0
93.5
90.7
91.6
86.7
97.3 1.8
Hopper-M-E-v2
53.9
69.2
107.6
101.9
102.1
98.0
105.4
91.5
104.2
Walker2d-M-E-v2
92.3
93.2
108.1
110.0
110.9
110.1
108.8
109.6
111.9 0.3
Total
429.1
455.6
672.6
710.1
716.0
677.4
608.5
692.4
757.0 +: 27.0
Table 2: Comparison between our Iterative IMo and SOTA methods on the AntMaze domain. We report the mean and standard deviation across 5 seeds for our method with each seed evaluating for 100 episodes. The performance for all baselines is directly reported from the IQL paper. Our Iterative IMG outperforms all baselines on 5 out of tasks and obtains the best overall performance.
Dataset
BC
Onestep RI.
TD3+RC
CQL
IQL.
lterative TMG
antmaze-u-v0
78.6
74.0
87.5
90.2 1
antmaze-u-d-v0
45.6
49.3
60.7
71.4
84.0
62.2
58.6
15.2
antmaze-m-p-x0
0.0
0.0
0.3
10.6
61.2
71.2
75.2 + 6.9
antmaze-m-d-vv
0.0
0.7
0.0
3.0
53.7
70.0
72.2 + 7.3
antmaze-l-p-v0
0.0
0.0
0.0
0.2
15.8
39.6
51.4
7.7
antmaze-l-d-v0
0.0
1.0
0.0
0.0
14.9
47.5
52.4 + 10.9
Total
100.2
112.2
125.3
163.8
303.6
378.0
400.0 + 52.0
only Iearn a Q function via SARSA without bootstrapping from OOD action value. These methods further apply an policy improvement operator (Wu al., 2019: Peng al. 2019) extract a policy. We also instantiate one-step algorithm with our CFPI operator and evaluate on standard benchmarks.
5
EXPERIMENTS
Our experiments aim to demonstrate the effectiveness of our CFPI operators. Firstly, on the standard offline RL benchmark D4RL (Fu al.) (020), we show that instantiating offline RL algorithms with our CFPI operators in both one-step and iterative manners outperforms SOTA methods (Sec. [5.1. Secondly, we show that our CFPI operator can improve a policy learned by other algorithms (Sec. 5.2). Ablation studies in Sec. 5.33 further shows our superiority over the other policy improvement operators and demonstrate the benefit of modeling the behavior policy as a Gaussian Mixture.
5.1 COMPARISON WITH SOTA OFFLINE RL ALGORITHMS
We instantiate one-step offiin RL algorithm from Algorithm with our policy improvement operator IMG. We learned a Gaussian Mixture bascline policy ž via behavior cloning. We employed the IQN (Dabney al. 2018a) architecture to model the value network for its better gencralizability. as we need to estimate out-of buffer Q(s, a) during policy deployment. We trained the Qu with SARSA algorithm (Sutton & Barto 2018 Parisotto et al. 2015). Appendix R. includes detailed training procedures of स and Qo with full HP settings. We obtain our one-step policy IMG (ft., T):
We evaluate the effectiveness of our one-step algorithm on the D4RL benchmark focusing on the Gym- MuJoCo domain, which contains locomotion tasks with dense rewards. Table 1 compares our one-step algorithm with SOTA methods, including the other one-step actor-critic methods IQL (Kostrikov et al. 2021), OneStepRL (Brandfonbrener et al. 2021), BCPO method TD3+BC (Fujimoto & Gu 2021). conservative method CQL (Kumar et al. 2020b), and trajectory optimization methods DT (Chen et al. 2021), TT (Janner et al. 2021). We also include the performance of two behavior policies
7

---

Under review as a conference paper at ICLR 2023
SG-BC and MG-BC modeled with Single Gaussian and Gaussian Mixture, respectively. We directly report results for IQL, BCQ, TD3+BC, CQL, and DT from the IQL paper, and TT's result from its own paper. Note that OneStepRL instantiates three different algorithms. We only report its (Rev. KI. Reg) result because this algorithm follows the BCPO paradigm and achieves the best overall performance. We highlight that OnesteRL. reports the results by tuning the HP for each dataset.
Results in Table demonstrate that our one-step algorithm outperforms the other algorithms by a significant margin without training policy to maximize its Q-value through SGD. We note that we use the same 7 for all datasets except Hopper-M-E. In Sec. 5.3 we will perform ablation studies and provide a fair comparison between our CFPI operators and the other policy improvement operators.
We further instantiate an iterative algorithm with IMG and evaluate its effectiveness on the challenging AntMaze domain of D4RL. The 6 tasks from AntMaze are more challenging due their sparse- reward nature and lack of optimal trajectories in the static datasets. [Tbll compares our Iterative IMG with SOTA algorithms on the AntMaze domain. Our method uses the same set of HP for all 6 tasks, outperforming all baselines on 5 out of 6 tasks and obtaining the best overall performance. Appendix C.I presents additional details with pseudo-codes and training curves.
5.2 IMPROVEMENT OVER A LEARNED POLICY
In this section, we show that our CFPI opera- tor Isg can further improve the performance of a Single Gaussian policy TIQL. learned by IQI. (Kostrikov et al., 202T) on the AntMaze domain. We first obtain the IQL policy TIQ and QIQL by training for 1M gradient steps using the PyTorch Implementation from RLkit (Berkeley). We emphasize that we fol- low the authors' exact training and evaluation protocols and include all training curves in Ap- pendix G.6. Interestingly, while the running average of the evaluation results during the course of training matches the reported results in the IQL paper. Table 3 shows that the evalu- ation of the final IM-step policy Tiol does not match the reported performance on all 6 tasks, echoing the training instability we are trying to resolve with our CFPI operators. This demonstrates how drastically performance can fluctuate across just dozens of epochs. Thanks to the tractability of Isa, directly obtain improved policy Isa (TOO. QIOL:T) that achieves better overall performance than both TOL (train) and (IM), as shown in Table 3] We tune the HP 7 using a small set of seeds for each task following the practice of (Brandfonbrener et al. 2021
Table 3: Our [ QIor) improves over the pol-
icy mot. learned by IQL on AntMaze. We report the mean and standard deviation 10 seeds. Each seed evaluates for 100 episodes.
Datasct
Togu. (train)
Sug. (IM)
Isa(STau Qua..)
unimazz-u- U-V0
87.4 1: 3.2
81.6 1:2
85.1 1:
antmaze-u-d-x0
58.0 5.7
55.8
9.10
untmaze m. p.v0
71.1 1: 5.43
64.2 E 13.2
75.5 1: 6.1
unimaze-m m--v0
70.0 6.36
66.8 1.44
3.8
untmaze-l-p-xt
34.4+
35.6
37.7 + 7.7
antmaze-l-d-x0
39.8 9.09
38.8
40. + 5.6
Total
361.7 1 35.6
344.7 1. 47.8
373.3 1. 37.5
Fu et al. 2020) and include more details in Appendix F2 and G.6
5.3 ABLATION STUDIES
We first provide fair comparison with the other policy improvement operators, demonstrating the effectiveness of solving the approximated BCPO (4) and modeling the behavior policy as a Gaussian Mixture. Additionally, we examine the sensitivity on ablate the number of Gaussian components, and discuss the limitation by ablating the Q network in Appendix G.2
G.3
G.4
respectively.
Effectiveness of our CFPI operators. In Table 4. we compare our CFPI operators with two policy improvement operators, namely, Easy BCQ (EBCQ) and Rev. KI. Reg from OneStepRL. (Brand fonbrener al. 2021). EBCQ doe not require training either, returning policy by selecting an action that maximizes a learned Q from Nneq actions randomly sampled from the behavior policy at. Rev. KL Reg sets (---)) in (2) as the reverse KL divergence and solves the problem via SGD, with a controlling the regularization strength. We omit the comparison with the other learning- based operator Exp. Weight, as Rev. KI. Reg achieves the best overall performance in OneStepRI..
For all methods, we present results with 4tf modeled by Single Gaussian (SG-) and Gaussian Mixture (MG-). To ensure a fair comparison, we employ the same Qn and R modeled and learned in the same way as in Sec. 5.T for all methods. Moreover, we tune Nhoq for EBCQ, for Rev. KI. Reg, and
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation studies of our Method on the Gym-MuJoCo domain. Again we report the mean and std of 10 seeds, each seed evaluates for 100 episodes.
Dataset
SG-EBCQ
MG-EBCQ
SG-Rev.
Reg
MG-Rev.
Reg
Isu
Imu
Cheetah.MMv2
$3.3 0.2
51.5 1 0.2
47.1 0.2
47.0 : 0.2
51.1 1 0.1
52.1 =: 0.3
Hop2
82.5
70.3
83.2 5.1
8.1
82.4 1.0
82.8 = 188
88.1
88.3
Cheetah- M.Rv2
43.5
44.3
4..8.10.44
Hopper-M-R-v2
88.5 12.2
10.6
99.7 + 1.0
9..4
87.7++8.7
93.6 -
M-R-x2
75.4
73.1
28.5
69.7
71.3.14.44
Cheetah-M-E-v2
81.
78.9
65.0=10.1
91.
97.3 1.8
Hopper-M-E.v2
40.0
6.2
76.6
79.4
7..3++8.9
73.0
10.5
Walker2d-M-E-v2
111.1 188
11
106.7 4.1
107.1 +0.0
111.1 1.1
1119 0.3
Total
663.5 1. 41.0
670.6 1. 31.9
669.7 70.3
671.2 11 89.1
688.9.1.331
725.8 1. 32.4
Median
IQM
Mean
Optimality Gap
SG-EBCO
84
84
72
81
27
30
Normelized Sture
Figure 2: Aggregate metrics (Agarwal et al. 2021 with 95% CIs based on results reported in Table 4 The CIs are estimated using the percentile bootstrap with stratified sampling. Higher median, IQM, and mean scores, and lower Optimality Gap correspond to better performance. Our IMG outperforms baselines by a significant margin based on all four metrics. Appendix F includes additional details.
for our methods. Each method uses the same set of HP for all datasets. As a result, the Hopper-M-E performance of IMG reported in Table is different from Table 1 Appendix F. includes further details on the HP tuning and corresponding experiment results in Table
and 12
As is shown in Table and Fig. [ our ZMG clearly outperforms all baselines by significant margin. The learning- based method Rev. KL Reg exhibits substantial amount of variance, again echoing the training instability we are trying to resolve. Moreover, our CFPI operators outperform their EBCQ counterparts, demonstrating the effectiveness of solving the approximated BCPO.
Effectiveness of Gaussian Mixture. As the three M-E datasets are collected by an expert and medium policy, we should recover an expert performance as long as we can 1) capture the two modes of the action distribution 2) and always select action from the expert mode. In other words, we can leverage the Qo learned by SARSA to select actions from the mean each Gaussian component, resulting in mode selection algorithm (MG-MS) that selects its action by
Pmode = arg max,, p. Qo(s,
s.t. {prlA, > {}} where Σm AN S.,) = n.. (16)
{ is set to filter out trivial components. Our MG-MS achieves expert performance Hopper-M-E
(104.2 + 5.1) and Walker2d-M-E (104.1 + 6.7), and matches SOTA algorithms in Cheetah-M-E (91.3 + 2.1). Appendix G.T includes the full results of MG-MS the Gym MuJoCo domain.
CONCLUSION AND LIMITATIONS
Motivated by the behavior constraint in the BCPO paradigm, we propose CFPI operators that perform policy improvement by solving an approximated BCPO closed form. As practical datasets are usually generated by heterogeneous policies, we use the Gaussian Mixture to model the data- generating policies and overcome extra optimization difficulties by leveraging the LogSumExp's LB and Jensen's 's Incquality. We instantiate one-step offline RL algorithm with CFPI operator and show that it can outperform SOTA algorithms on the Gym-MuJoCo domain the D4RL benchmark.
Our CFPI operators avoid the training instability incurred by policy improvement through SGD. However, our method still requires learning a good Q function. Specifically, our operators rely on the gradient information provided by the Q, and its accuracy largely impacts the effectiveness of our policy improvement. Therefore, one promising future direction for this work is to investigate ways to robustify the policy improvement given noisy Q.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We include our codes in the supplementary material.
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34:29304 29320,, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11556, 2022.
RAIL Berkeley, Rikit: Reinforcement leamning framework and algorithms implemented in pytorch. URL httpt::/tttthhhie http://al///eeeeeeeee/e//eaa
Christopher M Bishop and Markus Svensén. Bayesian hierarchical mixtures of experts. arXiv preprint a22:2:/2 2012.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. university press, 2004.
Cambridge
David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off off-policy evaluation. arXiv preprint (0.iv:200600800) 08909, 2021.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. A framework for data-driven robotics. arXiv preprint arXiv:1909. 12200, 2019.
James Callahan. Advanced calculus: a geometric view, volume 1. Springer, 2010.
Jinglin Chen and Nan Jiang. Information--theortiicc considerations in batch reinforcement learning. In International Conference on Machine Learning, pp. 1042-1051, PMLR, 2019.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 2021.
Kamil Ciosek, Quan Vuong, Robert Loftin, and Katja Hofmann. Better exploration with optimistic actor-critic, 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pp. 1096- 1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018b.
Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax optimmal off off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pp. 2701-2709, PMLR, 2020.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556 2005.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforeement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. In Thirty-Fifil Conference Neural Information Processing Systems, 2021.
10

---

Under review as a conference paper at ICLR 2023
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. CoRR, abs/1802.09477,, 2018. URLljtttp://axxw.crog/ /102
09477
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062, 2019.
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, and Shixiang Shane Gu. Why should i trust you, bellman? the bellman error poor replacement for value error. arXiv preprint arXiv:2201. 12417 2022.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected max q-learning operator for simple yet effective offline and online rl. In International Conference Machine Learning, pp. 3682-3691. PMLR, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, Ass//801.. .01 290, 2018. URL
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of bench- marked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv: 041333, 2017.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 0rXiv:1907.00456. 2019.
Chi Jin, Yuchen Zhang, Sivaraman Balakrishnan, Martin Wainwright, and Michael Jordan. Local maxima in the likelihood of gaussian mixture models: Structural results and algorithmic consequences. Advances in neural information processing systems, 29, 2016.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures experts and the em algorithm. Neural computation, 6(2): 1994.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference on Machine Learning. Citeseer, 2002.
Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1412,6694 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 14, 2013.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 2rXiv:21 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy gllaarning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Under review as a conference paper at ICLR 2023
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems, 33: 18560-18572, 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint S1tv:2000.04777, 2020b.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Interna- tional Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint ar Ttt::0005.0/445 2020.
Lihong Li, Thomas J Walsh, and Michael Littman. Towards unified theory of state abstraction for mdps. ISAIM, 4(5):9, 2006.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509002900
2015.
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004115547, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5187550):529-53, 2015.
Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):429 443, 1997.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic; Deep multitask and transfer reinforcement learning. CoRR, hts/1511.006422 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. Advances in neural information processing systems, 32, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 2019.
Theodore Perkins and Doina Precup. A convergent form of approximate policy iteration. Advances in neural information processing systems, 15, 2002.
Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep reinforcement learning framework for autonomous driving. Electronic Imaging, 2017(19) 2017.
John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1 502.0547777 2015. URI//ttp://arriv.oor//aass
05477
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, hbs/1707.06347, 2017. URL. httt:://arxii..e../ abs//
1707. 06347
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pp. 387-395. PMLR, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
12

---

Under review as a conference paper at ICLR 2023
Yunhao Tang, Michal Valko, and Rémi Munos. Taylor expansion policy optimization. In International Conference on Machine Learning, pp. 9397-9406. PMLR, 2020.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02645. 2018.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279-292 1992.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv: arXiv:1911.11361 2019.
Lei Xu, Michael Jordan, and Geoffrey E Hinton. An alternative model for mixtures of experts. Advances in neural information processing systems, 7, 1994.
Ming Yin and Yu-Xiang Wang. Asymptotically efficient off off-policy evaluation for tabular reinforce- ment learning. In International Conference on Artificial Intelligence and Statistics, pp. PMLR, 2020.
Ming Yin, Yaqi Duan, Mengdi Wang, and Yu-Xiang Wang. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. International Conference on Learning Representations, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 53:14129-1411 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34, 2021.
Ruiqi Zhang, Xuezhou Zhang, Chengzhuo Ni, and Mengdi Wang. Off-policy fitted q-evaluation with differentiable function approximators: Z-estimation and inference theory. International Conference Machine Learning, 2022.
Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems, 32, 2019.
13

---

Under review as a conference paper at ICLR 2023
Appendix
OUTLINE OF THE APPENDIX
In this Appendix, we organize the content in the following ways:
Appendix presents the missing proofs for Proposition 3.1 main paper.
3.2 3.3
Theorem 3.1 the
Appendix B justify one HP setting for Equation 15
Appendix discusses how to instantiate multi-step and iterative algorithms from our algorithm template Algorithm
Appendix provides the derivation of new CFPI operator that can work with both deterministic and VAE policy.
Appendix conducted reliable evaluation demonstrate the statistical significance of our methods and address statistical uncertainty.
Appendix gives experiment details, HP settings and corresponding experiment results.
Appendix provides additional ablation studies and experiment results.
Appendix H includes additional related work and discusses the relationship between our method and prior literature that leverage the Taylor expansion approach.
Our experiments are conducted on various types of 8GPUs machines. Different machines may have different GPU types, such NVIDIA GA100 and TU102. Training behavior policy for 500K gradient steps takes around 40 minutes, while training a Q network for 500K gradient steps takes around 50 minutes.
14

---

Under review as a conference paper at ICLR 2023
A PROOFS AND THEORETICAL RESULTS
A. PROOF OF PROPOSITION 3.1
Proposition 3.1. The optimization problem has a closed-form solution that is given by
v2 22go Eg [VQ ())s where ö = log dct(2nEg)) + log
(Tg(T) 14 +
VaQ(s, a) Es VoQ(s,a))ann
2
(17)
Proof.
The optimization problem (5) can be converted into the QCLP
max
p VaQ(s, (alQ((ss))aapa))) a)lam s.t. (u pe) 2 (p (B) log det (2+2p))))
(18)
"
Following similar procedure as is in OAC (Ciosek et al. 2019), we first derive the Lagrangian below:
L = n VoQ(s, a))=pg - (u(u u (1 - MB) - 8 ++ log det det (2
(19)
Taking the derivatives w.r.t μa, we get
V (v a))a -nEg (1- 14))
(20)
By setting V = 0, we get
1 Eg V (s, # uB
(21)
To satisfy the the KKT conditions [ooyd et al. 2004). we have > 0 and
(μ 140) Eg (μ me) 28 log dem
(22)
Finally with 21 and 22 get
(aQ(s,a)) م==u Eg (V
1
(23)
25 log (2x(
By setting log det(2 + log T and plugging 23 to 21, we obtain the final solution as
V2log Mg V Q s, a) 0a==
Msg (T) =44 +
(24)
(aQ(s, Σg (V.((s, a)
which completes the proof.
15

---

Under review as a conference paper at ICLR 2023
A.2 PROOF OF PROPOSITION 3.2
Proposition 3.2. By applying the first inequality Lemma 3.1 to the constraint of derive an optimization problem that lower bounds (10)
10 we can
n² (V,((sa))a-m
max
14
{
(25)
S.L.
max
1-200- p4) M, (p- us) 2
log dct((2=;)) + log
log 2; 2 -5,
and the closed-form solution to
is given by
1
pils (T) arg max
V. Q(, (s, s.t. 8 ==
min {log ); det((2 + log
(.(
2
(26)
2 + log A.) log det (T Ex)
where (6) = ps
E, V. Q(s,
(.Q((,,))a E, ((, a))am
Proof.
Recall that the Gaussian Mixture behavior policy is constructed by
TH Ne E.),
(27)
We first divide the optimization problem 25 into N sub-problems, with each sub-problem given by max μ² T [VQ al)=aa
(28)
S.L. (a u.) " (μ m.) 2 log det(( + log A 2 -ō,
which is equivalent to solving problem (5) for each Gaussian component with an additional constant term log A; and thus has unique closed-form solution.
Define the maximizer for each sub-problem as (5)) though (5) does not always exist. Whenever -1 log det(2x + log A < -8, there will be no μ satisfying the constraint as (4:))) Σ,- (n 14) is always greater than O. We thus set (;(5) to be None in this case. Next, we will show that there does not exist any i & {o)( = 1... N). s.t., ğ is the maximizer of (25). We can show this by contradiction. Suppose there exists & (p. one E {1,..., N} s.t.
=3
N...N. maximizing (55) there exists at least
1
(ā m3) 23 (ā m4)
log (2m2)))))) + log A -
(29)
2
2
Since jú is the maximizer of (55 it should also be maximizer of the sub-problem j. However, maximizer for sub-problem is given by (ā)(5) + j., contradicting with the fact that jú is the maximizer of the sub- sub-problem Therefore, the optimal solution to (25) has to be given
argmax,,
of [V [V, where A E -1.0)l==1.....
(30)
To solve each sub- aro-problem natural to set ag = p4, which reformulate the sub- prl-leoollmmmm as below
max T (V.((s
(31)
s.t. 5 (1) (1- 5 log det det(2 + log As
1
2
Note that problem (31) is also QCLP similar the problem ( Therefore, can derive its solution by following similar procedures as in Appendix A.] resulting in
2(8 + log As) log det (2T
(6) = ps +
Σ, (V.
(32)
[v.Q(s, a)) Σ, V.Q(s,
We complete the proof by further setting 8 min; flog A; det() + log
16

---

Under review as a conference paper at ICLR 2023
A.3 PROOF OF PROPOSITION 3.3
Proposition 3.3. By applying the second inequality of Lemma 3.1 to the constraint of derive an optimization problem that lower bounds (10
10
we can
V (v)((s,a))a=
max
N
(33)
s.I.
Σ
Ne
(-2
1 log det (e
-2(1 - 2; (u mi) 2 -
and the closed-form solution to
13 is given by
2logr Σm Aun x M4 1
Hjjoson (T) 1at
a))==
(aQ( (=
a a)) la=p
N
N Σ
N A log del (2+T
where E =
Σ
E ui
8 log r +
2
(34)
Proof. Note that problem (33) is also a QCLP. Before deciding the value of ag. we first derive its Lagrangian with a general ag below
1 2 log (2)) + 7(1 2,) (u mi) -
L- n (V.((,)))
Σ A
2
(35)
Taking the derivatives w.r.t μa, we get
N
L = (V)((s -
Σ
Ae (Σ, (11 - mi)
(36)
By setting V L. = 0, we get
N
N
N
11 =
Σ
D 14
+
Σ
(a]((s
"
i=1
i=1
= A
- E V Q,,,((s,, ((ssa))l==aa
(37)
N Σ
where Σ
P
p4
Equation 37 shows that the final solution to the problem Therefore, setting ag becomes natural choice.
33
will be a shift from the pseudo-mean ja.
Furthermore, by satisfying the KKT conditions, we have v) > and
N
N
A (M (p- D 1 (p 14) 28
Σ
X log 22
(38)
Plugging 33 into
38 gives the equation below
1
T ((V, - 14.
) E a+ (EIVo((s,0) 14 m1)
Xx
p
-14)
n
(39)
N
= 28
A log dot (2.
17

---

Under review as a conference paper at ICLR 2023
The LHS of (39
can be reformulated
N
r E V 0al== -11 -M:) Σ -
Σ Xi
a
( - vo - Hi)
1=1
[
( a) b=p Σ (IIVoQ(s,a))sz)
112 1=1
(40)
2
(I]Vo((s.u))==p))) ) x7' 65'(n-1 (1= 1b)
A
"
N
+
Σ
(ū
- (ū
(4) ps)
1=1
We note that the second line of 40 's RHS can be reduced to
A (I)
"=,''
Σ, " (1-11)) -11)
N - Σ 2123
N
1=1
(41)
Σ 5-
Σ Σ
2123 14
1)
i=1
2 )
1---E1 )
0
Therefore, 44 can be further reformulated as
T Σ,¹ (p+ 14 -Hs)
Σ X ++
V
1
N
[c((s.. ET (E(Voc((...0))
+
X (ji
/4:) T 1 (īt (7) pi)
1
V Q6, (10 )7 ExDr')) (
(2)
マ
(42)
+
Σ
( - (()-ma)- E ( 1ū-14 m1)
1=1
1
E[VaQ(s, a)))==)))) "''VV_
(ElVoQ(s,a)),)
2
d (ī ( E ( pi)
+
N + Σ
[aQ(s, EVVo((s,
Ao (ī - -=)
Σو- (ī ps)
1=1
18

---

Under review as a conference paper at ICLR 2023
can be reformulated as To this point, (39
VaQ(s, [
Σ
Aa (
Ex)
x ( p4)
tp
i=1
(43)
N
26
X log de
We can thus express y) as below
[al(s,a)) E
(44)
7
\
25 E d log (2 - En X4 (i - 14)² x (īt 14)
By setting 8 = Em di log det(( + log we have
[VaQ(s, a) _ E ViV
7
log Em d (ī m1) Σj (ī (1))
[aQ a Tip E V a) (l
2logr Em A ā + 2F Σ X us Em dop Dj pi
(45)
VoQ(s, Σ V. (s, a))==
2 log Enn 11t Σ
ET x Ei AuNT "M
[V.((s, (x Σ
1==B
2log E²''" jt Σi= i= Aapie 21²
Finally, plugging
45 into 37
with ag fi, we have
2oo - EN 11 A E. H. + + P
Pjensen (T) = ++
(46)
[= V/V.Q( a)oo=p
which completes the proof.
19

---

Under review as a conference paper at ICLR 2023
A.4 PROOF OF THEOREM 3.1
Algorithm [foloow the approximate policy ileration (API) (Perkins & Precup 2002) by iterating over the policy evaluation (E step, Line 4) and policy improvement (Z step, Line 5). Therefore, to verify & provides the improvement, we need to first show policy evaluation Q is accurate. In particular, we focus on the SARSA updates (Line 2). which is a form of on-policy Fitted Q-Iteration (Sutton Barto 2018). Fortunately, it is known that FQI is statistically efficient (e.g. (Chen & Jiang, (201)) under the mild condition for the function approximation class. Its linear counterpart, least-square value iteration, is also shown to be efficient for offline reinforcement learning (Jin et al.) 2021 Yin et al. 2022).
In this section we prove the safe policy improvement presented in Section 3.3
Recently, (Zou et al., 2019) shows the finite sample convergence guarantee for SARSA under the standard the mean square error loss.
Next, to show the performance improvement, we leverage the performance difference lemma to show algorithm achieves the desired goal.
Lemma A.1 (Performance Difference Lemma), For any policy π,, x', it holds that
J(T) J(x' I-7 Bandd
Ba (s) (s) A (s.
where (s, (s,a))==")) (s - (s) is the advantage function.
Similar to (Kumar et al. 2020b), we focus on the discrete case where the number of states [S] and actions| |A| are finite (note in the continuous case, the D(s, a. would be 0 for most locations, and thus the bound becomes less interesting). The adaptation to the continuous space can leverage standard techniques like state abstraction (Li et al. 2006) and covering arguments.
Next, we define the learning coefficient C,,5 of SARSA as
Cr. VD(,a))
( - (f*
Vx,aESxA.
Define the first-order approximation error
(s, == (a ap) V (s
(
a=us
then the approximation error is defined as:
CCFPr(s, = 12 ( a) (s,(s)))) = (a - (a-as)² vo()*(s,
+ +s, - 2 (s, a))
a=ay
Under the constraint D (T(- s), A- 8)) < 8 4 (or equivalently action is close az), the first-order approximation provides good estimation for the QTa
Theorem A.1 (Restatement Theorem 3.1). Assume the state and action spaces are discrete. Let ीो be the policy obtained after the CFPI update (Line of Algorithm 7 Then with probubility 1 8,
1
J(tt)-
- -(tt)) >
1 Eswat
E
10*0 - (s,(
2
C7,8 VD(,,a)
1- Exndtr Banir((ss)
E
+ CCFFi(s,a) =
For multi-step T iterative update, we similarly have with probability 1 1-8, 8,
T
T
J(Hr) - J(tg) = J(a)) ((t))J((f,))) J(H, 2
t=1
1=1
where D(s,0) a) denotes number of samples at s,a, the learning coefficient of SARSA is defined as Cr,5 max (11(AAAA v V
R (s, a (9. ag)² Var [V*a (s') 8, a). using CFPI, i.e. Cerri(s, a) = (a ta-age az) [v. + + (s, ($) When ag. (c) 0.
22h 1h
and C)Fri(s,a) denotes the error from the first-order approximation
20

---

Under review as a conference paper at ICLR 2023
proof of Theorem 3.7 We focus the first update, which is from ns f1. According to the
Sarsa update, have (* (s,
Cr. Vs,a E 8 x A with probability
VD(a,a)' 1 - and this due to previous on-policy evaluation result (e.g. (ou et al. 2019) ) Also denote
11 arg max, a
By Lemma A.1
((1) - (+t)
E soudi (1s)
1
E Bont
Baat (1o= V (s)]
1 ?
E Biddi
(1 12 (s,
1
1
2
Cz.
E Bannth
(s, (0
1 Egndii Bawas(1s))
1
(D(
1
2
Cz,8 VD(s,a)
E Bendin (3) 1-
E Bandir (ow+t((1))
+ CCFPI(s,a)
1
0
where the first inequality uses - ($) x
-
Cy.s and the last inequality VD(s,a)
t == arg maxx t+
Here
C5.8 = max V 2lm(122AA/)
22h
n0 (8, also, an) Var (Văg (s') | 8, a)
$0.00
h=D 3,a
Similarly, if the number of iteration
1, then Denote
Re Ph (s, also, ao)2 Var [V& (s") ss
(1) max (2ln(12SA/8)) 7.8
2h
50.40
Rt-1 Hh
(s, als0, an)
A= 8.a
then we have with probability 1 - 8, by the Corollary of Duan et al. 2020). the OPE estimation follows
c
a)) Q* (0,a0)))
(Ds
and
- ( 2 1-7
E Egndts 1
c(t)
CCFPI(s, a)) = (()),
E Exnais (awite(-)))
1
(D(,,a)
then for multi-step iterative algorithm, by union bound, have with probability 1 - 8
J(HT) ((+g) Σ - J(
(0
On the learning coefficient of SARSA. The learning of SARSA is known to be statistically efficient from existing off- policy evaluation (OPE) literature, for instance (Duan et al. 2020 Yin & Wang 2020). This is due to the on on-policy SARSA scheme is just a special case of OPE task by choosing n..
21

---

Under review as a conference paper at ICLR 2023
Concretely, translate the finite sample enror bound in Corollary of (Duan al. infinite horizon discounted setting as: for any initial state,action Su, ag. with probability 6,
2020 the
I (80, an) -Q) ao)l
221n(12/0)
Σ A= 8.0
Σ
2h
#3 1h (8, also, ao) Var [V** (6') 8,
V(so, ao)
Note the original statement in (Duan et al. 2020 is for a 6 here we conduct the version for *+ - D& instead and this can be readily obtained by fixing the initial state action Sp, do for e As a result, by a union bound (over S, A) it is valid to define
DC
Cy.6 max 22nnn22222///
E h=D a,a
ΣΣ
2h 1h (s, also, ag) Var [ (s') s, a)
and this makes sure the statistical guarantee in Theorem 3.1 follows through.
Similarly, for the multi-step case, the OPE estimator hold with the corresponding coefficient
ctt == max (21n(12AA/0) 9,8
22h
1h (s, als0, an)? Var [V& (s") s, s.
50.40
Hh #t-1
as,also,
0= 8.a
Lastly, even the assumption on the state-action space to be finite is not essential for Theorem 3.1 since, for more general function approximations, recent literature for OPE (Zhang et al. 2022 shows SARSA update in Algorithm is still statistically efficient.
22

---

Under review as a conference paper at ICLR 2023
B DETAILED PROCEDURES TO OBTAIN EQUATION 15
We first highlight that we set the HP å differently for Proposition 3.2 and 3.3 With the same т, we generate the two different ó for the two different settings. Specifically,
81se (T) -log + min
log det (2 - log A j.
(Proposition 3.2
2
(47)
Ojensen (T) -log +
Σ
A log det (2+
(Proposition 3.3
We next provide intuition for the design choices policy is constructed by
47
Recall that the Gaussian Mixture behavior
T = Σ A,
(48)
With the mixture weights A;=1..,, define the scaled probability #;(a) of the i-th Gaussian component evaluated at
# (1:) (()T(() x det(( exp( - (0 21)",, pi)
(49)
where (a) = (a; ,1, Σ,) denotes the probability of the i-th Gaussian component evaluated at a.
Therefore, have log # (p:) log A z/2 log det(2n which implies that
dis = log + min
log
- loog de
2
1
max
log d -
log det (et -log7)
(50)
1 log - (u(( 7
max
By setting (T) in this way, Fy ((we(T)) will satisfy the following condition whenever Py is valid solution to the sub rub-problem,
28
due to the KKT conditions, Vj ( ,I....,Ny N}.
1 ( m1,) "", (ig -1g)
log (et + log X, = (Be(TT
2
(51)
log म, (ūz)) max { log (
(, 1
max (#)(1)))
To elaborate the design of (jensen(F)) (T), first recall that the constraint of problem 13 is given by
N Σ
- log del (2x
3 - E. (n mn) 2 -djjnscm (T)
(52)
1=1
Note that the LHS of (52 is a concave function w.r.t μ. Thus, we can obtain its maximum by setting its derivatives 53 to zero
1 log det((
1 2(1 I. (H p.)) p.))
Σ Ai (-2
2
(53)
X ( m:) - r E
Interestingly, we can find that the solution given by μ = f. Plugging μ f into the LHS of (2). we can obtain its maximum as below
X dot((2
Map ((a--))20 - pi)
(54)
N
N
A
log dot((2T.))
Σ
X log (144
2
i=1
23

---

Under review as a conference paper at ICLR 2023
The inequality holds as the covariance matrix Σ, is a positive semi-definite matrix for i [1....N. Therefore, our choice of 6jensen( (T) can interpreted as
N
8jensen (T) = log
A log det (2 =
A log m. (u)) - log
(55)
2
1=1
i=1
24

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Iterative TMG
1: Input: Learned behavior policy e. Q network parameters Φ1, Φ2, target Q network parameters
1u 2,urg,,, dataset D, parameter т
2: repeat
3: Randomly sample batch of transitions, B = {[s, a, 8, from
4: Compute target actions
a' (s') clip [MG Ts') clip((,-c,c), -c, aLows (Hig))
where Q min(Q Qoa); and o0)
5: Compute targets
v(r, s = = 1111011111000 min Rown. d
6:
Update Q-functions by one step of gradient descent using
Σ
(8, v(r, ,d))
for i 1,2
B
(s,a,r,af J)EB
7: Update target networks with
Gung, f poturg, + +(1 p)d:
for = 1,2
8: until convergence
9: Output: IMG (tt. T:
C MULTI-STEP AND ITERATIVE ALGORITHMS
By setting O, we can derive multi-step and iterative algorithms. Thanks the tractability of our CFPI operators Isa and IMG. we can always perform the policy improvement step in-closed form. Therefore, there no significant gap between multi-step and iterative algorithms with our CFPI operators. One can differentiate our multi-step and iterative algorithms by whether an algorithm trains the policy evaluation step E(Qu-1: t1, D) to convergence not.
As for the policy evaluation operator E, the fitted Q evaluation (Ernst et al. Fujimoto al. 2022 with target network (Mnih al., 2015) has been demonstrated to be effective and successful paradigm to perform policy evaluation (Kumar et al.
2005 Le et al. 2019
2019 Fujimoto & Gu 2021; Haarnoja al. 2018 [illicrap et al. 2015 Fujimoto al. (0188) in deep (offline) RL. When
instantiating a multi-step or iterative al algorithm from Algorithm one can also consider the other policy evaluation operators by incorporating more optimization techniques.
In the rest of this section, we will instantiate an iterative algorithm with our CFPI operators performing the policy improvement step and evaluate its effectiveness on the challenging AntMaze domains,
C. ITERATIVE ALGORITHM WITH OUR CFPI OPERATORS
In Sec. [.1 we instantiate an iterative algorithm Iterative ING with our CFPI operator IMG. Algorithm
2 presents the corresponding pseudo-codes that learn a set of Q-function networks for simplicity.
Without loss of generality, we can easily generalize the algorithm to learn the action-value distribution
Z (s, a) as is defined in (88).
For each task, we learn Gaussian Mixture behavior policy Ωे with behavior cloning. Similar to Sec.5.1 we employed the IQN (Dabney et al. 2018a) architecture to model the Q-value network for its better generalizability. As our CFPI operator ZMG returns a deterministic policy, we follow the
TD3 (Fjjimoto al. 2018) to perform policy smoothing by adding noise to the action a' (s') in Line
4. After convergence, Algorithm outputs improved policy IMG (ft Q:
Table 3 compares our Iterative IMG with SOTA algorithms on the AntMaze domain. The performance for all baseline methods is directly reported from the IQL. paper (Kostrikov al. 2021 Our method
25

---

Under review as a conference paper at ICLR 2023
Table Comparison between our iterative algorithm and SOTA methods on the AntMaze domain of D4RL. We report the mean and standard deviation across seeds for our methods. Our Iterative IMG outperforms all baselines on 5 the effectiveness of our CFPI operator when instantiating iterative algorithm. "u" stands for umaze, "m" stands for medium, stands for large, "p" stands for play, and "d" stands for diverse.
tasks and obtaining the best overall performance, demonstrating
Dataset
BC
DT
Onestep RL
TD3+BC
CQL
IQL
Iterative IMG
antmaze-u-vO
54.6
59.2
64.3
78.6
74.0
87.5
90.2 + 3.9
antmaze-u-d-vo
45.6
49.3
60.7
71.4
84.0
62.2
58.6 + 15.2
antmaze-m-ppvv
0.0
0.0
0.3
10.6
61.2
71.2
75.2 = 6.9
antmaze-m-d-vo
0.0
0.7
0.0
3.0
53.7
70.0
72.2 = 7.3
antmaze-l-p-v0
0.0
0.0
0.0
0.2
15.8
39.6
51.4 7.7
antmaze-l-d-v0
0.0
1.0
0.0
0.0
14.9
47.5
52.4 10.9
Total
100.2
112.2
125.3
163.8
303.6
378.0
400.0 + 52.0
outperforms all baseline methods on on 5 out of 6 tasks and obtaining the best overall performance. The training curves shown in Fig. with the HP settings detailed in Table 6 We did not perform much HP tuning, and thus one should expect performance improvement after conducting fine-grained HP tuning.
antmaze-umaze-v0
antmaze-umaze-vv
antmaazelarge--laayyoo
Episode Return
Episode Return
Episode Return
60
80
80
60
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8
1.0
Gradient Steps (x10*)
Gradient Steps (x10*)
Gradient Steps ix10*)
antrmaze-umaze-diierrs--
antmaze-medium- -ivess-
antmaze-large-diierse-o
80
Episode Return
Episode Return
Episode Return
60
60
40
40
40
20
20
0.0
0.2
0.4
0.6
0,8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
Gradient Steps (x109)
Gradient Steps (x109)
Gradient Steps Ix10")
Figure 3: Iterative IMG training results on AntMaze. Shaded area denotes standard deviation.
26

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
Shared HP
Optimizer
Adam KKnggma Ba 2014)
Normalize states
False
activation function
ReLU
Mini-batch size
256
Gaussian components (N)
8
Number of gradient steps
500K
Policy architecture
MLP
MG-BC
Policy learning rate
le-4
Policy hidden layers
3
Policy hidden dim
256
Threshold g in (16)
0.05
Number of gradient steps
1M
Critic architecture
IQN
Dabney et al. 2018a
Critic hidden dim
256
Critic hidden layers
3
Critic learning rate
3e-4
Iterative IG
Number quantiles N₂
8
Number of cosine basis elements
64
Discount factor
0.99
Target update rate
5e-3
Target update period
1
log
1.5
Table 6: Hyperparameters for Iterative ING-
D CFPI BEYOND GAUSSIAN POLICIES
In the main paper, we mainly discuss the scenario when the behavior policy Ty is from the Gaussian family and develop two CFPI operators. However, methods can also work with non-Gaussian Ty. Next, we derive a new CFPI operator IDET that can work with deterministic иy. We then show that IDET also be leveraged improve general stochastic policy Ty without knowing its actual expression, as long as we can sample from it.
D.1 DETERMINISTIC BEHAVIOR POLICY
When modeling both π It and ny ng as deterministic policies, we can derive the following BCPO from the problem by setting -)-..) as the mean squared error.
max 4 [V a)la= s.t.
= b| - 4 12 8
(56)
Problem 56 has similar form as the problem 18 μ (d) as below
We thus obtain its closed-form solution
25 (s,
Mdee (6) p3 +
Q a))
(57)
[
a) #=
Therefore, we can derive a new CFPI operator IDET (T Q; that returns policy with action selected by (57).
We further note that the problem (6) can be seen linear approximation of the objectives used in TD3 + BC (Fujimoto & Gu 2021
D.2 BEYOND DETERMINISTIC BEHAVIOR POLICY
Though assume ne to be deterministic policy during the derivation of IDET. indeed leverage IDET to tackle the more general case when we can only sample from Ta without knowing its actual expression.
27

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Policy improvement of ZOET with stochastic ng
Input: State s, stochastic policy тe, value function Q, 6, number of candidate actions to sample M
1: Sample candidate actions {a1,..., Car) from T
2: Obtain the EBCQ policy TEBCQ with action selected by TEBCO(S) = arg max Q(s, am)
3:
Return Iper TTBBQQ Q; 6) by calculating (57)
Table 7: ZDET results on the Gym-MuJoCo domain. We report the mean and standard deviation seeds and each seed evaluates for 100 episodes.
Dataset
DET-BC
VAE-BC
VAE-EBCQ
DEE with Wiet
TDE with Tyac
Walker20-Meeiuu--2
71.2 1: 2.0
70.6 + 3.0
70.6 3.4
79.5 = 12.9
86.5 1 6.3
Walker22-Mccduu--Replay-22
19.5 12.6
19.4 + 2.9
33.5 7.3
57.1 11.6
62.6
Walker2d Walker2d-Medium Expert-Ex
74.4 0.4
74.9 + 7.6
82.7 11.9
111.2 1.8
111.1 0.9
Algorithm 3 details the procedures to perform the policy improvement step for stochastic behavior policy aT. We first obtain its EBCQ policy TEBCQ in Line 1-2. As TEBCQ is deterministic, we further plug it in IEE in Line 3 to return an improved policy.
D.3 EXPERIMENT RESULTS
To evaluate the performance of IDET, we first learn two behavior policies with two different models. Specifically, we model Taa with three-layer MLP that outputs deterministic policy and T,ae with the Variational auto-encoder (VAE) (Kingma & Welling 2013) from BCQ (Fujimoto et al. 2019). Moreover, we reused the same value function Qn as Section We present the results Table DET-BC and VAE denote the performance of Tdet and Tae respectively. VAE-EBCQ denotes the EBCQ performance of Tvae with M = 50 candidate actions. Since is deterministic, its EBCQ performance is the same as DET-BC. As for our two methods, we set ô = 0.1 for all datasets. We can observe that both our IEE with Jea and ZDET with Tae largely improve over the baseline methods. Moreover, IDET with Tye outperforms VAE-EBCQ by a significant margins on three datasets, demonstrating the effectiveness of our CFPI operator.
Indeed, our method benefits from an accurate and expressive behavior policy, as IDET with The achieves higher average performance compared to IDET with Tiet, while maintaining a lower standard deviation on all three datasets.
We also note that we did not spend too much effort optimizing the HP, e.g., the VAE architectures, learning rates, and the value of T.
28

---

Under review as a conference paper at ICLR 2023
E RELIABLE EVALUATION TO ADDRESS THE STATISTICAL UNCERTAINTY
Figure 4: Comparison between our methods and baselines using reliable evaluation methods proposed in (Agarwal al. 2021). We re-examine the results in Table 40 the 9 tasks from the D4RL. MuloCo Gym domain. Hach metric is calculated with a 95% CT bootstrap based on 9 tasks and 10 seeds for each task. Each seed further evaluates each method for 100 episodes. The interquartile mean (IQM) discards the top and bottom 25% data points and calculates the mean across the remaining 50% runs. The IQM more robust an estimator to outliers than the mean while maintaining less variance than the median. Higher median, IQM, mean scores, and lower Optimality Gap correspond to better performance. Our Ima outperforms the baseline methods by significant margin based on four metrics.
Median
IQM
Mean
Optimality Gap
ING Ise
MG-Rev. KL Reg SG-Rev. KL Reg
MG-EBCQ
SG-EBCQ
78
78
27
30
Normalized Score
Figure 5: Performance profiles (score distributions) for all methods on the tasks from the D4RL. MuloCo Gym domain. The average score is calculated by averaging all runs within one task. Each task contains 10 seeds, and each seed evaluates for 100 episodes. Shaded area denotes 95% confidence bands based on percentile bootstrap and stratified sampling (Agarwal et al. 2021). The value where the curves intersect with the dashed horizontal line y = 0.5 corresponds to the median, while the area under the performance curves corresponds to the mean.
SG-EBCQ
MG MG-EBCQ
SG RG-Rev-KL-Reg
MG MG-Rev Rev-KL-Reg
SG-PAC
MG PAC
1.00
Fraction of tasks with score > n
Fraction of runs with score
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
80
100
20
80
100
Normalized Score (n)
Normalized Score (n)
To demonstrate the superiority of our methods over the baselines and provide reliable evaluation results, we follow the evaluation protocols proposed in (Agarwal et al., (021) to re-examine the results in Table Specifically, we adopt the evaluation methods for all methods with Nusks X N runs in total.
Moreover, we obtain the performance profile of each method, revealing its score distribution and variability,. In particular, the score distribution shows the fraction of runs above a certain threshold and is given by
Nus
N
((n) = (17: 21.. I:Nwa))
[mm.n 2 n?]
Nusks m==1
N
n=1
Evaluation results in Fig. and Fig. 1 demonstrate that our IMa outperforms the baseline methods by significant margin based all four reliable metrics.
29

---

Under review as a conference paper at ICLR 2023
F HYPER-PARAMETER SETTINGS AND TRAINING DETAILS
obtain the mean and standard deviation of each method across 10 seeds. Each seed contains individual training process and evaluates the policy for 100 episodes.
For all methods we proposed in Table I Table 3 and Table 4
F.1 HP AND TRAINING DETAILS FOR METHODS IN TABLE AND TABLE 4
Table 8 includes the HP of methods evaluated on the Gym-MuloCo domain. We use Adam (Kingma & Ba 2014) optimizer for all learning algorithms and normalize the states in each dataset following the practice of TD3+BC (Fujimoto Gu [02T]. Note that our one-step offline RI. algorithms presented in Table I (Our IMG) and Table ([MG. IsG, MG-EBCQ. SG-EBCQ, MG-MS) require learning behavior policy and the value function Qo Therefore, will first describe the detailed procedures for learning Single Gaussian (SG-BC) and Gaussian Mixture (MG-BC) behavior policies. We next describe our SARSA-style training procedures to estimate Q" Finally, we will present the details for each one-step algorithm.
Hyperparameter
Value
Shared HP
Optimizer
Adam Kingma & Ba 2014
Normalize states
True MLP
Policy architecture
Policy learning rate
le-4
Policy hidden layers
Policy hidden dim
256
Policy activation function
ReLU
Threshold ę in (66
0.05
Gaussian components (N)
4
Number of gradient steps Mini-batch size
500K
MG-BC
256
Number of gradient steps Mini-batch size
500K
SG-BC
512
Number of gradient steps
Table 16
Critic architecture
IQN (Dabney et al.
2018a
Critic hidden dim
256
Critic hidden layers
Critic activation function
ReLU
SARSA
Number of quantiles Ng
Number of cosine basis elements
64
Discount factor
0.99
Target update rate
5e-3
Target update period
Our IMG
Table
log
0 for Hopper-M-B:
0.5 for the others
IMG & Isa Table 4)
log T
0.5 for all tasks
MG-EBCQ
Number of candidate actions Nncq
SG-EBCQ
Number of candidate actions Nbog
10
MG-Rev. KL Reg & SG-Rev. KL Reg
3.0
Number of gradient steps
100K
Table 8: Hyperparameters for our methods in Table 1 and Table 4
MG-BC. We parameterize the policy 3-layer MLP, which outputs the tanh of Gaussian Mixture with = 4 Gaussian components. For each Gaussian component, we learn the state- dependent diagonal covariance matrix. While existing methods suggest learning Gaussian Mixture
30

---

Under review as a conference paper at ICLR 2023
via expectation maximization (Jordan & Jacobs 1994 Xu et al. 1994 Jin al. 2016) variational Bayes (Bishop & Svensén, 2012). we empirically find that directly minimizing the negative log- likelihood of actions sampled from the offfine datasets achieves satisfactory performance, as is shown in Table 1] We train the policy for 500K gradient steps. We emphasize that we do not aim to propose better algorithm for learning Gaussian Mixture behavior policy. Instead, future work may use more advanced algorithm to capture the underlying behavior policy better.
SG-BC. We parameterize the policy as a 3-layer MLP, which outputs the tanh of a Single Gaussian with the state-dependent diagonal covariance matrix (Fu et al. 2020 Haarnoja et al. 2018). We train the policy for 500K gradient steps.
SARSA. We parameterize the value function with the IQN (Dabney ct al. 20 8a) architecture and train it to model the distribution Z S x A Z of the behavior return via quantile regression, where 2 is the action- distributional space (Ma et al. (020) defined as
(58) We define the CDF function of 2 as Fzz (z) Pr(z < 2). leading to the quantile func- tion (uüller 1997) (p) == inf(2 E R: p S Fz (2)) as the inverse CDF function, where p denotes the quantile fraction. We further denote z F (p) ease the notation.
2 = {2: : S A + 3 (R) E [(. a) < 0o, P(
To obtain z,, we leverage the empirical distributional bellman operator Th Z + Z defined as T57(s,a) _ +2 (6',') ,a) (8, a, 8') DD
(59)
where 12 B implies the random variables A and B are governed by the same distribution. We note that 7/ helps to construct a Huber quantile regression loss (Dabney et al. 2018a Ma et al. 2020: Dabney et al., 2018b), and we can finally learn 2 by minimizing the quantile regression loss following similar procedures as in (Ma al.) 2020).
To achieve the goal, we approximate 7 by Ng quantile fractions {pr E [0, 1] i i = - NN) with Po = O, PN2 and p Pj. Vi < j. We further denote P. (p; + 1/2, and use random sampling (Dabney et al. (a) to generate the quantile fractions. By further parameterizing 2 (s, a)
as 272 (s, az 0) with parameter 0, we can derive the loss function J2 (0) as
-
(z) = ((
Σ
Σ
(pi-1 - pi) (d:s)
1=
1=0
where
dzz= = dig (8, a, T, d') = 7Zp (s','';0) - Zp, (8, az 0)
(60)
20
if |6 S
and
(6,) 10 T (8. ( with L (Beg)
16
1
otherwise.
i is the parameter of the target network (Lillicrap et al. 2015) given by the Polyak averaging of A. We refer interested readers to (Dabney et al. 2018a| Ma et al. 2020) for further details.
The training procedures above returns 2/0- Vp E [o, 1]. With the learned 2 our one-step methods presented in Table and Table 4 extract the value function by setting Q E, (2/) Q as the expectation of 2," which is equivalent to the conventional action-value function Q Specifically, we use N = 32 fixed quantile fractions with p; = i/N,, = O...N. Given state-action pair (s, a), we calculate Qo(s, a) = (s, as
1 N Qo(s. a) (=,00)=
Σ 28 (s, A -= + Pi-1
(61)
N
2
Since our methods still need to query out-of- action values during rollout, we employed the conventional double Q-learning (Fujimoto et al. 2018) technique to prevent potential overestimation
without clipping. Specifically, we initialize Q! and Q3 differently and train them minimize (60)
With the learned Q! and Q?, we set the value of Qu(s, a) as
Qo(s,a)= = min (s,aa
(62)
k=1,2
31

---

Under review as a conference paper at ICLR 2023
Table 9: HP search for MG-EBCQ. We report the mean and std of 10 seeds, and each seed evaluates for 100 episodes.
Dataset
Moqq =
Noc =
Nbg = 10
Niag = 20
Noq = 50
Nog = 100
Cheetah-M-v2
47.2 1 0.3
51.5 = 0.2
53.3 0.3
0.3
55.3 0.4
55.8 0.4
Hopper-M-v2
63.3 2.3
82.5=19
88.3
90.8
92.1 7.6
91.3
Walker2d-M-v2
78.6
85.2
6..0+
73.4+11.0
67.6 14.8
62.7 15.8
Cheetah-M-R-v2
38.8
43.0=
4..
0.5
44.8 0.8
44.6
Hopper-M-R-v2
6.94
83.6+ 10.3
82.8
15.9
77.5
76.0 16.7
Walker2d-M-R-v2
55.1 3.4
73.1 + 5.2
75.6+
77.6
7..0++566
78.5 + 4.5
Choctah-M-E-v2
3.2
4.6
8.2
7.3
6.44
08.8
Hopper-M-F-v2
13.6
56.1 6.2
4..9
3.6
29.8
25.3 3.3
Walker2d-M-Bwv2
107.1 1:7.181888
111.1 1.111100 1.0
1114115
111.4
4.0
107.2 6.0
Total
597.2 27.4
670.6 = 31.9
0641+43.1
649.5 + 53.5
628.0 60.2
610.2 = 62.9
Table 10: HP search for SG-EBCQ. We report the mean and std of 10 seeds, and each seed evaluates for 100 episodes.
Dataset
Noeq
Nig =
Nog = 10
Nicg 20
Noq = 50
Nog = 100
Cheetah-M-v2
47.1 1.2
51.5 0.1
53.3 0.2
54.4 0.3 0.3
55.3 1 0.3
55.8 0.4
Hopper-M-v2
6..7
78.6 4.0
86.8 5.2
89.1
89.8 8.8
89.8 9.8
Walker2d-M-v2
2.8
86.9 1.8
85.2
81.5
76.6 11.8
72.4 13.8
Cheetah-M-R-v2
42.3 0.6
4..6
44.3
44.1 1.1
43.6 0.9
Hopper-M-R-v2
58.7 5.8
85.2 9.0
88.5 12.2
89.1+11.7
83.9
82.1 16.1
Walker2d-M-R-w2
54.0
72.2 1: 5.2
75.4.144 4.6
1.8
77.5 17.88.8..8
74.9 6.2
Cheetah-M-E-v2
71.8
81.9 4.8
81.8
6.96
71.5
68.2
Hopper-M-E.v2
66.4
49.8 = 6.2
40.0
34.9
29.0
25.2 +
Walker2d-M-E-v2
106.6 L 1.6
111.0
111.1 1.11118
110.0 3.7
107.2 007.11..8 7.8
106.0 1 9.0
Total
581.6 + 27.7
659.4 = 32.7
665.5 41.0
41.0
51..
634.7 + 63.9
618.1 + 67.5
for every (s, a) pair. Note that the double Q-learning technique is only used during policy evaluation.
As for deciding the number of gradient steps, detail our procedures in Appendix G.5 number gradient steps for each dataset can be found in Table 16
And the
Recall that our CFPI operator aMa QoiT) requires to learn a Gaussian Mixture behavior policy f. and a value function Qo- We train à and Qo according to the procedures
Our IMG Table 1
listed in MG-BC and SARSA, respectively. By following the practice of (Brandfonbrener et al.) Fu et al. 2020), perform grid search log {0, 0.5, 1.0, 1.5, 2.0, using 3 seeds. We note that we manually reduce IMG to MG-MS when log 7 = 0 by only considering the mean each non-trivial Gaussian component. Our results show that setting log
2021
0.5 achieves the best overall performance while Hopper-M-E requires an extremely small log T to perform well as shown in Appendix G.2 Therefore, decide set log datasets. We then obtain the results for the other 7 seeds with these HP settings and report the results on the 10 seeds in total.
for Hopper-M-F and log 0.5 for the other 8
IMG TTbll 4) & Isa (ablle 4). Different from the results in Table we use the same log r = 0.5 for all datasets including Hopper-M-E obtain the performance ZAaG in Table 4 In this way, we aim to understand the effectiveness of each component of our methods better. To fairly compare IMG and Isg. we tune the for Isa in a similar way by performing grid search on log T (0.5, 1.0, 1.5, 2.0) with 3 seeds and finally set log = 0.5 for all datasets. We then obtain the results for the other 7 seeds and report the results with seeds in total.
MG-EBCQ & SG-EBCQ. We tune the number of candidate actions Nayy from the same range {2, 5, 10, 20, 50, 100} as is in (Brandfonbrener et al. 2021). For each Nneq. we obtain its average performance for all tasks across T0 seeds and select the best performing Nooq for each method. We separately tune the Nozq for MG-EBCQ and SG-EBCQ. As result, we set Nocq = for MG-EBCQ and Nocq 10 for SG-EBCQ. Moreover, we highlight that MG-EBCQ (SG-EBCQ) uses the same behavior policy and value function as is in IMGG (IsG). We include the full search results in Table 9 and Table 10
32

---

Under review as a conference paper at ICLR 2023
Table 11: HP search for MG-Rev, KL Reg, We report the mean and std 10 seeds, and each seed evaluates for 100 episodes.
Dataset
0.03
a= 0.1
a= 0.3
a= =
a= 3.0
o= 10.0
Cheetah-M-v2
58.3 1: 1.1
58.1 1.2
5..
50.6 1: 0.3
47.0
44.3 1: 0.2
Hopper-M-v2
14.4 13.6
31.
89.4
99.7
7..3
58.5
Walker2d-M-v2
5.8
18.4+21.9
34.2
8
8.
76.9
Cheetah-M-R-v2
46.7
47.5 1.6
4..1
46.4
44.4
43.1 0.4
Hopper-M-R-v2
33.8
86.6 26.3
1.8.1
1014
9.1
77.6+
Walker2d-M-R-v2
73.7 28.8
65.4 33.8
64.0 39.9
65.4 35.8
69.7 - 30.9
+7.7
Cheetah-M-EEvv2
0.4
1.9
4.
6.3
65.0= 10.1
86.2
Hopper-M-F:v2
2.6
16.2
22.5+ 10.7
57.4 23.6
79.4 32.6
86.8 + 15.7
Walker2d-MMB-v22
10.4 15.3
25.5 38.1
93.5 1.54:34.5 34.5
109.8 0.6
107.1 + 4.0
97.4 7.0
Total
283.2 # 103.0
359.9 = 163.5
5143 = 138.5
637.8 77.2
671.2 1112:89.. 89.1
628.6 76.4
Table 12: HP search for SG-Rev. KL Reg. We report the mean and std of 10 seeds, and each seed evaluates for 100 episodes.
Dataset
0.03
a= 0.1
a 0.3
1.0
3.0
10.0
Cheetah-M-v2
58.6 + 1.3
$7.9
35.2 0.5
5.5
47.1 0.2
4.3
Hopper-M-v2
18.7 + 15.6
40.2
83.2 19.6
2.0
70.3
57.2 4.6
Walker2d-M-v2
3.5
26.2 27.1
37.0 27.6
83.3 7.5
82.4
77.1 1.2
Cheetah-M-R-v2
46.1 3.6
1.3
4..8
0..
4..3
42.5 1: 0.6
Hoppar-M-R-v2
+
60.8 = 27.7
92.0
100.7
99.7
70.3 19.2
Walker24-M-R-v2
59.5
72.7 38.8
+
75.1+25.3
63.6 28.5
19.7+21..5
Chectah-M-E-v2
1.1
3.4
7.1
18.9
9.8
89.1
Hopper-M-E-v2
5.0
2.6
2.9
43.8 23.6
76.6 18.3
67.6
Walker2d-M-F.vv
1.7
13.4
63.2
109.9
+6.8
Total
274.0 85.3
342. 165.9
505.9 150.5
647.2 79.5
669.7 70.3
604. 89.5
MG-Rev. KL Reg & SG-Rev. KL Reg. We tune the regularization strength ar from the same range {0.03, 0.1, 0.3, 1.0, 3.0, 10.0} as is in (Brandfonbrener et al. 2021). For each o, we obtain its average performance for all tasks across 10 seeds and select the best performing for each method. We separately tune the for MG-Rev. KL Reg & SG-Rev. KL Reg, although best overall performance both methods. Moreover, we highlight that MG-Rev. KL Reg (SG-Rev. KI. Reg) uses the same behavior policy and value function is in IMG (IsG). We include the full hyper-parameter search results in Table and Table 12
3.0 achieves the
33

---

Under review as a conference paper at ICLR 2023
F.2 HP AND TRAINING DETAILS FOR METHODS IN
TABLE
Hyperparameter
Value
Shared HP
Normalize states
False
Optimizer
Adam
Kingma & Ba 2014
Number of gradient steps
1M
Mini-batch size
256
Policy learning rate
3e-4
Policy hidden dim
256
Policy hidden layers
IQL HP
Policy activation function
ReLU
Critic architecture
MLP
Critie learning rate
3e-4
Critic hidden dim
256
Critic hidden layers
Critic activation function
ReLU
Target update rate
5e-3
Target update period
quantile
0.9
temperature
10.0
Isa (TIQL QIQL)
log
selected from
Table 13: Hyperparameters for methods
Table 3
Table 13 includes the HP for experiments in Sec. 5.2 The IQL. We use the same HP for the IQL training is reported in the IQL. paper. We obtain the IQL policy TOL and QiQL by training for 1M gradient steps using the PyTorch Implementation from RLkit (Berkeley). a widely used RL library. We emphasize that we follow the authors' exact training and evaluation protocol. We include the training curves for all tasks from the AntMaze domain in Appendix G.6
Note that IQL Kostrikov al. 2021) reported inconsistent offline experiment results on AntMaze in its paper's ' Table 1, Table 2, Table 5, and Table We suspect that these results are obtained from different sets of random seeds. In Appendix G.6 we present all these results in Table 17
To obtain the performance for Isa IIQL, QiQL). we follow the practice of (Brandfonbrener et al. 2021 Fu al., (2020) and perform grid search on log E {0.1, (0.1,0.2 0.2, 2.0} using seeds for each dataset. We then evaluate the best choice for each dataset by obtaining corresponding results on the other seeds. We finally report the results with 10 seeds in total.
Link to the IQI. paper. IQL's Table & are presented in the supplementary material
34

---

Under review as a conference paper at ICLR 2023
G ADDITIONAL EXPERIMENTS
G.1 COMPLETE EXPERIMENT RESULTS FOR MG-MS
Table 14 provides the results of MG-MS on the 9 tasks from the MuJoCo Gym domain in compensa- tion for the results in Sec. 5.3
Table 14:
Results of MG-MS on the MuJoCo Gym domain. We report the mean and standard deviation across 10 seeds, and each seed evaluates for 100 episodes.
Dataset
MG-MS I6
Cheetah-M-v2
43.6 + 0.2
Hopper-M-v2
55.3 + 6.3
Walker2d-M-v2
73.6 2.2
Cheetah-M-R-v2
42.4 0.4
Hopper-M-R-v2
61.5 15.1
Walker2d-M-R--2
65.0 + 10.4
Cheetah-M-B-v2
91.3 2.1
Hopper-M-E-v2
104.2 1
Walker2d-MMEE-v
104.1 + 6.7
Total
641.1 48.5
35

---

Under review as a conference paper at ICLR 2023
G.2
COMPLETE EXPERIMENT RESULTS THE EFFECT THE HP
Fig. 6 presents additional results in compensation for the results in Sec. 5.3 We note that Hopper-
Medrum-Expert--2 requires a much smaller log than the other tasks to perform well.
halfchhetahh-mddiumv22
halchhetaa-meeiiu--epaaayy
halfchetaa-mmeliunexxrrtzzzz
54.0
44.8
95
44.6
53.0
44.2
85
44.0
80
43.8
52.0
75
0.5
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
hopperr-edium-v2
hopper-meduu-repllayy-
hopper-meduummexppert-
95
95
90
85
88
85
80
86
80
70
0.5
1.5
2.0
1.0
1.5
2.0
1e-4
1e-1
log
walker2d-medium-v2
wallerr2d-meduum-eepaayyz
waller2ddmeddumeexpett//
113
85
82
80
78
109
108
log
log
log
Figure 6: Performance of IMG with varying log T. The other HP can be found in Table Each variant averages returns 10 seeds, and each seed contains 100 evaluation episodes. The shaded area denotes bootstrapped 95% Cl.
36

---

Under review as a conference paper at ICLR 2023
G.3 ABLATION STUDY ON THE NUMBER GAUSSIAN COMPONENTS
In this section, we explore whether increasing the number of Gaussian components will result in performance boost. We the same settings in Table 1eexcept modeling 7 with 8 Gaussian instead of 4. We hypothesize the performance gain should most likely happen on the three Medium- Replay datasets, as these datasets are collected by diverse policies. However, Table 15 shows that simply increasing the number of Gaussian components from 4 to 8 hardly results in performance boost, as increasing the number of Gaussian components will induce extra optimization difficulties during behavior cloning (Jin et al. (0166).
Table 15: Comparison between setting the number of Gaussian components to and 8 for our IMG on the three Medium-Replay datasets. We report the mean and standard deviation across seeds, and each seed evaluates for 100 episodes.
Dataset
components Table 1
8 components
Cheetah-M-R-v2
44.5 0.4
44.3 + 0.3
Hopper-M-R-v2 Walker2d-M-R-v2
93.6 = 7.9
90.6 + 11.6
78.2 = 5.6
79.4 + 4.5
37

---

Under review as a conference paper at ICLR 2023
G.4
MODELING THE VALUE NETWORK WITH CONVENTIONAL MLP
wallerzb-mmdiuum-eepayy22
hepger-meziuum-oppaayy22
ha halicheetah-mediim--raaayyy
100
48
a5
Normalized Score
80
Nomallzed Score
47
75
Normalized
46
70
90
45
55
5 Ensemble Size
1
5 Ensemble Size
10
1
2
10
Ensemble Size
Figure 7: Performance of IMG with varying ensemble sizes. Each variant averages returns over seeds, and each seed contains 100 evaluation episode. Each Q-value network is modeled by a 3-layer MLP. The shaded area denotes bootstrapped 95% CI.
Ensemble Size
400
350
350
300
Count Reconts
250
200
200
Count of
100
100
200-
50
50-
10
20
30
40
50
10
20
30
40
50
60
80
90
100
normalized score (binned)
normalized score (binned)
Ensemble Size
Ensemble Size 10
600
500
Count of Reconds
Count of Records
400
10
20
20
40
50
60
70
100
normallzzed score (binned)
normalized
Figure 8: Performance of IMG with varying ensemble sizes Walker2d-Medium- Maaiee2adMMiiummRRepaa2-2222 Each variant aggregates returns over 8 seeds, and each seed evaluates for 100 episodes. Each Q-value network is modeled by 3-layer MLP. With lower ensemble size, the performance exhibits large variance across different episodes.
Replay-v2.
Our experiments in Sec. 5.1 rely on learning a Q value function with the IQN (Dabney et al. (0118a) architecture. In this section, we examine the effectiveness of our CFPI operator IMG when working with an ensemble of conventional MLP Q-value networks with varying ensemble sizes M.
Each Q-value network Q uses ReLU activation and is parameterized with 0g. including 3 hidden layers of width 256. We train each @MP by minimizing the bellman error below
L(Ox) = E "( .D)D r SOLPP(S) (s',a'; 02) SMLP (8, az 0z)
(63)
38

---

Under review as a conference paper at ICLR 2023
where 0k is the parameter of target network given by the Polyak averaging of 8. We set (s. a). We further note that Equation 62 can be reformulated as
(ML 0k) Q
1 Qo(s,a)= min (8, a)( Q8 a)l 2 1b((, a) (8( a)
k=1.2
(64)
= (s, a) as)
where Do and oo calculate the mean and standard deviation of Q value (Ciosek et al. 2019). In the case with an ensemble of Q, we obtain Qo a) by generalizing (64) as below
@o(s,a)= = 122
(QMLP(S, as AMLP
M
k=1
(65)
where
MLP PQ
1 M
Σ as)
Other than the Q-value network, applied the same setting IMG in Table Fig. presents the results with different ensemble sizes, showing that the performance generally increases with the ensemble size. Such phenomenon illustrates limitation of our CFPI operator IMG, as it heavily relies on accurate gradient information (Q, a))a=ay-
A large ensemble of Q is more likely to provide accurate gradient information, thus leading to better performance. contrast, small ensemble size provides noisy gradient information, resulting in high variance across different rollout, as is shown in Fig. g
39

---

Under review as a conference paper at ICLR 2023
G.5
How TO DECIDE THE NUMBER OF GRADIENT STEPS FOR SARSA TRAINING?
halfcheetah--mddiun-2
haiichetthh-medtun-rrelaaay
halicheeza--meeluneeeeerzz/
120
50
110
Validasion less
45
Valicason loss
Validaatio
40
200
14-
3.
35-
25
200
600
800 1600
3402
200
600
860
1000
1200
1000
400
1200 1060 Gradient snops
600
1000
Gradiont stops
Gradiont stops
hopper-medlum--v2
hopper-medllm-rrelaayyyz
hopper/medlum-expprrtv
2.0
22
1.8
Validation lass
Validation loss
14-
14-
3.5
Validation
12-
12
1.6
10
0.8
0.6-
12-
Id00
I000 Gradient steps
Gradient steps
Gradient steps
wallker2d-medlum-v2
walkerr2-mmediium--eeeeya-22
walker22--mediuu--xxpettvz
16
11
Validation less
Validation looos
10
Validation
8.
30
1000
3000
Gradient steps
steps
sceps
Figure Lval on each dataset from the Gym-MuJoCo domain. We can observe that the model overfits to the training set when training for too may gradient steps. Each figure averages the validation loss over folds with the same training seed. The shaded area denotes one standard deviation.
Deciding the number of gradient steps is a non-trivial problem in offline RL. While we use a fixed number of gradient steps for behavior cloning, design rigorous procedure to decide the gradient steps for SARSA training, inspired by the success of k-fold validation.
In our preliminary experiments, we first train 2 using all data from each dataset for 2M gradient steps. We model the Q (s, a) as 3-layer MLP and train following Appendix G.4 By training in this way, we treat Q (s, a)) as the ground truth Q* (s, a) for (s, a) sampled the dataset D. Next, randomly split the dataset with the ratio 95/5 to create the trainining set Durii validation set D define the validation loss as
We then train new Q the SARSA training on D Therefore,
Lval E (s.o)~D..alll (s,
(s, a)ll
(66)
Fig. presents the Lval each dataset from the Gym-MuJoCo domain. We clearly observe that Q generally overfits the Drain when training for too many gradient steps. We evaluate two folds
40

---

Under review as a conference paper at ICLR 2023
Table 16: Gradient steps for the SARSA training
Dataset
Gradient steps
HalfCheetah-Medium-yy
200
Hopper-Mediumn-V2 Medium-v2
400
Balker2d-Medium-a
700
HallCheetah- -ee--MM-eedlaayyee
1500
Miedium-Replay--ye
300
Walker2d Walker2d-Medium Medium Medium-Roplay-v2
1100
400
Hopper- Medium Expertt-2
400
Walker2d -Meddium- Medium-Experr-w Expert-v2
400
with one seed. Therefore, we can decide the gradient steps of each dataset for the SARSA training according to the results in Fig. aa listed in Table 16
41

---

Under review as a conference paper at ICLR 2023
G.6 OUR REPRODUCED IQL TRAINING CURVES
We use the PyTorch (Paszke et al. 2019) Implementation of IQL from RLkit (Berkeley) to obtain its policy mQQ. and value function Qioi.. We do not use the official implementation open-sourced by the authors because our CFPI operators are also based on PyTorch. Fig. T0 presents our reproduced training curves of IQL on the datasets from the AntMaze domain.
We note that the IQL paper does not report consistent results in their paper for the offline experiment performance the AntMaze, as shown in Table 17, We suspect that these results are obtained from different sets of random seeds. Therefore, we can conclude that our reproduced results match the results reported the IQL paper. We believe our reproduction results of IQL are reasonable, even if we do not use the official implementation open-sourced by the authors.
antmaze-umaze-v0
antmaze-meddiumplayyye
antmazz-largg-play--o
80
40
Episode Return
75
Episode Return
60
Episode Return
30
50
40
20
0.5
1.0
0.5
0.0
0.5
1.0
Gradient Steps (1M)
Gradient Steps (1M)
Gradient Steps (1M)
sawnnaz--uazze-dieeee
antmaze- midiiiz--eeddummddveeeeeeeeee
antmaze-large-diverseww
Episode Return
60
Episode Return
60
Episode Return
40
40
20
20
20
0.0
0.5
1.0
0.0
1.0
Gradient Steps (1M)
Gradient Steps (1M)
Gradient Steps (1M)
Figure 10: IQL offline training results AntMaze. Shaded area denotes one standard deviation.
Table 17: Offline experiment results on AntMaze reported in different tables from the IQL paper
Dataset
Table 1 & 6
Table 2
Table 5
antmaze-u-v0
87.5 + 2.6
88.0
86.7 75.0
antmaze-u-d-vo
62.2 + 13.8
67.0
antmaze-m-pp-o
71.2 + 7.3
69.0
72.0
antmaze-m-d-vo
70.0 + 10.9
71.8
68.3
antmaze-l-p-v0
39.6 + 5.8
36.8
25.5
antmaze-l-d-v0
47.5 + 9.5
42.2
42.6
Total
378.0 + 49.9
374.8
370.1
https: ptttps://githae httpthb.tttttt///// com/ wtttm/ikotriikee cwt.://ksttrikoo////iiiiiiii lei_gpiiiittttalam_m__
Link to the IQI. paper. IQL's Table & are presented in the supplementary material
42

---

Under review as a conference paper at ICLR 2023
G.7 IMPROVE THE POLICY LEARNED BY CQL
In this section, we show that our CFPI operators can also improve the policy learned by CQL (ummar al. 2020b) the MuJoCo Gym Domain. We first obtain the CQL policy #CQI. and Qcor. by training for 1M gradient steps using the official CQL implementation We obtain improved policy (TCL, QCoL: T) that slightly outperforms TCQL overall, as shown in Table 18 For all 6 tasks, we log 0.1.
Table 18: Improving the policy learned by IQL with our CFPI operator IsG
Dataset
TCQL (IM)
Isa QCoL)
HaliCheetab-Mediun-22
45.5 0.3
47.1 + 1.5
Hopper-Medium--2
65.4 +:1 3.5
70.1 + 4.9
Walker2d-Mediums
81.4 0.6
81.6 + 1.1
44.6
45.9 + 1.7
Hopper- Mestee-Mediuu---eee Replay-v2
95.2
94.6 + 1.6
Walker2d- Walker2d-Medium--Be Repllay-v2
2.6
78.8 3.2
Total
9.4
418.2 + 13.9
43

---

Under review as a conference paper at ICLR 2023
H ADDITIONAL RELATED WORK
There has been a history of leveraging the Taylor expansion to construct efficient RL algorithms.
Kakade Langford 22002 proposed the conservative policy iteration that optimizes a mixture of policies towards its policy objective's lower bound, which is constructed by performing first-order
Taylor expansion on the mixture coefficient. Later, SOTA deep RL algorithms TRPO (Schulman ct al.
2015) and PPO (Schulman et al.] (017) extend the results to work with trust region policy constraints and learn a stochastic policy parameterized by a neural network. More recently, [Tang 2020) developed second-order Taylor expansion approach under similar online RI. settings.
et al.
At a high level, both our works and previous methods propose to create a surrogate of the original policy objective by leveraging the Taylor expansion approach. However, our motivation to use Taylor expansion is fundamentally different from the previous works (Kakade & Langford 2002 Schulman
2015 2017 Tang et al. 2020). which leverage the Taylor expansion to construct a lower bound of the policy objective so that optimizing towards the lower bound translates into guaranteed policy improvement. However, these methods do not result in closed-form solution to the policy and still require iterative policy updates.
al.
On the other hand, our method leverages the Taylor expansion to construct linear approximation of the policy objective, enabling the derivation a closed-form solution to the policy improvement step and thus avoiding performing policy improvement via SGD. We highlight that our closed-form policy update cannot be possible without directly optimizing the parameter of the policy distribution. In particular, the parameter should belong to the action space. We note that this is significant conceptual difference between our method and previous works.
Specifically, PDL Kakade & Langford 2002 parameterizes the mixture coefficient of mixture policy as 0. TRPO [Schulman et al. 2013) and PPO (Schulman et al. 2017] set 0 as the parameter of a neural network that outputs the parameters of a Gaussian distribution. In contrast, our methods learn deterministic policy T(s) Dirac(O and directly optimize the parameter 0(s). We aim to learn greedy ı by solving (()) arg max,, ((s, a). However, obtaining a greedy и continuous control is problematic (Silver et al.) 2014). Given the requirement of limited distribution shift in the offline RL, we thus leverage the first-order Taylor expansion to relax the problem into a more tractable form
((s) arg max (s,, a; 0z), s.t. - log Ta(ls) S
(67)
where is defined in Equation 3
By modeling n, as a Single Gaussian or Gaussian Mixture, we further transform the problem into QCLP and thus derive the closed-form solution.
Finally, we note that both the trust region methods TRPO and PPO and our methods constrain the divergence between the learned policy and behavior policy. However, the behavior policy always remains unchanged in our offline RL settings. As TRPO and PPO are designed for the online RL tasks, the updated policy will be used to collect new data and becomes the new behavior policy in future training iteration.
44


Reviews:
[{"review_id": "pxisCqpO_7", "reviewer": "Reviewer_CVbg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a method for constraining a policy for offline reinforcement learning. In particular, based on the first order Taylor approximation of the objective, the authors introduce closer form policy improvement operators.", "summary_of_the_review": "The paper is interesting and introduces a simple method that outperforms the baselines. Therefore, I recommend this paper for acceptance. I will increase my score if the authors also add the results on antmaze for the method.", "strength_and_weaknesses": "# Strength\n* The idea of having a closed-form solution for the policy improvement step is exciting and novel.\n* The idea is easy to implement and can be used as a standalone method and in combination with offline reinforcement learning methods such as OnestepRL and IQL, which do not require fitting a policy for value learning.\n* The method improves over the baselines on a standard benchmark for offline reinforcement learning. \n\n# Weaknesses\n* The method outperforms the policy extraction scheme used in the original implementation of IQL on the antmaze tasks. However, the experimental evaluation needs to be extended to demonstrate how the method combined with the iterative policy evaluation performs on antmaze tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nvwLDGMwKTF", "reviewer": "Reviewer_ypTb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on behavior-constrained policy optimization in offline RL problems. The authors use first-order Taylor approximation to get a closed-form solution for the policy update objective with behavior constraint. The behavior policy can be modeled as a single Gaussian or Gaussian mixture model and this paper provides a closed-form policy improvement operator in both cases. \n\nThe authors described a detailed theoretical derivation of this closed-form operator to map the behavior policy to a higher-valued policy. The policy-improved operator can be incorporated into general actor-critic offline RL algorithms to replace the policy update step. The experimental results empirically demonstrate that the proposed method is better than the existing SOTA methods on D4RL.", "summary_of_the_review": "A novel optimization approach for policy update with behavior constraints in offline RL, well supported by theoretical derivation and extensive evaluation.", "strength_and_weaknesses": "Strengths:\nThe authors provide both theoretical foundation and empirical results to demonstrate the efficacy of the proposed method.\nThe evaluation shows convincing metrics in Figure 2.\nThe ablative study is extensive.\nThe proposed method is novel because all prior works conduct the policy update step in offline RL via SGD, but this paper introduces a closed-form solution.\n\nWeaknesses:\n\nAs shown in Algorithm 1, the closed-form policy improvement operator should be compatible with the general actor-critic offline RL algorithm. Could you please show the performance when the operator is combined with other offline RL algorithms, such as CQL and TD3-BC? Does it always bring improvement no matter what's the base offline RL algorithm?\n\nOne strong motivation for this work is that \"training stability poses a major challenge in offline RL\". Could you provide a specific example where existing offline RL algorithms to learn value function and policy via SGD really suffers from the instability issue, and the proposed method directly solved this? Although there are experimental results showing the proposed method is better than existing methods, it is unclear whether the advantage really comes from better training stability. So the claim in the introduction about \"mitigate the issue of learning instability by leveraging optimization techniques\" is not fully supported.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "atevwlcCxcC", "reviewer": "Reviewer_m3D5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a closed form update for offline RL. A few major assumptions and steps of derivations apply: (1) policy is Gaussian or mixture of Gaussians; (2) apply Taylor expansion to the optimization objective. The paper shows some theoretical properties of the proposed method, and empirical improvements over prior work baselines.", "summary_of_the_review": "A few main questions.\n\n=== **Taylor expansion based approach to RL** ===\n\nUsing Taylor expansion is not novel in the RL community in general. As the authors may have noted, PDL [1] itself arises from the application of Taylor expansion to the value function objective. Renowned algorithms such as TRPO and PPO have applied the results to trust region setting, where there is a regularization from the behavior policy (or any regularized policy chosen by the algo designers) [2]. Lately, Tang et al. [3] has developed second order Taylor expansion approach to a similar setup in online RL.\n\nAdmittedly, there is some difference between the setup in the current paper and aforementioned work, most notably, here the Taylor expansion is applied to the action directly (mainly because the policy is deterministic) whereas prior work mostly handles stochastic policy. In prior work, the expansion takes place in the policy distribution space; here, the expansion takes place in the action space. Such technical differences aside, I think it is worth discussing current work's relation with such prior work.\n\n[1] Approximately optimal approximate RL, Kakade et al, 2007\n[2] Proximal policy optimization, Schulman et al, 2015\n[3] Taylor expansion policy optimization, Tang et al, 2020\n\n=== **Why closed form update is useful in offline RL case** ===\n\nThough experiments have showed performance gains, I think it helps the paper can be greatly improved to clarify more on the motivational question: why is using closed form update useful in the offline RL case. Intuitively, I think this might make sense because compared to usual SGD based updates, closed form update may reduce the stochastic jittering a bit and hence make the update more stable. Can we characterize the effect in theory? Or can we have some clean simple setup to illustrate this point? This question I think is quite fundamentally interesting overall but is a bit lacking in the current paper.\n\n=== **Theory is a bit weak** ===\n\nThe flagship theoretical result in the paper is Thm 1, where they show an one-step improvement bound on the objective. On the technical side, I think showing the theory does require adaptations from the original PDL, due to the difference between expanding on the action space or policy space, but overall the adaptation is pretty straightforward. \n\nThe main issue I have is that the connection between theory and practice is very weak. The efficiency of any offline RL algorithms over baseline methods, should be due to its statistical efficiency, i.e., given finite amount of offline RL data, can the algorithm achieve good performance. This setup underlies the challenge of offline RL where no new data can be collected. However, Thm 3.1 deals with the \"expectation\" behavior of the algorithm, i.e., how the algorithm behaves when there is infinite amount of data such that finite sample bounds turn into expected improvements. This is not very insightful because by studying the expected behavior of the algorithm, we bypass much of the challenge that underlies offline RL itself.\n\nThere should arguably, be comparison to sample complexity bound mentioned in [4]. If we obtain the finite sample bound version of Thm 3.1, does it improve over the bound derived in CQL? How do they compare? They have different dependencies on diff hyper-parameters on the env, what do they mean in practice?\n\n[4] Conservative Q-learning for offline RL, Kumar et al, 2020\n\n=== **Experiment** ===\n\nThe experiments show that the new algo obtains benefits in certain envs over the SOTA baselines, but understandably the improvements are not uniform and not always. This is expected because different algorithms behave differently under the same env, and we may not expect the same algo to outperform all other SOTA at the same time. I think what's valuable is to elucidate why the new algo performs better than the previous SOTA, e.g., IQL in M and ME dataset, but not as much in the MR dataset (in Table 1). Such a comparison would be valuable to understand relative strengths of diff methods.\n\nAs a side note, I think adding a \"total\" row in the table is not very informative, because aggregating statistics across all games does not help understand the performance much in this specific case.", "strength_and_weaknesses": "=== Strength ===\n\nThe paper provides an application of Taylor expansion based objective + behavior regularization in the offline RL setting, and has shown empirical improvements over baseline algorithms. This might motivate carrying out analytic updates (based on Taylor expansion derived objective) in future investigation on offline RL.\n\n=== Weakness ===\n\nThe theoretical and conceptual contributions of the paper are a bit weak. Though the contribution of deriving analytic updates based on Taylor expansion + Gaussian policy assumption is technically solid, it is not clear why carrying out such updates should be useful in the offline RL setting in the first place. Is this due to statistical efficiency? Does it help stabilize the update? The theory front of the paper is weak in that much of the results come from technical derivation, or Thm 3.1 is a fairly straightforward application of PDL (admittedly, certain adaptations are needed since in this case the action is being expanded in the first order, but not the policy as in the original PDL case). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject