Behavior Proximal Policy Optimization 
Authors: Zifeng Zhuang
Kun LEI
Jinxin Liu
Donglin Wang
Yilang Guo
Venue: ICLR 2023 poster (2023)
Keywords: Offline Reinforcement Learning
Monotonic Policy Improvement

Abstract: Offline reinforcement learning (RL) is a challenging setting where existing off-policy actor-critic methods perform poorly due to overestimating of out-of-distribution state-action pairs. Thus, various additional augmentations are proposed to keep the learned policy close to the offline dataset (or the behavior policy). In this work, starting from the analysis of offline monotonic policy improvement, we reach a surprising conclusion that online on-policy algorithms are naturally able to solve offline RL. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to overcome the overestimation. Based on this, we propose Behavior Proximal Policy Optimization (BPPO), which solves offline RL without any extra constraint or regularization introduced compared to PPO. Extensive experiments on the D4RL benchmark empirically show this extremely succinct method outperforms state-of-the-art offline RL algorithms. Our implementation is available at https://github.com/Dragon-Zhuang/BPPO.

Published as a conference paper at ICLR 2023
BEHAVIOR PROXIMAL POLICY OPTIMIZATION
Zifeng Zhuang 12- Kun Lei
Jinxin Liu Donglin Wang 28t Yilang Guo
Zhejiang University. School of Engineering, Westlake University.
3 Institute of Advanced Technology, Westlake Institute for Advanced Sudyy.
School of Software Engineering. Beijing Jiaotong University.
(zhuangzifenge leikun, diujinxinn wangdong in) @westlake edu. cn, 203011306bjtu.cm www.:eeetttuuuu..aoo.moo
ABSTRACT
Offline reinforcement learning (RL) is challenging setting where existing off off-policy actor-critic methods perform poorly due to overestimating of out-of- distribution state-action pairs. Thus, various additional augmentations to keep the learned policy close to the offline dataset (or the behavior policy). In this work, starting from the analysis of offline monotonic policy improvement, a surprising conclusion that online on-policy algorithms are naturally able to solve offline RI. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to overcome the overestimation. Based on this, we propose Behavior Proximal Policy Optimization (BPPO), which solves of- fline RL without any extra constraint or regularization introduced compared to PPO. Extensive experiments on the D4RL benchmark empirically show this extremely succinet method outperforms -tate-of-the-art offiin RL algorithms. Our implemen- tation is available at atttpss //attp:://gtthbb. wttp:uu..ooomoooooooooe ctw.aaagoommmm//o/ooo SwarrrgonnZhuunngen Zhuang/BPPO
proposed
reach
INTRODUCTION
Typically, reinforcement learning (RL) thought of as a paradigm for online learning, where the agent interacts with the environment to collect experiences and then uses them to improve itself (Sutton et al., 1998). This online process poses the biggest obstacles to real-world RL applications because of expensive or even risky data collection in some fields (such navigation (Mirowski et al. 2018) and healthcare (Yu et al. [a)) As an alternative, offline RL eliminates the online interaction and learns from fixed dataset collected by some arbitrary and possibly unknown process (Lange ct al., 2012; Fu et al. 2020). The prospect of this data-driven mode (Levine et al, 2020) is pretty encouraging and has been placed with great expectations for solving real-world RL applications.
Unfortunately, the major superiority of offline RL, the lack of online interaction, also raises another challenge. The classical off-policy iterative algorithms tend to underperform due to overestimating out-of-distribution (shorted as OOD) state-action pairs, even though offline RL can be viewed as an extreme off-policy case. More specifically, when Q-function poorly estimates the value of OOD state-action pairs during policy evaluation, the agent tends to take OOD actions with erroncously estimated high values, resulting in low-performance after policy improvement (Fujimoto et al.) 2019). Thus, to overcome the overestimation issue, some solutions keep the learned policy close to the behavior policy (or the offline dataset) (Fujimoto et al. 2019 Wu et al. 2019 Fujimoto & Gu 2021).
Most offline RL algorithms adopt online interactions to select hyperparameters. This is because offline hyperparameter selection, which selects hyperparameters without online interactions, is always an open problem lacking satisfactory solutions (Paine et al., 2020 Zhang & Jiang (021). Deploying the policy learned by offline RL potentially risky in certain areas (Mirowski et al. 2018 Yu et al. 202 since the performance is unknown. However, the risk during online interactions will be greatly reduced if the deployed policy can guarantee better performance than the behavior policy. consider how to offline dataset improve behavior policy with monotonic performance guarantee. We formulate this problem as offline monotonic policy improvement.
This inspires
Equaal contribution. 1 Corresponning author.

---

Published as a conference paper at ICLR 2023
To analyze offline monotonic policy improvement, introduce the Performance Difference Theorem (Kakade & Langford 2002). During analysis, we find that the offline setting does make the monotonic policy improvement more complicated, but the way monotonically improve policy remains unchanged. This indicates the algorithms derived from online monotonic policy improvement (such as Proximal Policy Optimization) can also achieve offline monotonic policy improvement. In other words, PPO can naturally solve offline RL. Based on this surprising discovery, we propose Behavior Proximal Policy Optimization (BPPO), an offline algorithm that monotonically improves behavior policy in the manner of PPO. Owing to the inherent conservatism of PPO, BPPO restricts the ratio of learned policy and behavior policy within a certain range, similar the offline RL methods which make the learned policy close to the behavior policy. As offline algorithms are becoming more and more sophisticated, TD3+BC (Fujimoto & Gu (02T), which augments TD3 (Fujimoto et al. 2018) with behavior cloning (Pomerleau] 1988), reminds us to revisit the simple alternatives with potentially good performance. BPPO is such a "most simple" alternative without introducing any extra constraint or regularization on the basis of PPO. Extensive experiments on the D4RL benchmark (Fu et al. 2020) empirically shows that BPPO outperforms ofte-of-thhean offline RL algorithms.
2
PRELIMINARIES
2.1 REINFORCEMENT LEARNING
Reinforcement Learning (RL) is framework of sequential decision. Typically, this problem is formulated by a Markov decision process (MDP) M (S, r, p. do, 7}. with state space S, action space A, scalar reward function r, transition dynamics p, initial state distribution do (so) and discount factor 7 (Sutton et al., 1998). The objective RL is to learn a policy, which defines a distribution over action conditioned
states (ai (t)) at timestep where a A. 8t E S. Given this definition, the trajectory T = (Sg. ap, ... ST, ar) generated by the agent's interaction with environment M can be described as distribution P. (r) do(so) I[ ' (ag|st) ($1+1 (St, az), where T is the length of the trajectory, and it can be infinite. Then, the goal of RL can written as an expectation under the trajectory distribution J (T)) (.)
Σ 7 >'r(st, az) This objective can also be measured by a state-action value function Q₂ (s, a), the expected discounted return given the action a in state
Qx (s, a) = (r)sa())ssa)) ΣT 7'r at) |s0 s, ap = a Similarly, the value function V (s) is the expected discounted return of certain state Va (s) = (r(a)) Σ 2rr(st, 0z) 8 Then, we can define the advantage function: Ar (s, a) = Q+ (s. a) Va (s).
2.2 OFFLINE REINFORCEMENT LEARNING
= {((, az S141, r
}
In offline RL, the agent only has access to a fixed dataset with transitions D =
collected by behavior policy TT. Without interacting with environment M, offline RL expects the agent to infer policy from the dataset. Behavior cloning (BC) (Pomerleau 1988), approach of imitation learning, can directly imitate the action each state with supervised learning:
f argmax E(
[log π (als)].
(1)
Note that the performance of ne trained by behavior cloning highly depends the quality of transitions, also the collection process of behavior policy πą. In the rest of this paper, improving behavior policy actually refers improving the estimated behavior policy 7.. because Ty is unknown.
2.3 PERFORMANCE DIFFERENCE THEOREM
Theorem 1. (akade & Langford 2002) Let the discounted unnormalized visitation frriioonrreeeeiiie as Pn (s) Σt 7²t P (st = s)m) and P (s+ s)m) represents the probability of the t-th state equals to in trajectories generated by policy w. For any two policies n and n²' the performance difference
Ja (I ()',))= J (x') J (T) can be measured by the advantage function:
T
JA (T', T) = Br-PEr(T)
Σ x A An ax) 1==
= EEopp [A a))
(2)
2

---

Published as a conference paper at ICLR 2023
This theorem implies that improving policy from to N' can be achieved by maximizing (2. From this theorem, Trust Region Policy Optimization (TRPO) (Schulman et al. 2013a) is derived, which guarantee the monotonic improvement of performance. We also apply this theorem to formulate offline monotonic policy improvement.
Derivation detail is presented in Appendix A
3 OFFLINE MONOTONIC IMPROVEMENT OVER BEHAVIOR POLICY
In this section, we theoretically analyze offline monotonic policy improvement based on Theorem namely improving the i generated by behavior cloning ① with offline dataset D. Applying the Performance Difference Theorem to the estimated behavior policy 7t. we can get
JA (x, it) = (e (),exx((ee)) [Axp (s,
Maximizing this equation can obtain policy better than behavior policy n.. But the above equation not tractable due to the dependence of the new policy's state distribution Pr. (s). For standard online method, Pn. (s) is replaced by the old state distribution Paş (s). But in the offline setting, Pay (s) cannot be obtained through interactions with the environment like the online situation. We use the state distribution recovered by the offline dataset pD (s) for replacement, where PD (s) = Ei P (st = s)D) and P (s, s represents the probability of the t-th state equals to in the offline dataset. Therefore, the approximation of Ja (T,, TT)) can be written as:
Ja (T, nt) (x-pp(())
(sx
To measure the difference between JA (IT, #t) and its approximation JA (TT, #g), we introduce midterm Es-psy (x))))(()))) [A a)] with the state distribution P#₁ (s). During the proof, the
commonly-used total variational divergence DTv (=|(*t) [s] f T (als) - nे (a|s) between policy म, n. at state is necessary. For the total variational divergence between the offline dataset D and the estimated behavior policy ag. it may not be straightforward. We can view the offline dataset D = (st, Ge, 8t)1, as deterministic distribution, and then the distance is:
Proposition 1.
For offline dataset D { (8t, ae, re+1; and policy S the total variational divergence can be expressed as Drv (DI [st]
ft (arls:)).
Detailed derivation process is presented in Appendix
B Now ready to measure the difference: (1 fe (azls:)). we can
Theorem 2. Given the distance Drv (T] [s] and Drv (DIl derive the following bound:
JA (T)) 2 JA Tt) 4y max Drv ( ( [s]
[Dy (T]) [s]]
s>peg(t)
-2yhng max Drv (ा|(
[s]
E ()pp())
- (als)],
here Ang max Aag (8, a)). The proof is presented in Appendix
Compared to the theorem in the online setting (Schulman et al. 2015a Achiam et al. 2017 Queeney et al., 202I), the second right term of Equation (5) is similar while the third term is unique for the offline. E
Å.. (a|s)] represents the difference caused by the mismatch between offline
s>-p(()) dataset D and ft.. When ž is determined, this term is one constant. And because the inequality max, Drv (4 Z
[Drv (T]((tg) [s]] holds, we can claim the following conclusion:
s (-)
Conclusion
To guarantee the true objective JA (I,, ftg) non-decreasing. we should simultancously maxi- mize (amppp)-aan<(te [Aay (s, a)] and minimize [max, Drv (π|(*t) [s]], which means the offline dataset D is capable of monotonically improving the estimated behavior policy At.
3

---

Published as a conference paper at ICLR 2023
Suppose have improved the behavior policy àt and get a policy Tg. The above theorem only guarantees that Fą has a higher performance than ng but Ty may not be optimal. If the offline dataset D can still improve the policy Te to get better policy Tk+1, + must be closer to the optimal policy. Thus, we further analyze the monotonic policy improvement over policy Ty.- Applying Performance Difference Theorem to the policy Tgg
JA (T T&) (a [Axx (s, a)].
(6)
To approximate the above equation, the common manner replacing P= with the old policy state distribution Pr.. But in the offline RL, тı is forbidden from acting in the environment. As result, the state distribution Pru is impossible estimate. Thus, the only choice without any other alternative is replacing Prr. by the state distribution from the offline dataset D:
Ja (T, m = Exrppp (sspp())anm((l)) [Ar, [s)
Intuitively, this replacement is reasonable if Th, Tg are similar which means this approximation must be related to the distance Drv (f [s]. Concretely, the gap can be formulated as follows:
Theorem 3. Given the distance DTv (T]|mT) [s], Drv (Tal)
[s] and Drv (D])at) [s]
tt (als)), we can derive the following bound:
JA (T, mh) 2 Jo (TT, TK) 47 max Drv (T][TT) [s]
(T)
[Drv (π][)T) [s]]
8
p
-y max DTv (T]| ) [s]
DDv (Tx (t) [+]]
s ())
2y max Drv (T] [s]
E
- Ao (als))
(8)
(-pp(())
here AT.
max 8,0
Am (s,a)): The proof is presented in Appendix
Compared to the theorem 2 one additional term related the distance of Th Ag has been introduced.
The distance
DDV (#)] [s]] is irrelevant to the target policy π which can also be viewed
srupa
as one constant. Besides, theorem is specific case of this theorem if TH = म... Thus, we set 00 = nt since मे the first policy to improved and the following section we will no longer deliberately distinguish t,. Th. Similarly, we can derive the following conclusion:
Conclusion
To guarantee the true objective JA (T, T&) non-decreasing, we should simultaneously maxi- mize Eew<p(+)) [Ar, (s, a)] and minimize [max, Drv (T((TT)) [s]]. which means the
offline dataset D capable of monotonically improving the policy Tg. where 0, 1.2.....
BEHAVIOR PROXIMAL POLICY OPTIMIZATION
In this section, we derive a practical algorithm based on the theoretical results, And surprisingly, the loss function of this algorithm is the same the online on-policy method Proximal Policy Optimization (PPO) (Schulman et al. 2017). Furthermore, this algorithm highly depends on the behavior policy so we name it as Behavior Proximal Policy Optimization, shorted as BPPO.
According to the Conclusion 2, to monotonically improve policy тh. we should jointly optimize:
Maximize Ex-po (-).a [Ax, a)) & Minimize max Drv (T|]TT) [s],
(9)
here 1 = 1, and uu fg. But minimizing the total divergence between æ and Te results in trivial solution _ = n which is impossible to make improvement over Tg. A more reasonable optimization objective is to maximize JA (T,, TK) while constraining the divergence:
Maximize Es-pp() [Ax, (s, a)] s.t. max Drv (T]])T)) [s] Sc
(10)
4

---

Published as a conference paper at ICLR 2023
For the term to be maximized, adopt importance sampling to make the expectation only depends on the action distribution of the old policy Ty rather than the new policy æ.
(a|s) Trk (a|s) An (s,
(xxx)p()) [Ame (s, a)] (s-pp()) (-)a)
(11)
In this way, we could estimate this term by sampling states from offline the dataset PD (-) then sampling actions with old policy a ~ Tkk (.1s). For the total variational divergence, we rewrite as
max Drv (TT])TT)) [s] max / π (als)
TT (als) da
Ja
1
π (als)
(als)
max 2 Th (als)
1 da
max E
(12)
(T& (a|s)
2
(-1a)
Ty (a|s)
In the offline setting, only states PD (.) are available and other states are inaccessible. So the operation max, can also be expressed as max When comparing Equation
(TT and (12). we find
s<pp(-)
that the state distribution, the action distribution and the policy ratio appear in both. Thus we consider The following constraints are equivalent:
how to insert the divergence constraint into Equation (II).
π (als)
max Drv (T (T)) suppo(-)
[s] SE
max s-ppo(-)
E
s 2e
Th (a]s)
(als) Th (als)*
max se-pp())
E
clip
2c, 20
clip (a,1,u) min (max (z, (13)
aw (
Here the max operation is impractical solve, we adopt heuristic approximation (Schulman ct al. 2015a) that changes max into expectation. Then divergence constraint (13) can be inserted:
(a|s) TTk (a|s) A (s,
(als) T (a|s)
1 2e, + 26) Ar, (0: (ea(ssa))
La (T) (e)pp()))amaaa min
(14)
where the operation min makes this objective become the lower bound of Equation IT).. This loss function is quite similar to PPO (Schulman et al. (007)) and the only difference is the state distribution. Therefore, we claim that online on-policy algorithms are naturally able to solve offline RL.
DISCUSSIONS AND IMPLEMENTATION DETAILS
In this section, first directly highlight why BPPO can solve offline reinforcement learning, namely, how to overcome the overestimation issue. Then we discuss some implementation details, especially, the approximation of the advantage An, (s. a). Finally, we analyze the relation between BPPO and previous algorithms including Onestep RL and iterative methods.
Why BPPO solve offline RL? According the final loss (14 and Equation 13 BPPO
actually constrains the closeness by the expectation of the total variational divergence:
π (a|s) Ty (a|s)
< 2c.
(15)
If A = O, this equation ensures the closeness between learned policy π and behavior policy 7t. When > O, one issue worthy of attention is whether the closeness between learned policy π and T can indirectly constrain the closeness between and ft. To achieve this, also prevent the learned policy л completely away from n.s, we introduce a technique called clip ratio decay. As the policy updates, the clip ratio gradually decreases until reaching a certain training step (such 200 steps):
= x (a)' IF < 200 ELSE
= 6200
(16)
here denotes the training steps, 6g denotes the initial clip ratio, and o E (0, the decay coefficient.
5

---

Published as a conference paper at ICLR 2023
From Figure I(a] and [(b)
can
find that the ratio Tk n may be out of the certain range [1 2e, 1 + 2c] (the region surrounded by the dotted pink and purple line) without clip ratio de- cay technique (also = 1). But the ra- tio stays within the range stably when the decay is applied which means the Equation (15) can ensure the close- ness between the final learned policy by BPPO and behavior policy.
hesterc
(a) hopper-medium
(b) hopper-medium-replyy
Visualization of the importance weight between the updated policy Ty and the estimated behavior policy a--
Figure I:
How to approximate the advantage? When calculating the loss function (14). the only difference from the online situation is the approximation of advantage As, (s, a). In online RL, GAE (General- ized Advantage Estimation) (Schulman et al. 2015b) approximates the As,, using the data collected by policy тk- Obviously, GAE is inappropriate in the offline situations due to the existence of online interaction. As result, BPPO has calculate the advantage A=, QT.. V in an off-policy manner where Qn is calculated by Q-learning (Watkins & Dayan 1992) using offline dataset 'D and Vu is caleulated by fitting returns E 7'r(st, at) using the MSE loss. Note that the value function is Vr rather than Vr, since the state distribution has been changed into s PD (.) in Theorem 2
Algorithm 1 Behavior Proximal Policy Optimization (BPPO)
Besides, we have another simple choice based on the results that close to the TH with the help of clip ratio decay. We can replace all the Ase with the Agg. which may introduce some error but the benefit is that An, must be more accurate than Are since off- off-policy estimation is potentially dangerous, especially in the offline setting. We conduct a series of experiments in Section 7.2 to compare these two implementations and find that the latter one, advantage replacement, is better. Based on the above im- plementation details, we summa- rize the whole workflow of BPPO Algorithm
1: Estimate behavior policy से by behavior cloning;
2: Calculate Q-function Qn by SARSA;
3: Calculate value function Vx by fitting returns;
4: Initialize kt = 0 and set T& Th & Quz Quai
5: for 0, 1, ... do
6: AT, = QT. Vi
7:
Update the policy by maximizing La (T);
8:
if J(x) > (TT) then
9:
Set k = k +1 & The t
10:
advantage replacement then QT, Qu
11:
12:
else
Calculate QT by Q-learning;
14:
end if
end if
16:
end for
What is the relation between BPPO, Onestep RL and iterative methods? Since BPPO is highly related to on-policy algorithms, it is naturally associated with On- estep RL (Brandfonbrener et al.) 2021) that solves offline RL without off off-policy evaluation. If we remove lines 8~~15 in Algorithm [] we get Onestep version of BPPO, which means only the behavior policy H is improved. In contrast, BPPO also improves TR, the policy that has been improved over a... The right figure shows the dif- ference between BPPO and its Onestep version: Onestep strictly requires the new policy close to ita while BPPO appropriately loosens this restriction.
Co
Fo
E1
62
41
Th (
(
12
Figure 2: The difference between Onestep BPPO (left) and BPPO (right), where the decreasing circle corresponds to decay.
we calculate the Q-function in off-policy manner, namely, line 13 Algorithm [ the method switches to an iterative style. If adopt advantage replacement, line 11, BPPO only estimates the advantage function once but updates many policies, from ftg to T Onestep RL estimates the Q-function once and use it to update estimated behavior policy. Iterative methods estimate Q-function several times and then update the corresponding policy. Strictly speaking. BPPO is neither an Onestep nor iterative method. BPPO is special case between these types.

---

Published as a conference paper at ICLR 2023
RELATED WORK
Offline Reinforcement Learning Most of the online off-policy methods fail or underperforn in offline RL due to extrapolation error (Fujimoto al.) 2019) distributional shift (Levine al. 2020p. Thus most offline algorithms typically augment existing off-policy algorithms with penalty measuring divergence between the policy and the offline data (or behavior policy). Depending on how to implement this penalty, variety of methods were proposed such as batch constrained (Fujimoto
al. 2019), KL-control (Jaques al., 2019 Liu et al., 20226), bchavior-regularize, (Wu et al. 20199
Fujimoto & Gu 202T) and policy constraint (Kumar et al. 2019; Levine et al. 2020 Kostrikov et al.
2021). Other methods augment BC with a weight to make the policy favor high advantage actions
(Wang et al.) 2018 Siegel et al. 2020; Peng et al. 2019 Wang ct al. 2020). Some methods extra
introduced Uncertainty estimation (An et al. 2021b lb Bai et al. 2022) or conservative (Kumar et al.
2020
Yu et al.
2021b Nachum al. 2019) estimation to overcome overestimation.
Monotonic Policy Improvement Monotonic policy improvement in online RI. was first introduced by Kakade Langford (2002). On this basis, two classical on-policy methods Trust Region Policy Optimization (TRPO) (Schulman et al., (2015a) and Proximal Policy Optimization (PPO) (Schulman
al. 2017) were proposed. Afterwards, monotonic policy improvement has been extended to constraind MDP (Achiam et al. (017). model- based method (Luo et al., 2018) and off- aff-policy RL (Qeenney et al., 2021 Meng al. (02T). The main idea behind BPPO to regularize each policy update by restricting the divergence. Such regularization is often used in unsupervised skill learning (Liu et al. [202]] 2022a Tian et al. 202T) and imitation learning (Xiao et al. 2019 Kang et al.) 2021). Xu et al. (2221) mentions that offline algorithms lack guaranteed performance improvement over the behavior policy but we are the first to introduce monotonic policy improvement to solve offline RL.
EXPERIMENTS
We conduct series experiments on the Gym (v2), Adroit (v1), Kitchen (v0) and Antmaze (v2) from D4RL (Fu et al. 2020) to evaluate the performance and analyze the design choice of Behavior Proximal Policy Optimization (BPPO). Specifically, we aim to answer: 1) How does BPPO compare with previous Onestep and iterative methods? 2) What is the superiority of BPPO over its Onestep and iterative version? 3) What is the influence of hyperparameters clip ratio and clip ratio decay o?
Table The normalized results on D4RL Gym, Adroit, and Kitchen. We bold best results and BPPO is calculated by averaging mean returns over 10 evaluation trajectories and five random seeds. The symbol* specifies that the results are reproduced by running the offical open-source code.
Iterative methods
Onestep methods
Suite
Enviromment
CQL
TD3+BC
Onestep RL
IQL
BC (Ours)
BPPO (Ours)
hulfcheetah medium v2
44.0
48.3
48.4
47.4
43.5.10.1
44.0.10.2
hopper- merper-mdiim-22
61,3433.2
93,941:99
walker2d-medium
72,5
83.7
81.8
78.3
74,2=4466
83,6++099
halfcheetah medium- medium-roplay-r2
45.5
44.6
38.1
44.2
40.120.1
41.0+::.6
hopper medium replay-v2
95.0
60.9
97.5
94.7
66.0.1 18.3
92.5.13.4
Gym
walker2d medium- medium-replay-ve replay-v2
77.2
$1.8
49.5
73.9
33.44:1.2
77.617..88
hullcheetah medium expert-v2
92.5.119
hopper medium expert x22
105.4
98.0
103.3
91.5
64.917.7
112.8117
woolle/2dmeddum-xxppee -xpprtteexper--22
108.8
113.0
109.6
107.743.5
113.1+2244
Gym focomotion-12 total
698.5
677.4
684.5
692.4
553.1=:57.2
751.0++21.8
pen- not-human--l
61.61997
117.8:11199
hammer human. vl
4.4
2.0*
0.2*
1.4
2.0.1..9
14.9
human-vl
9.9
-0.1
7,8-3.55
25.9-1-7.5
relocatc-human--1
0.2
-0,3"
2.1
0.1
0.1
4.8-22.2
Adroit
pen-cloned-v1
39.2
41.5*
60.0
3T.3
18.8
110.::6.3
hammer cloned-vv
2.1
0.8*
2.0
0.5
8.915.11
door-cloned-vl
0.4
-0.4"
0.4
1.6
0.9.0.88
6.2-1166
relocate cloned-vl
0.1110.0
admit-v total
93.6
52.2
155.2
FIR.1
131.6-131.1
291.4
kitchen-eunng Mlichen-omppltte-vd
43.8
0.0*
2.0
62.5
55.0
91.3.899
kitchen caarhen-partial-ve partial-vO
49.8
22.5*
35.5*
46.3
44.0149
$7.0-12.4
Kitchen
kitchen mixed-vl
62.5.66.7
lätchen. vi total
144.6
47.5
63.5
159.8
144.0 1800
211.0
locomotion+ kitche+
936.7
777.1
905.3
970.3
831.14100.33
1253.44:78.6
7

---

Published as a conference paper at ICLR 2023
7.1 RESULTS ON D4RL BENCHMARKS
We first compare BPPO with iterative methods including CQL (Kumar et al. 2020 and TD3+BC (Fujimoto & Gu 2021). and Onestep methods including Onestep RL. (Brandfonbrener et al. 2021 and IQL (Kostrikov et al. 2021). Most results of Onestep RL, IQL, CQL, TD3+BC are extracted from the paper IQL and the results with symbol * are reproduced by ourselves. Since BPPO first estimates a behavior policy and then improves it, we list the results of BC on the left side of BPPO.
From Table I we find BPPO achieves comparable performance outperforms when considering the total performance. For Adroit and Kitchen, BPPO prominently outperforms other methods. Compared BC, BPPO achieves 51 % performance improvement on all D4RL tasks. Interestingly, our implemented BC on Adroit and Kitchen nearly outperform the baselines, which may imply improving behavior policy rather than learning from scratch is better.
each task of Gym and slightly
Next, we evaluate whether BPPO can solve more difficult tasks with sparse reward. For Antmaze tasks, we also compare BPPO with Decision Transformer (DT) (Chen et al. (021), RvS-G and RvS-R (Emmons et al. 202T). DT conditions on past trajectories to predict future actions using Transformer. RyS-G and RyS-R condition on goals or rewards learn policy via supervised learning.
Table 2: The normalized results on D4RL Antmaze tasks. The results of CQL and IQL are extracted from paper IQL while others extracted from paper RvS. In the BC column, symbol * specifies the (Emmons et al. 2021) which removes the failed trajectories instead of standard BC.
Filtered BC
Enviromment
CQL
TD3+BC
Onestep
IQL
DT
RyS-R
RyS-G
BC (Ours)
BPPO (Ours)
Umaze-v2
74.0
78.6
64.3
87.5
65.6
64.4
65.4
$1.7-12.4
95.0
Umaze-diverse-v2
84.0
71.4
60.7
62.2
51.2
70.1
48.3-47.22
91.7-14.1
Medium--pla--v2
61.2
10.6
0.3
71.2
1.0
4,5
58.1
16.7=5.2*
51.7=7.5
Medium--diverseee
53.7
3.0
0.0
70.0
0.6
7.7
67.3
33.3::10.3*
70.0:16.3
Large play v2
15.8
0,2
0.0
39.6
0.0
3.5
32.4
11.7
86.7.88.2
Large-divers--2
14.9
0.0
0.0
47.5
0.2
3.7
36.9
46.7:20079
88.3:14.11
Total
303.6
163.8
61.0
378.0
118.6
153.9
321.0
245.0+885.5
483.3=35.7
As shown in Table 2 BPPO can outperform most tasks and is significantly better than other algorithms in the total performance all tasks. We adopt Filtered BC last four tasks, where only the successful trajectories is selected for behavior cloning. The performance of CQL and IQL is very impressive since no additional operations or information is introduced. RyS-G uses the goal to overcome the sparse reward challenge. The superior performance demonstrates BPPO also considerably improve the policy performance based on (Filtered) BC on tasks with sparse reward.
7.2 THE SUPERIORITY OF BPPO OVER ONESTEP AND ITERATIVE VERSION
BPPO v.s. Onestep BPPO We choose to improve policy тı after it has been improved over behavior policy ftg because Theorem 2 provides no guarantee of optimality. Besides, BPPO and Onestep RL are easily to be connected because BPPO is based online method while Onestep RL solves offline RL without off-policy evaluation. Although Figure gives an intuitive interpretation to show the advantage of BPPO over its Onestep version, the soundness is relatively weak. We further analyze the superiority of BPPO over its Onestep version empirically.
80
Norroto Roars
5 e
8
BAPO
SPPO
130 Dkp (+9)
150
193
100 (lop
190
Gadderi Stop (43)
(a) hopper-medium--2
(b) hopper-medium--eelay--2
(c) hopper-medium-exxpertyy
Figure 3: The comparison between BPPO and Onestep BPPO. The hyperparameters of both methods tuned through the grid search, and then exhibit their learning curves with the best performance.
8

---

Published as a conference paper at ICLR 2023
In 3 observe that both BPPO and Onestep BPPO can outperform BC (the orange dotted line). This indicates both of them can achieve monotonic improvement over behavior policy fg. Another important result is that BPPO is consitently better than Onestep BPPO and this demonstrates two key points: First, improving Ty to fully utilize information necessary. Second, compared to strictly restricting the learned policy close the behavior policy, appropriate looseness is useful.
When approximating the advantage Ar., we have two implementation choices. One is advantage replacement (line 11 in Algorithm [T]. The other one is off off-policy
BPPO v.S. iterative BPPO
Q-estimation (line 13 in Algorithm [). corresponding to iterative BPPO. Both of them will introduce extra error compared true A The error of the former comes from replacement Axe < A while the latter comes from the off-policy estimation itself. We compare BPPO with iterative BPPO Figure and find that advantage replacement. namely BPPO, is obviously better.
(a) halfchhettah medium--rpplyg
walker2d mediam replamr
(c) halfcheelah mediumn expert
walker2d medium expert
The comparison between BPPO (the green curves) and its iterative versions in which we update the Q network to approximate 2T, instead of Q+y using in BPPO. In particular, we "BPPOoff=5" to denote that we update Q network for 5 gradient steps per policy training step.
Figure 4:
7.3 ABLATION STUDY OF DIFFERENT HYPERPARAMETERS
In this section, we evaluate the influence of clip ratio e and its decay rate a. Clip ratio restricts the policy close behavior policy and it directly solves the offline overestimation. Since also appears in PPO, we can set it properly to avoid catastrophic performance, which is the unique feature of BPPO. gradually tightens this restriction during policy improvement. We show how these coefficients contribute to the performance of BPPO and more ablations can be found in Appendix G and
8
00
(a) hopper-medium--eplay (b) hopper-medium-expert (c) hopper-mediumm-rpllyy (d) hopper-medium-exppet
Figure 5: Ablation study on clip ratio
(5() 5(b) and clip ratio decay a (5(c) 5(d).
Firstly, we analyze five values of the clip coefficient (0.05, 0.1, 0.2, 0.25, 0.3). In most envi- ronment, like hopper-mediun-expprr [5b) different shows no significant difference so we choose = 0.25, while only 0.1 is obviously better than others for cooper-mediuu--eplle demonstrate how the clip ratio decay (a = 0.90, 0.94, 0.96, 0.98, 1.00) affects the performance of BPPO. As shown in Figure 5(c) low decay rate (o = 0.90) or no decay (o 1.00) may cause crash during training. We use σ 0.96 to achieve stable policy improvement for all environments.
We then
CONCLUSION
Behavior Proximal Policy Optimization (BPPO) starts from offline monotonic policy improvement, using the loss function of PPO to elegantly solve offline RL without any extra constraint or reg- ularization introduced. Theoretical derivations and extensive experiments show that the inherent conservatism from the on-policy method PPO is naturally suitable to overcome overestimation in offline RI. BPPO is simple to implement and achieves superior performance the D4RI. dataset.

---

Published as a conference paper at ICLR 2023
9 ACKNOWLEDGEMENTS
This work was supported by the National Science and Technology Innovation 2030 -Mjor Project (Grant No. 20222D020880)), and NSFC General Program (Grant No. 62176215). We really appreciate Li He and Yachen Kang for helpful discussions and writing polishing.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline
reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021a.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021b.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202. 11566, 2022.
David Brandfoobrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933 -9466, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084-15097, 2021.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang. Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable wwrantage-weightee policy optimization for offline arXiv preprint arXiv:2203.0890 2022.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. arXiv preprint #rXiv:2202.02446 2022.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:21 2021.
Logan Engssrrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2020. URL httlittttp//ppeerettt
net foet/fo-ae Etrum 2d- rew.ddrrreeetrtt
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems,
2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 1wXX.v:907.0000 2019.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proc. 19th International Conference Machine Learning. Citeseer, 2002.
10

---

Published as a conference paper at ICLR 2023
Yachen Kang. Jinxin Liu, Xin Cao, and Donglin Wang. Off-dynamics inverse reinforcement learning from hetero-domain. arXiv preprint arXIv:2110.11443, 2021.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint
2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 1199-1191, 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint w.Xiv:2005.01643 2020.
Jinxin Liu, Hao Shen, Donglin Wang. Yachen Kang, and Qiangxing Tian. Unsupervised domain adap- tation with dynamics-aware rewards in reinforcement learning. Advances in Neural Information Processing Systems, 34:28784-28797 2021.
Jinxin Liu, Donglin Wang, Qiangxing Tian, and Zhengyu Chen. Learn goal-conditioned policy with intrinsic motivation for deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7558-7566, 2022a.
Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. arXiv preprint 2rXi::2003.06663 2022b.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. arXiv preprint vrXiv:1807.03856 2018
Wenjia Meng, Qian Zheng, Yue Shi, and Gang Pan. An off off-policy trust region policy optimiza- tion method with monotonic improvement guarantee for deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 33(5):2223-2233, 2021.
Piotr Mirowski, Matt Grimes, Mateusz. Malinowski, Karl Moritz Hermann, Keith Anderson, Denis Teplyashin, Karen Simonyan, Andrew Zisserman, Raia Hadsell, et al. Learning to navigate in cities without map. Advances in Neural Information Processing Systems, 31, 2018.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint
2019.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcchre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.00055 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perfone peshorerformacc deep learning library. Advances in neural information processing systems, 32, 2019.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 1910.00/77 2019.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
James Queeney, Yannis Paschalidis, and Christos G Cassandras. Generalized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems, 34:1 14:11 11919, 2021.
11

---

Published as a conference paper at ICLR 2023
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprini arXiv: 1506.020620000 506602238, 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Noah Y Siegel, Jost Tobias Springenberg. Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.0839% 2020.
Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning. 1998.
Qiangxing Tian, Guanchu Wang, Jinxin Liu, Donglin Wang, and Yachen Kang. Independent skill transfer for deep reinforcement learning. In Proceedings of the Twenty-Ninth International Conference International Joint Conferences Artificial Intelligence, pp. 2901-2907, 2021.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 23:7768-77777 2020.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint / 1361, 2019.
Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein adversarial imitation learning. arXiv preprint arXiv: www.. 08110, 2019.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. arXiv preprint arXiv:21 2021.
Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. arXiv preprint arXiv:2206.02880 2022.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey, ACM Computing Surveys (CSUR), 55(1):1-36, 2021a.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-basee policy optimization. Advances in neural information processing systems, 34:28954-28967 202
Siyuan Zhang and Nan Jiang. Towards hyperparameter- free policy selection for offline reinforcement learning. Advances in Neural Information Processing Systems, 34: 12864-1287) 2021.
12

---

Published as a conference paper at ICLR 2023
A PROOF OF PERFORMANCE DIFFERENCE THEOREM 1
Proof. First note that Az a) = ( (r(s,a) + +>V= (6) - V+())) Therefore,
T
> Ax (st, as)
Pp
T
=Ba~Pa'
7 (r (st, az) + vV2 (8141) Vz (st))
1=
=ByNPa/
- -v (su) + Σ 1=0
r'r (i)
Exg [V (8o)] + ETIPY
22rr(su,
(अ) + (IT)
JA (T)
(17)
Now the first equation in has been proved. For the proof of second equation, we decompose the expectation over the trajectory into the sum of expectation over state-action pairs:
T Σ 1m0
E
S Ar (s,, ar)
P (st s|(T) Bawn(( A A (s, a)]
t=0
P (sz s(m') Ea [A (s, (s)
P (B (11) [r (s,a)]
= = (11) [An as)
(18)
B PROOF OF PROPOSITION
Proof. For state-action pair (st, at) DD can be viewed as deterministic policy that satisfies
TD (a = ae(st) = 1 and TD (a # az (s) = 0. So
Drv (DI [st] Drv (TD ( [sc]
Ea bTD (als:) # (alsz)
2
/ (P(ar) TTD (ar (t) न (arls:)) + P (a + az) TT (alss) t8 (als:) da
- 2
1
/ (
Rの (a + a. arls+)] da
-
/
[P (1 - (1- + ( P (aa)) (1 - ズे (a. (e)ls))))) da
-
(1 #. (ac(8))))
(19)
2
13

---

Published as a conference paper at ICLR 2023
C
PROOF OF THEOREM 2
The definition of Ax (s) is as follows:
An, (s) (( [Ax, (s,a)]
(20)
Note that the expectation of advantage function As (s, a) depends on another policy и rather than n.. so Ax (s) # O. Furthermore, given the An,a (s), the performance difference in Theorem can be rewritten
JA (t)) Ey (111)))) [Ax (s, a)] Bxyppat)) [Ar,ta (s) JA (T,, ft) = Esppo (.aw))))))) [Aa, (s, a)] = Es-po()) [Ar (s)]
(21)
(22)
Lemma 1, For all state s.
A (s) - 2max ARa (s,a)l Drv (x|( [s]
(23)
Proof.
The expectation of advantage function A (8, a) over its policy a equals zero:
Bann [A=(s, a)] = Eawn [0x(8, a)) - V++ (8)] = Eawr (Qx(s, - V (*) =0
(24)
Thus, with the help of Hölder's incquality, we get
A (s) = [e-a((1s)) [Asa (s, a]] Eanag(()s)) [Aa (s.
| (als) - Ro (als)]l, || (s, al -2 (I| [s) max AA, (s, .4s
(25)
Lemma 2. ((Achiam et al.) 2017)) The divergence between two unnormalized visitation frequencies, | (:) Px' (+)ll1, is bounded by an average total variational divergence of the policies т and nt:
ll (.) Px (-)l 22
E smp2()) [Drv [=][=({{8)
(26)
Given this powerful lemma and other preparation, now we are able to derive the bound of JA +3) JA (Ir, #s)
JA (T, # JA (T, #t)) = Bomppr) [Ar,oo (+)] - BeNpo() [Ar (+)]
(BenpH() [Am (8)] Eenpe, ( AAr (0)]) + (Bomnap [Am (6)] Esupo() [Ar (0)]
(27)
Based on Hölder's inequality and lemma 2 we can bound the first term as follows:
(Bempe() [Am (s)] E Egnpey (-) [A (s)]
S Px (.) - ps (-) || (s)
<2" smppy(()
Drv ( ())) [s]] max An, (s)
(28)
For the second term, we can derive similar bound and furthermore let Drv (DI1#p) [s] (1 - na (a)(s))). Finally, using lemmal we get
Ja (T, Ja (IT, #t)
22 max Am (s)
E' snppag (-)
Drv (Λ [s]] +
E e-po(()
DrV (DIlag) [s]]
1
-27 max Am,to (6)
I amip (-)
Drv (T]
[s]]
E
1 t3 (a|s)]
(-)
1
<4> max Axx (8, a)) max Drv (s)
E (ouppn()) (.)
Drv (T||)T)
[s]] +
E
[1 Ao (als))
s-)p
(29)
14

---

Published as a conference paper at ICLR 2023
D PROOF OF THEOREM
As an extension of Theorem 2 the proof process of Theorem 3 similar. Based on the Equation
28 we can directly derive the final bound:
Ja (IT, T&) - JJ (T,, TT)
Bernn (11))((1)) [rna (s, [s) Bsmp((),aaa(aa)) [Axa (s, [s.a)]
- (Bompa() [Ar,TA (s)] - Berppee) (A (8)] (Boroae [A (s)] BoNpas ( [AT (5)1)
+ (Eamps [Axmm (s)] - Eewpo() [A (6)]
22 max An,zs (s)l
E empt (-)
Drv [s]]
+
E
Drv (Tal)*t) +
E [1 to (a|s)]
smppo (-)
s-)pp()
54 max AAr a)) max Drv (x [s]
3,a
E (+)
DrV (T]])T])
(Drv (Th (s]]
+
E
[1 Ro (a|s)]
)mpeg())
p-vpo(-)2
(30)
E WHY GAE IS UNAVAILABLE IN OFFLINE SETTING?
In traditional online situation, advantage An (s, a) is estimated by Generalized Advantage Estimation (GAE) (Schulman et al. 2015b) using the data collected by policy иæ But in offline RI, only offline
dataset D (st, ae, St+1, rt) from true behavior policy xg is available. The advantage of (8t, 0:) calculated by GAE as follow:
Axs ar) = (G2)) (rets yVas (s1+1+1) - Va (SL+i))
(31)
GAE can only calculate the advantage of (st, az) E D. For (st, āt) D. where ā is in- distribution action sampling but (st, āg) & D. GAE is unable to give any estimation. This is because the calculation process of GAE depends on the trajectory and does not have the ability to generalize to unseen state-action pairs. Therefore, GAE is not satisfactory choice for offline RI. Offline RL forbids the interaction with environment, data usage should be more efficient. Concretely, we expect advantage approximation method can not only calculate the advantage of (st,a,), but also (st, ā.). As result, we directly estimate advantage with the definition Ang (8, a) Qas (8, a) Vag (s). where Q-function is estimated by SARSA and value function by fitting returns MSE loss. This function approximation method can generalize to the advantage of (8t, a.).
Ei "1=07r(66 "r((s, a:) with
F THEORETICAL ANALYSIS FOR Advantage Replacement
We choose to replace all Ax, with trustworthy Ax then theoretically measure the difference rather than empirically make Ar, learned by Q-learning more accurate. The difference caused by replacing the Arg in JA (IT, TK) with Ang (s, a) can be measured the following theorem:
Theorem 4. Given the distance DTV (TT](TT) [s] and assume the reward function satisfies Ir (s,a)) S Peman for all 8, a, then
JA (TT, TK) (.. (rx (s. (s.a)] s 22 + 1) R E s
[Drv (TH (TA]|TT)) [s]]
(32)
15

---

Published as a conference paper at ICLR 2023
Proof. First note that Ar a) Bx (r(s,a) + (5") V=(s)]. Then we have
Pampp(-)aaati (Ar, (s, [s) Pespp())ava(()) [Aro (s,
R (o) Bee (p(ole)))) 7 (VTn (5') -V (5')) (Van (8) Vi (8))
(Ba (amplll VV (s') Vap (s') + V (s) Vap [s)
(33)
Similarly to Equation [8). the value function can be rewritten V (s) = Espppts [r (8)] Then the
difference between two value function can be measured using Hölder's inequality and lemma 2
VT (8) ViT. [s) = Eempacc) (r (s)] Ba-peges (s)]
< PPT (-) PTo (-)1 ||r (s)ll S 27 IR
Drv (T<|l(TT) [s]] max Ir (s)
(34)
smpag() (+)
Thus, the final bound
E Bs=pp(),az((11))
Am (s,
Exp ( [Ans (s, (s.a)]
<E <
27² s' sumpp (.)
E
[Drv (TT) [s']] max Ir (s')l
+
E
[DVv (TT|(TT)) [s]] max /r (s)
srupra (-)
max |r (s)
E ()
Drv (TH| [s]]
(35)
Note that the right end term of the equation is irrelevant the policy and can be viewed constant when optimizing т. Combining the result of Theorem 3 and 4 we get the following corollary:
Corollary 1. Given the distance Drv (m][=T) [s). Drv (FullFt) [s] and Drv (D) Rt)) [s]
(1 - fts (als)), we can derive the following bound:
JA (π,TH) TK) 2 Esspo(),antt [Amo (s, a)]
4y,,, max Drv (x (x]| [s]
E
[Drv [||| [s]]
Pap (-)
- -4 max Drv (x (T))
[s]
E
Drv (#x)( [s]]
pnppp (-)
2. max Drv (T|[TT) [s]
E supp(.)
[] म. (a|s)]
CRATT
(36)
where Am, max Ar (s, a)) and CrEITS 27 (7 1)
max |r (s, a)l E amppg (-)
DrV (TTllTTT) [s]]
8,a
Conclusion 3
To guarantee the true objective JA (TT, TK) nen-decra,,ing, we can also simultancously maxi- mize
[Ary (s, a)] and minimize [max, Drv (T][)T)) [s]],
1,2,,...
G ABLATION STUDY ON AN ASYMMETRIC COEFFICIENT
In this section, we give the details of all hyperparameter selections in our experiments. addition the aforementioned clip ratio and its clip decay coefficient a, introduce the E (0, as an asymmetric coefficient to adjust the advantage An based on the positive or negative of advantage:
A Iw 1(Are < 0) Ang-
(37)
16

---

Published as a conference paper at ICLR 2023
For w 0.5, that downweights the contributions of the state-action value Qr smaller than it's expectation, i.e. Va while distributing more weights to larger Qu,- The asymmetric coefficient can adjust the weight of advantage based the Q performance, which downweights the contributions of the state-action value Q smaller than its expectation while distributing more weights to advantage with larger value. We analyze how the three coefficients affect the performance of BPPO.
We analyze three values of the asymmetric coefficient w = (0.5, 0.9) in three Gym environments. Figure [ shows that 0.9 best for these tasks, especially in hepper-mediim--12 and hopper- mediuu-replaa--.. With a larger value w. the policy improvement can be guided in a better direction, leading better performance Gym environments. Based on the performance of different coefficient values above, we use the asymmetric advantage coefficient w = 0.9 for the Gym dataset training and 0.7 for the Adroit, Antmaze, and Kitchen datasets training, respectively.
103
100
Nomaliee Beur
Nomatre teer
Nomaliei
5
BG
120
153
100 150 Step
30
102
150
(XC5)
(a) hopper-medium-VV
(b) Sooper-medium--eplaay
(c) hopper eetprer-mmdium-xeee-----..
Figure 6: Ablation study on coefficient w. We optimize the hyperparameters through the grid search, then we fix the value of other coefficients with the best performance and change the value of the asymmetric coefficient to analyze how affects the BPPO. In particular, w = 0.5 denotes without the asymmetric coefficient during the training phase (contributing equal value to all Advantages).
H IMPORTANCE RATIO DURING TRAINING
In this section, we consider exploring whether the importance weight between the improved policy Th and the behavior policy πg will be arbitrarily large. To this end, we quantify this importance weight in the training phase in Figure 7 In F we often observe that the ratio of the BPPO with decay always stays in the clipped region (the region surrounded by the dotted yellow and red line).
However, the BPPO without decay is beyond the region in Figure 7(a)and) [(b) That demonstrates the improved policy without decay farther away from the behavior policy than the case of BPPO
with decay. It may cause unstable performance and even crashing, shown in Figure 5(c) 5(d and
10 when o = 1.00 (i.e., without decay).
SPPO uithout
ayponiness Rarb male
Incon mamoo Raonni
223
1.0
8.
1.0
koppsate
(a) hopper-medium-v2
(b) hopper pooper-mediim--eplayy
Sooper--edium-eexee
Figure 7: Visualization of the importance weight between the updated policy and the behavior policy trained by BC. When the performance of the policy is improved, we calculate the importance weight (i.e., the probability ratio) between the improved policy and the behavior policy.
17

---

Published as a conference paper at ICLR 2023
I COEFFICIENT PLOTS OF ONESTEP BPPO
In this section, we exhibit the learning curves and coefficient plots of Onestep BPPO, As shown in Figure 8 and 9 0.25 and w 0.9 are best for those tasks. Figure I0 shows how the clip coefficient decay affects the performance the Onestep BPPO. We can observe that the performance of the curve without decay with low decay is unstable over three tasks and even crash during training the shapper-mediun-replyy improvement for Onestep BPPO. that We use the coefficients with the best performance to compare with the BPPO in Figure
task. Thus, we select 0.96 achieve stable policy
Nomearee Rwiu
Nomatree S2ATT
Nomatoe
80
e+0.2
e0200
8
200.25
25
ex0.30
50
BC
103
100
150
50
103
(205)
(a) hopper-medium-v2
(b) hoppar-mcdiuu--eplay--22
(c) hopper-medium-epprrr
Figure Ablation study Onestep BPPO on coefficient e. We optimize the hyperparameters through the grid search, then we fix the value of other coefficients with the best performance and change the value the clip coefficient to analyze how affects the Onestep BPPO.
100
Komatzed Ronu
Nomatzed Reeum
Nomotzid
89
LP0 um12
15
Gradier
Gradinnt Slep
Grndeet
(a) hopper-medium-v2
(b) hooper--ediuu--eellay-22
(c) hloper-meddium-exper--
Figure 9: Ablation study of Onestep BPPO on coefficient w. We optimize the hyperparameters through the grid search, then we fix the value of other coefficients with the best performance and change the value of the asymmetric coefficient analyze how it affects the Onestep BPPO.
Nomaree Ream
Nomatee Donum Dosm
000
8
1-d 50
6
04234
0v0
18
5
2
Grodi ent Stos (x6)
Grackent 9koo (x5)
Gradiatt Skup (+1)
(a) hopper-medium--2
(b) hopper-mediun-eeplay-222
hopper--edduum-exxpee--
Figure 10: Ablation study of Onestep BPPO clip coefficient decay and its decay rate. We optimize the hyperparameters through the grid search, then we fix the value of other coefficients with the best performance and change the value of the clip decay coefficient analyze how it affects the Onestep BPPO. In particular, a = 1.00 denotes without the decay coefficient during the training phase.
18

---

Published as a conference paper at ICLR 2023
J EXTRA COMPARISONS
In this section, we have added the EDAC (An et al. 2021a), LAPO (Chen et al. 2022), RORL YYang et al., 2022), and ATAC (Cheng al. 2022) the comparison baselines to further evaluate the superionity of the BPPO. Although the performance of the BPPO slightly worse than the SOTA methods on Gym environment, the BPPO significantly outperforms all methods the Adroit, Kitchen, and Antmaze datasets and has the best overall performance over all datasets.
Table 3: The normalized results of all algorithms on Gym locomotion and Adroit datasets. The results of the EDAC, RORL, and ATAC are extracted from their original articles.
Environment/meethod
EDAC
RORL
ATAC
Ours
nalfcheetab-mediun--2
65.9
66.8
54.3
44.0.E0.2
hopper-medium-.
101.6
104,8
102.8
93.9#3.9
92.5
102.4
91.0
83.6
halfcheetah
61.3
61.9
49.5
41.0+0.6
hopper--eedium-replayaaa
101
102.8
102.8
92.5++3.4
87.1
90.4
94.1
77.6+7.8
malfihheetah-mmeciu---eeee
106.3
107.8
95.5
92.5#1.9
nooper--medium---xxee--2222222
110.7
112.7
112.6
112.811.7
wallee22dmmdiium-xxpoe
114.7
121.2
116.3
113.1+2.4
Cym locomotion-v2 total
841.1
870.8
818.9
751.0++21.8
pen-human-vl
52.1
33.7
79.3
117.8:11199
hammer-humang
0.8
2.3
6.7
14.943.2
door-human-vl
10.7
3.8
8.7
25.9
relocate-human-vl
0.1
0
0.3
2.2
pen-cloned-vl
68.2
35.7
73.9
110.8:16.3
hammer-cloned-vy
0.3
1.7
2.3
8.945..1
door-cloned-v1
9.6
-0.1
8.2
6,2
relocate-clonede adroit-vl total locomotion adroit total
0
0
0.8
1.9+
141.8
77.1
180.2
291.4++3..8
982.9
947.9
999.1
1042.4160.6
Table 4: The normalized results of all algorithms on Kitchen dataset. The results of the LAPO are extracted from its original article.
Environment//ettod
LAPO
Ours
fitchen-complete-ve
53.2
91.5.18.9
kitchen sacchenpartial-ver
53.7
57.0
ditchen-mixedo
62.4
62.5+66.7
kitchen-vO total
169.3
211.0+18.0
Table The normalized results of all algorithms on Antmaze dataset. The results of the RORL are extracted from its original article.
Environment/mettodd
RORL
Ours
Umaze-v2
96.7
95.0++5.55
1maze-diverse--2
90.7
91.7.14.1
Medium-play-ve
76.3
51.7+77.5
Medium-diven diverse-v2
69.3
70.0
Large- Large-play-v2
16.3
86.7-1822
Large- divers--v v2
41.0
88.3
Antmaze-v2 total
390.3
483.34335.7
19

---

Published as a conference paper at ICLR 2023
K IMPLEMENTATION AND EXPERIMENT DETAILS
Following the online PPO method, we use tricks called 'code-level optimization" including learning rate decay, orthogonal initialization, and normalization of the advantage in each mini-batch, which are considered very important to the success of the online PPO algorithm (Engstrom et al. 2020). We clip the concatenated gradient of all parameters such that the 'global L2 norm does exceed 0.5. We use 2 layers MLP with 1024 hidden units for the Q and policy networks, and use layers MLP with 512 hidden units for value function V. Our method is constructed by Pytorch (Paszke et al. 2019). Next, we introduce the training details of the Q, V. (estimated) behavior policy Tt and target policy n, respectively.
Q and V networks training: we run 2 x 10 steps for fitting value Q and V functions using learning rate 10 10-4, respectively.
(Estimated) behavior policy R training: we run 5 x 10at steps for a cloning using learning rate 10-4
Target policy π training: during policy improvement, we use the learning rate decay, i.c., decaying in each interval step in the first 200 gradient steps and then remaining the learning rate (decay rate σ = 0.96). We run 1,000 gradient steps for policy improvement for Gym, Adroit, and Kitchen tasks and run 100 gradient steps for Antmaze tasks. The selections of the initial policy learning rate, initial clip ratio, and asymmetric coefficient are listed in Table respectively.
Table 6: The selections of part of hyperparameters during policy improvement phase.
Hyperparameter
Task
Value
Gym locomotion and cloned tasks of Adroit
10 10-5 5
Initial policy learning rate
Kitchen, Antmaze, and human tasks of Adroit
Hopper-mediumm-eeppay--22
0.1
Initial clip ratio
Antmaze
0.5
Others
0.25
Gym locomotion
0.9
Asymmetric coefficient w
Others
0.7
20


Reviews:
[{"review_id": "zxVJNEQR3PU", "reviewer": "Reviewer_ksVK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, starting from the analysis of offline monotonic policy improvement, this paper gets a surprising finding that some online on-policy algorithms are naturally able to solve offline RL. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to accomplish the closeness. Based on this, this paper designs an algorithm called Behavior Proximal Policy Optimization (BPPO), which successfully solves offline RL without any extra constraint or regularization introduced. Extensive experiments on the D4RL benchmark indicate this extremely succinct method outperforms state-of-the-art offline RL algorithms.\n", "summary_of_the_review": "Please answer my question in above.", "strength_and_weaknesses": "Strength: This paper provides theoretical analysis for monotonic policy improvement of the BPPO. Empirically, BPPO works for the vast majority of D4RL dataset. \n\nWeakness: \n\n1. Assumption 1 seems a little strong. Assuming $\\hat{\\pi}_\\beta=\\pi_\\beta$ is not very reasonable since $\\hat{\\pi}_\\beta$ is learned from the data. Ideally, there should be an expression that bounds the difference between $\\pi$ and $\\hat{\\pi}$ using data.  \n\n2. equation (4), it is not reasonable to directly replace $\\rho_\\pi$ with state data distribution since the main challenge of offline is the data shift. As a result, there should be an importance ratio in front of it and should be estimated. Simply replacing the expression is not principled. \n\n3. While policy improvement guarantee is nice, it cannot be guaranteed to converge to the optimal policy. Can you further show that your BPPO converge to the optimal policy after $K$ iterations?\n\n4. Empirically, how will BPPO work for the sparse reward setting? one recent offline RL works all tackle the Antmaze environment as well. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rKYJGwx2vMg", "reviewer": "Reviewer_QTZM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an empirical finding regarding conservatism of on-policy online RL, and uses this insight to propose an offline RL algorithm, Behavior PPO that monotonically improves the performance over behavior policy in the same manner as PPO during online training. The main contribution is that the proposed BPPO algorithm is simple to implement, achieves results comparable to prior works like CQL and IQL on D4RL benchmark, and has nice theoretical properties of monotonic improvement during training. ", "summary_of_the_review": "The paper provides an interesting theoretical insight and develop a practical simple algorithm, which is great, but the results are not at all convincing regarding why the proposed algorithm should be favored over prior works like CQL, IQL that also have neat theoretical insights (conservatism) and perform as well as the proposed algorithm on the D4RL benchmark, and prior works like TD3+BC that are equally simple to implement. \n\n--- AFTER REBUTTAL RESPONSES ---\n\nThe authors have clarified some concerns regarding the experiments, and provided some missing results in the tables. As such I am updating my score to weak accept, but with low confidence, because the significance of the results are still unclear (minor improvements) and I am not convinced that the method is simpler to implement compared to the baselines (CQL, TD3+BC), and that the theoretical bounds on improvement are better than CQL. ", "strength_and_weaknesses": "Strengths\n\n- The proposed algorithm is simple to implement and is only a slight modification to existing algorithm PPO.\n- The insight of conservatism naturally emerging from online on-policy RL is neat, and interesting.\n- The experiments are on a standard benchmark which is good from the perspective of head-to-head comparison and reproducibility\n\nWeaknesses\n\n- Assumption 1 is likely to be violated in several practical settings if the value of $\\zeta$ is low. The experiments section doesnt provide any insight on the feasibility of this assumption while learning the behavior policy. Since experiments are all in simulation, it should be possible to analyze this.\n\n- The experimental insights are not very interesting. The results show that BPPO is either very slightly better or slightly worse in all the environments of D4RL , so it is unclear why this algorithm should be preferred in practice over existing baselines. Also, D4RL seems to be a very saturated benchmark, particularly the Gym environments, so it is unclear whether a gain of reward by 1 point on Walker is a meaningful difference. Results for several algorithms on several environments are not presented in Table 1 - it should be simple to run them and perform comparisons, right?\n\n- Offline RL depends on the quality of the offline dataset. There is no analysis showing how the proposed algorithm compares with baselines in different regimes of the optimality of offline data. I think this would be particularly important because the analysis depends on being able to recover the underlying behavior policy. \n\n- In order to show that the proposed algorithm leads to non-trivial gains or is otherwise interesting empirically, I think there needs to be results in a practical setting where BPPO is able to \"solve\" a task while prior approaches fail to solve it. Alternatively, any other comparison that shows why BPPO should be used by practitioners - I am not convinced that simplicity alone can be the reason given prior works like TD3+BC are equally if not more simple and achieve nearly the same performance on the D4RL benchmark. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "iE3Drd4u-KP", "reviewer": "Reviewer_7X1S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies monotonic policy improvement in offline RL, aiming at demonstrating the effectiveness of online monotonic policy improvement algorithm in solving offline RL. Following the vein of TRPO and PPO, this work proposes Behavior Proximal Policy Optimization (BPPO) by adjusting the policy improvement lower bound based on the states in offline buffer and the advantage estimates of behavior policy. The proposed algorithm is evaluated on the D4RL benchmark, consisting of the performance comparsion with several onestep offline RL and iterative offline RL algorithms and the analysis of two hyperparameters.", "summary_of_the_review": "According to my detailed review above, I think this paper is below the acceptance threshold (actually I would give a 4 if there was). \n\nThis is mainly due to the overlap between the proposed theories in this work and GePPO (and other off-policy TRPO/PPO advances), my concern on the soundness of the advantage approximation and the lack of important hyparameter analysis in the experiments.\n", "strength_and_weaknesses": "$\\textbf{Strengths:}$\n+ I appreciate the authors' efforts in study the monotonic policy improvement in offline RL. To my knowledge, it is novel.\n+ The paper is well written and the presentation of the proposed algorithm is very clear.\n+ The experiments are conducted from multiple aspects.\n\n&nbsp;\n\n$\\textbf{Weaknesses:}$\n\nAlthough I think the authors effort in studying monotonic policy improvement in offline RL is novel, I view it as a specific case of Generalized Off-policy Monotonic Policy Improvement. \n\nTwo representative prior works are:\n- [1] James Queeney, Yannis Paschalidis, Christos G. Cassandras. Generalized Proximal Policy Optimization with Sample Reuse. NeurIPS 2021\n- [2] Wenjia Meng, Qian Zheng, Yue Shi, Gang Pan. An Off-Policy Trust Region Policy Optimization Method With Monotonic Improvement Guarantee for Deep Reinforcement Learning. IEEE Trans. Neural Networks Learn. Syst. 33(5): 2223-2235 (2022)\n\n&nbsp;\n\n\nIn my view, Theorem 1 and 2 presented in this paper are specific forms of the Generalized Policy Improvement Lower Bound as presented by Theorem 1 in GePPO [1] (similar ones can also be found in [2]). Concretely, if we use the offline buffer for the off-policy state distribution considered in Theorem 1 in GePPO, we can derive very similar form to Theorem 1 and 2 (maybe only differ at the use of $\\xi$-coupled policy for expression).\n\nTherefore, the theoretical results are not new to me. I think it is very necessary to include the two works mentioned in the related work and maybe the methodology of this paper, as they are not mentioned in the “Monotonic Policy Improvement” paragraph.\n\n\n&nbsp;\n\n\nFor using the advantage estimates of the behavior policy to bypass the advantage estimation of iterative $\\pi_k$ is new to me. However, I worry about the soundness since the approximation makes the policy improvement lower bound even looser.\n\nThe significant performance difference between $\\omega=0.9$ and $\\omega=0.5$ of the asymmetric coefficient in Figure 4 may support my concern on the soundness of using $\\bar A_{\\hat{\\pi}_{\\beta}}$.\n\n&nbsp;\n\n\nTheorem 2 and 3 are based on Assumption 1. It seems that there is mismatch between the state-action support considered in Assumption 1 and in Lemma 1, 2, i.e., $\\xi$-coupled or $\\alpha$-coupled policy for the finite states in the offline buffer v.s. for all states. Is this mismatch a small issue in the derivation?\n\n&nbsp;\n\n\nBesides, what does $\\pi \\le \\pi_k$ condition at Line 7 in Algorithm 1 mean?\n\n\n&nbsp;\n\nFor the expeirments, in addition to the two hyperparameters investigated in Section 6.3, I think readers are also interested in how the iterative step number $K$ in Algorithm 1 and the decay coefficient/decay schedule are selected and designed, and how different choices of them influence the performance.\n\n\nI appreciate the empirical comparison in Section 6.2. Since Onestep BPPO is the variant let $K=0$ in Algorithm 1, one question here is, what if we update policy $\\pi$ by maximizing $\\hat{L}_{\\beta}(\\pi)$, i.e., Equation 10, for the same number of BPPO? I think some additional results for this will strengthen the comparison in Section 6.2.\n\n\nTo make the performance comparison in Section 6.1 more complete, I recommend the authors to consider EDAC [3], LAPO [4] and RORL [5] as SOTA baselines later.\n\n\n- [3] Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. NeurIPS 2021\n- [4] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, Chongjie Zhang. Latent-Variable Advantage-Weighted Policy Optimization for Offline RL. arXiv: 2203.08949 (2022)\n- [5] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, Lei Han:\nRORL: Robust Offline Reinforcement Learning via Conservative Smoothing. arXiv:2206.02829 (2022)\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AU5TEbf2WE", "reviewer": "Reviewer_qeAt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of learning in the offline RL scenario. While existing approaches need to regularize objectives in order to learn a policy in proximity to the one used to collect data, the authors explore the use of online on-policy algorithms in order to solve such tasks. Their focus is on the policy improvement theorem, from which notable algorithms like TRPO or PPO can be derived. They first extend such a theorem to the offline setting, deriving a lower bound on the true performance that depends on additional terms related to the offline dataset. Then they extend such theorem to consider an improvement over an arbitrary learned policy. Based on these results, they derive an iterative algorithm that optimizes the PPO objective, but with a change in Importance Sampling ratios, where the denominator is constantly replaced by the current learned policy. To prevent this optimization process to diverge, they introduce a clipping decay mechanism.\n\nExperiments show that this approach is effective in solving offline tasks. Moreover, an ablation on different hyperparameters demonstrates the robustness of the method.", "summary_of_the_review": "This paper introduces an offline version of PPO that can be used to solve offline tasks. It is based on an offline version of the policy improvement theorem. Although the experiments show improvement, there are a few concerns regarding the soundness of the proposed approach and the experimental choice before acceptance.", "strength_and_weaknesses": "Strengths:\n- The theoretical results seem correct and not trivial. The authors state the assumptions well before each theorem.\n- The problem is significant. In particular, recent work showed that applying on-policy algorithms to offline RL problems can lead to surprising results. This paper goes one step further in such a direction.\n- The experimental results show higher performance with respect to the baselines\n\nWeaknesses:\n- Some parts of the paper seem written in a hurry and are unclear. For example, the first 5 lines after Eq. 11 are confusing. Other parts of the paper seem too colloquial. For instance \"simply augmenting TD3 (...) with behavior cloning (...) reminds us to revisit\", or \"the most tough thing is the existence of A\".\n- There are claims that are not verified through experiments. In the abstract and in the introduction, it is stressed that existing off-policy actor-critic methods overestimate out-of-distribution actions. This motivates the use of a proximal objective. However, the paper is lacking an experiment showing that the proposed method does not suffer from this issue.\n- I have a few concerns regarding the soundness of the method. The sequential optimization of bound (8) would lead to policy improvement. However, the approximation in Eq 13 has no guarantees to improve upon the behavioral policy. The only guarantees are given by Theorem 4, which does not involve the true objective. Can the authors clarify this?\n- The comparison with Onestep BPPO, which consists of Behavioral Cloning + PPO, is not convincing. Did the authors take the tuned hyperparameters they found for their method and applied them to the Onestep method?\n- The paper is lacking an important ablation on the clip coefficient decay and the asymmetric coefficient for the advantage: How does the method perform without such tricks? How does it perform with respect to Onestep PPO, when both methods are without tricks? \n- I kind of disagree with the authors in the main claim of the paper, which is that they discover that an online, on-policy method can solve offline RL problems. What the paper proposed, is instead an off-policy version of PPO that works for offline RL. Please note that PPO can already be considered an off-policy algorithm, since at each iteration, after the first policy gradient step, it is learning about a policy that was not used to collect the data. The clipped importance weight helps keep the optimization to be near-on-policy. BPPO further relaxes this constraint, letting the learned policy be able to be far from the behavioral policy. \n\nOther questions:\n- The experiments involve continuous action spaces. How does this affect Assumption 1? I expect the value of $\\xi$ to be 1.\n- What is the value of the bounds proposed during training? What is the magnitude of each component in the bound during the experiments?\n- At each iteration, the importance weight between the current policy and the previous one is clipped. Can the authors quantify instead the importance weight between the final learned policy and the behavioral policy? Can this value be arbitrarily large?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-wCSlmSBIKx", "reviewer": "Reviewer_t4J4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper finds that some online on-policy algorithms are naturally able to solve offline RL by theoretical analysis. PPO method is extended to the offline setting by an additional model of advantage. Experiments show that the proposed method achieves good performance without expensive online interaction.", "summary_of_the_review": "The paper is well-written and meaningful. The finding is interesting.  I would like to ask some questions about the method:\n1. Since a model is introduced to estimate the advantage, are there any experiments to show the accuracy of the model?\n2. For Eq. 13, the loss of \\pi_k is only related to the behavior policy and \\pi_k itself. In other words, \\pi_k is not improved based on \\pi_k-1. Then, why does the algorithm need K steps to get the best policy? \n3. For Algorithm 1, what is the meaning of \\pi <= \\pi_k in line 7? is it measured by the return of the policy?", "strength_and_weaknesses": "interesting finding, good theoretical analysis, and sound experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster