Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addresses the problem of learning in the offline RL scenario. While existing approaches need to regularize objectives in order to learn a policy in proximity to the one used to collect data, the authors explore the use of online on-policy algorithms in order to solve such tasks. Their focus is on the policy improvement theorem, from which notable algorithms like TRPO or PPO can be derived. They first extend such a theorem to the offline setting, deriving a lower bound on the true performance that depends on additional terms related to the offline dataset. Then they extend such theorem to consider an improvement over an arbitrary learned policy. Based on these results, they derive an iterative algorithm that optimizes the PPO objective, but with a change in Importance Sampling ratios, where the denominator is constantly replaced by the current learned policy. To prevent this optimization process to diverge, they introduce a clipping decay mechanism.

Experiments show that this approach is effective in solving offline tasks. Moreover, an ablation on different hyperparameters demonstrates the robustness of the method.
strength_and_weaknesses: Strengths:
- The theoretical results seem correct and not trivial. The authors state the assumptions well before each theorem.
- The problem is significant. In particular, recent work showed that applying on-policy algorithms to offline RL problems can lead to surprising results. This paper goes one step further in such a direction.
- The experimental results show higher performance with respect to the baselines

Weaknesses:
- Some parts of the paper seem written in a hurry and are unclear. For example, the first 5 lines after Eq. 11 are confusing. Other parts of the paper seem too colloquial. For instance "simply augmenting TD3 (...) with behavior cloning (...) reminds us to revisit", or "the most tough thing is the existence of A".
- There are claims that are not verified through experiments. In the abstract and in the introduction, it is stressed that existing off-policy actor-critic methods overestimate out-of-distribution actions. This motivates the use of a proximal objective. However, the paper is lacking an experiment showing that the proposed method does not suffer from this issue.
- I have a few concerns regarding the soundness of the method. The sequential optimization of bound (8) would lead to policy improvement. However, the approximation in Eq 13 has no guarantees to improve upon the behavioral policy. The only guarantees are given by Theorem 4, which does not involve the true objective. Can the authors clarify this?
- The comparison with Onestep BPPO, which consists of Behavioral Cloning + PPO, is not convincing. Did the authors take the tuned hyperparameters they found for their method and applied them to the Onestep method?
- The paper is lacking an important ablation on the clip coefficient decay and the asymmetric coefficient for the advantage: How does the method perform without such tricks? How does it perform with respect to Onestep PPO, when both methods are without tricks? 
- I kind of disagree with the authors in the main claim of the paper, which is that they discover that an online, on-policy method can solve offline RL problems. What the paper proposed, is instead an off-policy version of PPO that works for offline RL. Please note that PPO can already be considered an off-policy algorithm, since at each iteration, after the first policy gradient step, it is learning about a policy that was not used to collect the data. The clipped importance weight helps keep the optimization to be near-on-policy. BPPO further relaxes this constraint, letting the learned policy be able to be far from the behavioral policy. 

Other questions:
- The experiments involve continuous action spaces. How does this affect Assumption 1? I expect the value of $\xi$ to be 1.
- What is the value of the bounds proposed during training? What is the magnitude of each component in the bound during the experiments?
- At each iteration, the importance weight between the current policy and the previous one is clipped. Can the authors quantify instead the importance weight between the final learned policy and the behavioral policy? Can this value be arbitrarily large?
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, apart from some sections that need to be further polished (see weaknesses). The theory seems of high quality, but the practical approximations make the proposed algorithm less sound. Further experiments are needed to improve the quality of the paper. To my knowledge, the results are novel. The authors specify the hyperparameters used in their implementation.
summary_of_the_review: This paper introduces an offline version of PPO that can be used to solve offline tasks. It is based on an offline version of the policy improvement theorem. Although the experiments show improvement, there are a few concerns regarding the soundness of the proposed approach and the experimental choice before acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies monotonic policy improvement in offline RL, aiming at demonstrating the effectiveness of online monotonic policy improvement algorithm in solving offline RL. Following the vein of TRPO and PPO, this work proposes Behavior Proximal Policy Optimization (BPPO) by adjusting the policy improvement lower bound based on the states in offline buffer and the advantage estimates of behavior policy. The proposed algorithm is evaluated on the D4RL benchmark, consisting of the performance comparsion with several onestep offline RL and iterative offline RL algorithms and the analysis of two hyperparameters.
strength_and_weaknesses: $\textbf{Strengths:}$
+ I appreciate the authors' efforts in study the monotonic policy improvement in offline RL. To my knowledge, it is novel.
+ The paper is well written and the presentation of the proposed algorithm is very clear.
+ The experiments are conducted from multiple aspects.

&nbsp;

$\textbf{Weaknesses:}$

Although I think the authors effort in studying monotonic policy improvement in offline RL is novel, I view it as a specific case of Generalized Off-policy Monotonic Policy Improvement. 

Two representative prior works are:
- [1] James Queeney, Yannis Paschalidis, Christos G. Cassandras. Generalized Proximal Policy Optimization with Sample Reuse. NeurIPS 2021
- [2] Wenjia Meng, Qian Zheng, Yue Shi, Gang Pan. An Off-Policy Trust Region Policy Optimization Method With Monotonic Improvement Guarantee for Deep Reinforcement Learning. IEEE Trans. Neural Networks Learn. Syst. 33(5): 2223-2235 (2022)

&nbsp;


In my view, Theorem 1 and 2 presented in this paper are specific forms of the Generalized Policy Improvement Lower Bound as presented by Theorem 1 in GePPO [1] (similar ones can also be found in [2]). Concretely, if we use the offline buffer for the off-policy state distribution considered in Theorem 1 in GePPO, we can derive very similar form to Theorem 1 and 2 (maybe only differ at the use of $\xi$-coupled policy for expression).

Therefore, the theoretical results are not new to me. I think it is very necessary to include the two works mentioned in the related work and maybe the methodology of this paper, as they are not mentioned in the “Monotonic Policy Improvement” paragraph.


&nbsp;


For using the advantage estimates of the behavior policy to bypass the advantage estimation of iterative $\pi_k$ is new to me. However, I worry about the soundness since the approximation makes the policy improvement lower bound even looser.

The significant performance difference between $\omega=0.9$ and $\omega=0.5$ of the asymmetric coefficient in Figure 4 may support my concern on the soundness of using $\bar A_{\hat{\pi}_{\beta}}$.

&nbsp;


Theorem 2 and 3 are based on Assumption 1. It seems that there is mismatch between the state-action support considered in Assumption 1 and in Lemma 1, 2, i.e., $\xi$-coupled or $\alpha$-coupled policy for the finite states in the offline buffer v.s. for all states. Is this mismatch a small issue in the derivation?

&nbsp;


Besides, what does $\pi \le \pi_k$ condition at Line 7 in Algorithm 1 mean?


&nbsp;

For the expeirments, in addition to the two hyperparameters investigated in Section 6.3, I think readers are also interested in how the iterative step number $K$ in Algorithm 1 and the decay coefficient/decay schedule are selected and designed, and how different choices of them influence the performance.


I appreciate the empirical comparison in Section 6.2. Since Onestep BPPO is the variant let $K=0$ in Algorithm 1, one question here is, what if we update policy $\pi$ by maximizing $\hat{L}_{\beta}(\pi)$, i.e., Equation 10, for the same number of BPPO? I think some additional results for this will strengthen the comparison in Section 6.2.


To make the performance comparison in Section 6.1 more complete, I recommend the authors to consider EDAC [3], LAPO [4] and RORL [5] as SOTA baselines later.


- [3] Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble. NeurIPS 2021
- [4] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, Chongjie Zhang. Latent-Variable Advantage-Weighted Policy Optimization for Offline RL. arXiv: 2203.08949 (2022)
- [5] Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, Lei Han:
RORL: Robust Offline Reinforcement Learning via Conservative Smoothing. arXiv:2206.02829 (2022)




clarity,_quality,_novelty_and_reproducibility: $\textbf{Clarity: }$

The paper is well written and the presentation of the proposed algorithm is very clear.

$\textbf{Novelty: }$

To my knowledge, most part of the theoretical results and algorithm designs are not new to me, since I view them as a concrete form in offline RL which can be easily derived from GePPO (i.e., when the offline buffer is considered as the off-policy data). Using the advantage estimates of behavior policy to bypass the need of estimating the advantage of current policies is novel. However, I have a major concern on the soundness of this replacement.

$\textbf{Quality: }$

Assumption 1 is difficult to achieve in practice to my knowledge. Moreover, Theorem 2 may have a small issue in derivation due to a mismatch between the required condition and Assumption 1. The proposed algorithm is estabilished on two approximation choices which can be very loose in the policy improvement bound. For the experiments,


$\textbf{Reproductibility: }$

The proposed algorithm is clear and it seems to be easy to implement. However, the source codes are not provided.

summary_of_the_review: According to my detailed review above, I think this paper is below the acceptance threshold (actually I would give a 4 if there was). 

This is mainly due to the overlap between the proposed theories in this work and GePPO (and other off-policy TRPO/PPO advances), my concern on the soundness of the advantage approximation and the lack of important hyparameter analysis in the experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper finds that some online on-policy algorithms are naturally able to solve offline RL by theoretical analysis. PPO method is extended to the offline setting by an additional model of advantage. Experiments show that the proposed method achieves good performance without expensive online interaction.
strength_and_weaknesses: interesting finding, good theoretical analysis, and sound experiments.
clarity,_quality,_novelty_and_reproducibility: good except that the code is not provided.
summary_of_the_review: The paper is well-written and meaningful. The finding is interesting.  I would like to ask some questions about the method:
1. Since a model is introduced to estimate the advantage, are there any experiments to show the accuracy of the model?
2. For Eq. 13, the loss of \pi_k is only related to the behavior policy and \pi_k itself. In other words, \pi_k is not improved based on \pi_k-1. Then, why does the algorithm need K steps to get the best policy? 
3. For Algorithm 1, what is the meaning of \pi <= \pi_k in line 7? is it measured by the return of the policy?
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes an empirical finding regarding conservatism of on-policy online RL, and uses this insight to propose an offline RL algorithm, Behavior PPO that monotonically improves the performance over behavior policy in the same manner as PPO during online training. The main contribution is that the proposed BPPO algorithm is simple to implement, achieves results comparable to prior works like CQL and IQL on D4RL benchmark, and has nice theoretical properties of monotonic improvement during training. 
strength_and_weaknesses: Strengths

- The proposed algorithm is simple to implement and is only a slight modification to existing algorithm PPO.
- The insight of conservatism naturally emerging from online on-policy RL is neat, and interesting.
- The experiments are on a standard benchmark which is good from the perspective of head-to-head comparison and reproducibility

Weaknesses

- Assumption 1 is likely to be violated in several practical settings if the value of $\zeta$ is low. The experiments section doesnt provide any insight on the feasibility of this assumption while learning the behavior policy. Since experiments are all in simulation, it should be possible to analyze this.

- The experimental insights are not very interesting. The results show that BPPO is either very slightly better or slightly worse in all the environments of D4RL , so it is unclear why this algorithm should be preferred in practice over existing baselines. Also, D4RL seems to be a very saturated benchmark, particularly the Gym environments, so it is unclear whether a gain of reward by 1 point on Walker is a meaningful difference. Results for several algorithms on several environments are not presented in Table 1 - it should be simple to run them and perform comparisons, right?

- Offline RL depends on the quality of the offline dataset. There is no analysis showing how the proposed algorithm compares with baselines in different regimes of the optimality of offline data. I think this would be particularly important because the analysis depends on being able to recover the underlying behavior policy. 

- In order to show that the proposed algorithm leads to non-trivial gains or is otherwise interesting empirically, I think there needs to be results in a practical setting where BPPO is able to "solve" a task while prior approaches fail to solve it. Alternatively, any other comparison that shows why BPPO should be used by practitioners - I am not convinced that simplicity alone can be the reason given prior works like TD3+BC are equally if not more simple and achieve nearly the same performance on the D4RL benchmark. 
clarity,_quality,_novelty_and_reproducibility: The paper is clear to read, easy to follow and understand. The work is reasonably high quality combining an interesting theoretical insight with practical execution through a simple algorithm. The work is not necessarily novel, as prior approaches like Onestep RL are very similar in principle to the proposed algorithm. I haven't carefully looked at the proofs of the theorems.
summary_of_the_review: The paper provides an interesting theoretical insight and develop a practical simple algorithm, which is great, but the results are not at all convincing regarding why the proposed algorithm should be favored over prior works like CQL, IQL that also have neat theoretical insights (conservatism) and perform as well as the proposed algorithm on the D4RL benchmark, and prior works like TD3+BC that are equally simple to implement. 

--- AFTER REBUTTAL RESPONSES ---

The authors have clarified some concerns regarding the experiments, and provided some missing results in the tables. As such I am updating my score to weak accept, but with low confidence, because the significance of the results are still unclear (minor improvements) and I am not convinced that the method is simpler to implement compared to the baselines (CQL, TD3+BC), and that the theoretical bounds on improvement are better than CQL.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, starting from the analysis of offline monotonic policy improvement, this paper gets a surprising finding that some online on-policy algorithms are naturally able to solve offline RL. Specifically, the inherent conservatism of these on-policy algorithms is exactly what the offline RL method needs to accomplish the closeness. Based on this, this paper designs an algorithm called Behavior Proximal Policy Optimization (BPPO), which successfully solves offline RL without any extra constraint or regularization introduced. Extensive experiments on the D4RL benchmark indicate this extremely succinct method outperforms state-of-the-art offline RL algorithms.

strength_and_weaknesses: Strength: This paper provides theoretical analysis for monotonic policy improvement of the BPPO. Empirically, BPPO works for the vast majority of D4RL dataset. 

Weakness: 

1. Assumption 1 seems a little strong. Assuming $\hat{\pi}_\beta=\pi_\beta$ is not very reasonable since $\hat{\pi}_\beta$ is learned from the data. Ideally, there should be an expression that bounds the difference between $\pi$ and $\hat{\pi}$ using data.  

2. equation (4), it is not reasonable to directly replace $\rho_\pi$ with state data distribution since the main challenge of offline is the data shift. As a result, there should be an importance ratio in front of it and should be estimated. Simply replacing the expression is not principled. 

3. While policy improvement guarantee is nice, it cannot be guaranteed to converge to the optimal policy. Can you further show that your BPPO converge to the optimal policy after $K$ iterations?

4. Empirically, how will BPPO work for the sparse reward setting? one recent offline RL works all tackle the Antmaze environment as well. 

clarity,_quality,_novelty_and_reproducibility: Overall speaking, the writing the paper is clear.
summary_of_the_review: Please answer my question in above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper is based on the observation that online on-policy algorithms are naturally able to solve offline RL problems. It provides a theoretical explanation and extensive experimental results. While reviewers have brought up different concerns, the author's have provided detailed responses with additional experimental results. I recommend accepting this paper for two main reasons: 1) the authors seem to have addressed the majority of concerns in their response, 2) the fact that "online on-policy algorithms are naturally able to solve offline RL problems" is a finding and insight that is likely of interest to the community, even if reviewers may feel that the method is not strongly superior in performance to more standard offline RL algorithms. For the latter, this is even likelier, given that the paper appears to be very well-written.
justification_for_why_not_higher_score: The paper does not show performance that is much better than different approaches. It is more of interest to the community because it shows that online on-policy algorithms seem to be naturally able to solve offline RL problems.
justification_for_why_not_lower_score: The results and findings are likely of interest to the community.
Decision: Accept: poster