HP3O: Hybrid-Policy Proximal Policy Optimization with Best Trajectory

Authors: Qisai Liu, Zhanhong Jiang, Hsin-Jung Yang, Mahsa Khosravi, Joshua Russell Waite, Soumik Sarkar
Primary area: reinforcement learning
Keywords: Reinforcement learning, proximal policy optimization, hybrid policy, trajectory replay buffer, variance reduction

Abstract:
Proximal policy optimization (PPO) is one of the most popular state-of-the-art on-policy algorithms that has become a standard baseline in modern reinforcement learning with applications in numerous fields. Though it delivers stable performance with theoretical policy improvement guarantees, high variance and high sample complexity still remain critical challenges in on-policy algorithms. To alleviate these issues, we propose Hybrid-Policy Proximal Policy Optimization (HP3O), which utilizes a trajectory replay buffer to make efficient use of trajectories generated by recent policies. Particularly, the buffer applies the "first in, first out" (FIFO) strategy so as to keep only the recent trajectories to attenuate the data distribution drift. A batch consisting of the trajectory with the best return and other randomly sampled ones from the buffer is used for updating the policy networks. The strategy helps the agent to improve its capability on top of the most recent best performance and in turn reduce variance empirically. We theoretically construct the policy improvement guarantees for the proposed algorithm. HP3O is validated and compared against several baseline algorithms using multiple continuous control environments. Our code is available here.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729579012981

thread_type:
official_review

<thread_object>
creator: reviewer_ye97

<content>
summary:
This paper proposes a new hybrid-policy RL method called HP3O for continuous control tasks, which employs a trajectory replay buffer to reduce variance  and improve sample efficiency. HP3O is simple but effective with theoretical guarantees on policy improvement. The experiments on Mujoco control tasks demonstrate its effectiveness empirically. This paper is well written and the method is clearly described.

strengths:
1. The new method HP3O(+) is simple and easy to implement. The effectiveness of HP3O policy improvement is analyzed theoretically in detail. 
2. The PPO is widely used in applications. Thus, this method can be utilized to improve the performance of downstream tasks such as robotics control tasks.
3. The paper is well-written with clear descriptions of the method and the experiments.

weaknesses:
The main concerns lie in the experiments:

1. More experiment results with (more) baseline methods are expected. In this paper, authors compare HP3O(+) with PPO, A2C, P3O, and SAC. However, some results are not shown in the current version: 

   1. The learning curves of SAC & HP3O+ in Fig. 2 are absent. It is recommended that Fig. 2 and Fig. 3 can be merged, with more results of HP3O+ in more experimental tasks.

   2. Similarly, HP3O+ in absent in Fig. 4(a), which is important to demonstrate the performance of HP3O+ on variance.

   3. More baseline methods are expected, such as on-policy methods constructed based on PPO. Besides, comparisons with off-policy methods such as SAC are also meaningful. As illustrated in the paper, HP3O achieves superior sample efficiency compared to SAC, which is normal and acceptable.
2. This paper claims that SAC achieves higher sample efficiency at the cost of runtime complexity. Fig. 4(b) gives the training times of methods in *HalfCheetah* with 1M steps. However, it is more reasonable to give the training times of each method with *same episode returns*, which is more meaningful in practical applications. Besides, more results in other tasks are expected to demonstrate this claim. It is recommended that the curve of episode return - training time is clearer than Fig. 4(b).

3. Some latex errors \ref in the manuscripts.

questions:
1. Does the experiment utilize parallel gymnasium environment settings, or is each method trained with single environment? In practical applications, parallel environments are widely used such as ISAAC simulator with thousands  of robots trained simultaneously, which reduce training times with higher computation costs. Does this method still achieve similar higher performance than other methods? 
2. In this paper, the author claims that ``A notable disadvantage of SAC is that it only works with continuous action spaces. " However, SAC can be utilized in discrete action spaces [1]. In this paper, HP3O(+) is implemented in continuous action spaces. Is this method effective in discrete action space?
3. The experiments are conducted in several Mujoco continuous control tasks. Is this method effective in locomotion tasks with higher action spaces, such as Humanoid-v2 task? 

[1] Christodoulou P. Soft actor-critic for discrete action settings[J]. arXiv preprint arXiv:1910.07207, 2019.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696883445

thread_type:
official_review

<thread_object>
creator: reviewer_iv5k

<content>
summary:
This paper introduces Hybrid-Policy PPO (HP3O), a variant of Proximal Policy Optimization (PPO) designed to improve sample efficiency and reduce high variance. HP3O integrates on-policy and off-policy learning through a replay buffer that stores recent trajectories, using a “first in, first out” (FIFO) strategy to retain only the most recent experiences. By incorporating the trajectory with the highest return along with randomly sampled trajectories from the buffer for policy updates, HP3O empirically reduces variance while preserving a policy improvement guarantee. The authors provide rigorous proofs for the policy improvement lower bounds of both HP3O and its variant, HP3O+. Empirical evaluations across continuous control environments demonstrate the effectiveness of HP3O compared to other baselines.

strengths:
1. The hybrid-policy approach of HP3O is an innovative extension of PPO, blending on-policy stability with off-policy sample efficiency by storing recent trajectories in a replay buffer and selectively sampling high-reward experiences.
2. The paper is rigorous both in theory and empirical validation. It presents a clear theoretical framework, including policy improvement bounds that account for the hybrid on-policy/off-policy nature of HP3O.
3. The paper is well-structured and provides detailed explanations of key concepts.

weaknesses:
1. While the paper incorporates the replay buffer concept from off-policy reinforcement learning into PPO, the scope of this structure modification shows limited originality.
2. The training curves in Figures 1 and 2 do not appear to have fully converged, making it difficult to assess the ultimate performance that each model could achieve.
3. The impact of the replay buffer size has not been discussed theoretically or evaluated empirically, leaving an important aspect of the method unexplored.

questions:
1. In Algorithm 1 Line 16, should $G_t−V_\phi(s_t)$ be replaced by $\hat{A}_t^{\pi_k}$ to be consistent with HP3O+?
2. Could you provide the final, converged reward for each algorithm in the experiments to better demonstrate their performance?
3. Could you include a discussion on the impact of replay buffer size, either through theoretical analysis or additional simulation results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706114023

thread_type:
official_review

<thread_object>
creator: reviewer_zrpn

<content>
summary:
This paper proposes Hybrid-Policy Proximal Policy Optimization (HP3O) and a variant HP3O+, which incorporate a replay buffer with standard PPO, to improve the sample efficiency while mitigating the high variance common in off-policy algorithms. They derived the policy improvement lower bound for both HP3O and HP3O+. Experiments conducted on continuous control environments demonstrate that the proposed algorithms perform comparably or better than baselines, including PPO, A2C [Peng et al. 2018] and P3O [Chen et al. 2023].

strengths:
- the proposed approach combines the strengths of on-policy and off-policy methods
- offers theoretical insights
- validated the efficacy in continuous control experiments

weaknesses:
1. The baselines in the experiments are limited: six baseline methods are listed in Table 1 as related works but only one of them is used as a baseline in the experiments. The paper explains that other methods were excluded because “corresponding code bases are either inaccessible or problematic”. However,  at least three of these methods have publicly available codebases, including the most relevant work GePPO.  It would be beneficial to clarify the specific issues that prevents their inclusion:

  - GePPO (official): https://github.com/jqueeney/geppo
  - Policy-on-off PPO (official): https://github.com/rasoolfa/P3O
  - Off-policy PPO (unofficial): https://github.com/Ladun/OffPolicy-PPO

    Given that GePPO and Off-policy PPO are both hybrid methods with theoretical guarantees,  comparing with them would better illustrate the benefits of using a trajectory buffer.

2. Since HP3O+ is a proposed algorithm, its learning curve should ideally be included in Fig. 2. It would strengthen the paper if the authors add the results or elaborate why they’re unnecessary.

3. It seems that the learning curves only report performance of the training episodes. If so, I think that it’s better to report the evaluation performance instead, as is done in SAC.

4. In the limitation section, it is mentioned that dequeuing replay buffer may cause issues in sparse reward setting. On a related note, conducting ablation studies on the buffer size in general settings would provide insights into its impact on performance and stability.

Minor:
- For future work, it would be useful to explore the performance of HP3O in discrete action spaces, such as Atari, as done in PPO.
- Typo on line 282, “the the”

questions:
- If the best trajectory is not added to the replay buffer, which part of the theoretical results no longer holds?
- In Theorem 2, there’s a factor of 2 in the last two terms on the right-hand side. Should these factors be canceled out instead?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731560010282

thread_type:
official_review

<thread_object>
creator: reviewer_xcpf

<content>
summary:
This paper focuses on the issues of high variance and high sample complexity in proximal policy optimization (PPO) algorithm. To mitigate the issues, an approach of Hybrid-Policy Proximal Policy Optimization (HP3O) is proposed by utilizing a trajectory replay buffer with a "first in, first out" (FIFO) strategy. Further, another variant termed HP3O+ is proposed by leveraging a new baseline to enhance the model performance. Theoretical analysis is also provided to show the policy improvement lower bounds for the proposed algorithms. Finally, experiments were conducted on continuous control environments from Gymnasium (Walker, HalfCheetah, Lunarlander, Swimmer, Hopper, and InvertedPendulum), comparing HP3O/HP3O+ with PPO, A2C, and P3O.

strengths:
- High variance in on-policy policy gradient methods, such as PPO, remains a challenge. This work aims to mitigate this issue by proposing a trajectory replay buffer with FIFO stategy so that both on-policy and off-policy experience can be utilized to balance the trade-off between sample efficiency and training stability.  
- Both theoretical analysis and empirical evaluation on continuous control environments are provided to show its effectiveness.

weaknesses:
- The contribution of this paper looks a little bit incremental to me. The idea of using FIFO strategy for replay buffer is quite straightforward, this is OK, but I don't see why and how this will make a big difference. It's more interesting to compare and analyze FIFO with other strategies, such as prioritized replay buffer, and show why FIFO could work well or much better than the other ones. From the theoretical perspective, what's the improvement of sample complexity over other policy gradient methods? From the empirical evaluation, the current results are insufficient to support the claim. Although the paper acknowledged that fully off-policy method, such as SAC, could work better, it's still necessary to compare the proposed methods with both the state-of-the-art methods and off-policy methods (TD3, SAC, etc.). Currently, HP3O showed the advantage over PPO, A2C, P3O on some domains, but why it can make success and why it fails, it's not clear yet. For the ablation study, it's more expected to see the proposed method with or without FIFO replay buffer and best trajectory.
- There are some missing cross-references, such as "Appendix ?? and ??".

questions:
- See my comments in the weaknesses.
- How to define or measure the best trajectory in the replay buffer?
- How to implement the proposed replay buffer (with FIFO) but withoug padding? How much memory it would consume?
- Why the variance of HalfCheetach becomes larger when training goes longer?
- How do you calculate the explained variance? what are the inputs and groundtruth?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732688759602

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the updated revision:

comment:
We greatly appreciate all the insightful comments and feedback from the reviewers and would like to provide specific details on the revisions we have conducted based on the comments and suggestions from all reviewers, given the rebuttal period. Please see the following list of precise modifications in the revised draft.

$• $  Added Remark 5: We included a new remark to explain why we conducted sample efficiency analysis rather than exact sample complexity analysis, strengthening the theoretical foundation of our proposed algorithms. For more details, please refer to section A.4 in the Appendix. Additionally, we added a theoretical comparison between HP3O and HP3O+ in section A.5 to highlight how HP3O+ improves upon HP3O in terms of the policy improvement lower bound.

$• $  Expanded Comparative Evaluation: We added baseline methods (GEPPO and OffPolicy) and HP3O+ to the comparative evaluation in section 6.1. This addresses the major concern about the lack of sufficient baseline comparisons. Specifically, we implemented GEPPO, OffPolicy PPO (abbreviated as OffPolicy), and HP3O+ for all environments and incorporated the results in Figure 2. The discussion in section 6.1 has also been updated to reflect these additions. We included baseline methods (GEPPO and OffPolicy) and HP3O+ in the ablation studies in section 6.2. This revision is to extend the variance and time complexity plots to include more baselines and HP3O+, both Figure 3(a) and Figure 3(b). We also adjusted the discussion in section 6.2.

$• $  Enhanced Evaluation on Cartpole and Humanoid: We have added baselines methods (GEPPO and OffPolicy) and HP3O+ in Figure 5 for the Cartpole environment and the results for the Humanoid task in section A.6. We updated our evaluation on the Cartpole environment by adding GEPPO and OffPolicy results in Figure 5 in section A.6. Additionally, the result for the Humanoid task with different methods is also incorporated in this section. However, obtaining the result for the Humanoid task is quite time-consuming. Given the limited time within the rebuttal period, we could only obtain results for HP3O, SAC, A2C, and PPO, which still give us a preliminary evaluation of this complex continuous environment. For the rest of the methods, we will include them in the final version of this paper.

$• $ Converged Reward Table: Added converged reward table in section A.12. The converged performance of each algorithm is critical to evaluate them. Therefore, to address the comments from the reviewer and based on their suggestions, we have added a table (Table 2 in section A.12) to include the mean and standard deviation of the episodic reward at or close to the converged stage.

$• $ Episodic Return-Time Cost Analysis: Added figures in section A.13 to show the “episodic return-time cost of diverse methods. To show the precise wall-clock time of each method reaching a certain episodic return, we have added the results in Figure 12 in section A.13. However, due to the time limit, we only showed one environment: HalfCheetach, and will include results for all other environments in the final version.

$• $  Buffer Size and Memory Discussion: Added the technical discussion for the buffer size and memory calculation in section A.14. We added an extensive discussion on the impact of the buffer size on the model performance to reveal the relationship between replay buffer and model performance qualitatively. Additionally, we also provided the memory usage estimate in this section.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a variation of PPO based on a trajectory buffer to reduce the noise in PPO and improve the learning efficiency. This paper is on a positive borderline. The main remaining concerns are the proposed scheme appears to be incremental and the theoretical and experimental results are limited. After reviewing the discussion, I think the novelty issue is minor. Nonetheless, I agree with reviewers (XcPF, zRpn, Ye97) more baselines are needed in the experiments, especially off-policy methods. The authors made some attempts to include results of other PPO variants, but other common baselines are needed to establish how the algorithm performs. In addition, another weakness of the paper is that only results on Mujoco problems are presented. I would encourage to also experiment with non-mujoco problems especially those with high-dimensional observations or discrete actions.

Considering these points, I would not recommend acceptance of the paper at its current status.