CHPO: Constrained Hybrid-action Policy Optimization for Reinforcement Learning
Authors: Ao Zhou
Jiayi Guan
Li Shen
Fan Lu
Sanqing Qu
Junqiao Zhao
Ziqiao Wang
Ya Wu
Guang Chen
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement learning
parameterized action spaces
constrained policy optimization.
Primary area: reinforcement_learning
TL;DR: In this work, we propose a constrained hybrid-action policy optimization algorithm to address the tasks involving cost constraints and hybrid actions.

Abstract: Constrained hybrid-action reinforcement learning (RL) promises to learn a safe policy within a parameterized action space, which is particularly valuable for safety-critical applications involving discrete-continuous hybrid action spaces. However, existing hybrid-action RL algorithms primarily focus on reward maximization, which faces significant challenges for tasks involving both cost constraints and hybrid action spaces. In this work, we propose a novel Constrained Hybrid-action Policy Optimization algorithm (CHPO) to address the problems of constrained hybrid-action RL. Concretely, we rethink the limitations of hybrid-action RL in handling safe tasks with parameterized action spaces and reframe the objective of constrained hybrid-action RL by introducing the concept of Constrained Parameterized-action Markov Decision Process (CPMDP). Subsequently, we present a constrained hybrid-action policy optimization algorithm to confront the constrained hybrid-action problems and conduct theoretical analyses demonstrating that the CHPO converges to the optimal solution while satisfying safety constraints. Finally, extensive experiments demonstrate that the CHPO achieves competitive performance across multiple experimental tasks.

CHPO: Constrained Hybrid-action Policy
Optimization for Reinforcement Learning
Ao Zh Jiayi Guan
Li Shen
Fan Lu
Sanqing Qu Guang Chen
Junqiao Zhao'
Ziqiao Wang
Ya Wu
'Tonji University
Shanghai Innovation Institute
SSun Yat-Sen University
*CNNC Equipment Technology Research (Shanghai) Co., Ltd.
(2211107, guanjiayi, lufan, sanqinggu)@toongi edu, cn {Zhaoj juqiao, ziqiaowang, guangchen) @tongjiinn........... edu. mathshenl i @gmaall.......................ae com qpo1448163. com
Abstract
Constrained hybrid-action reinforcement learning (RL) promises learn a safe policy within parameterized action space, which is particularly valuable for safety-critical applications involving discrete-continuous hybrid action spaces. However, existing hybrid-action RL algorithms primarily focus reward max- imization, which faces significant challenges for tasks involving both cost con- straints and hybrid action spaces. In this work, propose novel Constrained Hybrid-action Policy Optimization algorithm (CHPO) to address the problems of constrained hybrid-action RL. Concretely, rethink the limitations of hybrid- action RL in handling safe tasks with parameterized action spaces and reframe the objective of constrained hybrid-action RL by introducing the concept of Con- strained Parameterized-actioo Markov Decision Process (CPMDP), Subsequently, we present constrained hybrid-action policy optimization algorithm to confront the constrained hybrid-action problems and conduct theoretical analyses demon- strating that the CHPO converges to the optimal solution while satisfying safety constraints. Finally, extensive experiments demonstrate that the CHPO achieves competitive performance across multiple experimental tasks. Our code is available at github. CHPO
Introduction
Hybrid-action reinforcement learning (RL) aims handle tasks with parameterized action spaces, combining discrete and continuous actions to enable effeectvve dicisiionmmainng in complex scenar- ios 1.3. Recent works of hybrid-action RL have achieved remarkable achievements in the domains of policy games 4 3. robotics (6, resource allocation [2]. and autonomous driving 13 14] However, safety concerns remain primary challenge to the real-world deployment of hybrid- action RL, particularly in scenarios with high safety requirements [75 For instance, in the context of autonomous driving, agents need learn reward- maximizing policy within parameterized action spaces while simultaneously ensuring collision avoidance [18-20). Constrained hybrid-action RI. is a promising and potentially effective approach to address the aforementioned issues. learns policies that satisfy safety constraints in tasks with parameterized action spaces.
"Coreesooniigg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Although substantial progress is being made by leveraging the Parameterized Action Markov Decision Process (PAMDP) framework [2T] for policy learning in hybrid action tasks 2
S 22 its adaptation to handle safety constraints in such settings remains challenging and unresolved 23 For the
aforementioned cost constraint problems, common methods, like the Lagrange multiplier and penalty function methods [26H29], are sensitive to initial values and require extra coefficients, leading to training instability and costly hyperparameter tuning. Furthermore, since hybrid- aaciinaction RL requires the simultancous optimization of discrete actions and continuous parameters within policy networks, this typically necessitates the design of evaluation strategies for both components to jointly guide policy updates, which increases the difficulty of convergence compared to RL for a single action space. To make matters worse, the additional hyperparameters introduced by safety constraints further exacerbate the instability of model training. Therefore, this work focuses solving the problems of constrained hybrid-action RI. involving hybrid actions and cost constraints.
To solve the aforementioned constrained hybrid-action optimization problem in parameterized action spaces, we propose a novel Constrained Hybrid-action Policy Optimization algorithm (CHPO). Concretely, we introduce the Constrained Parameterized-action Markov Decision Process (CPMDP) to represent the cost constraints in parameterized action spaces. Based on this, we redefine a novel constrained hybrid-action RL objective for tasks involving hybrid actions and cost constraints. Subsequently, we propose a constrained hybrid-action policy optimization algorithm to address the constrained hybrid-action RL tasks. Furthermore, we present theoretical analyses the convergence and safety of CHPO, demonstrating its capability to adaptively learn safe policies for parameterized action spaces under given safety constraints. Finally, extensive experiments demonstrate that the CHPO algorithm achieves competitive performance across multiple experimental tasks, particularly outperforming baseline algorithms in maximizing rewards while ensuring that the average cost across multiple seeds satisfies the safety constraint. The main contributions of this work are listed as follows:
To the best of my knowledge, we are the first to formulate a constrained hybrid-action RL objective for the policy optimization problem involving hybrid actions and cost constraints.
We develop a constrained hybrid-action policy optimization method to solve the constrained hybrid-action RL tasks, which not only learns safe policies that satisfy safety constraints but also improves the stability of the algorithm.
Theoretical analyses provide the convergence and safety guarantees of CHPO, indicating CHPO can learn safe policies for hybrid action spaces under given safety constraints.
Extensive comparisons and ablation experiments demonstrate that the CHPO algorithm deliv- ers competitive performance, particularly outperforming baseline algorithms in maximizing rewards while ensuring that average costs across multiple seeds satisfy safety constraints.
2 Related Work
In this section, we extensively discuss the related work on constrained hybrid-action RL. We primarily focus on hybrid-action RL and constrained RL. In addition, we provide supplementary discussion of the related work on RL in Appendix A
Hybrid-action RL is an approach to address parameterized actions composed of discrete actions with continuous action parameters. The straightforward method to directly discretize the continuous action spaces and transform them into a large discrete set, which often results in an excessively large discrete action space (30]. Currently, the PAMDP (21 framework has been widely adopted to better address the policy learning challenges in RI. with parameterized action spaces. Based on it, PADDPG M applics the DDPG [31] algorithm directly by relaxing discrete action spaces into continuous set, and PDQN 4 combines the spirits of both DQN [32 for discrete action spaces and DDPG for continuous action spaces. Furthermore, MPDQN (5) utilizes multi-pass deep
Q-networks to separate continuous action parameters, and HPPO 2 uses the hybrid actor-critic architecture flexible to the structure of the action space. Moreover, some studies focus on considering the dependency between discrete and continuous actions solve RL tasks with hybrid actions g 33
Constrained RL aims to solve policy optimization problems where cost constraints are enforced alongside maximizing rewards. After the proposal of the Constrained Markov Decision Pro- cess (CMDP) 34 framework, numerous constrained RI. algorithms have been developed to learn safe policies satisfying cost constraints [35] 36 16 37 Among them, some works directly use

---

Lagrange multipliers and show outstanding performance in satisfying constraints [3] Editor policies and safe exploration have also achieved promising results as alternatives to Lagrangian Furthermore, there have been successful attempts in addressing constrained RI.
multipliers 42 43
tasks by employing constrained methods such as interior poiioor-pontttt [26]. conditional value-at-risk [+ 4d.
penalty function methods (28], and world models [47 48) Moreover, the algorithms based on the
primal-dual methods have been widely applied to solve constrained RL problems, such as PDO [49], CVPO (50), and RCPO [51]. Extensive research offline safe RL further enhances the safety of
the training process (5-557] These diverse methodologies underline the versatility and growing
sophistication of constrained RL research.
In summary, compared to purely discrete or continuous action spaces, parameterized action spaces are more challenging to handle in RL. This difficulty arises because parameterized action spaces include both discrete actions and continuous parameters, whereas most RI. models are designed specifically for either discrete or continuous action spaces. More critically, the difficulty further exacerbates the challenge of selecting safe discrete actions and continuous action parameters in safety-critic parameterized action spaces when both cost and reward are involved. When extending constrained RL algorithms to hybrid action spaces naively, directly relaxing them into a continuous space substantially increases action complexity and degrades performance, making existing constrained PPO variants (2) inapplicable. To the best of our knowledge, our work is the first attempt to simultaneously address parameterized actions and cost constraints.
3 Preliminaries
This section presents fundamental concepts of RL, hybrid-action RL, and constrained RL. Sub- sequently, we rethink the existing works and highlight the purpose and significance of our work.
3.1 Concepts and Background
Standard RL employs the Markov Decision Process (MDP) framework defined by the tuple (S,A, A, P.r. Po: 7). where S E R" is the state space,
R" is the single discrete or continu- ous action space, P: S x A x S + [0, 1] denotes the state transition probability p(st+1 (s. az) from state s to state St+1 under the action ag, r S A E represents the reward function abbreviated as r = r(s ot), Po S + [0. 1] is the distribution of initial states, and E (0, 1] is the discount factor. The policy π is a probability distribution mapping the state 8, to the action dz. The common objective of standard RL is to maximize the cumulative rewards:
n arg max ETOT
where the T = {s0, ag. } ~ क denotes the trajectory. Moreover, constrained RL based on the CMDP framework to define tuple (S, P, P,r, P), where the added C' is the set of costs {q+ : x A + i 1,2,... , m for violating m constraints. We use shorthand C = c,(s,, a. for simplicity. The goal of constrained RL is to maximize the cumulative rewards n² arg max, E E, 1 ' while satisfying safety constraints E E I' ) SC, where č is the cost threshold of the i-th cost constraint.
On the other hand, hybrid-action RL utilizes the PAMDP to define the parameterized tuple AS, Ap,, po. 7). The whole parameterized action space Ap includes finite set of discrete actions Ad = {01, (01, A2, 0&) } and set of real-valued continuous parameters A.. C AR-A2 corresponding to each discrete action 0d E Aa. In this way, complete action step is composed of aa, and actt: where Ad, E Aa is a discrete action and ac,t E A. is the chosen parameter to execute with the discrete action ad,t. The whole action space Ap is the union of each discrete action with all possible parameters for that action:
Ap = U((aa lad E Ad, ag E Ac).
The parameterized policy T ap(a,, ())) is to select the discrete action ad,t and the continuous parameter ac.t. which are unified as ae (ad,t, ac,t). Similar to standard RL, the objective of hybrid-actin RL also maximize rewards:
80
T = arg max E EETOT
Σ
I''
(3)
Tp
3

---

3.2 Rethinking to Hybrid-action RL
We rethink the application requirements of hybrid-action RL algorithms in real-world scenarios. In the autonomous driving parking task [58] 59], the intelligent vehicle needs select the direction of travel and the distance to travel in that direction to successfully park, but the vehicle must avoid collisions with obstacles ensuring safety throughout the process. In this work, we establish and test a parameterized Parking task with specific details provided in Appendix [.I] Similarly, in robotic manipulation, both the selection of operations and the corresponding operation parameters must be considered, along with constraints related to hazardous areas. These cases require us to simultaneously account for parameterized actions and cost constraints.
Moving
Moving
HardMove
HardMove
Reward of episode o
Cost of episade
1006
Peooooo of presan
POON
Itarations-1e5]
(tr5))oos((1e55)
(tarations(1ee5)
Ierations
Figure The figure depicts the reward and cost curves for the Moving and HardMove tasks. The shaded curves represent the mean and variance of online testing during the training process with three different random seeds. The cost limit is set as č =
We conduct experiments on the hybrid-action RL algorithms PAMDP in the typical experimental scenarios Moving 2 22 33 60) and HardMove (22 33 60) of DI-engine [60] where introduce costs and hazardous areas into these tasks. A detailed description of the tasks is provided in Appendix D.I] The experimental results shown in Fig- [ The curve of the HPPO [2] algorithm indicates that, while the current hybrid-action RI algorithm can achieve high rewards, it often violates safety constraints
significant extent. Additionally, inspired by the RCPO (5]] algorithm, we attempt to use the cost as negative reward to learn policies that satisfy safety constraints. We combine the PDQN algorithm [4] with reward-constrained policy optimization to form PDQN-Rco. The result of PDQN-Roo shows that although the ewward-constraing method can learn policies that adhere to safety constraints, the rewards are relatively poor. This suggests that the wwward-constrained method for hybrid action spaces tends to drive the policy towards local optima, making it challenging ensure optimal performance.
In summary, for parameterized action spaces, the current hybrid-action RL algorithms and the reward- constrained approaches insufficient to adequately meet the requirement of maximizing rewards while satisfying safety constraints. Inspired by CMDP and PAMDP, we propose a novel constrained hybrid-action RL task for addressing the simultaneous treatment of parameterized action spaces and cost constraints in RL application scenarios.
Methodology
In this section, we provide detailed exposition of constrained hybrid-action policy optimization for constrained hybrid-action RL. Firstly, we define the constrained hybrid-action RL task and reframe the objective of constrained hybrid-action RL by introducing the concept of CPMDP. Subsequently, we construct constrained hybrid-action actor-critic architecture to address scenarios involving costs and hybrid actions and propose constrained hybrid-action policy optimization algorithm for constrained hybrid-action RL within the architecture. Additionally, we present theoretical analyses demonstrating that CHPO guarantees the convergence of learning safe policies under safety constraints.
4.1 Constrained Hybrid-action RL
Definition 4.1. Constrained hybrid-action RL tasks refer to tasks that involve parameterized ac- tion spaces and cost constraints, with the objective of maximizing rewards while satisfying safety constraints. This is formulated as:
x T'r s.t. E
8 D' Sts
T = arg Max ET
ErawT
(4)
1=0

---

Based on the analysis and discussion in Section 3.2 it is known that existing hybrid-action RL algorithms and the cost constraints and parameterized actions. Therefore, we introduce the cost to define the constrained parameterized tuple (S.Ap. C, P.r, (o, 7) as the CPMDP framework, where the added C' is the set of costs {or S x Ap > RR, i 1,2, . for violating constraints in parameterized action spaces and we also use shorthand Ci,t = ci(St, ax) = c; (St, (ad,t, dc,))) for simplicity.
method face challenges when dealing with tasks that involve
4.2 Constrained Hybrid-action Policy Optimization
Due to the difficulty in directly handling the objective stated in Definition 4. we are inspired by HPPO to define the state-value for reward on the parameterized policy Tp VT())
E7-EE 2t=0 (8t (x) = s] and the state-value for cost is defined similarly as V = Erwep 1 ( as) = s]. Furthermore, since the policy Tp outputs two types of actions, we define the discrete policy Ra outputting discrete actions ag and the continuous policy Tee outputting continuous parameters separately, which together form the policy Tp for clearer explanation. Therefore, the Eq. (4) is rewritten as:
o = arg max E VV. (s)], s.t. ES (ves, [
(5)
S6p
where o, and de, represent the parameters of the state-value functions, and 0p = (0s, 0z) is the parameters of the policy Ty consisting of 02 and 0,.
To handle a class of discrete-continuous hybrid action spaces with safety constraints, we propose a constrained hybrid-action actor-critic architecture capable of learning safe policies in both discrete and continuous action spaces under cost constraints. The architecture comprises two critic networks and two actor networks: the reward critic network represents the reward state-value function V2, (s), the cost critie network represents the cost value function V (s), the discrete actor network de. defines the discrete policy function Te.,, and the continuous actor network defines the continuous policy function .T... In our architecture, the cost critic network provides the estimation of the cost advantage function Aç: -V((s)) + G.t + 70,,++ + + ------ -1 + - -T-i (sT) to allow the constrained hybrid-action actor-critic architecture to flexibly adapt to constrained policy optimization methods, where T is timesteps for the policy up and much less than the length of an episode, and I E [0, T) is the timestep index. The computation of the reward advantage function Å", similar to Å is given by A; = (s:) objective of the cost state-value update is written as:
(st) + re + TT++1 + ... + ------rr-- rr-1 + + TTVV"((s)) The
((0c,)=arg min Ear ((C()))))) - V&, (.))²}
(6)
where Ci(s)= Ei=o ( aets) is the target cost return computed by the Monte Carlo method and 8t Similarly, the objective of the reward state-value update is expressed as:
((Ør) arg min Esw. (f((s) (V)())-VV(())) (+))
where i(s)) is the target reward return computed using R((s) - 1¹ (e ($t+j, ax+;) and 8t s.
The constrained hybrid-action optimization described in Eq. (5) is conventionally tackled using the Lagrange multiplier method, which converts the constrained formulation into an unconstrained optimization problem by incorporating penalty terms into the objective function. While effective in principle, this primal-dual approach often suffers from issues such as sensitivity to the initialization of Lagrange multipliers and the careful selection of learning rates, making it computationally expensive due to the extensive hyperparameter tuning required. Inspired by CRPO [61]. we directly tackle the constrained hybrid-action problem using the policy gradient of the primal objective based on stochastic approximation theory, effectively avoiding the instability and inefficiency associated with auxiliary Lagrange multipliers. Theoretical analyses in Section 4.3 indicate that the direct optimization method ensures stable convergence and enhances the practicality of solving constrained bybrid-action RL problems.
Based on the above analysis, we propose a Constrained Hybrid-action Policy Optimization (CHPO) for RL with parameterized action spaces to solve constrained hybrid-action RI. tasks. The param- eterized policy update for the CHPO algorithm is performed within the constrained hybrid-action
5

---

actor-critic architecture. We first evaluate whether the current policy satisfies the cost constraints based on the cost state-value and then update the parameterized policy by maximizing the reward or minimizing the cost. Specifically, when the cost constraints in parameterized action spaces are satisfied (i.e., E V
I cj), the constrained hybrid- accion optimization described in Eq. (5) is (s)
transformed into unconstrained optimization problem, requiring consideration of reward maximiza- tion within the parameterized action spaces. If the cost constraints are not satisfied, the constrained hybrid-action optimization in Eq. (5) is converted into minimizing the cost in discrete-continuous hybrid action spaces. Therefore, the objective of the policy update is expressed as:
To, (als) Top (als)
-Ino, (Imn, - Ion, Rx Att)
[(0p) arg max ES 8p
(8)
where Vs represents the safe policy, Ta, E Fg indicates that the policy To, satisfies safety constraints, while тe, $ wa denotes that the policy ma, violates the constraints. IaT, is sign function that takes the value one when тo, satisfies safety constraints; otherwise, it takes zero. Similarly, when To, violates safety constraints, I En. is equal to one; otherwise, it equals zero. 6% is the parameters of the discrete and continuous actor networks at k-th update including 04 and 0% The policy To, update involves separately updating the discrete policy ne, and the continuous policy Te,, which includes handling 0₂ and 0c. A detailed update example is provided in Section 4.4
4.3 Theoretical Analysis
To validate the convergence properties and cost safety of CHPO, we derive the policy and cost boundary. Before deriving the following, we review the policy boundary of the natural stochastic gradient, as presented in Lemma 4.2
Lemma 4.2. The boundary of the policy based natural stochastic gradient updates 167
((TO,) [(TOz)) S 3(1 + ((Fmax)
110% Qilla + 2ar S
(1-7)3 a + E. SD (TT5))
7)2 (1-
(9)
(TK,(Tō,)1 ((oT11))
where mmax represents the maximum reward, Q2 and Qu denote the evaluated value and the estimated value of the action-state at k-th update, Tak: and Tah+1 represent the policies at step k and i -
1, respectively. The parameter a (1-7)1.5 1-5/)//////// /SSIAA |S] and [A] represent parameters associated with the dimensions of the observation state s and the action space a, respectively. K denotes the maximum number of policy update steps.
Based on the policy boundary presented in Lemma 4.2 and the update method of the CHPO, we
derive the policy and cost boundary shown in Propositions 4.3 and 4.4 The propositions 4.3 and 4.4 are presented and discussed detail in Appendices B.2 and
Proposition 4.3. After updating the policy for the K steps according the method described in Eq. (8) the policy converges to a boundary.
V SIA
[( - E[
(10)
(1 (1-77) (1-7)1- 155 /
Proposition 4.4. After updating the policy for K steps according to the method described in Eq. { the policy cost converges to boundary.
(1 K
B[[(TT)))] - 5 56
(11)
The proposition 4.3 shows that the policy converges to the optimal policy after K iterations, with the convergence bound dependent on the dimensions of the state space (S], the action space [A], and the number of update steps K. The proposition 4.4 demonstrates that, after K iterations, the policy

---

cost converges to within a bounded range of the cost threshold, where the bound depends on the dimensions of the state space |S], the action space |A,, and the number of iterations K. The number of iterations K consists of the number of iterations [W,] for maximizing the reward and the number Σr [NV.
of iterations |Ne,1 for minimizing the cost. We define C/A ratio=
ΣT [N,, H+W, as the proportion of update steps dedicated to cost minimization relative to the total number of policy.
Remark 4.5. The proportion (C/A ratio) between the update iterations allocated to reward maxi- mization and cost minimization influences the final optimal policy [f([(T0;)).
detailed discussion of Remark 4.5 is presented in Appendix B.4, with corresponding experiments further supplemented in the ablation study.
4.4 Practical Algorithm
To facilitate the understanding of the implementation process of the CHPO algorithm, we provide detailed explanation of practical instance of the CHPO algorithm. The pseudo-code for the CHPO algorithm is shown in Aigorithmoot Appendix In the state-value estimation step, we employ Eq. ( and Eq. (7) in the constrained hybrid-action actor-critic architecture to separately update the state-values of reward and cost. In the policy update step, we evaluate whether the cost constraints are satisfied and update the policy network parameters 6, including 0₂ and 0.. Since the cost state-value function Ves (s) estimates the cost return with the discount factor 7, utilizing Oce
the cost vtaue in Eq. ( to determine whether the safety constraints met may lead to the neglect of some unsafe circumstances. Therefore, we use the cost return without the discount factor fee (TTp)) B a Ei=oc.. instead of Vo, (s) in Eq. (5) decide whether cost constraints satisfied and the objective of policy update, where тHp represents the parameterized policy Te, at k-th update. Concretely, for the parameterized policy Te, update including the discrete policy no, and continuous policy ge, update, if cost constraints are satisfied, the objective of policy update in Eq. (8) is transformed into:
[(0p) = arg max ESNS min To, (als) A clip( To, (als) Tay (als) 1 -,, e, (11+)))) )
(12)
Op
Tap (a|s)
where of is the parameters of the parameterized policy network at k-th update, and a includes the discrete action ag and the continuous action de. Conversely, if the safety constraints are not satisfied, the objective of the policy update is expressed as:
To (als) Å clip( Top (a|s)
To (als)
[(0p)) = arg min Ea-S min 0p
Toz (als) 1 c, + e)
(13)
That is to say, the discrete policy Te, and continuous policy no, constitute the parameterized policy To, to determine complete actions, achieving reward maximization while satisfying safety constraints.
5
Experimental Evaluation
In this section, we conduct comprehensive comparative experiments between CHPO and previous hybrid-action RL methods in tasks with different hybrid action spaces and observation dimensions.
5.1 Task and Baseline
Task. To assess the performance of CHPO in various tasks with parameterized action spaces, we select three widely adopted tasks from DI-engine (60] and establish Parking task with parameterized action spaces as experimental tasks in this work. Concretely, we choose the Moving 2 22 33 60).
60 tasks, all of which require agents to perform both discrete and continuous actions to reach a target area. For instance, in the Moving task, the agent can take discrete actions such as turn, accel, or break, accompanied by two continuous parameters: acceleration and steering angle. To the best of my knowledge, there are currently no standardized tasks for constrained hybrid-action RI. To facilitate further research and reproducibility of this work, we modify the aforementioned standard experimental scenarios of DI-engine for bybrid-action RL
Sliding 22, [60) and HardMove [2
7

---

Moving
Sliding
HardMove
Parking
Cost of apisade Reward of episade
-20
-1.3
AAR NESTIO
(terations("165)
Itrrstions(1ee)
Iterations(11e5)
(erlations(1e5)
Figure 2: The figure depicts the reward and cost curves of CHPO and the baseline algorithms in the comparison experiment to showcase the performance of each algorithm. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds. The cost limit of the four tasks is set at c = 1.
by introducing costs and hazardous areas into these tasks. Additionally, in the Parking task, the intelligent vehicle selects from three discrete directional actions, each associated with specific movement distance, to park in designated areas while avoiding collisions with obstacles and other vehicles. A more detailed description of four task settings can be found in Appendix D.I
Baselines. Due to the lack of hybrid-action RL algorithms that can simultaneously handle parameter- ized action spaces and cost constraints, we enhance existing hybrid-action RL methods to develop constrained Abrid-action RL algorithms as the experimental baselines. PADDPG-Lag is an improved hybrid-action RI algorithm that combines the Lagrange multiplier method with PADDPG [1] to enable the optimization of constrained hybrid-action policies in parameterized action spaces. That is say, the online safe RL algorithm DDPG-Lag handles these tasks by relaxing discrete action spaces into a continuous set. Similarly, we introduce the Lagrange multiplier method based on HPPO [2] to relax the constrained hybrid-action problem into an unconstrained optimization problem and obtain an improved algorithm HPPO-Lag. Moreover, we incorporate the into PDQN 4 which combines the spirits of both DQN for discrete action spaces and DDPG for continuous action spaces, resulting in an enhanced algorithm PDQN-Rco.
method [51)
5.2 Performance Comparison Experiment
Performance various tasks. To evaluate the performance of the CHPO algorithm across various safe tasks with parameterized action spaces, we conduct comprehensive comparative experiments. The results involve comparing the performance of the CHPO algorithm with the baseline algorithms in the four tasks: Moving, Sliding, HardMove, and Parking. Fig. 2 iillustrates the reward and cost curves for the baseline algorithms and the CHPO algorithm in the four tasks. From the illustrated results, it can be observed that compared to the baseline algorithms, CHPO maintains the average costs within the specified cost limit while achieving higher rewards. Furthermore, the CHPO algorithm demonstrates a more stable performance compared to other baseline algorithms. Notably, CHPO provides the highest rewards across all four tasks while adhering to the cost limit. The above results indicate that CHPO consistently enforces policy compliance with safety constraints across various safe tasks involving parameterized action spaces while delivering competitive reward returns.
Performance on different cost limits. To evaluate the performance of the CHPO algorithm under different cost limits, we conduct comparative experiments between the CHPO algorithm and the baseline algorithms at multiple cost limits. The results in Table [ demonstrate that the CHPO algorithm ensures adherence to cost constraints while providing competitive reward returns across the four tasks with varying cost limits. Additionally, from the results in the table, it is evident that an increase in the cost limit and more relaxed safety constraints lead higher mean values of reward for the CHPO algorithm and the baseline algorithms in the four tasks. Notably, compared to other baseline algorithms, the CHPO algorithm effectively adjusts the policy based on different cost limits to maximize the reward as much as possible. Moreover, the reward and cost curves for the CHPO algorithm and the baseline algorithms under these cost limits & 1.5 and ã = 2 during online testing are provided in Fig. and Fig. of Appendix D.2 The analysis of the above results indicates that the CHPO algorithm can effectively learn policies that satisfy various safety constraints across tasks with parameterized action spaces while achieving excellent rewards.
8

---

Table 1: The performance of CHPO and the baseline algorithmss is evaluated across different cost limits. The results from experiments involving 40 episodes are conducted with 3 random seeds. These cost limits of the four tasks are set at C = 1, c 1.5 and é; = 2. The costs satisfying safety constraints are shaded gray, with the highest-reward entries highlighted in bold.
Moving
Sliding
Method
Metries
& 1.3
G
1.5
Reward f
-0.4320.55
-0.07=0.89
0.8920.86
-0.93=0.16
-0.9240. 18
-0.67=0.30
IIPPO-Lag
Cost L
0.17+6.58
0,43+8.22
1.09=1.41
0.28#1.12
1.04#3,95
1.3243.61
Reward f
1.30#0.67
1.41=0.54
1,50#0.52
-0.67=0.33
-0.61#0.32
-0.4620.33
PADDPG-Lag
Cost L
0.36+1.59
1.10#3.57
1,59
1. 1.09&1.68
1.17
1.29+1.96
Reward f
-0.83=0025
0.22
0.58+0.92
-0 -0.99=0.09
-0.93+0.22 2.2
~0.89+0.39
PDQN-Rco
Cost 4
0.31+0.62
0.63+5.20
0.39+3.24 2.24
0. 0.1221.00
1.2242.02
1.7445.69
Reward f
1.41=0.55
1.5220.46
1.6020.36
-0.37=0.52
0.3420.43
+0.30+0.51
CHPO
Cost L
0.22+0.99
1.42+4.03
1885+2.11
0.92=1.60
1.24=1.67
1.93+3,45
HardMove
Parking
Method
Metrics
8, = 1.5
&
&
1.5
Reward f
-0,51
0.02=0.64
0.21
20.01#25.21
23.22+13.03
23.02+16.92
IIPPO-Lag
Cost 4
0.67+1.22
1.08=1.95
1.33+1.53
0.82+1.26
1.2623.72
1.75+5.24
Reward f
-0.29=0.59
0.80=1.20
1.28=0.81
16.66
35.21+5.52
37,24=3.00
PADDPG-Lag
Cost 4
0.35#1.11
1.18=1.76
1.73+3.39
0.94=2.15
1.21#2.62
2.2742.73
Reward f
-0.8820.20
-0.8420.23
-0.81#0.24
32.72+7 44
34.95+5.30
35,33=5.34
PDQN-Reo
Cost L
0.37+0.89
0.38+1.04
0.43+1.10
0.21
0.3420.75
0.8742.30
Reward f
0.89=0,59
1.0220,54
1.3620,33
37.63+3.04
37.98+2.47
37.73+3.15
CHPO
Cost +
0.97x1.41
1,48+2.13
1.73+2.14
0.93+3.17
1.39+5.31
1.88+2.27
5.3
Ablation Experiment
Performance with and without the constraint module. We evaluate the impact of the constraint module in the CHPO algorithm by removing it during policy updates. The HPO represents CHPO without the constraint module during policy updates. Fig. dispplas the reward and cost curves for the CHPO algorithm and the CHPO algorithm without the constraint module in the Moving and HardMove tasks. From the figure, it can be observed that when the constraint module is not used to handle safety tasks with parameterized action spaces, HPO achieves higher rewards compared to CHPO. However, the costs achieved by HPO significantly exceed the predefined cost limit 6; = 1, resulting in substantial violations of safety constraints. Meanwhile, CHPO effectively adheres to the safety constraints with only minimal loss in rewards. Additionally, the results this ablation experiments for the Sliding and Parking tasks presented in F of Appendix D.2 This demonstrates that the constraint module in the CHPO algorithm effectively enforces safety constraints tasks with hybrid action spaces while maintaining satisfactory reward performance.
Maving
Moving
HardMove
HardMove
Reward epiode
1.5 1.0
Cost of opiode
Ronoon of seppi Booog
epicade
Garline
Iterations(1eeS)
[terationss1ee)
Rerations/*1e5)
Iteratioons!1es)
Figure 3: The figure depicts the reward and cost curves in ablation experiments regarding the constraint module. The HPO represents CHPO without the constraint module. The shadowed areas represent the variance of the test results for three random seeds and the cost limit is set at é; 1.
Performance on different C/A ratios. To evaluate the impact of the C/A ratios on the performance of CHPO, we experiment with four different C/A ratios tailored to the practical training scenarios of four distinct tasks. Fig. of Appendix D.2 presents the mean and standard deviation of CHPO's performance on safety tasks with parametenized action spaces across different C/A ratios. The results in the figure indicate that as the C/A ratios increase which reflects higher frequency of cost minimization policy updates, the mean costs generally follow decreasing trend. However, when the number of cost minimization policy updates becomes excessively high, the mean costs no longer decrease and instead begin to rise, accompanied by reduction in rewards. This effect particularly pronounced in the Moving, Sliding, and Parking tasks, where excessive cost minimization

---

policy updates result in higher cost values, lower mean rewards, and increased reward variance. Additionally, the reward and cost curves corresponding these observations are presented in Fig.B Appendix D.2 The above results indicate that the CHPO algorithm can achieve satisfactory rewards while ensuring that average costs satisfy safety constraints within a broad range of C/A ratios.
Conclusion
In this work, we propose novel constrained hybrid-action policy optimization algorithm for con- strained hybrid-action reinforcement learning. Concretely, we first rethink the requirements of hybrid-action RL in real-world applications and redefine the constrained hybrid-action RL objective for tasks involving hybrid actions and cost constraints by introducing the CPMDP. Subsequently, present constrained hybrid-action policy optimization algorithm to address the constrained hybrid-action RL tasks within the constrained hybrid-action actor-critic architecture. Additionally, we present theoretical analyses demonstrating that our method guarantees the convergence of learn- ing safe policies for parameterized action spaces under given safety constraints. Finally, extensive experiments illustrate that the CHPO algorithm is capable of handling the constrained hybrid-action RL tasks and providing competitive performance, particularly in terms of safety.
Acknowledgments and Disclosure of Funding
This work was supported by the National Key Research and Development Program of China (No. 2024YFE021 1000), in part by the National Natural Science Foundation of China (No. 62372329, No. 62576364), part by the Shanghai Scientific Innovation Foundation (No. 23DZ1203400), in part by the China Postdoctoral Science Foundation (No. BX20250383, GZB20250385, 2025M771530, 0025M771539) in part by Tongii-Qomolo Autonomous Driving Commercial Vehicle Joint Lab Project, and in part by Xiaomi Young Talents Program.
References
[1] Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143 2015.
[2] Zhou Fan, Rui Su, Weinan Zhang, and Yong Yu. Hybrid actor-critic reinforcement learning in parameterized action space. arXiv preprint 1903:0/003..1444 2019.
[3] Xiaofeng Li, Lu Dong, Lei Xue, and Changyin Sun. Hybrid reinforcement learning for optimal control of non-linear switching system. IEEE Transactions on Neural Networks and Learning Systems, 11::111111-1970,, 2022.
[4] Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu. Parametrized deep q-networks learning: Reinforcement learning with
hybrid action space. arXiv preprint arXiv:1810.06.0 2018.
[5] Craig Bester, Steven D James, and George D Konidaris. Multi-past g-networks for deep reinforcement learning with parameterised action spaces. arXiv preprint 140808880 2019.
[6] Mehdi Khamassi, George Velentzas, Theodore Tsitsimis, and Costas Tzafestas. Active explo- ration and parameterized reinforcement learning applied to simulated human-robot interaction task. In 2017 First IEEE International Conference on Robotic Computing (IRC), pages 28-35. IEEE, 2017.
[7] Mehdi Khamassi, George Velentzas, Theodore Tsitsimis, and Costas Tzafestas. Robot fast adaptation to changes in human engagement during simulated dynamic social interaction with active exploration in parameterized reinforcement learning. IEEE Transactions on Cognitive and Developmental Systems, 10(4):881-88) 2018
[8] Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Tobias Springen- berg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller. Continuous-discrett reinforcement learning for hybrid control in robotics. In Conference on Robot Learning, pages 735-751. PMLR, 2020.
10

---

[9] Murtaza Dalal, Deepak Pathak, and Russ R Salakhutdinov. Accelerating robotic reinforcement learning parameterized action primitives. Advances Neural Information Processing Systems, 34:21 147-2184728 2021.
[10] Yongyi Ran, Han Hu, Yonggang Wen, and Xin Zhou. Optimizing energy efficiency for data center via parameterized deep reinforcement learning. IEEE Transactions on Services 2022.
Computing, 16(2):1310-)
[11] Amna Mughees, Mohammad Tahir, Muhammad Aman Sheikh, Angela Amphawan, Yap Kian Meng, Abdul Ahad, and Kazem Chamran. Energy-efficient joint resource allocation in 5g hetnet using multi-agent parameterized deep reinforcement learning. Physical Communication, 61: 102206, 2023.
[12] Ting Wang, Xi Fan, Kai Cheng, Xiao Du, Haibin Cai, and Yang Wang. Parameterized deep reinforcement learning with hybrid action space for energy efficient data center networks. Computer Networks, 235:109989, 2023.
[13] Yuyang Xia, Shuncheng Liu, Quanlin Yu, Liwei Deng, You Zhang, Han Su, and Kai Zheng. Parameterized decision-making with multi-modality perception for autonomous driving. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 1463-4476 IEEE, 2024.
[14] Waleed Rafique, Jikai Wang, and Zhonghai Chen. Robust decision making with multi-modal perception for autonomus driving in hybrid action space. In 2024 IEEE 25th China Conference on System Simulation Technology and its Application (CCSSTA), pages 484-492. IEEE, 2024.
[15] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1199-1191, 2020.
[16] Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances Neural Information Processing Systems, 34:13859- 118899-1386,,,18 2021.
[17] Jing Hou, Guang Chen, Zhijun Li, Wei He, Shangding Gu, Alois Knoll, and Changjun Jiang. Hybrid residual multiexpert reinforcement learning for spatial scheduling of high-density parking lots. IEEE transactions on cybernetics, 2023.
[18] Jun Xie, Zhaocheng He, and Yiting Zhu. A drl based cooperative approach for parking space allocation in an automated valet parking system. Applied Intelligence, 53(5):5368 (165))5368--877 2023.
[19] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853-17862, 2023.
[20] Xintao Yan, Zhengxia Zou, Shuo Feng, Haojie Zhu, Haowei Sun, and Henry X Liu. Learning naturalistic driving environment with statistical realism. Nature communications, 14(1):2037, 2023.
[21] Warwick Masson, Pravesh Ranchod, and George Konidaris. Reinforcement learning with pa- rameterized actions. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
[22] Renhao Zhang, Haotian Fu, Yilin Miao, and George Konidaris. Model-based reinforcement learning for parameterized action spaces. arXiv preprint arXiv:2404.033377, 2024.
(23) Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
[24] Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv: 1902.04633 2019.
11

---

[25] Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A review of safe reinforcement learning: Methods, theories and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[26] Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4940 2020.
[27] Santiago Paternain, Miguel Calvo-Fullana, Luiz FO Chamon, and Alejandro Ribeiro. Safe policies for reinforcement learning via primal-dual methods. IEEE Transactions Automatic Control, 68(3): 2022.
[28) Linrui Zhang, Li Shen, Long Yang. Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint arXiv:2205.118144 2022.
[29] Ashish Jayant and Shalabh Bhatnagar. Model-based safe deep reinforcement learning via constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems, 22224422224444 2022.
[30] Alexander A Sherstov and Peter Stone. Function approximation via tile coding: Automating parameter choice. In International symposium on absiraction, reformulation, and approximation, pages 194-205. Springer, 2005.
[31] TP Lillicrap. Continuous control with deep reinforcement learning. (tti:::509.097,,,, 2015.
arXiv preprint
[32] Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. International conference on machine learning, pages 1995-2003. PMLR, 2016.
[33] Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete sascrete-ootinuuus action reinforcement learning via hybrid action representation. arXiv preprint
2021.
[34] Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
[35] Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
[36] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[37] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36: 18644-1888,,, 2023.
[38] Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning. arXiv preprint arXiv: 1xXiv:1910.017008 7(1):2, 2019.
[39] Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning, pages 3703-3712. PMLR, 2019.
[40] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-9143, PMLR, 2020.
[41] Ruichen Xu, Jinming Xu, Xiao Liu, and Yuan Lin. Safe hybrid-action reinforcement learning- based decision and control for diseretionary lane change. Machines, 12(4):252, 2024.
[42] Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. Advances in Neural Information Processing Systems, 35:2608-262.. 2022.
12

---

[43] Akifumi Wachi, Wataru Hashimoto, Xun Shen, and Kazumune Hashimoto. Safe exploration in reinforcement learning: A generalized formulation and algorithms. Advances in Neural Information Processing Systems, 36:29252-2222777 2023.
[44] LA Prashanth, Policy gradients for cvar-constrained mdps. In International Conference on Algorithmic Learning Theory, pages 155-169. Springer, 2014.
[45] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. Advances in neural information processing systems, 28, 2015.
[46] Chengyang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. To- wards safe reinforcement learning via constraining conditional value-at-risk. arXiv preprint arXiv:2206.04433 2022.
[47] Yarden As, Ilnur Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy opti- mization via bayesian world models. arXiv preprint (xXi::200100900) 2022.
[48] Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. Safedreamer: Safe reinforcement learning with world models. arXiv preprint arXiv:2307.07776. 2023.
[49] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk cisk-constr reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
[50] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-13668. PMLR, 2022.
[51] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arxiv:/8055 1074, 2018
[52] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 2161 1-21000. PMLR, 2023.
[53] Jiayi Guan, Guang Chen, Jiaming Ji, Long Yang. Zhijun Li, et al. Voce: Variational optimization with conservative estimation for offiin safe reinforcement learning. Advances in Neural Information Processing Systems, -3778888888 2023.
[54] Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. When should we prefer decision transformers for offline reinforcement learning? arXiv preprint arXiv:2305. 14550, 2023.
[55] Yinan Zheng. Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guice -uided diffusion model. arXiv preprint arXiv:2401. 10700, 2024.
[56] Jayy Guan, Li Shen, Ao Zhou, Lusong Li, Han Hu, Xiaodong He, Guang Chen, and Changjun Jiang. Poce: Primal policy optimization with conservative estimation for multi-constraint offline reinforcement learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2024.
Pattern Recognition, pages
[57] Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25(285):1-6, 2024.
[58] Jong Min Kim, Kyung Il Lim, and Jung Ha Kim. Auto parking path planning system using modified reeds-shepp curve algorithm. In 2014 11th International Conference Ubiquitous Robots and Ambient Intelligence (URAI), pages 311 311-315. IEEE, 2014.
[59] Qiang Zhang, Yafei Wang, Xulei Liu, and Yichen Zhang. Parking curve optimization for oblique slots in multi-obstacle environment. In Eighth International Conference on Electtomechunical Control Technology and Transportation (ICECTT 2023), volume 12790, pages 342-351. SPIE, 2023.
13

---

[60] Yazhe Niu, Jingxin Xu, Yuan Pu, Yumpeng Nie, Jinouwen Zhang, Shuai Hu, Liangxuan Zhao, Ming Zhang, and Yu Liu. Di-engine: universal system/engine for decision intelligence. 2021.
https:// thub. com/ httmm//ppaaaa/D-eeneeneee
[61] Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pages 1 11480-1 1491. PMLR, 2021.
[62] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
[63] Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv:1312.5602, 2013.
arXiv preprint
[64] Volodymyr Mnih. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783, 2016.
[65] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[66] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[67] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
[68] John Schulman. Trust region policy optimization. arXiv preprint urXiv:1502.05477 2015.
[69] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[70] Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. arXiv preprint arXiv:2004. 12956, 2020.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect contributions and scope. We present our four contributions, including problem proposal, algorithm, theoretical analysis, and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification: Please refer to Appendix E In summary, our algorithm primarily focuses on simulation environments and relies on online interactions, which are often constrained by low data sampling efficiency. We aim to deploy our method in real- wal-world applications and explore offline RL. methods for constrained hybrid-action policy optimization in the future.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All proof's can be found in Appendix B We have included complete, correct, and detailed proof for this theorem in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
15

---

Justification: We provide all the information needed to reproduce the main experimental results of the paper. The implementation process the CHPO algorithm is provided in Section 4.4 and Appendix ( Experimental results and details can be found in Section 5 and Appendix We also provide our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code necessary to run our own method (CHPO), as well as implementations the environments on which the algorithms were run. All other baselines are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
gablic/guidess http://eeeeeeoeedbiiieeeeeeeeees woweeuumisssooooooooo.....
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips http://///ppiiioeeeo////////aaaa
for more details.
16

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting can be found in Appendix D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The statistical significance of all presented empirical results is justified. In particular, we include standard deviation values for all results presented Table [ In addition, use the shaded areas in the figures represent the variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
17

---

Answer: [Yes]
Justification: We present the details of the hardware and other resources used for our experiments in Appendix D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e-g.. preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httnghhurpg.///orrpr...oooooo//p htt.ai//ehthcsccccennieess
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics carefully and affirm that our work adheres to all listed requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: To the best of our knowledge, our work cannot be misused in any way to cause any form of negative social impact. The impact statements can be found in Appendix D
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
18

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The research does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper already cited related packages that we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets with this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
19

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy (https://neun) http://pps.pcpcoaaeeeee/e/////// for what should or should not be described.
20

---

A Related Work
In this section, we further discuss the related work on standard RL algorithms to illustrate the challenges in handling hybrid action spaces.
RL seeks to learn award-maximiziigg policies through interactions with environments characterized by a single action space. On the one hand, RL for discrete action spaces has been extensively studied, focusing tasks where the set of possible actions is finite. Based on the Q-learning [62] algorithm which represents the expected cumulative reward of taking action, some variations of
DQN
63
are widely used in discrete action spaces, including asynchronous DQN [64], double
DQN
655 and dueling DQN B2). On the other hand, policy gradient [66] methods, such as
PPO [67 and DDPG [31]. have shown effectiveness in handling continuous action spaces by directly optimizing policies using gradient. Extensions like TRPO [68] and SAC [69] improve stability and efficiency through constraints on policy updates and entropy regularization. Since most RL models are specifically designed for either discrete or continuous action spaces, they struggle to simultaneously handle action spaces that contain both discrete and continuous components.
B Proofs and Discussions
Lemma B.1. (Policy gap (6l1) The policy gap for the natural stochastic gradient updated policy is:
[(Tū,) - ((TAy)) 3(1 + (TTmax)
2ar | 112 -2x
1
max
+
(1-
(1 71-
(14)
-DKL -TKL((T,llToott) ((T
where maax represents the maximum reward, Qu and Qu denote the evaluated value and the estimated value of the action-state k-th update, Toy and g represent the policies at step l and ls 1.
respectively. The parameter a =11 (11-))2// /
Proposition B.2. After updating the policy for the K steps according to the method described in Eq. ( the policy converges to a boundary.
V SIA
((T0,) ( 50
(15)
- (1--))LRKK
Proof. The relationship between the policy after K steps of updates and the optimal policy is represented as ((Tă,) ((Tog).
((T8,) B([((TO))) )]
m
m
(E KENe
Σ
+ Σ ((TT,)
Σ
(TT(T))))) - Σ L. (TTy))
E Waat + LNST
BENr
KENG
WEN,
ETH E
ΣREN,
[(Tū,)) Los (Toy))
+ En We [W!
((TØO)) (.(()))
En Way + NNl
Eno EKEN.,
D 1 Eans (DKI ((o3)) -Doc) (To, ((Tgges))
2002 2ac 1S|A
+
EnD We,1 + Nn
71-
3(1 + (CCmax)
ΣLEN.
I Eans ( (DKI(Tr'l1
11
-2 112
+ E W [NGT
(1-7)2
a
-DKI(TO, (1
2ar max
|S||AA
3(1 + ammax)
112 -Rilla
(1-7)3 7)3
(1-
1i ENEN., + +KHNN
Esws (DKL (DKL(TT, (1T0g) -Dz (DKz (D [Tof
E
2Th. We, + +WW1l
a
Emo ELEN.
EnD ELENS
3(1 + ((Cmax)
110 -2
+ En [Na. + Wil
(1 213
En Wa + [Wrl
(1 11-1 7)2
21

---

ΣXEN, Ei=0 No.11 + NN
2ar maxe |S|AA|
+ 3(1 + ar mmx
110 QQilla
7)3
(1 7)2
Eews (DKI (Toog) - D 11
0 |NG++|N,
E/= EKEN.
2ar max |
((1+0))
11 -Q%
+
1== [aall + NN
(1 11-79 -7)3
(1-
ΣREN.
2ar max S||LA|
3 3(1+
112% -Qille
i= (e ||,1
(1 7)3
(1-7)2
160
Bens (DKL TTag) - DEL ( (18,, (1 1)
== | + W
m
E1= ERENe,
LarRe 2 max ||
3(1 + alemax)
110 -Q% ll2
1=0 [aa W
7)3
(1 7)2
ΣXEN.
RaRe max |S||A|
3(1 + axem
110% -Qiil2
E [Wa,1 + NGTT
(1 (1-7) 7)3
(1 11-7) 7)2
Eans (DKL ((TOg) - DKL(TĀ, [ITT,...))]
m
1= INN
10 EKEN, + EKEN.
RaRe max S||AA
EnEN.
B(ItaRemar
10 Qiill
(1 (1-
(1 y)-
EI=0 ERENe,
3(1 + aeemax)
11 - l1a
i= Wai + WII
(1 7)2
(1))
2a Ra 18|1A
Enns (Dx7. |TTOy) DK (
+
1= Wa, + WTl
a
(1-7)3
1 ΣKEN., + EMEN.
3(1 + RRema)
v(1 - (11-7188114
E We + WW
71-
VK
1 Bows (D&L ( - DEL ( (T0,
10= + INA IWNT
3(1 + alRmaax
(1-7)3
VK
(13
(20)
Eows (DKI(TT, (1rog) D ( 1Tog 1)) + 3(1 + a Re
v(1 - (1-7)15) 7)|51| ||A
Ka
(1 71-
VK
2a RaR max ||
7)3
(0)
Esns( (DKI (KI((T,,lTTO)) - Dxr((TT, (T)- + RaRe [S||A||
3(1 + Remax)
/ /SAA
max
Ka
(1 7/3
(1 (1-7) 7)).
K
[ui)
VIAA
2 2R max V
(1 - 11))1.55 KK Eaws
(Tt 11 +
(1 - 1.55KK
3(1 + Remaxx) ) V SIA
(1- (1-7)1.5 1.5KKK
Eaws (DKz(T) mlmg)) DKL( (l( + 2R max + +-3 + 3Remax
(1 (1-7)L55VK
(16)
where N, represents the set of policies updated based on maximizing rewards and |N, represents the number of steps taken to update based on maximizing rewards. No, and [Ne,] represent the set of policy updates based on the i-th cost and the number of update steps based on the i-th cost, respectively.
22

---

|S] and |A| are the dimensions of the observation space and the action space, respectively. K is the maximum number of steps for policy updates. Based on the relationship of the policy boundaries updated by natural gradient as shown in Lemma B.1 the relationship of the inequalities {i) is obtained. Set the mamax max (Tmax: Cmax), the inequality { is derived. Based on the relation
11Qx Quel2 S V(==>]SSIA the inequality { is obtained. Substitute K = Ei= [ + [W
VR
the equation {iv} is derived. Where (1 7) //S////STT
(0, 1), the inequality {v} is
obtained, and substituting a (1 7/.5 / S/SIIA results in inequality {vi}. Based on Eq. (16
VISIIA thus proving Proposition B.2
follows that ((TO,) [I<(Toy))30
(1-7)*-5K
Proposition B.3. After updating the policy for K steps according to the method described in Eq. the policy cost converges to boundary.
(
(
B[C(TTO)) ) Es 5
(17)
K
Proof. After K updates of the policy, the cost constraint relationship is expressed follows:
E E[ (oy c
Le (ToR)) + Σ (ToT))
-
Σ E INA
1=0 KEN
NEN.
Ca (ToT)) + E CT(TT))) -
2 Σ
C
20 NV II
1= KEN
REN,
1=0 KEN
REN,
En EKEN.
ΣKEN,
Co. (TOO)) -
mm +
(T(())) -
110 N
[Wa [N71
-i=0
EL= EMEN.
Ce, (TOT)) -ca -ca
Well + |NG1
0 EKENe,
(CC.(TT))) Lcs (ToO)) + (Co) - -c:) 2))
E. Ne. + WNl
En EkENe, Σ Wall+
E/= EKEN., +
Le, (Toz)) Lea (Trog)
Les (Toz)) 2.
E Warl 1all+ [N+l
(18)
When the policy updated based on rewards, the policy satisfies the cost restriction fully, resulting in the inequality (i}. Eq. (18) is further simplified as shown in Eq. 19.
E B(((TTO)))
-c:
EnD EREN.
Ene EXEN.,
Las (Toz) Dos (TOz))
+
Wo + WNT Lo (TTO))
TT Wo + |NN
mm Dim
(1)
Emo ERENS,
Le, (ToO)) Len (Frog)
D We + INF
Em Din EREN.,
10 - -Qx ll2
We + [N.
(0)) EnD ELEN.
/ (1-2) (1
En Na + WNl
K
1
2
K
(19)
23

---

After K steps of policy updates from lines 14 to 17 in Algorithm 1 assuming that the condition
La (Top)) C S 0, the inequality relation of equation {i} derived. Based on the relation [70]
Lo (TOY)) Le, (TOz) ) S 11 Qx
||2, the inequality relation of equation { is derived. From the
inequality relation I|Qu Qull2 S
/(->)| |A! VK
inequality { is obtained. Based on Eq. 19).
[To() E S θ
thus proving Proposition B.3
Remark B.4. The proportion (C/A ratio) between the update iterations allocated to reward maxi- mization and cost minimization influences the final optimal policy TEI (l[((To)))
Proof. Based on Eq. (6). the relationship between the final policy Toz after k iterations and the optimal policy Tep is derived.
( - EE(C [E(((TTO))))
A
$ (1 11-))15/
Esns (DEL (Tă, ((TTg) DKL(TT)) (10,11100))))) + 2Re max + +3+ 3.e
Σ T8 log L8n
Σ
Top log
+ 2 max +3+ 3 Rex
(1 (1-))18VK
T8o
T8b+1
VISIA
T
log
log
+ 2e max + 3Re max
(1 (1-1) (1-))15/
T8O
T T
L88
VISIA
Tppt1
Σ
B log
2 max
+ 3 3.emax
(1 - ->)
TT
VISIA
S
p log
(2Re R +3 (3e3memax))
(1 (1-7)L5VK
TTC.
(1.5)
(20)
In the above Eq. 20 S.A.y, and Remax are constants. Therefore, the deviation between the policy
after the K update steps and the optimal policy depends not only on ViS||A| but also on the
(1->)1.3/K)
policy Tot
which is closely influenced by the number of update steps allocated to the minimization
of costs and maximization of rewards. The proportion (C/A ratio) between the update iterations allocated to reward maximization and cost minimization influences the final optimal policy. We Σ. [Nr. define C/A ratio= Σ. N., as the proportion of update steps dedicated to cost minimization relative to the total number of policy, and conduct comprehensive evaluations of its effect through ablation experiments.
C Practical Algorithm
To facilitate the understanding of the CHPO algorithm's implementation, we provide its pseudo-code shown in Algorithm After estimating the reward and cost advantage functions Áī and Áç we first employ Eq. (6) and Eq. 7 update the cost state-value function dc, (s) and the reward state-value function Vg, (s). respectively. Subsequently, to more accurately assess unsafe situations, we compute the cost return without the discount factor ry as L., (Tgz)) = E,-rat IE Ci,1] in Eq. (5) to decide whether cost constraints are satisfied. When the cost constraints are satisfied, the parameterized policy ne, is updated according to Eq. (12) with the objective of maximizing the reward. the cost constraints are not satisfied, the update objective of the parameterized policy ma, is to minimize the cost through Eq. (13). In summary, we evaluate whether the cost constraints are satisfied and update the policy network parameters 0p. including 0g and O., in the policy update step.
24

---

Algorithm CHPO
Input:
The tuple Ap, C.P,r.po.7)
2: Output: Policy networks parameters Op including discrete policy parameters 02 and continuous
policy parameters 0z:
3: Parameters for the reward and cost state-value Φr.Pc.
4: for each epoch do
5: Run the policy No, for T timesteps, collecting the buffer D = {(s,{{ as, ae a,, 1,
6: for each batch do
7: Sample batch of (st; Od,t, Se,t, rt,, Ci,e) from buffer D;
8:
Estimate reward and cost advantages Až and A6;; I/ State-value update step:
9:
10:
Update the cost state-value de; via the Eq. (6) Update the reward state-value Ø, via the Eq. 7 II Policy update step:
11:
13:
Compute the cost return without the discount factor Le, (TTO)) = E7,x, [Ei= Ci,]; L., (To²) I C3 then
14:
15:
The policy TH, is updated by maximizing the reward as described in Eq. 12
16:
else
17:
The policy Tæ, is updated by minimizing the cost through Eq. 13 end if
19:
end for
20:
end for
D Experimental Details
D.1 Task
Task. To evaluate the performance of the CHPO algorithm in various safety tasks with hybrid action spaces across different domains, we select three widely adopted tasks including Moving (2] 22 33 60), Sliding [2] (60, and HardMove [22] 33 60 from DI-engine [60] and create a Parking task requiring safety considerations as the experimental tasks in this work. To the best of our knowledge, these tasks the DI-engine are not well-suited for constrained hybrid-action RL tasks. Therefore, to support the further study of constrained hybrid-action RL algorithms, we incorporate danger areas and costs into these tasks of DI-engine. Additionally, to enhance the comprehensiveness of the experiments, design custom Parking task that involves selecting both discrete and continuous actions while ensuring collision avoidance. The detailed descriptions tasks are provided below:
Moving, In this task, the goal of the agent is to navigate to the target area while avoiding dangerous areas. The agent can choose from discrete actions such as turn, accel, or break, which are combined with two continuous parameters acceleration value and steering angle to determine its movement. The movement of the agent is always in the direction of its current direction. the agent reaches the target area, moves out of the field, or exceeds the maximum step 200. The reward is calculated based on the reduction in distance to the target area between consecutive steps. Additionally, the agent receives bonus reward if it successfully stops within the target area. A cost is incurred each time the agent enters the dangerous areas. The task parameters are as follows: the field is square with side length of 2, the target area is circle with radius of 0. and the danger zones are multiple circles, each with a radius of 0.07.
episode ends
Sliding. In this scenario, the parameterized action spaces available to the agent and the objective of the agent are identical those of the Moving task. The underlying physics in the Sliding task differs, as it considers the conservation of inertia, whereas the Moving task does not account for inertia conservation. That is to say, the movement of the agent is determined by the vector sum of two polar vectors, influenced by the current action and the previous movement of the agent. In this task, the agent incurs a cost each time enters a dangerous area. Aside from this, all parameters and rewards are same between the two tasks.
25

---

Cosl of episodee Roward apisode
Moving
Sliding
HardMove
Parking
iterations("leh
(terations(1155)
iteaations(1e55)
Figure 4: The figure depicts the reward and cost curves of CHPO and the baseline algorithms in the comparison experiment on different cost limits showcase the performance of each algorithm. The
shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds. The cost limit of the four tasks is set & 1.5.
Moving
Sliding
HardMove
Parking
Cosl of episode. Reward of epioode
6.
-03
Exro
CIP
CXIP
INEOR La
Ierationss1ee5)
[terations(*1e5)
Iterations(1e5)
Ieeaations(1e5)
Figure 5: The figure depicts the reward and cost curves of CHPO and the baseline algorithms in the comparison experiment on different cost limits to showcase the performance each algorithm. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds. The cost limit of the four tasks is set at č =
HardMove. The task is contiiuc- hybrid action space RL task designed to evaluate the performance of RL algorithms in high-dimensional hybrid action spaces. In this task, the agent controls n uniformly distributed actuators with the objective of using their actions to move the agent to the target area while avoiding dangerous areas. Each actuator has two options- on and off resulting in 2" discrete actions. For actuators that are turned on, the agent must select continuous parameters specifying the distance to move. As the dimensionality of the discrete action space increases exponentially with n, and each discrete action is intricately linked to its corresponding continuous parameter, the task presents significant challenges for the scalability and adaptability of constrained hybrid-action RL algorithms. The reward comprises two components: the first is based on the reduction in distance to the target area, and the second is an additional bonus awarded upon reaching the target area. The agent receives a cost for entering the dangerous areas. The episode terminates if one of the three conditions is filled: the agent stops inside the target area, the agent leaves the field, or the step count is higher than the limit 25. In this task, the field is square with a side length of 2, the target area is circle with radius of 0. and the dangerous areas are represented by multiple circles with radius of 0.1.
The task simulates an RS-curve parking scheme, a commonly used parking method in confined spaces. In the RS-curve parking scheme, discrete actions correspond to the selection of specific curve types, while continuous actions represent the distance traveled along the chosen curve. The scheme effectively models the decomposition of actions required for the parking process. The RS-curve parking trajectory is composed of a series of standardized curves including straight segments and are segments. Based on the current position of the vehicle and the target parking spot, an appropriate curve combination is selected and these curve types form the discrete action spaces, After selecting curve type, the movement distance of the vehicle along the curve needs be determined, which constitutes the continuous action spaces. The objective is to park the vehicle in the target area and avoid collisions with barriers. The rewards in the Parking task include the reduction in distance to the target area, the reduction in heading deviation, and a reward for successful parking. The costs include the distance between the vehicle and the barriers, along with a penalty incurred for collisions with barriers. An episode terminates when the vehicle either successfully completes the
Parking,
26

---

Sliding
Sliding
Parking
Parking
of upioode
Cost of apizede
Cost of eptcode
Ceons Di
Iterations-1ee)
Iterations(11e5)
Iterations: 11e5)
Figure The figure depicts the reward and cost curves in ablation experiments regarding the constraint module. The HPO represents CHPO without the constraint module. The shadowed areas represent the variance of the test results for three random seeds and the cost limit set at C; Sliding
Moving
HardMove
Parking
20
1.5
3.5
Reward
2.0
-0.
33
0.
-1.0
0:35
9.75
0.25
0.35
0.0
Cost of apicdd
0:95
0.30
0.25
0.20
0:35
0.20
2.25
0.50
0.35
5.22
0.25
0.30
0.35
0.40
0.30
0.25
2.20
C/A ratio
C/A ratio
C/A ratio
Figure 7: The line chart with variance representations illustrates the performance of CHPO under different ratios of cost minimization policy update counts to all policy update counts (C/A ratio). The results averaged 120 episodes from 3 random seeds and the cost limit of the four tasks is set at & = Each task is set with four different C/A ratios based on the actual training conditions.
Moving
Sliding
HardMove
Parking
of apizode Roward of splade
Lak Mi to Cal
1BC CA
[terations(11e5)
[terations(1ee5)
Iterations(1ee5)
Iterations
Figure 8: The figure depicts the reward and cost curves of CHPO under different ratios of cost minimization policy update counts to all policy update counts (C/A ratio). Each task is set with four different C/A ratios based on the actual training conditions. The shadowed areas represent the variance of the test results for three different random seeds and the cost limit is set at ē; = 1.
task, goes out of bounds, or exceeds the maximum step limit of 50. In this task, the dimensions the vehicle specified follows: width of 2, length of 4.3, and front-to-reat axle distance of 2.7. The turning radius of curves set to 5, while the parking area has a width of 2.5 and a length of 5.3.
D.2 Experimental Results
To facilitate the analysis of various performance trends during the algorithm training process, we record the reward and cost curves for CHPO and baseline algorithms throughout the training process. These experimental results serve as supplement to the comparative and ablation experiments in the manuscript, offering a clear understanding of the CHPO's performance during the training process.
Performance on different cost limits. Fig. and Fig. illlustrate the testing curves of the CHPO algorithm and the baseline algorithms for the four constrained hybrid-action RL tasks under different cost limits. As shown in the figures, the test curves of CHPO during training demonstrate stable behavior, with the cost curves under different cost limits consistently remaining within the safety constraints. Additionally, compared with the baseline algorithms, the cost curves of CHPO are positioned near their respective cost limits, indicating that the CHPO algorithm effectively adapts to the various constraints without overfitting. At the same time, the cost limit ē increases from
27

---

Table 2: The hyper-pas ppermparaeeters of the CHPO algorithm model. Where s, and de denote the dimensions the state, discrete action, and continuous action respectively. The batch size set to 320 for the Moving and Sliding tasks, and 64 for the HardMove and Parking tasks.
Sort
syper-parameters
Setting
Number of neurons
2x2 128 62864XX4X ReLu
Activation function
State-value(r)
Number of networks
Learning rate Optimizer
3.00e-04 Adam
Number of neurons
8x 128 x 64 x4xx6xx ReLu
Activation function
State-value(ss)
Number of networks
Learning rate
3.00e-04
Optimizer
Adam
Number neurons(encoder)
8x 256 x x1
Number (Ta)
encoder x 64 aa ad encoder 64 0. ReLu
Number of neurons(n.))
Policy(mp)
Activation fanction
Number of networks
Learning rate
3.00e-04
Optimizer
Adam 320 or 64
Batch size
Others
Discount factor Clip ratio
0.99
0.2
1 to 2, the reward curves of the CHPO algorithm exhibit noticeable improvement across all four tasks compared to the baseline algorithms. The above results demonstrate that CHPO can address constrained hybrid-action RL tasks under various cost constraints, delivering competitive rewards while ensuring compliance with safety constraints.
Performance with and without the constraint module. Fig. 6 illustrates the reward and cost curves for the CHPO algorithm and the CHPO algorithm without the constraint module in the Sliding and Parking tasks. From the results in the figure, it can be observed that HPO achieves higher rewards in tasks with parameterized action spaces by disregarding safety constraints, leading to significant violations. In contrast, the CHPO algorithm effectively learns policies that adhere safety constraints while maintaining a reasonable trade-off in reward. The above analysis of results shows that CHPO successfully handles safety constraints in hybrid action spaces and provides acceptable rewards.
Performance on different C/A ratios. Fig. & iillustrates the testing curves of the CHPO algorithm for the four constrained hybrid-action RL tasks under different ratios of cost minimization policy update counts to all policy update counts (C/A ratio). The results shown in the figure reveal that smaller C/A ratios often fail satisfy safety constraints, while larger C/A ratios, although effective in meeting safety constraints, tend to induce significant fluctuations in the reward curves. Furthermore, we observe that across all four tasks, the CHPO algorithm can learn policies that satisfy safety constraints and deliver satisfactory rewards under a wide range of C/A ratios. In addition, the mean and standard deviation results presented in Fig. also illustrate that CHPO can achieve satisfactory rewards while ensuring that the average cost satisfies the safety constraint within a broad range of C/A ratios.
D.3 Experimental Setting
Experiments are run on machines that consist of AMD Ryzen Threadripper 3960X cores and RTX 3090. We provide a detailed explanation of the experimental tasks in Section D.I Table 2 displays the parameters of the neural network model utilized in our CHPO algorithm. Additionally, detailed configuration information for the testing environment is provided in the code appendix. You can refer to the README file in the appendix code for instructions on installing and configuring the training and testing environment for the CHPO model.
E Impact and Limitation Statements
This paper presents work aiming to advance the field of hybrid-action RL, and we believe the study can significantly benefit constrained hybrid-action policy optimization for RL. Moreover, this work can provide theoretical foundation for certain applications, such as
Impact Statements:
28

---

parameterized safe path planning in autonomous driving and parameterized safe planning and control for robots (agents). To the best of our knowledge. our work cannot be misused in any way to cause any form of negative social impact.
Limitation Statements: The study currently focuses online interactions within simulation experiments, which often suffer from low data sampling efficiency. In the future, we aim to deploy our method in real-world applications and explore offline RL methods for constrained hybrid-action policy optimization.
29


Reviews:
[{"review_id": "X7n6qOwAej", "reviewer": "Reviewer_iyh9", "summary": "This paper addresses the problem of RL in environments with hybrid action spaces under safety constraints. In real-world applications like robotics and autonomous driving, traditional hybrid-action RL methods focus mainly on maximizing reward and often fail to ensure safety requirements in complex parameterized action spaces. The authors try to solve the problem by incorporating the safety constraints and using online RL to solve the constrained optimization problem.", "questions": "None", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strength\n- The authors formalize the Constrained Parameterized-action Markov Decision Process, which models RL tasks involving both hybrid action spaces and cost constraints. The idea is novel.\n\n- The authors propose Constrained Hybrid-Action Policy Optimization, a new RL algorithm that can optimize policies in hybrid action spaces while strictly enforcing safety constraints. The proposed algorithm adopts a dual actor-critic architecture, and uses a direct policy gradient approach to avoid unstable Lagrange multiplier tuning.\n\n- This paper provides theoretical guarantees for the convergence and safety of CHPO, showing that it learns policies that maximize reward without violating the cost constraints. Experiments on four tasks show that CHPO outperforms existing baselines in balancing reward maximization and safety costs.\n\n# Major weaknesses\n- Although online RL under safety constraints can solve some problems, the probability of risk occurrence at the initial state cannot be avoided, which is important in the real world since some occurrences of risk may make the training process stop. Moreover, as Figures 2, 4 and 5 show, the costs of CHPO are larger than the baselines and limitations for a long iteration, which may be intolerable in the real world. The usage in the real world may be limited. In my opinion, offline RL algorithms can achieve good performance in the tasks with safety constraints. From this perspective, the novelty of the motivation may be limited.\n\n- The results in Table 1 show that CHPO can maximize the average reward under the safety constraints, but the variance may be too large, leading to the costs surpassing the limitations. As Figure 2 shows, CHPO can achieve much larger rewards when the costs surpass the limitation, which means the results may not be strong enough to confirm the theory of the authors.\n\n- The baselines are out of date, with the most recent year being 2019. Is there any new method?\n\n# Minor weaknesses\n- The computation of the reward advantage function in Line 190 can be described in the appendix.", "was_revised": true, "final_justification": "The authors responses have addressed many of my concerns, so I raise my score  to 4.", "rebuttal": "We appreciate the time and effort you are dedicated to providing feedback on our paper and are grateful for the meaningful comments.\n\n**Q1: The probability of risk occurrence at the initial state cannot be avoided.**\n\nThank you for bringing our attention to the offline setting. You raise a highly valuable and thought-provoking question. Safety constraints in hybrid action spaces are also in high demand in real-world applications, such as in gaming[1][2] and in recent IL + Online RL studies[3][4]. Our work simultaneously addresses the problem of safe policy optimization in hybrid action spaces, providing both a theoretical foundation and technical support for future research on constrained hybrid-action policy optimization in offline settings. Moreover, real-world data is used to build high-fidelity scenarios through world models, providing a feasible paradigm for online closed-loop interactions in safe RL. This makes it possible to sample from real-world environments with high safety requirements.\n\nThe relatively high cost of CHPO in a long iteration is intentional to ensure global optimality. We incorporate the C/A ratio during training to ensure that the cost decreases gradually, allowing the algorithm to explore and achieve the maximum reward while satisfying the safety constraints. Furthermore, if higher policy safety is desired, we adjust the hyperparameters to balance the trade-off between reward and safety constraints, thereby ensuring strict adherence to safety constraints.\n\n**Q2: The variance may be too large.**\n\nThanks for the kind suggestion. Although the theory guarantees the safety of the policy, in practical implementations the bias in the value estimation of the cost value network occasionally cause the policy to exceed the cost threshold. However, if strict adherence to the cost constraint is required, our algorithm achieves this by tuning the hyperparameters to ensure that the final policy strictly satisfies the cost constraint. Furthermore, to enhance the rigor of our statements, we have adopted your suggestion and revised the manuscript to clarify that “CHPO maximizes the average reward while ensuring that the average cost across multiple seeds satisfies the safety constraint.”\n\n**Q3: The baselines are out of date.**\n\nThank you for your thoughtful review. The baselines we use are classical approaches, which have also been referenced by other recent studies [5][6]. Since there is no existing algorithm specifically for constrained hybrid-action RL, we combine well-established hybrid-action RL methods with stable constrained RL algorithms.\n\n**Q4: The computation of the reward advantage function in Line 190 can be described in the appendix.**\n\nWe appreciate your careful reading. We provide the computation of the reward advantage function in the appendix: $\\hat{A}^{r}_t=-V^{r}(s_t)+r_t+\\gamma  r\\_{t+1}+\\cdots+{\\gamma}^{T-t-1}r\\_{T-1}+{\\gamma}^{T-t}V^{r}(s_T)$.\n\n**Reference**\n\n[1] Hausknecht, M., & Stone, P. (2015). Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143.\n\n[2] Fan, Z., Su, R., Zhang, W., & Yu, Y. (2019). Hybrid actor-critic reinforcement learning in parameterized action space. arXiv preprint arXiv:1903.01344.\n\n[3] Gao, D., Wang, H., Zhou, H., Ammar, N., Mishra, S., Moradipari, A., ... & Zhang, J. (2025). IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning. arXiv preprint arXiv:2505.10442.\n\n[4] Albaba, M., Christen, S., Langarek, T., Gebhardt, C., Hilliges, O., & Black, M. J. (2024). Rile: Reinforced imitation learning. arXiv preprint arXiv:2406.08472.\n\n[5] Niu, Y., Xu, J., Pu, Y., Nie, Y., Zhang, J., Hu, S., ... & Liu, Y. (2021). Di-engine: A universal ai system/engine for decision intelligence.\n\n[6] Zhang, R., Fu, H., Miao, Y., & Konidaris, G. (2024). Model-based reinforcement learning for parameterized action spaces. arXiv preprint arXiv:2404.03037."}, {"review_id": "GxoG79jf0J", "reviewer": "Reviewer_33v4", "summary": "The paper studies constrained hybrid-action reinforcement learning, where the goal is to learn a safe policy within a parameterized hybrid action space. The authors propose a novel Constrained Hybrid-action Policy Optimization algorithm (CHPO), which optimizes policies directly using a primal approach without Lagrangian multipliers. Experiments on multiple tasks show that CHPO achieves higher rewards while strictly satisfying safety constraints and outperforming existing baselines.", "questions": "What the connection is between the proposed algorithm and the hybrid-action setting in this paper?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well written. The algorithm is simple but intuitively correct and effective.\n2. The paper provides a convergence analysis, which helps understand the stability and reliability of the proposed CHPO algorithm.\n3. The authors provide enough experiment results to illustrate the advantages of their algorithm compared to existing works.\n\n\n**Weaknesses**\n\n1. The algorithm requires two actor/critic networks, which doubles the computational burden and makes it difficult to apply to large neural networks, i.e., LLMs.\n2. This paper does not demonstrate any algorithmic justification for using a “hybrid-action” setup. Essentially, $\\theta_d$ and $\\theta_c$ can be combined and treated as a single continuous policy. The authors need to explain what is special about their hybrid setting.\n3. The authors should compare to existing contained PPO algorithm [1] and discuss the difference to it.\n\n\n[1] Jayant, A. K., & Bhatnagar, S. (2022). Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems, 35, 24432-24445.", "was_revised": true, "final_justification": "Most of my concerns are addressed.", "rebuttal": "We sincerely thank the reviewer for their insightful feedback, and we’re grateful for the opportunity to improve and clarify our work through answering your detailed questions and concerns.\n\n**Q1: The algorithm requires two actor/critic networks, which doubles the computational burden and makes it difficult to apply to large neural networks, i.e., LLMs.**\n\nThank you for catching this. The addition of the cost value estimation network indeed increases some computational overhead. However, in our study, the policy network is not composed of two independent networks. Instead, the discrete and continuous action policies share a common backbone network, with two separate output heads generating the discrete actions and the continuous parameters, respectively.\n\nOur work primarily focuses on the constrained policy optimization problem for agents that take structured parameters as input and output actions in hybrid action spaces. If this approach is extended to large neural networks, the constrained policy optimization of our algorithm is still applicable. By leveraging methods such as GRPO [1] and replacing value estimation networks with statistical sampling, the critic networks are removed.\n\n**Q2: This paper does not demonstrate any algorithmic justification for using a “hybrid-action” setup.**\n\nThanks for the comment. Hybrid-action RL algorithms are designed to handle special scenarios where discrete actions and their associated continuous parameters are selected simultaneously. Conventional RL is limited to either purely discrete or continuous action spaces and cannot address hybrid-action problems. In real-world scenarios, such as a soccer game, the agent selects a discrete action (e.g., dash, turn, tackle, or kick) and determines continuous actions such as power and direction to successfully perform a shooting task. Conventional RL methods do not simultaneously handle the selection of discrete actions and the determination of continuous actions. In contrast, hybrid-action RL policy outputs both the selected discrete action and its corresponding continuous parameters simultaneously.\n\nThe parameters $\\theta_d$ and $\\theta_c$ are not treated as a single continuous policy. Unlike PADDPG, which converts discrete actions into continuous action spaces, our policy network shares a single backbone network with two distinct output heads: one outputs the discrete actions and the other outputs the continuous actions. The differences between the two actor networks arise solely from their separate output heads, and we denote their parameters as $\\theta_d$ and $\\theta_c$ to distinguish them. Since the optimization objectives of $\\theta_d$ and $\\theta_c$ are aligned, we unify them under the parameter $\\theta_p$ to jointly update both discrete and continuous policies.\n\nIn our constrained hybrid-action setting, the agent not only selects both the discrete action and the continuous action simultaneously but also satisfies safety constraints. For example, in the *Parking* task, the agent needs to determine the curve type and its arc length while avoiding collisions with surrounding obstacles. \n\n**Q3: The authors should compare to existing contained PPO algorithm [2] and discuss the difference to it.**\n\nThanks for the kind suggestion. Compared with the existing contained PPO algorithm, our approach handles constrained policy optimization in hybrid action spaces. The existing contained PPO algorithm focuses on maximizing rewards under safety constraints within a single action space and does not account for hybrid-action scenarios. In contrast, our method achieves constrained policy optimization in discrete–continuous hybrid action spaces. To clearly highlight the differences from [2], we have revised the related work section of the manuscript to explicitly discuss that the “existing constrained PPO algorithm [2] focuses on constrained policy optimization in a single action space and does not consider hybrid-action scenarios.”\n\n**Q4: What the connection is between the proposed algorithm and the hybrid-action setting in this paper?**\n\nBuilding on the foundation of constrained hybrid-action spaces, our work further addresses the critical issue of safety. CHPO integrates cost constraints into the hybrid-action policy optimization process, ensuring that the policy not only achieves high rewards but also satisfies safety constraints when jointly optimizing discrete actions and continuous actions. In our hybrid-action setting, the agent simultaneously considers both the selection of discrete actions and the determination of continuous actions while ensuring safety. As illustrated in Fig. 1 of the manuscript, conventional hybrid-action RL methods cannot learn safe policies in discrete-continuous hybrid action spaces. Our proposed algorithm specifically targets the problem of constrained policy optimization in the hybrid-action setting.\n\n**Reference**\n\n[1] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., ... & Guo, D. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.\n\n[2] Jayant, A. K., & Bhatnagar, S. (2022). Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm. Advances in Neural Information Processing Systems, 35, 24432-24445."}, {"review_id": "rWSRh2xUkK", "reviewer": "Reviewer_jBVV", "summary": "The paper introduces Constrained Hybrid-action Policy Optimization (CHPO), a novel algorithm for reinforcement learning in parameterized action spaces under safety (cost) constraints. The authors first formulate the problem as a Constrained Parameterized‐action Markov Decision Process (CPMDP), extending the standard PAMDP framework to include cost functions alongside rewards. They then propose a two‐critic, two‐actor hybrid actor–critic architecture: one critic and actor pair for rewards, and another for costs. By dynamically switching between reward maximization and cost minimization depending on whether the current policy violates safety thresholds, CHPO directly optimizes the constrained objective without relying on auxiliary Lagrange multipliers. Theoretical analysis proves that CHPO converges to an optimal safe policy within a bounded error that scales with the state/action dimensionality and number of updates. Empirically, CHPO consistently maintains costs below the specified limits while achieving higher or comparable rewards relative to four baselines.", "questions": "1. What is the run-time/memory overhead of maintaining two critics and two actors compared to standard PAMDP methods?\n2. Evaluations are limited to four relatively low-dimensional tasks; performance on more complex robotics or vision-based environments remains untested.\n3. Convergence bounds assume accurate value estimates and tabular |S|,|A|. However, since |A| includes continuous parameters, how is this handled in your analysis?\n4. In theoretical analysis, the author use the notation of |A| and $\\pi_{\\theta_p}$, which correspond to a standard MDP formulation. How is the PAMDP structure incorporated into the analysis?\n5. What is the backbone of CHPO? Is it based on HPPO, PDQN, or PADDPG?\n6. In the empirical section, the authors compare CHPO with HPPO-Lag, PADDPG-Lag, and PDQN-Rco. Why were HPPO-Rco, PADDPG-Rco, and PDQN-Lag not included for a more balanced comparison?", "limitations": "Yes, in Appendix E.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- First to define the CPMDP framework for constrained hybrid‐action RL, neatly integrating cost constraints into PAMDPs.\n- Rigorous theoretical bounds lend confidence in the stability and safety of learning.\n\n\nWeaknesses\n- The method relies heavily on the PAMDP structure, but the theoretical analysis does not appear to explicitly address or leverage this structure.\n- The evaluated environments lack diversity and are quite similar in nature, which limits the generalizability of the results.", "was_revised": true, "final_justification": "I would keep my original score of “borderline accept,” as my major concerns remain: their theory does not leverage the special architecture of PAMDPs, and they ultimately did not present experimental results on the current SOTA PAMDP algorithm, DLPA. That said, this is the first paper to consider safety RL in the PAMDP domain, which does provide a degree of novelty.", "rebuttal": "Thank you for your valuable feedback. We appreciate the time and effort you have dedicated to reviewing our paper. We will address the weaknesses and questions you raised below.\n\n**Q1: The theoretical analysis does not appear to explicitly address or leverage PAMDP structure.**\n\nThanks for the comment. Based on PAMDP, we divide the policy network into a discrete policy $\\pi_{\\theta_d}$ and a continuous policy $\\pi_{\\theta_c}$, which share the same backbone but differ in the output heads: one outputs discrete actions, while the other outputs continuous actions. For clarity, we denote their parameters as $\\theta_d$ and $\\theta_c$, respectively. Since $\\theta_d$ and $\\theta_c$ share the same optimization objective, we unify them as $\\theta_p$. Our theoretical analysis is consistently conducted on the hybrid policy parameter $\\theta_p$ derived from PAMDP. We define $|\\mathcal{A}| := |\\mathcal{A}_d| + |\\mathcal{A}_c|$, which unifies the discrete and continuous action spaces in PAMDP into $|\\mathcal{A}|$ for theoretical analysis. We have further clarified in the paper the relationship among $\\theta_p$, $|\\mathcal{A}|$, and PAMDP in theoretical analysis.\n\n**Q2: The evaluated environments lack diversity and are quite similar in nature.**\n\nThanks for the kind suggestion. Currently, there is no existing simulation environment specifically designed for the constrained hybrid-action setting, so we have to modify and construct tasks for evaluation. We are applying our approach to real-world exploration environments, specifically to the RM65 robotic arm, where high-level motion primitives (fixed discrete actions) are manually designed, such as Lift (raising the end-effector upward) and Twist (rotating the gripper or object along a certain axis). The execution details of each discrete action are controlled by continuous parameters. We are in the process of applying CHPO to this robotic arm to accomplish hybrid-action tasks while ensuring safety.\n\n**Q3: What is the run-time/memory overhead of maintaining two critics and two actors compared to standard PAMDP methods?**\n\nThank you for bringing our attention to the efficiency. We have recorded the run-time and memory usage of the standard PAMDP method (HPPO) and CHPO, as shown in the table below. Additionally, we have supplemented the run-time and memory usage of other baseline algorithms.\n|   Method   | Runtime | Memory | Parameter count |\n|:----------:|:-------:|:------:|:---------------:|\n|    HPPO   |  1.66ms |  490MB |      113222     |\n|  HPPO-Lag  |  2.42ms |  539MB |      167623     |\n| PADDPG-Lag |  1.02ms |  481MB |      22599      |\n|  PDQN-Rco  |  1.51ms |  473MB |      67823      |\n|    CHPO    |  2.43ms |  503MB |      167623     |\n\n**Q4: Evaluations are limited to four relatively low-dimensional tasks; performance on more complex robotics or vision-based environments remains untested.**\n\nThank you for pointing this out. Our experiments are currently limited to low-dimensional tasks because there is no existing simulation environment for the hybrid-action settings. We modify and build tasks for evaluation. We are actively applying our approach to real-world RM65 robotic arm exploration. In this setting, the discrete actions of the robotic arm include operations such as lift, twist, and grasp, while the continuous actions correspond to the execution parameters for each discrete action. We perform constrained hybrid-action policy optimization on the robotic arm to accomplish tasks while ensuring safety.\n\n**Q5: Convergence bounds assume accurate value estimates and tabular |S|,|A|. However, since |A| includes continuous parameters, how is this handled in your analysis?**\n\nAccording to related work [1], the boundary of the critic network is related to the dimensions of $|\\mathcal{S}|$ and $|\\mathcal{A}|$, and the relationship holds in both discrete and continuous spaces $||Q_k - \\hat{Q}_k ||_2  \\leq \\frac{\\sqrt{(1-\\gamma)|\\mathcal{S}||\\mathcal{A}|}}{\\sqrt{K}}$. Building on [1][2], this work further defines $|\\mathcal{A}| := |\\mathcal{A}_d| + |\\mathcal{A}_c|$, which remains applicable to the convergence analysis of our method. We have added clarifications regarding this in the revised manuscript.\n\nMoreover, our work is based on tabular-form derivations, primarily focusing on analyzing and illustrating the decisive parameters affecting the convergence boundary of the CHPO algorithm. Therefore, we have not derived the convergence for network-based representations. If such a derivation is required, one directly replaces the parameterized representation with the weights and biases of the policy network, in which case the convergence boundary will incorporate the estimation error of the policy network. We have added this clarification to the revised manuscript.\n\n**Q6: How is the PAMDP structure incorporated into the analysis?**\n\nThanks for the comment. Based on the PAMDP framework, the hybrid action space consists of a discrete action space $\\mathcal{A}\\_d$ and a continuous action space $\\mathcal{A}\\_c$. We define $|\\mathcal{A}|:= |\\mathcal{A}\\_d| + |\\mathcal{A}\\_c|$ to represent this hybrid action space for subsequent theoretical analysis. Under the PAMDP framework, we employ a discrete policy $\\pi_{\\theta_d}$ to output discrete actions and a continuous policy $\\pi_{\\theta_c}$ to output continuous actions. In the implementation of the CHPO algorithm, these two policies share the same backbone network but are distinguished at the output heads to reduce computational overhead. Since the optimization objectives of $\\pi_{\\theta_d}$ and $\\pi_{\\theta_c}$ are identical, we unify them into the hybrid-action policy $\\pi_{\\theta_p}$ for the subsequent theoretical analysis, where $\\theta_p$ is composed of $\\theta_d$ and $\\theta_c$.\n\n**Q7: What is the backbone of CHPO? Is it based on HPPO, PDQN, or PADDPG?**\n\nThank you for catching this. CHPO is built upon HPPO. We chose HPPO because, among the three algorithms, HPPO demonstrates superior performance in handling purely hybrid-action tasks. PADDPG addresses hybrid-action tasks by converting the discrete action space into a continuous one, while PDQN separately applies DQN for discrete actions and DDPG for continuous actions. Compared to HPPO, both PDQN and PADDPG involve more complex handling of hybrid action spaces and show inferior performance. We also evaluate the performance of these three algorithms in terms of reward maximization without cost consideration. The results averaged over 3 seeds in *Moving* are as follows:\n| Method |    HPPO   |   PADDPG  |    PDQN   |\n|:------:|:---------:|:---------:|:---------:|\n| Reward | 1.86±0.28 | 1.47±0.54 | 1.64±0.31 |\n\nThese results indicate that HPPO performs better on hybrid action tasks when safety constraints are not considered.\n\n**Q8: Why were HPPO-Rco, PADDPG-Rco, and PDQN-Lag not included for a more balanced comparison?**\n\nThank you for catching this. PDQN cannot be combined with the Lag method to form PDQN-Lag because PDQN consists of two separate components: DQN for selecting discrete actions and DDPG for selecting continuous actions, making it infeasible to integrate the Lag method into both parts simultaneously. Therefore, we choose to combine PDQN with RCPO to form PDQN-Rco. In addition, we combine both HPPO and PADDPG with RCPO and conduct tests in the *Parking* environment. The comparative results under the cost limit $\\bar{c}_i = 1$ are as follows:\n| Method | HPPO-Rco    | PADDPG-Rco  | HPPO-Lag    | PADDPG-Lag  | PDQN-Rco   | CHPO                |\n|--------|-------------|-------------|-------------|-------------|------------|---------------------|\n| Reward | 20.04±22.04 | 21.74±16.02 | 20.01±25.21 | 26.66±16.39 | 32.72±7.44 | **37.63±3.04**      |\n| Cost   | 1.55±4.05   | 0.46±2.35   | 0.82±1.26   | 0.94±2.15   | 0.21±0.35  | **0.93±3.17**  \n\nTheir performance is inferior to PDQN-Rco; therefore, we ultimately select PDQN-Rco as the baseline instead of HPPO-Rco and PADDPG-Rco.\n\n**Reference**\n\n[1] Dalal, G., Szorenyi, B., & Thoppe, G. (2020, April). A tale of two-timescale reinforcement learning with the tightest finite-time bound. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 3701-3708).\n\n[2] Xu, T., Liang, Y., & Lan, G. (2021, July). Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning (pp. 11480-11491). PMLR."}, {"review_id": "xS7ASudISe", "reviewer": "Reviewer_HetW", "summary": "This paper introduces the Constrained Parameterized-action Markov Decision Process (CPMDP) framework to formalize the problem of maximizing rewards under safety constraints in such hybrid action spaces. They propose the Constrained Hybrid-action Policy Optimization (CHPO) algorithm, which employs a constrained actor-critic architecture with separate critic networks for reward and cost estimation, and dynamically adjusts policy updates. Theoretical analyses demonstrate that CHPO converges to near-optimal solutions while guaranteeing safety constraint satisfaction, with convergence bounds derived for both policy and cost.", "questions": "(1)Theoretical proofs assume accurate state-value function approximation. How does CHPO perform when critic networks have high estimation errors (common in high-dimensional or noisy environments)? Are there ablation studies on critic architecture depth or noise injection? Perhaps adding an ablation test with noise cost/reward signals (e.g., adding Gaussian noise to $c_{i,t}$) to verify the robustness can solve this problem. And please consider how the approximation error affects the derived convergence boundary (e.g., the sensitivity of $\\|Q_k - \\hat{Q}_k\\|_2$ in Proposition 4.3).\n\n(2) Acronyms \"C/A ratio\" is introduced in Remark 4.5 but not defined until Appendix B.4.\n\n(3) The paper states that CHPO is \"inspired by HPPO [2]\" but does not clearly contrast their architectures. It makes me a little bit confused to identify how CHPO differ from HPPO design.", "limitations": "The dual actor-critic architecture (discrete/continuous actors, reward/cost critics) introduces higher computational complexity compared to single-modality constrained RL methods. The paper does not explicitly analyze computational efficiency (e.g., training time, parameter count), which could be a concern for real-time applications. Baselines like HPPO-Lag or PDQN-Rco may be more lightweight, though less effective.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n(1) The paper is well-written and well-organized. Technical details (e.g., actor-critic architecture, policy update rules) are clearly articulated. \n\n(2) Adequate theories and complete proof processes reflect the theoretical rationality of this method, which is a significant advantage in a field often dominated by empirical methods.\n\nWeaknesses：\n\n （1）While the baselines (HPPO-Lag, PDQN-Rco) are adapted from existing hybrid-action RL methods, they may not represent the strongest possible constrained RL baselines for hybrid spaces. For instance, integrating modern constrained RL techniques (e.g., CVPO, RCPO) with hybrid action frameworks could provide stiffer competition. The paper’s claim of “first attempt” to address hybrid actions and constraints is valid, but stronger baselines would strengthen the novelty argument.\n\n(2) The convergence proofs rely on idealized assumptions, such as perfect state-value estimation. For example, in Lemma B.1 and Proposition B.2, assume that $V^r(s)$ and $V^{c_i}(s)$ are accurately approximated without estimation errors. The theoretical bounds of the paper (e.g., $\\Theta(\\frac{\\sqrt{|S||A|}}{(1-\\gamma)^{1.5}K})$) depend on these idealized conditions, which may limit the practical scalability.", "was_revised": true, "final_justification": "The experiments and data supplemented by the author have made the conclusion more reliable. The responses have resolved most of my doubts. I have no other questions and decide to increase the score after careful consideration.", "rebuttal": "We appreciate the thoughtful review and the constructive feedback. We hope the following clarifications and experiments address your questions.\n\n**Q1: Stronger baselines would strengthen the novelty argument.**\n\nThanks for the kind suggestion. RCPO has been covered in our baseline PDQN-Rco. The Lagrangian dual method we adopt is conceptually similar to CVPO in addressing constrained problems; however, CVPO first computes a variational distribution and then solves the parameterized network based on this distribution, which introduces additional network estimation errors and is not well-suited for constrained hybrid-action policy optimization. We have discussed CVPO in our paper.\n\n**Q2: The theoretical bounds of the paper (e.g., $\\Theta \\left(\\frac{\\sqrt{|\\mathcal{S}||\\mathcal{A}|}}{(1-\\gamma)^{1.5}\\sqrt{K}}\\right)$) depend on these idealized conditions, which may limit the practical scalability.**\n\nThanks for the kind suggestion. Although our theoretical analysis is based on the assumption of accurate value estimation, we have already incorporated scaling in the analysis, as shown in the inequalities (i), (ii), (iii), and (v) in Eq. (16). Intermittent input noise affects training stability, while persistent noise slows down the convergence rate, but it does not have a significant impact on the final results.\n\n**Q3: How does CHPO perform when critic networks have high estimation errors (common in high-dimensional or noisy environments)?**\n\nThank you for bringing our attention to the robustness. We add ablation tests with noisy cost and reward signals, where Gaussian noise is injected into the cost and reward signals to evaluate the robustness of CHPO when the critic networks exhibit higher estimation errors. The experimental results of *Parking* are presented below, showing the results under 1% and 5% persistent noise as well as random noise.\n| Gaussian noise |     1%     |     5%     |   random   |      0     |\n|:--------------:|:----------:|:----------:|:----------:|:----------:|\n|     Reward     | 37.27±3.03 | 38.17±2.85 | 37.28±3.26 | 37.63±3.04 |\n|      Cost      |  0.89±3.01 |  0.80±2.79 | 0.78±3.16  | 0.93±3.17  |\n\nBased on the above experiments, we conclude that sporadic random noise does not affect the final results but causes fluctuations during training, while persistent noise slows down the convergence rate without impacting the final stable convergence. Moreover, in theoretical analysis, we apply scaling as shown in the inequalities (i), (ii), (iii), and (v) of Eq. (16). Based on both experimental and theoretical analysis, we believe that the approximation errors do not affect the final stable convergence.\n\n**Q4: Acronyms \"C/A ratio\" is introduced in Remark 4.5 but not defined until Appendix B.4.**\n\nThanks for the kind suggestion. We have added the definition of the C/A ratio at line 246 of the manuscript, i.e.,“We define C/A ratio $= \\frac{\\sum_{i}^{m}|\\mathcal{N}{c_i}|}{\\sum_{i}^{m}|\\mathcal{N}\\_{c_i}| + |\\mathcal{N}\\_r|}$ as the proportion of update steps dedicated to cost minimization relative to the total number of policy updates.”\n\n**Q5: The paper states that CHPO is \"inspired by HPPO [2]\" but does not clearly contrast their architectures. It makes me a little bit confused to identify how CHPO differ from HPPO design.**\n\nThanks for the comment. HPPO is a policy optimization method designed for hybrid action spaces, focusing primarily on reward maximization. However, it does not account for safety constraints, which often leads to cost violations when applied to safety-critical tasks.\n\nCHPO builds upon the hybrid-action policy optimization framework of HPPO by introducing Constrained Parameterized-action Markov Decision Process (CPMDP), where safety constraints are explicitly modeled through a cost function $C$. CHPO proposes a constrained hybrid-action actor-critic architecture, consisting of two actors and two critics, where the two critics are responsible for learning the reward and cost state-value functions, respectively. During policy updates, CHPO accounts for safety constraints and rewards, employing a two-stage update mechanism of reward maximization and cost minimization, whereas HPPO optimizes solely for rewards. Although CHPO draws inspiration from HPPO in value function design, its core innovations lie in introducing CPMDP, the constrained hybrid-action actor-critic architecture, and constrained hybrid-action policy optimization, that are absent in HPPO design.\n\n**Q6: The paper does not explicitly analyze computational efficiency (e.g., training time, parameter count), which could be a concern for real-time applications.**\n\nThank you for bringing our attention to the efficiency. The training time and parameter count of CHPO and the baselines are as follows:\n\n|   Method   | Runtime | Parameter count | Memory | Convergence steps |\n|:----------:|:-------:|:---------------:|:------:|:-----------------:|\n|  HPPO-Lag  |  2.42ms |      167623     |  539MB |      ≥2.5*1e5      |\n| PADDPG-Lag |  1.02ms |      22599      |  481MB |      ≥3.5*1e5      |\n|  PDQN-Rco  |  1.51ms |      67823      |  473MB |      ≥3.0*1e5      |\n|    CHPO    |  2.43ms |      167623     |  503MB |      ≥3.5*1e5      |\n\nAlthough CHPO has a longer runtime and a larger number of parameters compared to the baselines, their memory usage is similar, and CHPO achieves better performance.\n\n**Q7: The paper uses inconsistent formats and some spaces in the citation format have been ignored.**\n\nThank you for your careful review. We have checked similar issues and revised the formatting of equation and figure references in the manuscript: “Eq. 5” has been changed to “Eq. (5)” at line 613, and “Fig.6” has been changed to “Fig. 6” at line 334. Additionally, we have added a period at the end of Eq. (20) between lines 599 and 600.\n\n**Q8: Remark 4.5 in the text uses a formatting style that does not align with the naming conventions of other structured elements (e.g., Lemma 4.2, Proposition 4.3).**\n\nThank you for your careful review. We have adjusted the formatting style of Remark 4.5 in the text to match the naming conventions of Lemma 4.2 and Proposition 4.3."}]

Author rebuttal:

Meta-review: The paper proposes Constrained Hybrid-action Policy Optimization (CHPO), addressing reinforcement learning with hybrid discrete–continuous actions under safety constraints by formulating the CPMDP framework. The strengths include good motivation, strong theoretical guarantees, and convincing experimental results showing improved reward–safety trade-offs compared to baselines. Weaknesses lie mainly in a little limited environments and reliance on stylized assumptions in the proofs, but the authors provided clarifications, ablations, and additional results during the rebuttal that mitigated these concerns. The reviewers acknowledged that most of their concerns were resolved, and some raised their scores. Overall, the novelty, analytical results, and impact justify acceptance.
Decision: Accept (poster)