Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs
Authors: Wei Hung
Shao-Hua Sun
Ping-Chun Hsieh
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement learning
action constraints
Primary area: reinforcement learning

Abstract: Action-constrained reinforcement learning (ACRL) is a generic framework for learning control policies with zero action constraint violation, which is required by various safety-critical and resource-constrained applications. The existing ACRL methods can typically achieve favorable constraint satisfaction but at the cost of either high computational burden incurred by the quadratic programs (QP) or increased architectural complexity due to the use of sophisticated generative models. In this paper, we propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifications: (i) To enforce the action constraints, we leverage the classic acceptance-rejection method, where we treat the unconstrained policy as the proposal distribution and derive a modified policy with feasible actions. (ii) To improve the acceptance rate of the proposal distribution, we construct an augmented two-objective Markov decision process (MDP), which include additional self-loop state transitions and a penalty signal for the rejected actions. This augmented MDP incentives the learned policy to stay close to the feasible action sets. Through extensive experiments in both robot control and resource allocation domains, we demonstrate that the proposed framework enjoys faster training progress, better constraint satisfaction, and a lower action inference time simultaneously than the state-of-the-art ACRL methods. We have made the source code publicly available to encourage further research in this direction.

Published as a conference paper at ICLR 2025
EFFICIENT ACTION-CONSTRAINED REINFORCEMENT LEARNING VIA SICEPTANCE-REJECTIOO METHOD AND AUGMENTED MDPS
Wei Hung' Shao-Hua Sun? Ping-Chun Hsich'
National Yang Ming Chiao Tung University, Hsinchu, Taiwan National Taiwan University, Taipei, Taiwan
[hwe11048576 csca, (inghsieh/@nycus edu.. tw
ABSTRACT
serion-onsttraiedd reinforcement learning (ACRL) is a generic framework for learning control policies with zero action constraint violation, which required by various safety-critical and resource-comee sssource-constrained applications. The existing ACRL methods can typically achieve favorable constraint satisfaction but at the cost of either a high computational burden incurred by the quadratic programs (QP) increased architectural complexity due to the use sophisticated generative mod- els. In this paper, propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifi- cations: (i) To enforee the action constraints, we leverage the classic acceptance- rejection method, where we treat the unconstrained policy as the proposal distribu- tion and derive modified policy with feasible actions. (ii) To improve the accep- tance rate of the proposal distribution, we construct an augmented two-objective Markov decision process (MDP), which includes additional self-loop state tran- sitions and a penalty signal for the rejected actions. This augmented MDP in- centivizes the learned policy to stay close to the feasible action sets. Through extensive experiments in both robot control and resource allocation domains, we demonstrate that the proposed framework enjoys faster training progress, better constraint satisfaction, and a lower action inference time simultaneously than the ACRL methods. We have made the source code publicly available* to encourage further research in this direction.
INTRODUCTION
Action-constrained reinforcement learning (ACRL) aims to find an optimal policy maximizing ex- pected cumulative return while satisfying constraints imposed on the action space and has served as generic framework for learning sequential decision making in both safety-critical and resource- constrained applications. As classic example, robot control usually subject the inherent kine- matic constraints of the robots, e.g., torque output power, which need to be satisfied throughout the training and the inference stages to avoid damage to physical components (Singletary et al., 2021; Tang et al., 2024; Liu et al., 2024). Another example is dynamic resource allocation for networked systems (Chen et al., 2023; Jay et al., 2019; Chen et al., 2021), such as communication networks and bike sharing systems (Zhang al., 2022; 2021), which involve the capacity constraints the communication links and the docking facilities, respectively. To prevent network congestion and resource ver-utilization. these constraints must be considered at each step of training and deploy- ment. Given its wide applicability, developing practical ACRL algorithms that can learn policies accruing high returns with minimal constraint violation is essential.
Existing ACRL methods have explored the following techniques: (i) Action projection: As a con- ceptually simple and widely-used technique, action projection finds a feasible action closest the original unconstrained action produced by the policy. The projection step can be used in action post-processing (Kasaura al., 2023) or implemented by a differentiable projection layer (Amos
'https:/ http:///////bb...////g
thub. com wwt.///.///./// NNCCN Handit L.ab/aab/aa ARAM

---

Published as a conference paper at ICLR 2025
& Kolter, 2017) part of the policy network of standard deep RL algorithm for end-to-end training (Pham et al., 2018; Dalal et al., 2018; Bhatia et al., 2019). Despite the simplicity, to find close feasible actions, action projection needs to solve quadratic program (QP), which is compu- tationally costly and scales poorly to high-dimensional action spaces (Ichnowski et al., 2021). (ii) Frank- Wolfe search: Lin et al. (2021) propose decouple policy updates from action constraints by Frank-Wolfe search subroutine. Despite its effectiveness, ing multiple QPs per training iteration and therefore suffers from substantially higher training time. (iii) Generative models: To replace the projection step, generative models, such as Normalizing Flows (Kobyzev et al., 2020), have been employed as learnable projection layer that is trained to satisfy the constraints (Brahmanage et al., 2023; Chen et al., 2023).
Wolfe method requires solv-
Despite the recent advancement in ACRL,, existing algorithms suffer from significant computational overhead induced by solving QPs or learning sophisticated models, such as Normalizing Flows. Table summarizes the limitations of existing methods interaction time, training load, and action violation rate. This motivates to develop more efficient ACRL approach.
Table I: A qualitative comparison of ACRL algorithms.
Algorithm
Action violation
Generative model
Interaction Training load
Remarks
time
rate
OptLayer (Pham al., 2018)
High
High
Low
Zero- Zero-gradient issue
ApprÜpt (Bhatia et ul., 2)
High
X
High
Low
Zero-grudien issue
NEWPO (Lin et al., 2021)
Low
X
Low
High
Rely heavily on QPs
FlowPG (Brahmanage et al., 2023)
Low
Low
High
Require pre-training
DPre+ (Kasaura et al., 2023)
High
X
High
Low
SPre+ (Kasaura al., 2023)
High
High
Low
IAR-A2C (Chen 2023)
Low
Low
High
Support only discrete actions
ARAM (Ours)
Low
X
Low
Low
We propose a framework called ARAM, which augments standard deep RL algorithm with two modifications: Acceptance-Rejeet Rejection method and Augmented MDPs. (i) Acceptance-reeeectio method: Given a policy network, ARAM enforces the action constraints by rethinking ACRL through acceptance-rejeccioon sampling, ie.. first sampling actions from the unconstrained policy and then only accepting those that are in the feasible action set. This sampling strategy can substan- tially reduce the need for solving QPs, compared to the methods built on the action projection step or the Frank-Wolfe search. (ii) Augmented unconstrained two-objective Markov decision process: One technical issue of the acceptance-rejeectoo method is the possibly low acceptance rate, which is likely to occur in the early training stage. Under low acceptance rate, the sampling process could take excessively long and thereby incur high training overhead. To improve the acceptance rate, we augment the original MDP with additional self-loop transitions and penalty function induced by the event whether the action accepted. Through this augmented MDP, we leverage the penalty induced by constraint violation to guide the policy distribution towards regions of higher acceptance rate. Notably, these two modifications can be combined with any standard deep RL algorithm. In this paper, we take the Soft Actor Critic (SAC) as the base RL algorithm. Moreover, to obviate the need for hyperparameter tuning of the penalty weight, we directly leverage the multi-objective extension of SAC that can learn policies under all the penalty weights.
We evaluate the proposed ARAM in various ACRL benchmarks, including the MuJoCo locomotion tasks and resource allocation of communication networks and bike sharing systems. The experi- mental results show that: (i) ARAM enjoys faster learning progress than the state-of-the-art ACRL methods, measured either in environment steps or wall clock time. Moreover, the difference is par- ticularly significant under wall clock time thanks to the low training time of our design. (ii) ARAM requires significantly fewer QP operations than the other ACRL benchmark methods, mostly by 2-5 orders of magnitude fewer. (iii) ARAM indeed achieves high action acceptance rate through the guidance of the augmented MDP. (iv) ARAM also enjoys the lowest per-action inference time as largely obviates the need for QP operations and learns without using a generative model.
2 RELATED WORK
Conit--nost RL. The first category focuses on ensuring that the actions meet the constraints at each step of the training and evaluation processes. To ensure zero constraint violation, one nat-
2

---

Published as a conference paper at ICLR 2025
ural and commonly-used technique is action projection, which finds feasible action closest to the original unconstrained action produced by the policy, To enable end-to-end training. multiple ACRL methods have incorporated the differentiable projection layer (Amos & Kolter, 2017) as the output layer of the policy network, such as Optl.ayr (Pham et al., 2018), Safety Layer (Dalal et al., 2018), and Approximate OptLayer (Bhatia al., 2019). However, these projection layers are known to suffer from slow learning in ACRL due to the zero-gradient issue (Lin et al., 2021).
To address the zero-gradient issue, several approaches have been adopted recently: (i) Frank-Wolfe search: Lin et al. (2021) propose Neural Frank-Wolft Policy Optimization (NFWPO), which de- couples policy updates from action constraints by a Frank- Wolle search subroutine. Despite its effectiveness, Frank-Wolld search involves solving multiple quadratic programs (QP) per training iteration, and this is computationally very costly and is known to scale poorly to high-dimensional action spaces (Ichnowski et al., 2021). (ii) Removing projection layers and using pre-projected actions for critic learning: Without using differentiable projection layers, Kasaura et al. (2023) pro- pose DDPG-based DPre+ and SAC-based SPre+, which use QP-based projection step for action post-procsssing, train the critic with pre-projected actions, and apply a penalty term to reduce con- straint violation. Notably, DPre+ and SPre+ achieve fairly strong reward performance but still at the cost of high computational overhead incurred by QPs. (iii) Generative models: To replace the projection layer, generative models, e-g., normalizing flow, have been integrated into the policy net- work, such as FlowPG for continuous control (Brahmanage et al., 2023) and IAR-A2C for discrete control (Chen al.. 2023), to generate multi-modal action distributions that can better satisfy the constraints. Despite the effectiveness, learning sophisticated generative model adds substantial de- sign complexity to ACRL. By contrast, the proposed ARAM largely removes the overhead of QPs and completely obviates the need for generative models.
RL for Constrained MDPs. The other class of methods focuses on ensuring the long-term av- erage action safety by defining a cost function and modeling the problem as a Constrained MDP (CMDP) (Altman, 2021). For example, Constrained Policy Optimization (CPO) (Achiam et al., 2017) is the first policy gradient method developed to solve CMDPs. It uses the Fisher informa- tion matrix and second-order Taylor expansion to ensure safety constraints, but it is computation- ally expensive and requires more samples, potentially reducing efficiency. To address these, Tessler et al. (2019) propose Reward Constrained Policy Optimization (RCPO), which leverages primal-dual methods improve both the efficiency and effectiveness of policy optimization under constraints. Building on similar goals of improving efficiency, FOCOPS (Zhang et al., 2020) takes a different approach by using a first-order approximation for policy optimization. This reduces computational complexity but introduces convergence issues due to approximation errors in the first-order con- straints. The above list is by no means exhaustive and is only meant to provide an overview of this line of research. Please refer to (Liu et al., 2021) for more related prior works, While these methods can ensure that the long-term expected cost remains under a certain threshold, they fail to enforce action constraints at every environment step needed in ACRL throughout training and deployment.
Augmented Safety Mechanisms in RL. To improve safety in exploration, Eysenbach et al. (2018) introduced reset framework, which detects when the agent enters unsafe states and resets it, thereby improving both safety and sampling efficiency. Building on this, Thananjeyan et al. (2021) intro- duced the concept of learned recovery zones, ensuring that when the agent deviates from safe limits, it can autonomously return to a safe state, providing more robust safcty guarantees during explo- ration. In parallel, Thomas et al. (2021) proposed specialized Markov Decision Process (MDP) to constrain the training process, helping to further avoid unsafe actions during exploration. In addition to mechanisms that correct unsafe actions, Safety Augmented Value Estimation from Demonstra- tions (SAVED) (Thananjeyan et al., 2020) employs model predictive control (MPC) to proactively avoid unsafe actions by updating the policy, specifically preventing infeasible actions that could lead to dangerous situations. Complementing this, Yu et al. (2022) proposed Safety Editor (SEditor), a mechanism that transforms actions produced by a utility maximizer into safe alternatives, preventing violations of safety constraints during execution. While these methods can not fully guarantee that the policy's s actions always remain within safe regions, they offer an effective approach to signifi- cantly reduce unsafe behaviors, laying foundation to develop more robust safety mechanisms.
3

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES: ACTION-CONSTRAII REINFORCEMENT LEARNING
In ACRL, we consider an action-comme constrained Markov Decision Process (MDP). Given a set X. let (x)) denote the set of all probability distributions on X' An action-constrained MDP is defined by a tuple M == (S, A. P, Y, C), where S denotes the state space, A denotes the action space, P: P:S% x A - A(S) serves as the transition kernel, ? (0. 1) is the discount factor, r S x A - IR denotes the bounded reward function. Without loss of generality, we presume the reward r(s, a) to lie in the [0, 1] interval since we can rescale a bounded reward function the range of [0, 1] given the maximum and minimum possible reward values. For each 8 E S, there is a non-empty feasible action set C(s) S A induced by the underlying collection of action constraints. That is to say, no actions outside the feasible set C(s) can be applied to the environment, ensuring that only valid actions are considered within the system dynamics. Notably, we make no assumption on the structure of C(s) (and hence C(s) needs not be convex).
At each time E N {0}. the learner observes the current state st E S of the environment, selects a feasible action a₁ E C((z),, and receives reward r,. We use π : S + (AA) to denote Markov
stationary stochastic policy, which is updated iteratively by the learner. Given a policy H, the Q
functions Q( .T) : S x A R is defined as ((s, a; T) == E Σ tt== 7'rr|so 5,00 a; m] which can be characterized as the unique solution to the following Bellman equation:
((s, 0; R) = r(s,a) a) + "B [Q(s',(':=
(1)
To learn a policy and the corresponding Q function under large state and action spaces, parameterized functions TG S ((4) and Qo : S x A + R as function approximators, where and 0 typically denote the parameters of neural networks in the deep RL literature. Our goal is to learn an optimal policy x' such that Q(s, a: x²) 2 Q(s, a; R), for all s E S,a a E C(s) and π Ilc, where 1lc := { + A(C),, which denotes the set of all feasible policies. We also use II = { : S + (AA) to denote the set of all unconstrained Markov stationary policies.
the
Notations. Throughout the paper, use (ar, y) to denote the inner product of real vectors æ, y. Moreover, we use 1 to denote the d-dimensional vector of all ones.
4 ALGORITHM
To address ACRL, we devise and introduce two main modifications to existing deep RL. algorithms: the acceptance-rejeetion method (Section 4.1) and the augmented two-objective MDP (Section 4.2). Then, we describe practical multi-objective RL implementation of ARAM in Section 4.3.
4.1 ACCEPTANCE-REJECTION METHOD
To adapt a standard deep RL method to ACRL, we need to convert an unconstrained policy na E II into feasible policy =1 Ne.. Notably, the action constraints, in general, can be complex and take arbitrary forms of expression. As result, the feasible action sets C(s) are likely to be rather unstructured. To tackle this, we propose to rethink the constraint satisfaction in ACRL through the classic cereppance-reecction method (ARM), which is generic algorithm for sampling from general distributions (Kroese et al., 2013).
Using ARM in the context of ACRL. For didactic purposes, here focus the continuous control and assume that the action space A is a compact convex set despite that the same argument work seamlessly under discrete action spaces. Let f and g be two probability density functions over A. ARM can generate random variables that follow the target distribution f while drawing samples from another proposal distribution g. To put ARM in the context of ACRL, let us fix a state and take the constrained and the unconstrained policies as the target and the proposal distributions, respectively, i.e., f = 7 (s) E ACC(s)), (s) A(A). Clearly, we have क.
(als)) = 0 for all
& C(s). Let M > 0 be some constant such that M (a(as) 2 (a(sss) for all E A. ARM can generate an action that follows (s):
1. Generate a' E A from the unconstrained Rs(s).
2. If a' E C(s), accept a' with probability (a'ls)) (s) =T(("|s));: otherwise, if a' C(s), reject a' and return the first step.
4

---

Published as a conference paper at ICLR 2025
Choices of target distribution T... Note that have the freedom to configure the desired TīJ. for specific purposes, e.g., exploration. One convenient choice is to simply set (als) x T (a(ss)), for all C(s). In this case,
always accept action a' C(s) in the above step 2 if we set
M = 1/(Jaec()) T (d)s))da.
Salient features of ARM. The advantages of using ARM in ACRL are two-fold: (i) Efficiency: ARM is computationally efficient as only requires checking if an action satisfies the constraints. As result, ARM largely obviates the need to solve QPs or learn a generative model. (ii) Generality: ARM very general, i.e, can be integrated with any standard unconstrained RL method.
Issues of low acceptance rate under ARM. Despite the efficiency and generality of ARM, solely naively adopting ARM in ACRL can lead to keeping sampling invalid actions and getting stuck when the policy has low action acceptance rate. The issues are two-fold: (i) The ARM procedure could repeat indefinitely with a near-zero acceptance probability. In ACRL, this scenario is likely to happen at the early training stage since the randomly initialized action distribution can be drastically different from the feasible action set. (ii) A low acceptance rate typically implics a poor action coverage of the policy over C(s). This would significantly affect the performance in the cumulative reward. To address this issue by increasing the action acceptance rate through the course of learning, present solution in Section 4.2.
4.2 AUGMENTED UNCONSTRAINED TWO-OBIECTIVE MDP
To mitigate this issue of low ARM acceptance rate, we propose to apply ARM on an augmented unconstrained MDP, which guides the policy updates towards the feasible action set by a penalty signal induced by the action acceptance events of ARM, instead of on the original action-constrained MDP. Moreover, we show that these two MDPs are equivalent with respect to optimal policies.
Constructing augmented MDP. Based the origi- nal aution-consstrined MDP M = (S, A. P, Y; T, C),, we propose to construct Augmented Unconstrained Two- Objective MDP (AUTO-MDP) M == (S, A, P. 7. by adding additional self-loop state transitions and penalty signal for those actions a E C(s):
S
8'
pi P(r)(aa)
a
Ma,a) = (r(a,a).
p(aaa a) F(a,a) [0.c(s,a)
The AUTO-MDP M shares the same state and action spaces with the original MDP M.
The augmented reward function F S A - R2
Figure I: An illustration of AUTO- MDP, where a E C(s) and ā C(s).
returns a 2-dimensional reward vector ar)s c(s. a)] and is defined as: Let K. > 0 be a constant penalty. Then, we construct (i) For any (s, a) with a & C(s). F(s,a) = 10, -K]. (ii) For any (s, a) with a E C(s), F(s, a) == a),
The augmented transition kernel P is defined as follows: For any (s, with a E C(s), let ((s'ls,a) = ()) For any (s, a,ass) with a & C(s), let
p(s'(s.a) = = { 1, 5 s=s
(2)
0, otherwise
The idea of AUTO-MDP is illustrated in Figure 1.
Moreover, for any policy æ E A, we define the vector-valued Q function Q S A R of ((ssaa;m) = E D t= <rr:/so s, ag = a: m]. which is natural extension of the standard valued Q function. Clearly, the vector-valued Q function also satisfies the Bellman equation
no
Qs.. r(s, a) + /B (Q(s', d);
As in the standard multi-objective MDPs (MOMDP), the set of optimal policies depend on the preference over the objectives. In the MOMDP literature (Abels et al., 2019; Yang et al., 2019), this is typically characterized by using linear scalarization with a preference vector X = [),, Ae] such that the scalarized Q value of a policy is defined as Q (s, m) == (A, Q(, a: m)). Without loss
5

---

Policy update by (5): Published as a conference paper at ICLR 2025
of generality, we presume that À lies in two-dimensional probability simplex, ie., A, 2 0, Ae 0, and X, + Ae
With such design, obtain a policy with sufficiently high acceptance rate under ARM, the learner shall find a policy that maximizes cumulative reward while minimizing violation penalty.
Equivalence of AUTO-MDP and original MDP in terms of optimal policies. We show that the constructed AUTO-MDP and the original MDP are equivalent in the sense that they share the same set of optimal policies. This property can be formally stated in the following proposition.
Proposition 1 (Equivalence in optimality). Let 'T" E He be optimal policy among all the policies in Tee under the original action-constraine MDP M. Then, for any X E A, the policy " remains an optimal policy among all the policies in A under the AUTO-MDP M.
The proof of Proposition provided Ap- pendix A. This result suggests that solv- ing AUTO-MDP can achieve the same maxi- mum cumulative reward the original action- constrained MDP while providing incentives for higher action acceptance rate.
Algorithm 1: Practical Implementation of ARAM
Input Initial parameters o, 0, preference
sampling distribution px actor and critic learning rates En EQ
Remarks on the preference vector A. Given Initialize the replay buffer D,; the equivalence property in Proposition 1, to 2 solve the AUTO-MDP, one could select a pref- for each iteration j Sample À A according to pxi for each environment step t do
Initialize the augmented replay buffer Da;
erence vector À A and find a corresponding 4 optimal policy F such that X A(Q a; ) 2 5
A (s, T), for all s E S.a a E A. and 6 AA practice, different choices of X lead to distinct learning behaviors: (i) If A, is much larger than A., then the violation penalty 8 could be too small to enhance the acceptance 9 rate of ARM. (ii) X, much smaller than Aç. then high violation penalty could make the pol- 10 icy very conservative and lead to low cumulative 11 reward.
Sample 0t ~ ( A) by ARM;
7
Obtain augmented reward r₁ [rr, and next state $t41i if ar C(st) then
Store (st, (81,a,, ae, C, 86+1) in Dr
else
Store (st, (e, C,, S1+1) in Da;
12
for each gradient step 7 do
To find a proper X, one straightforward approach 13 to employ hyperparameter tuning over X, at the cost of several times more of environment 14
Draw mini-batch of samples from D, and Da:
Critic update by (4):
steps. To address this, we propose practi- cal implementation of ARAM based on multi- 15 objective RL, which can learn well-performing policies for all preferences simultancously, as described below.
O + 0 - EQ Vo))
x EN V (n))
4.3 A OULTIVORJECTIVE RL IMPLEMENTATION OF ARAM
This introduces a practical multi-objective RL implementation of ARAM. As described earlier, the two modifications ARM and AUTO-MDP are general in that they can be employed adapt any standard deep RL algorithm to ACRL. This work adopts SAC (Haarnoja et al., 2018) as the backbone showcase how to integrate the proposed modifications into an existing deep RL algorithm.
Solving AUTO-MDP via multi-objective RL. To learn policies for all preferences simultaneously, we adapt the multi-objective SAC (MOSAC) presented (Hung et al., 2023) to the AUTO-MDP:
Policy loss and critic loss. MOSAC also adopts an actor-critic architecture as in vanilla SAC. Let and 0 denote the parameters of the policy and the critic. MOSAC leams preference-dependee policy (ala; )) and the corresponding vector-valued Q function by preference-dependent critic network (0 X). Given state-action sampling distri-

---

Published as a conference paper at ICLR 2025
Update the policy network
Accosoonci Method
2. Augmennted Uncorsdreined Two-Obyeddvv MDP
Mub--bjective
Reinforcement Learning
Preference
Set (Containing
Policy Network
Ja(
Penallty
Augmentod
Ad
Replay
C(s,)
Buffer
Critic
ARM
Network
(0.
State
Og
E C(ac)
Environment
Reward
Replay
Buffer
Update the critic network
Figure 2: An illustration of the ARAM framework. ARAM is composed of three components: (1)
ARM: Use an oracle to verify whether the sampled action is in the feasible action set. (2) AUTO-
MDP: Assign penalties to invalid actions within augmented MDP framework, thereby reducing
the rate of action violations. (3) MORL: Use MORL to discover well-performing policies under all penalty weights simultaneously.
bution μ. the critic network updated iteratively by minimizing the following loss function
(Q( = B(a,a)m (12, (s, A) (8, + -Ee (Va (8': mT. A)1))
(4)
where Vo(s; = (Ao= a; uे. A) a log TH (als; A)1a] with entropy coefficient a, 9 denotes the parameters of the target critic network. Regarding the policy update, the policy Ts is updated by minimizing
(d A) = Esn.p Eaawa [o lol log (TH (a 8; A)) - (2, Q (8, To. [
(5)
Notably, Equation (4) and Equation (5) can be viewed as the critic loss and the policy loss of vanilla SAC under the scalarized Q function.
Dual-buffer design. Like vanilla SAC, MOSAC is an off-policy algorithm and makes policy and critic updates based on the samples from experience replay buffers. To better address the augmented transitions in AUTO-MDP, we propose a dual-buffer design, where we store the feasible transitions and the augmented infeasible transitions in two separate replay buffers, namely a real replay buffer D, and an augmented replay buffer D.. This design offers more flexibility in balancing the number of updates by feasible and infeasible transitions, especially at the initial training stage when the action violation rate high.
Preference distribution. To update the policy and the critic for different preferences, the preference À is drawn from some distribution px. One natural choice is to set as a uniform distribution over the wwo-dimensional probability simplex, or essentially Dirichlet distribution with concentration parameter equal to 1.
The training process is illustrated in Figure 2, with the pseudo code provided in Algorithm 1.
5 EXPERIMENTS
Benchmark Methods. We compare ARAM with various recent benchmark ACRL algorithms, including NFWPO, DPre+, SPre+, and FlowPG. NFWPO (Lin et al., 2021) achieves favorable con- straint satisfaction at the cost of high QP overhead as it enforces action constraints by Frank- search. DPre+ and SPre+, proposed by (Kasaura et al., 2023). adapt the vanilla DDPG (Lillicrap et al., 2016) and SAC (Haarnoja et al., 2018) to ACRL by using QP-based projection step for action post-processing, learning the critic with pre-projected actions, and applying a penalty term to guide the policy updates. For a fair comparison, we use the official implementation and the hy- perparameter settings of DPre+, SPre+, and NFWPO provided by (Kasaura et al., 2023). FlowPG enforces action constraints via pre-trained Normalizing Flow model, and we use the official source code provided by (Brahmanage et al., 2023). For the testing of ARAM, we set X = [0.9. 0.1] as the default input preference of the policy network To (-IA). Moreover, during the testing of all the above algorithms, an auxiliary projection step is employed to guarantee that actions used for environment interaction always satisfy the action constraints.
7

---

Published as a conference paper at ICLR 2025
DPre+
SPre+
FlowPG
NFWPO
(Ours)
4K
1K
6K
Return
Return
Return
-8
Return
2K
-10
1K
2K
-12
-1K
Time (s)
Time (5)
Time
(5)
Hopper
(b) HopperVel
Reacher
(d) HalfCheetah
5K
15K
1KK
-14K
10K
-15K
Return
3K
Return
Return
2K
Return
5K
-17K
1K
-3K
-18K
0
-5K
Time
Ant
NSFnet
(g) BSS3z
(h) BSS5z
Figure 3: Learning curves of ARAM and the benchmark methods across different environments in terms of the evaluation return.
Evaluation Domains. We evaluate the algorithms in various benchmark domains widely used in the ACRL literature (Lin et al., 2021; Kasaura et al., 2023; Brahmanage et al., 2023): (i) MuJoCo loco- motion tasks (Todorov al., 2012): These tasks involve training robots to achieve specified goals, such as running forward and controlling their speed within certain limits. (ii) Resource allocation for networked systems: These tasks involve properly allocating resource under capacity constraints, including NSFnet and Bike Sharing System (BSS) (Ghosh & Varakantham, 2017). For NSFnet, the learner needs allocate packets of different flows multiple communication links. The action constraints are induced by the per-link maximum total assigned packet arrival rate. We follow the configuration provided by (Lin al., 2021) and use the open-source network simulator from PCC- RL (Jay et al., 2019). For BSS, the environment consists of m bikes and n stations, each with capacity limit of c. The learner needs to reallocate bikes to different stations based on current situation. We follow the experimental scenario of (Lin et al., 2021) and evaluate our approach on two tasks: BSS3z with n and m and BSS5z with n and m 150. Their capacities are both set to 40. A detailed description about these tasks is provided in Appendix D.
Performance Metrics. evaluate the performance in the following aspects:
Training efficiency: We record the evaluation returns at different training stages, in terms both the wall clock time and the environment steps. To ensure fair measurements of wall clock time, we run each algorithm independently using the same computing device. Moreover, we report the cumulative number of QP operations as indicator the training computational overhead.
Valid action rate: At the testing phase, we evaluate the valid action by sampling 100 actions from the policy network at each step of an episode. This metric reflects how effectively each method enforces the action constraints throughout the evaluation phase.
Per-action inference time: The efficiency of action inference reflects the design complexity, such as the need for generative models and QP operations, of each ACRL method. During evaluation, we measure the per-action inference time for I million actions. This inference time serves critical metric for ACRL deployment.
5.1 EXPERIMENTAL RESULTS
Our proposed method effectively reduces the use of costly QP operations, allowing us to address ACRL with lightweight framework. This subsection demonstrates the effectiveness of our
8

---

Published as a conference paper at ICLR 2025
method in training efficiency, valid action rate, and per-action inference time. Unless stated other- wise, all the results reported below are averaged over five random seeds.
Does the proposed method outperform other ACRL benchmark methods in cumulative wards? Figure 3 shows the evaluation reward versus wall clock time, and Table 2 and Figure 8 (in Appendix C) show the valid action rates. We observe that ARAM enjoys fast learning progress and high valid action rates simultaneously. On the other hand, due to the reliance on the projection layer, the projection- bassed methods like DPre+ and SPre+ have relatively very low valid action rates, which require a large number of QP operations and hence lead to longer training time. Addition- ally, they perform poorly in resource allocation environments since the optimal solutions for these problems cannot be directly found through projection. Regarding NPWPO and FlowPG, these two methods can both effectively learn policies that satisfy the constraints but not necessarily achieve a high average return. Moreover, the results regarding sample efficiency, measured by the evaluation return versus environment steps, can be found in Appendix C.
Table 2: Comparison of ARAM and the benchmark algorithms in terms of the valid action rate.
Environment
DPrc+
SPrc+
NFWPO
FlowPG
ARAM (Ours)
Hogper
0.97 1 0.03
0.40 0.01
0.99 1: 0.01
0.98
0.01
0,99 1 0.01
HopperVel
0.38 + 0.30
0.60
0.09
0.94 + 0.03
0.71
+ 0.19
0.80 + 0.09
Reacher
0.25 1 0.16
0.93 1.5110.00 0.03
0.97 1. 0.02
0.95
0.01
0,98 1 0.01
HallCheetah
0.35 :: 0.06
0,78
0.97 = 0.01
0.78 0.17
0.78 : 0.29
Ant
0.41 1: 0.38
0.29 0.07
0,99 1. 0.01
0.83
0.17
1.00 1 0.00
NSFnet
0.00 0.00
0.01
0.92 = 0.05
0,98 -: 0.01
0.94 1 0,04
BSS32
0.25 0.02
0.24
0.73 = 0.18
0.59
0.72 - 0.24
BSS52
0.31 0.11
0.16
0.84 = 0.14
0.77 + 0.16
Does our proposed method achieve lower training and inference overhead? QP operations are known to be computationally costly and thereby account for substantial fraction of training time in many ACRL methods. Figure 4 presents the log-scale plot of QP usage under different algorithms. It is evident that ARAM exhibits significantly lower QP computation compared to others. Due to its reduced dependency on QP operations, ARAM achieves higher training efficiency.
Moreover, Table 3 shows the average per-action inference time during evaluation. ARAM benefits from computationally efficient action inference due to almost QP-free design, as only a minimal subset of policy output actions that violate constraints requires the QP operator. By contrast, the projection-bas based methods like DPre+ and SPre+ and the flow-based FlowPG all suffer from much higher per-action inference time.
Table 3: Comparison of ARAM and the benchmark algorithms in terms of the evaluation time. The evaluation times are calculated through one million evaluation steps.
Environment
DPre+
SPre+
NFWPO (s)
FlowPG (6)
ARAM (Ouns)
Reucher
71.47 5.63
303.01 + 60.81
67.43 + 5.02
71.47 + 3.12
44.72 + 0.93
HalfCheetah
211.67 = 48.70
253.53 = 30.48
180.41 : 10.57
210.63 7,51
S7.14 ::: 8.13
Hopper
99.43 6.71
175.43 9.47
88.71
103.12 5.18
61.72
HopparVel
177.31 15.73
97.12 = 7.95
82.63 -: 7.13
91.43
63.81 + 10.04
Comparison: ARAM versus RL for constrained MDPs with projection. To make the compari- son even more comprehensive, we also adapt FOCOPS (Zhang et al., 2020), which is a popular RL approach designed for constrained MDPs address long-term discounted cost, ACRL by adding projection step. Table 4 and Table 5 present comparison of ARAM and FOCOPS in the eval- uation rewards and the valid action rates. These results suggest that ACRL requires fundamentally different solutions from RL. for constrained MDPs.
Ablation study on MORL. To investigate the benefits of using MORL, we perform an ablation study that compares the MORL implementation with single-objective variant with fixed prefer- ence (termed SOSAC below). From Figure 5, MORL can discover policies with both competitive final forward reward and favorable constraint satisfaction due to the implicit knowledge sharing across preferences during training. By contrast, SOSAC under a fixed preference fails to meet both criteria simultancously, and this suggests that direct hyperparameter tuning can be rather ineffective. We also provide the learning curves in Figure 11 in Appendix C.
CONCLUSION
In this paper, we introduced ARAM, a novel framework designed to address ACRL problems by augmenting standard deep RL. algorithms. By employing the acceptance-rejection method and an

---

Published as a conference paper at ICLR 2025
DPre+
SPre+
FlowPG
NFWPO
(Ours)
Number QP Operations
Number of QP Operations
Number of QP Operations
Number of QP operations
1.0
10
10
10
107
107
10
107
10a
10a
10a
10
10a
10a
10a
10
10a
10a
6
8
10
0
Environment Steps (x10%)
Environment Steps
(x10F)
Environment Steps (x10a)
Environment Steps (x10%)
(a) Hopper
(b) HopperVel
(c) Reacher
(d) HalfCheetah
Number of QP Operations
Number of QP Operations
Number of QP Operations
Number of QP Operations
10a
10
10
10
107
107
107
107
10
10a
10a
10
10a
102
10a
10a
10a
101
6
8
10
2
3
4
2
4
6
8
10
0
2
4
10
Environment Steps (x10?)
Environment Steps (x10²)
Environment Steps (x107)
Environment Steps (x10")
(e) Ant
(f) NSFnet
(g) BSS3z
(h) BSS5z
Figure 4: Cumulative number of QP operations of ARAM and the benchmark methods across dif-
ferent environments, with the y-axis on logarithmic scale.
SOSAC
Valid Action Rate
Action
indeerrca- 5.1 405AC
1053C polbesiie 201,
Forward Reward
Forward Reward
(a) HalfChectah
(b) Ant
Figure 5: Ablation study of MORL. We plot the tuples of forward reward and valid action rate of ARAM and the three single-objective SAC variants with fixed preferences X [0.9, 0.1], |0.5, 0.5], and [0.1, 0.9) in HalfCheetah and Ant. The five markers of each color refer to the results of the same algorithm over five distinct random seeds.
Table 4: Comparison of ARAM and FO- COPS in terms of the evaluation return.
Comparison of ARAM and FO- COPS in terms of the valid action rate.
Table 5:
Environment
FOCOPS
(Ours)
Favironment
POCOPS
(Ours)
Hopper (x10%)
2.27 =
3.07 0.24
Hopper
0.22
0.01
Vel (x10P)
0.07 = 3.13
6.49 2.31
VepperVel
0.32 = 0..11
0.80 1: 0.09
Reacher (x10P)
-5,80 1.14
-4.78 : 0.33
Reacher
0.87 = 0..0
0.98 1 0.01
HallCheetah (x10")
6.08 1 1.87
8.38 1.11
HallCheetah
0.50 1 0.13
0.78 0.29
(x10%)
0.32
NSFnet (x10f)
0.68 1: 0.16
1.32 0.08
NSFoet
0..66 1: 0.28
0.94 1 0.04
BSS32 (x10P)
-1.93 1. 0.20
-1.65 0.04
BSS3<
0.45 1. 0.19
0.72 1. 0.24
BSS5x (x109)
-1.61 + 0.05
-1.51 0.02
BSS5>
0.16 + 0.13
0.77 + 0.16
augmented MDP, ARAM effectively reduces the need for costly QP operations and improves valid action rates. Our experimental results demonstrate that ARAM can simultaneously achieve faster learning progress and require significantly fewer QP operations than the existing ACRL methods.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This material is based upon work partially supported by National Science and Technology Coun- cil (NSTC), Taiwan under Contract No. NSTC 113-2628-B-A49-028 and the Higher Education Sprout Project the National Yang Ming Chiao Tung University and Ministry of Education, Tai- wan. Shao-Hua Sun was supported by the Yushan Fellow Program by the Ministry of Education, Taiwan. We also thank the National Center for High-performance Computing (NCHC) for providing computational and storage resources.
REFERENCES
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objectivve deep reinforcement learning. In International Conference on Machine Learming, 2019.
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, 2017.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Brandon Amos and Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, 2017.
Abhinav Bhatia, Pradeep Varakantham, and Akshat Kumar. Resource constrained deep reinforce-
ment learning. In International Conference on Automated Planning and Scheduling, 2019.
Janaka Brahmanage, Jiajing Ling, and Akshat Kumar. FlowPG: Action-constrained Policy Gradient with Normalizing Flows. Advances in Neural Information Processing Systems, 2023.
Bingqing Chen, Priya L Donti, Kyri Baker, J Zico Kolter, and Mario Bergés. Enforcing policy feasi- bility constraints through differentiable projection for energy optimization. AnACM International Conference on Future Energy Systems, 2021.
Changyu Chen, Ramesha Karunasena, Thanh Nguyen, Arunesh Sinha, and Pradeep Varakantham. Generative modelling of stochastic actions with arbitrary constraints in reinforcement learning. Advances in Neural Information Processing Systems, 2023.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.087 087557,088777... 2018.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave trace: Learning to Reset for Safe and Autonomous Reinforcement Learning. In International Conference on Learning. 2018.
Supriyo Ghosh and Pradeep Varakantham. Incentivizing the of bike trailers for dynamic reposi- tioning in bike sharing systems. In International Conference on Automated Planning and Schedul- ing, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning. 2018
Wei Hung, Bo Kai Huang, Ping-Chun Hsieh, and Xi Liu. Q-Pensieve: Boosting sample efficiency of multi-objective RL. through memory sharing of Q-snapshots. In International Conference on Learning Representations, 2023.
Jeffrey Ichnowski, Paras Jain, Bartolomeo Stellato, Goran Banjac, Michael Luo. Francesco Borrelli, Joseph E Gonzalez, Ion Stoica, and Ken Goldberg. Accelerating quadratic optimization with reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Nathan Jay, Noga H Rotman, P Godfrey, Michael Schapira, and Aviv Tamar. A deep reinforce- ment learning perspective on internet congestion control. International Conference on Machine Learning, 2019.
11

---

Published as a conference paper at ICLR 2025
Kazumi Kasaura, Shuwa Miura, Tadashi Kozuno, Ryo Yonetani, Kenta Hoshino, and Yohei Hosoe. Benchmarking actor-criti deep reinforcement learning algorithms for robotics control with action constraints. Robotics and Automation Letters, 2023.
Ivan Kobyzev, Simon Prince, and Marcus A Brubaker. Normalizing Flows: introduction and review of current methods. IEEE Transactions on Paltern Analysis and Machine Intelligence, 2020.
Dirk P Kroese, Thomas Taimre, and Zdravko I Botev. Handbook of monte carlo methods. John Wiley & Sons, 2013.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Interna- tional Conference on Learning Representations, 2016.
Jyun-Li Lin, Wei Hung, Shang-Hsuan Yang. Ping-Chun Hsich, and Xi Liu. Escaping from zero gra- dient: Revisiting conbsrnnndtt reinforcement learning via Frank- Wolfe policy optimization. In Uncertainty in Artificial Intelligence, 2021.
Guanghui Liu, Qiang Li, Bohan Yang, Hualiang Zhang, and Lijin Fang. An efficient linear programming- time-optimal feedrate planning considering kinematic and dynamics con- straints of robots. IEEE Robotics and Automation Letters, 2024.
Yongshuai Liu, Avishai Halev, and Xin Liu. Policy learning with constraints in model-free rein- forcement learning: A survey. International Joint Conference on Artificial Intelligence, 2021.
Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlayer-practict constrained op- timization for deep reinforcement learning in the real world. In International Conference Robotics and Automation, 2018.
Andrew Singletary, Shishir Kolathaya, and Aaron D Ames. Safety-critical kinematic control of robotic systems. IEEE Control Systems Letters, 2021.
Richard S Sutton. Reinforcement learning: an introduction. A Bradford Book, 2018.
Yunxi Tang, Xiangyu Chu, Jing Huang, and KW Samuel Au. Learning- MPC with safety filter for constrained deformable linear object manipulation. IEEE Robotics and Automation Letters, 2024.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. International Conference Learning Representations, 2019.
Brijen Thananjeyan, Ashwin Balakrishna. Ugo Rosolia, Felix Li, Rowan McAllister, Joseph E Gon- zalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Safety augmented value estimation from demonstrations (SAVED): Safe deep model-based rl for sparse cost robotic tasks. Robotics and Automation Letters, 2020.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery RL: Safe reinforcement learning with learned recovery zones. Robotics and Automation Letters, 2021.
Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforeement learning by imagining the near future. Advances in Neural Information Processing Systems, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In International Conference On Intelligent Robots and Systems, 2012.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective reinforeement learning and policy adaptation. Advances in Neural Information Processing Sys- tems, 2019.
Haonan Yu, Wei Xu, and Haichao Zhang. Towards safe reinforcement learning with a safety editor policy. Advances in Neural Information Processing Systems, 2022.
12

---

Published as a conference paper at ICLR 2025
Cong Zhang, Fan Wu, He Wang, Bihua Tang, Wenhao Fan, and Yuanan Liu. A meta-learning algorithm for rebalancing the bike-sharing system in loT smart city. IEEE Internet of Things Journal, 2022.
Jie Zhang, Meng Meng, Yiik Diew Wong, Petros Ieromonachou, and David ZW Wang. A data- driven dynamic repositioning model in bicycle-sharing systems. International Journal of Produc- tion Economics, 2021.
Yiming Zhang. Quan Vuong, and Keith Ross, First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 2020.
13

---

Published as a conference paper at ICLR 2025
APPENDICES
Table of Contents
Proof Proposition
B Detailed Architecture of ARAM
15
C Additional Experimental Results
16
Comparison: ARAM versus the benchmark algorithms Terms Sample Effi ciency
16
C.2 Comparison: ARAM versus RL for constrained MDPs with projection.
17
C.3 Ablation Study: The learning curves of ARAM versus SOSAC
18
C.4 Sensitivity Analysis: Effect of Violation Penalty in ARAM
18
Sensitivity Analysis: Effect of Target Distribution ARAM
C.6 Additional Experiment: Scalability of ARAM
19
D Detailed Experimental Configurations
21
D.1 Environments and Action Constraints
21
D.2
Hyperparameters of ARAM
21
E Detailed introduction to the existing ACRL methods
21
E.1 Action Projection
21
E.2 Frank-Wolfe Policy Optimization (FWPO)
22
E.3 FlowPG
22
A PROOF OF PROPOSITION
For ease of exposition, we restate the proposition as follows.
Proposition (Equivalence in optimality). Let π E II be optimal policy among all the policies in II under the original aeciionconstrained MDP M. Then, for any A. the policy n remains an optimal policy among all the policies in AA under the AUTO-MDP M.
Proof. Let first fix a preference vector À A. Recall that X Ap de). Let # E A be a deterministic optimal policy of the AUTO-MDP under the preference vector A. By the classic literature of MDPs, we know such a n must exist. Then, we have
A (s a a)))) 2 (A, Q(s a))))
(6)
for all E S, for all A, and for all AA In the sequel, we slightly abuse the notation and use (s) to denote the deterministic action taken at state by a deterministic policy æ.
To prove the proposition, policies He.. We prove this by contradiction. Suppose क $ Ne.. Then, there must exist a state E S such that (5) & C(s). Then, know
just need show that such π must also be in the set of feasible
(A, (1),m (2, -K] 7 (5, ((())(1)
KC
(8)
1-
where Equation (7) follows from the Bellman equation and the self-loop transitions of AUTO-MDP, and Equation (8) holds by recursively rolling out the self-loop transitions. Let us pick feasible
action C(s). We know such action a must exist C(s) is assumed non-empty. Then,
14

---

Published as a conference paper at ICLR 2025
have
(A, (3,, = (A, o] + +7BxmD(45,) (Q(s)=
(9)
K
ス Ar a(5,a) + 7h 1-7
(10)
K
(11)
where Equation (9) follows from the Bellman equation, Equation (10) holds by considering the worst-case reward and penalty, and Equation (11) holds due to the non-negativity of the reward function and that ? 1. Therefore, by combining Equation (8) and Equation (1I), we know (3 0; TT) = (A, (s, a; mx)) > (x, (3, 7 (3); = Qa 7 (3); Tx). Then, by the argu- ment of the standard one-step greedy policy improvement, we know ग can be improved by taking action a at state 3 instead. Hence, o cannot be an optimal policy. This completes the proof.
B DETAILED ARCHITECTURE OF ARAM
Our implementation is based on Q-Pensieve (Hung et al., 2023), Figure 6 illustrates the complete training process of ARAM. MORL aims optimize policies under varying preferences, which poses challenges in sample efficiency. To address this, Q-Pensieve introduces memory-sharing technique via the Q Replay Buffer, which stores past critic-network snapshots. This approach enables lever- aging past experiences effectively, improving sample efficiency by allowing the agent to revisit and learn from previous interactions across different preference settings. By incorporating the Q Re- play Buffer, ARAM facilitates efficient policy learning, allowing the agent to balance performance improvement and constraint satisfaction.
Update policy for preference )
Sample proference
Set (0
Calculate the policy
(d(
Sample from the policy
Resample action from the policy
Roplay Buffer
ARM
Store the penalty
Ciss)
Augmented Replay
Network
Critic
Oa
C(a)
Augmented
Store the franeltion Roplay Bufter
Buffer
Add the critic snapshot Replay Buffer
Taka the adiin interact with the enviromeent
Emitronment
Crilie Network
(0
Sample the proference
Og
Calculate the critic loss
Update network for preference
Sample the proforances and Ay..
Figure 6: Overview the training process ARAM. When an action is sampled from the policy, if it belongs the feasible set, the transition is stored in the Replay Buffer (bluc). Otherwise, the corresponding penalty transition is stored in the Augmented Replay Buffer, and another action is resampled (red). These stored transitions are then used to update the policy and critic networks.
15

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTAL RESULTS
In this section, we compare ARAM with other baselines, including action-constrained RI. and RL for Constrained MDPs. The learning curves demonstrate that ARAM achieves better sample efficiency. Additionally, we conduct an ablation study on the MORL framework, investigating how varying preferences within MORL impact the performance of ARAM.
C.I COMPARISON: ARAM VERSUS THE BENCHMARK ALGORITHMS IN TERMS SAMPLE EFFICIENCY
Does ARAM have better sample efficiency? Figure 7 shows the evaluation rewards, and Figure displays the action acceptance rates. We can observe that ARAM achieves consistently the best reward performance while maintaining a low action violation rate. On the other hand, FlowPG and NFWPO can effectively control the actions within the feasible but their reward performance appears lower. In contrast, QP-based methods exhibit a significantly higher rate of action violations.
+
SPre+
FlowPG
NFWPO
ARAM (Ours)
1k
3k
Relurn
Return
Return
Return
6k
2k
-8)
4k
-10
0
4
10
-1k
2
8
%
8
10
Steps (x10")
Steps (x10")
Steps
Environment Steps (x10%)
(a) Hopper
(b) HopperVel
(c) Reacher
(d) HalfCheetah
10k
-15k
Relumn
Return
Retur
-2
Return
-16K
-3k
-5k
2
18k
Environment Steps (x10%)
Environment Steps (X10*)
Environment Steps (x10")
Environment Steps (x10*)
(c) Ant
(f) NSFnet
(g) BSS3z
(h) BSS5z
Figure 7: Learning curves of ARAM and the benchmark algorithms across different environments in terms of the evaluation return.
16

---

Published as a conference paper at ICLR 2025
+
S
FlowPG
NFWPO
ARAM (Ours)
1.0
0.8
Valid Action Rote
0.8
Valld Action Rate
0.8
0.6
Valld Action
0.6
0.4
0.4
0.4
0.4
0.2
0.0g
0.0g
10
0.0
0.0
10
Environment Steps (x10%)
Environment Steps (x10")
Environment Steps (x10")
Environment Steps (X10")
(a) Hopper
(b) HopperVel
Reacher
(d) HalfCheetah
1.0
0.8
Valid Action Rate
Velid Action Rate
0.8
Valld Action Rate
0.8
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.00
10 Environment Steps (xc1d")
2
8
0.0
2
3
0.0g
10
0.0g
Environment Steps (x10%)
Environment Steps (x10")
Environment Steps (x10")
(c) Ant
(f) NSFnet
(g) BSS3z
(h) BSS5z
Figure 8: Learning curves of ARAM and the benchmark algorithms across different environments terms of the valid action rate.
FOCOPS
ARAM (Ours)
1K
8K
-6
Return
Return
Return
2K
4K
-10
2K
00
-1K
2
-12
0
8
Environment Steps (x10*)
Environment Steps (x10%)
Enviranment Steps (x10*)
Environment Steps (x10*)
(a) Hopper
(b) HopperVel
(c) Reacher
(d) HalfCheetah
5K
15K
-1K
-14K
4K
-15K
Retum
Return
Return
Return
5K
-2K
16K
17K
6 8
10
5K
2
4
2
8
10
-18K
0
2
6
8
10
Environment Steps (X10*)
Environment Steps (K10")
Environment Steps (x10")
Environment Steps (x10")
(c) Ant
(f) NSFnet
gg) BSS3z
(h) BSS5z.
Figure 9: Comparison of the evaluation return of ARAM and FOCOPS across various environments.
C.2 COMPARISON: ARAM VERSUS RL FOR CONSTRAINED MDPs WITH PROJECTION.
Does ARAM more effectively reduce the number of action violations while achieving better performance? Figure and Figure 10 present the training curves of total return and valid action rate, respectively. From these curves, observe that ARAM consistently outperforms FOCOPS in both the evaluation return and the valid action rate, indicating its superior ability to handle action constraints while optimizing performance.
17

---

Published as a conference paper at ICLR 2025
FOCOPS
ARAM (Ours)
1.0
0.8
Volld Action R00
0.8
Valld Action Rate
C.8
0.6
Valid Action
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
6.2
0.00
0.0g
0.0
0.00
Environment Steps (x10*)
Environment Steps (x10*)
Environment Steps (x10*)
Environment Steps (x10²)
(a) Hopper
(b) HopperVel
Reacher
(d) HalfCheetah
1.0
0.8
Valid Action Rate
Valid Action Rate
0.8
Valid Action Rate
0.8
Action
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.00
8
0.0
2
0.0
10
0.0
4
Environment Steps (x10*)
Environment Steps IX10%)
Environment Steps (X10")
Environment Steps (X10")
(e) Ant
(f) NSFnet
(g) BSS3z
(h) BSS5z
Figure 10: Comparison of the valid action rate of ARAM and FOCOPS across various environments.
C.3 ABLATION STUDY: THE LEARNING CURVES OF ARAM VERSUS SOSAC
We present the learning curves of the total return and the valid action rate in Figure 11. We can observe that the ARAM consistently achieves higher returns compared to the SOSAC variant with fixed preferences.
SOSAC with preference [0.9, 0.1] SOSAC with preference [0.5, 0.5]
SOSAC with preference [0.1, 0.9]
ARAM (Ours)
5K
1.0
Valid Action Rate
4K
0.8
0.8
Return
6K
3K
Valid Action
0.6
0.6
2K
0.4
0.4
2K
1K
0.2
0.2
6 8
10
2
4
6
8
10
0.0
2
6
8
10 0.0
2
10
Environment Steps (x10²)
Environment Steps (x109)
Environment Steps (x109)
Environment Steps (x10²)
(a) HalfCheetah
(b) Ant
(c) HalfCheetah
(d) Ant
Figure 11: Ablation Study on MORL. These figures present the learning curves of ARAM and three single-objective SAC variants with different fixed preferences in terms of the evaluation return
(Figures 1 la and 1lb) and the valid action rate (Figures 1lc and lid) in the HopperVel and Ant environments.
C.4 SENSITIVITY ANALYSIS: EFFECT OF VIOLATION PENALTY IN ARAM
We present the results of a hyperparameter sensitivity study on the violation penalty in the Hop- Vel and Ant environments, shown in Figure 12. This study examines how different fixed values
18

---

Gaussian Published as a conference paper at ICLR 2025
of I. impact the return and valid action rate during training. The findings demonstrate that ARAM maintains stable performance across varying K. values, indicating that the model is largely insensi- tive to this hyperparameter. This supports our original experimental setup, where single K. value was used consistently across all tasks and environments without requiring task-specific tuning.
penalty weight = 0.05
penalty weight = 0.1
penalty weight 0.2
1.00
1e3
le3
1.0
1.0
4.5
0.75
Vaild Rate
0.8
Vaile Rate
0.8
Return
Return
3.0
0.6
0.6
0.50
0.4
0.4
0.25
0.2
0.2
0.00
0
6
10
0
2
6
8
10
0.0
4
8
10
0.0g
4
6
10
Environment Steps (x10")
Environment (x1d")
Environment (x10")
Enviranment Stops (x107)
(a) HopperVel
(b) Ant
(c) HopperVel
(d) Ant
Figure 12: Sensitivity Analysis of the Violation Penalty K. These figures provide comparison of the return and the valid action rate across training steps for different values of K. (K' = 0.05, 0.1, 0.2)
in the HopperVel and Ant environments. These figures show that ARAM is not highly sensitive to the penalty weight.
C.5 SENSITIVITY ANALYSIS: EFFECT OF TARGET DISTRIBUTION IN ARAM
We analyze the impact of different larget distributions in ARAM, specifically examining whether this choice significantly affects the final performance. Since ARAM is implemented based on the MOSAC framework, the policy network outputs the mean and standard deviation distribution, denoted as: Ta(als) ~ (N,, o²) Following the notation in the paper, we configure an alternative (constrained) target distribution, x (a|s), by leveraging the (unconstrained) Student-t dis- tribution: (a)a)) ~ (u, o²) where T, denotes a Student-t distribution with degrees of freedom v, mean n,, and variance o². The constrained target distribution then defined as:
(als)) x is (als) for a E C(s).
=|(ass) for & C(s).
Notably, the Gaussian distribution special case of the Student-t distribution with v co. The Student-t distribution has heavier tails compared to the Gaussian, potentially leading to more exploratory action distribution. For ARM, the scaling factor M can be set as: M Mo/ (Jaect.) #a(a)s)da) where Mo is constant dependent rate is shown on Figure 13. We observe that ARAM can have a slight performance improvement under this new target distribution (especially at the early training stage) and is not very sensitive to this choice. This also corroborates the nice flexibility in the choice of target distribution.
The return and valid action
C.6 ADDITIONAL. EXPERIMENT: SCALABILITY OF ARAM
To demonstrate the scalability of ARAM to siahber-immnnsonaal environments, we extend the NSFnet environment to a 20-dimensional action space called NSFnet20d. Specifically, the agent manages packet delivery across a classic T3 NSFNET network backbone by simultaneously handling 20 packet flows, each with distinct routing paths. The action is defined as the rate allocation of each flow along each candidate path. In total, there are 30 communication links shared by these 20 different flows. The action constraints ensure that the distribution of packets on these shared links stays within the bandwidth capacity, defined as 50 units for each link. Figure 14 presents the results, illustrating ARAM's superior performance in the evaluation return, the valid action rate, and the QP efficiency compared to ACRL benchmark methods. ARAM consistently achieves higher action
19

---

Published as a conference paper at ICLR 2025
ARAM with Student's t-distribution
ARAM with Normal distribution
le2
6.0
1.0
0.8
4.5
Vaild Action Rate
0.6
Return
3.0
0.4
1.5
0.2
0.0
1
2
4 5
0.0
0
1
2
3 4
Environment Steps (x109)
Environment Steps (x105)
Figure 13: Sensitivity Analysis of the Target Distribution in ARAM. These figures provide comparison of the return and the valid action rate across training steps for different target distribu- tions: Student's t-distribution and the Normal distribution. These figures show that ARAM is not highly sensitive to the choice of target distribution.
validity rate while maintaining robust learning under high-dimensional action spaces and complex constraints, demonstrating its adaptability and efficiency.
DPre+
SPre+
FlowPG
NEWPO
ARAM (Ours)
le4
1.0
10
Number of Operations
1.6
Vaild Action Rate
0.8
107
Return
0.8
0.6
10
0.4
0.0
10a
0.2
-0.8
10
0.0
2
4
1
3
1
2
4
5
Environment Steps (x10*)
Environment Steps (x10*)
Environment Steps (x10²)
Figure 14: Comparison of the evaluation return, the valid action rate, and the number of QP opera- tions the NSFnet20d environment.
20

---

Published as a conference paper at ICLR 2025
D DETAILED EXPERIMENTAL CONFIGURATIONS
D.1 ENVIRONMENTS AND ACTION CONSTRAINTS
In this section, we provide the details about the experimental setups and the action constraints con- sidered different environments.
The action defined as vector (01, , with each element representing the torque applied to specific joint, and (W1, "2.."." (() denotes the angular velocities of the joints. The tasks considered in our experiments include:
MuloCo:
Reacher, HalfCheetah, Hopper, and Ant: We use the experimental settings and objectives provided by OpenAI Gym V3 to control the agents these environments. Each environment presents unique challenges and objectives, such as reaching target location, maintaining balance, or achieving high-speed movement.
HopperVel: This task is specifically designed to control the robot to maintain target hori-
zontal velocity of m/s, differing from the standard Hopper's goal forward hopping.
NSFnet: Based on the T3 NSFNET Backbone discussed in (Lin et al., 2021), this network consists of different packet flows, each with distinct routing paths. There are communication links shared by different flows. The action is defined as the rate allocation of each flow along each candidate path. The action constraint to check the distribution of packets on these shared links stays within the bandwidth limits, defined as 50 units for each link. To define action constraints, we described the eight-tuple (linkj,.... links), each containing the total amount of flow that pass through that specific link.
BSS: The action is allocate bikes to stations under random demands.
BSS3z: There are 3 stations with a total of 90 bikes (m = 90, n = 3), and each station has capacity of 40 bikes.
BSS5z: This system comprises 5 stations with a total of 150 bikes (m 150, = 5), and each station also has capacity of 40 bikes.
Table 6: This table provides an overview of the action constraints applied in each experiment envi- ronment.
Environment
Action Constraint
HopperVel
max(w;a; 0) < 10
Hopper
max( 0) < 10
Reacher
0 + ag < 0.05
HalfCheetah
|w.a; < 20
Ant
a < 2
1=
NSFnet
Eielinky Oi S 50,
Vj E (1,
BSS3z
Σ aj 90
S 5,a; <= 40
BSS5z
Σ5 i= a; 150
<5, <= 40
D.2 HYPERPARAMETERS OF ARAM
We conduct all the experiments with the following hyperparameters.
E DETAILED INTRODUCTION TO THE EXISTING ACRL METHODS
E. ACTION PROJECTION
An action produced by the neural network is not guaranteed to remain within the feasible action set. To address this, there various frameworks map the output action to that satisfies the
21

---

Published as a conference paper at ICLR 2025
Table This table provides an overview of the hyperparameters used in ARAM.
Parameter
ARAM
Optimizer
Adam
Learning Rate
0.0003
Discount Factor
0.99
Replay Buffer Size
1000000
Number of Hidden Units per Layer
[256, 256]
Number of Samples per Minibatch
256
Nonlinearity
ReLU
Target Smoothing Coefficient
0.005
Target Update Interval
I
Gradient Steps
I
Sample Ratio for Augmented Replay Buffer (1)
0.2
Decay Interval for 1)
10,000
Decay Factor for 77
0.9
Table 8: This table provides an overview of the hyperparameters used in the bascline methods.
Parameter
DPre+
SPre+
NFWPO
FlowPG
Learning Rate
0.001
0.001
0.001
0.001
FW Learning Rate
0.01
Discount Factor
0.99
0.99
0.99
0.99
Replay Buffer Size
1000000
1000000
1000000
1000000
Number of Hidden Units
[256, 256]
[256, 256]
[256, 256]
[400, 300]
Number of Samples per Minibatch
256
256
256
100
Target Smoothing Coefficient
0.005
0.005
0.005
0.005
Target Update Interval
1
1
Gradient Steps
constraints. An intuitive approach is through a QP operation to find the closest feasible action to the original one within the acceptable action space. We refer this the QP-Solver:
QP-Solver(s, a, C(s)) arg min a' all2-
(12)
Kasaura al. (2023) describe family of algorithms based on the QP-Solver. However, as the complexity of the action space increases, the computation time for this type of approach can become lengthy. To ensure the feasibility of policy learning within reasonable timeframe, it is essential to minimize reliance on the QP-Solver.
E.2 FRANK-WOLFE POLICY OPTIMIZATION (FWPO)
Aside from the Qo-Solver, Lin al. (2021) employ the Frank-Wolff (FW) method update the policy and propose Frank-Wolfe Policy Optimization (FWPO). To describe the different policy pa- rameters, FWPO uses dR to denote the policy parameters in the k-th iteration. To update the policy with the action constraints, FWPO adopts a generalized policy iteration framework (Sutton, 2018), which consists of two subroutines:
(i) Policy update via state- state-wise FW: For each state s and the corresponding learning rate (s(s), search for feasible action, then guide the current policy parameters update.
(x(s) = arg max (c, VoQo a; To )a-=o, (s))) cEC(.)
(13)
T Nee (s) + (s) Toa (s)),
(14)
(ii) Evaluation of the current policy: Use any policy evaluation algorithm obtain Qa a; Tou Where (s) Tox (s) is the update direction.
E.3 FLOWPG
Brahmanage et al. (2023) adopt normalizing to create an invertible mapping between the support of simple distribution and the space of valid actions. Specifically, the focus is the conditional
22

---

Published as a conference paper at ICLR 2025
RealNVP model, which is well-suited for the general ACRL setting where the set of valid actions depends on the state variable. The conditional RealNVP extends the original RealNVP by incorpo- rating conditioning variable in both the prior distribution and the transformation functions. These transformation functions, implemented as affine coupling layers, enable efficient forward and back- ward propagation during model learning and sample generation.
Algorithm 2: FlowPG Algorithm
Input : Initial parameter vectors 0, 0, v
Initialize replay buffer B;
for episode = 1, M do
Initialize the random noise generator N for action exploration;
fort=1,.... do
Select action a; = (x)) Nr based on current policy and exploration noise; Apply flow and get the environment action Sy (ūe,, 8t);
0g is invalid then a, QP-Solver(st., a, C(s))):
Update actor policy using the sampled policy gradient:
Vod(TT) VaQu(st, Vala 8x) (o)
Learning the RL-Model. Integrating DDPG with normalizing flows involves using the learned mapping directly in the original policy network, which improves training speed and stability. The architecture of the policy network consists of the original DDPG policy network and the learned mapping function fø. The objective is to learn deterministic policy (T)(())) s) that gives the action a given a state s, maximizing J(Ts):
max J(ks) +
(15)
where B is the replay buffer and é represents the Q-function parameters, treated as constants dur- ing policy update. The policy update involves gradient ascent with respect to the policy network parameters 6c
V.. ( (mmo)))) (a..a)
(16)
The critic update follows the same rule as in DDPG, with actions stored in the replay buffer being either the flow model output or projected actions. The proposed method can be extended other RL algorithms like SAC or PPO, as normalizing flows enable the computation of log probabilities of actions required during training. The pseudo code FlowPG is provided in Algorithm 2.
23


Reviews:
[{"review_id": "kizZW2pOt9", "reviewer": "Reviewer_osn9", "summary": "The paper presents acceptance-rejection augmented MDPs, a novel framework proposed to improve action-constrained reinforcement learning. The method aims to address the computational inefficiencies and architectural complexities in existing Action-Constrained Reinforcement Learning (ACRL), by incorporating two key innovations: an acceptance-rejection mechanism to filter out infeasible actions, and an augmented MDP to optimize policy training towards feasible regions by penalizing constraint violations. The framework is empirically tested against state-of-the-art ACRL benchmarks in robotic control and resource allocation domains. Overall, the idea of combining the acceptance-rejection method with the augmented MDP is interesting, but the work could be improved in both the methodology and the experimental demonstration (see Weakness and Questions below for further details).", "questions": "1. How can the augmented MDP approach guarantee that actions remain within constraints? Even if complete avoidance of constraint violations is not feasible, it would be useful to establish an understanding, such as defining a convergence rate and probability that constraint violations will diminish to a certain margin of error over time.\n\n2. Can the scalability of the proposed ARAM framework be demonstrated in high-dimensional action spaces, through a comparative analysis with existing ACRL methods?\n\n3. Although the authors state \"... we directly leverage the multi-objective extension of SAC that can learn policies under all the penalty weights.\" (page 2), it seems that multi-objective reinforcement learning approaches might be sensitive to hyperparameters, such as the penalty weight. Is ARAM similarly sensitive to this hyperparameter, particularly in its convergence behavior? Additionally, how is the penalty weight selected in practice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The computational overhead caused by solving QPs is challenging in existing ACRL methods, particularly scaling to high-dimensional action spaces.\n\n2. The paper integrates a multitude of concepts such as action-constrained reinforcement learning, Acceptance-rejection\nmethod, and Augmented MDPs. These topics have been at the forefront of recent research trends.\n\n3. The paper is well-organized, with clear explanations of the background, the proposed methods, theoretical foundations (including Proposition 1), and detailed experimental setups.", "weaknesses": "1. The two main components, i.e., the acceptance-rejection method and augmented unconstrained MDPs, seem somewhat contradictory. They represent opposing optimization strategies: the acceptance-rejection method directly filters out actions outside the feasible set, potentially leading to overly conservative decisions; then, the augmented MDP penalizes constraint violations in a gradual way, providing a “soft” learning approach for the agent. However, this softer approach does not fully guarantee that actions remain within the feasible set.\n\n2. While some theoretical insights are provided, including optimality equivalence in Proposition 1, key theoretical gaps remain unaddressed. Specifically, for augmented MDPs, there is no guarantee that actions won’t violate constraints, i.e., a critical issue in action-constrained reinforcement learning.\n\n3. The introduction highlights the computational challenges of scaling quadratic programs to high-dimensional action spaces. However, the experimental analysis in Section 5 primarily explores action constraints in relatively lower-dimensional spaces. The performance of ARAM in high-dimensional action spaces (such as complex robotic manipulators) would benefit from more in-depth analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zQIYVlS9be", "reviewer": "Reviewer_g8E7", "summary": "This paper proposes a novel framework for action-constrained reinforcement learning, named ARAM, which integrates an acceptance-rejection method and an augmented Markov decision process to alleviate the low action acceptance rate problem under ARM. ARAM aims to avoid the computational burden associated with quadratic programs in existing ACRL methods while achieving less constraint violations. The authors demonstrate through experiments on robot control and resource allocation tasks that ARAM achieves faster training, better constraint satisfaction, and reduced action inference time compared to various recent ACRL algorithms.", "questions": "- Does the choice of different target distributions in ARM have a significant impact on the final results?\n- Is the ARM’s acceptance rate analyzed in detail across the different stages of training? It would be useful to understand if and how the acceptance rate varies in the proposed setup.\n- What role do quadratic programs (QP) play in ARAM? QPs are not included in Algorithm 1 or Figure 2.\n- How is MORL implemented in ARAM? Does it significantly increase computational requirements?\n- The layout of Algorithm 1 in the paper needs adjustment.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-written paper with clear objectives.\n- Although the two key improvements (ARM and AUTO-MDP) are straightforward, the experimental results demonstrate the elegance and efficiency of this method in addressing the ACRL problem.\n- The paper is well-supported with experiments that compare ARAM against various benchmarks, showcasing improvements in training speed, constraint satisfaction, and inference time.\n- The authors have the intention to share the code.", "weaknesses": "- The experiments are only conducted in simple simulation environments. Applying the method to real-world scenarios would make the results more impressive and convincing.\n- Lacks experimental comparisons of the ARM acceptance rate.\n- The paper lacks a discussion on the limitations of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hd1r0p2xWg", "reviewer": "Reviewer_meyg", "summary": "This paper aims to adapt the standard unconstrained RL method to ACRL via two proposed techniques: an acceptance-rejection method and an augmented two-objective MDP. It compares ARAM with various recent benchmark ACRL algorithms on MuJoCo tasks and Resource allocation for networked systems, showing the superiority of ARAM in terms of training efficiency, valid action rate, and per-action inference time.", "questions": "1. Could you provide the value of $M$ and $\\kappa$ for all the tasks?  Do you need extra tuning on these two hyperparameters? \n2. Could you provide the computation infrastructure?\n3. Is it possible that ARAM with a single setting of fixed $\\lambda$  outperforms other baselines across MuJoCo locomotion tasks? It is suggested that the authors conduct an experiment with a single fixed λ across all MuJoCo tasks and report those results, or explain why such an experiment might not be feasible or meaningful.\n4. I find in Table 2 that NFWPO enjoys the highest valid action rate. That indicates that it requires a smaller number of QP operations, as opposed to DPre+ and SPre+. Yet in Figure 4, NFWPO shows the highest number of QP operations, which seems conflict to with the analysis in Lines 456-462. Please clarify this apparent contradiction and, if necessary, correct either the data or the analysis.\n5. In the last row of Table 2, 0.77 is much lower than 0.84, thus I suggest not making it bold.\n6. As claimed in the abstract, \"We propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifications\". Could the authors provide ARAM techniques with another backbone standard RL algorithm to support this claim?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. I appreciate the experiments in terms of the considerable evaluation metrics.\n3. I am not very familiar with the ACRL field. From my perspective and limited knowledge of this field, I appreciate the proposed direction to solve the ACRL problem, I think it is a new try.", "weaknesses": "1. The proposed preference distribution tuning procedure seems a bit redundant. It is better to compare ARAM with fixed $\\lambda$  to the baselines for a fair comparison. Otherwise, it is hard to distinguish whether the main components (acceptance-rejection method and AUTO-MDP) are more important or the preference distribution tuning procedure. As we can observe in Figure 11, ARAM is not quite robust to the  $\\lambda$, and a single set of  $\\lambda$  could not achieve satisfactory performance across all tasks.  To make the comparison more fair and isolate the effects of the main components, it would be helpful to suggest that the authors include results for ARAM with a fixed λ alongside the current results in the main paper and hope to see the superiority of ARAM without $\\lambda$-tuning. This would allow readers to better understand the relative contributions of the acceptance-rejection method, AUTO-MDP, and preference tuning. Of course, you may not need to fully resolve this weakness, yet I suggest explicitly mentioning it as a limitation. \n2. Several experimental details have not been provided. For example, what is the value of M and $\\kappa$ for all the tasks, and other parameters used? They are quite important details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zOw20l2lcI", "reviewer": "Reviewer_edLJ", "summary": "This work presents a groundbreaking approach to learning control policies that adhere to strict action constraints, making it perfect for safety-critical and resource-constrained applications. The authors introduce a novel framework that enhances traditional reinforcement learning methods by utilizing the acceptance-rejection method and an augmented two-objective Markov decision process.", "questions": "1. What metrics were used to evaluate the acceptance rate during the experiments, and how were these metrics calculated?\n2. Is there a systematic approach to determine the optimal penalty weight, or is it left to empirical tuning?\n3. Conducting more experiments on hyperparameter sensitivity is better.\n4. Whether introducing AUTO-MDP in the theoretical aspect will affect the guarantee of convergence, while the current article only discusses the optimality.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The framework significantly reduces the computational burden associated with traditional ACRL methods, which often rely on solving complex quadratic programs (QPs). By utilizing the acceptance-rejection method, the need for QP operations is largely obviated, leading to faster training times.", "weaknesses": "The effectiveness of the acceptance-rejection method heavily relies on the quality of the initial policy. If the initial policy is far from optimal, it may result in a high number of rejected actions, which can hinder learning progress and increase computational overhead. While the augmented MDP approach aims to improve acceptance rates, it introduces additional complexity to the learning process. The design and tuning of the penalty function and self-loop transitions may require careful consideration, and improper tuning could lead to suboptimal performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)