Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking
Authors: Roland Stolz
Hanna Krasowski
Jakob Thumm
Michael Eichelbeck
Philipp Gassert
Matthias Althoff
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Policy Gradient
Action Masking
Robotics
Continuous Actions
Primary area: reinforcement_learning
TL;DR: This work introduces three action masking methods for continuous action spaces to focus the exploration of reinforcement learning on state-specific relevant actions, which enhances learning efficiency and effectiveness.

Abstract: Continuous action spaces in reinforcement learning (RL) are commonly defined as multidimensional intervals. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning because the typically large global action space leads to frequent exploration of irrelevant actions. Yet, little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions. Focusing learning on these relevant actions can significantly improve training efficiency and effectiveness. In this paper, we propose to focus learning on the set of relevant actions and introduce three continuous action masking methods for exactly mapping the action space to the state-dependent set of relevant actions. Thus, our methods ensure that only relevant actions are executed, enhancing the predictability of the RL agent and enabling its use in safety-critical applications. We further derive the implications of the proposed methods on the policy gradient. Using proximal policy optimization ( PPO), we evaluate our methods on four control tasks, where the relevant action set is computed based on the system dynamics and a relevant state set. Our experiments show that the three action masking methods achieve higher final rewards and converge faster than the baseline without action masking.

Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking
Roland Stolz!*
Hanna Krasowski 1.2,
Jakob Thumm!
Michael Eichelbeck
Philipp Gassert
Matthias Althoff
Technical University of Munich, University of California, Berkeley, "Munich Center for Machine Learning
{roland.stolz, hanna krasowski)@tumm
Abstract
Continuous action spaces in reinforcement learning (RL) are commonly defined multidimensional intervals. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning because the typically large global action space leads to frequent exploration of irrelevant actions. Yet, little task knowledge can be sufficient identify significantly smaller state-specific sets relevant actions. Focusing learning on these relevant actions can significantly improve training efficiency and effectiveness. In this paper, propose focus learning on the set of relevant actions and introduce three continuous action mask- ing methods for exactly mapping the action space the state-dependent relevant actions. Thus, our methods ensure that only relevant actions are executed, enhancing the predictability of the RL agent and enabling its use in safety-critic applications. We further derive the implications of the proposed methods on the policy gradient. Using proximal policy optimization (PPO), we evaluate our meth- ods on four control tasks, where the relevant action set is computed based on the system dynamics and a relevant state set. Our experiments show that the three action masking methods achieve higher final rewards and converge faster than the baseline without action masking.
of
Introduction
Reinforcement learning (RL) can solve complex tasks in areas such as robotics 13 games (35], and large language models (26). Yet, training RI. agents is often sample-ineffciiet due frequent exploration actions, which are irrelevant to learning good policy. Irrelevant actions are actions that are either physically impossible, forbidden due to some formal specification, or evidently counterproductive for solving the task. Since the global action space is typically large in relation to the relevant actions in each state, exploring these actions frequently can introduce unnecessary costs, lead to slow convergence, or even prevent the agent from learning suitable policy.
Action masking mitigates this problem by constraining the exploration the set of relevant state- specific actions, which can obtained based on task knowledge. For example, when there is opponent within reach in video games, attack actions masked from the action space (43). Leveraging task knowledge through action masking usually leads to faster convergence and also improves the predictability of the RI agent, especially when the set of relevant actions has specific
"The first two authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

notion, such as being the set of safe actions. For instance, if the set of relevant actions is set of verified safe actions, action masking can be used to provide safety guarantees [O T9).
When the relevant action set is easy to compute, action masking usually benefits RI. by improving the sample efficiency and is the quasi-standard for discrete action spaces [34 46). e-g., in motion planning
[8 18] 23 30 42), games [14 15| 43 and power systems (21] 38). However, real-worid systems operate in continuous space and discretizing might prevent learning optimal policies. Furthermore, simulation of real-world systems is computationally expensive, and developing RL agents for them often requires additional real-world training 45 Thus, sample efficiency is particularly valuable for
these applications.
In this work, propose three action masking methods for continuous action spaces. They can employ convex set representations, e.g., polytopes or zonotopes, for the relevant action set. Our action masking methods generalize previous work in [19), which is constrained to intervals as relevant action sets. This extends the applicability of continuous action masking to expressive convex relevant action sets, which is especially useful when action dimensions are coupled, e.g., for a thrust-controlled quadrotor. To integrate the relevant action set into RL, we introduce three methods: the generator mask, which exploits the generator representation of zonotope, the ray mask, which projects an action into the relevant action set based on radial directions, and the distributional mask, which truncates the policy distribution to the relevant action set. summary, our main contributions are:
We introduce continuous action masking based on convex sets representing the state- dependent relevant action sets;
We present three methods to utilize the relevant action sets and derive their integration in the backward pass of RL with stochastic policies;
We evaluate our approach on four benchmark environments that demonstrate the applicability of our continuous action masking approaches.
2
Related literature
Action masking has been mainly applied to discrete action spaces 88 10 14
19 18 18 30
38 41 42, 43 46). Huang et al. (15) derive implications on policy gradient RL and evaluate their theoretical findings real-time strategy games. They show that masking actions leads to higher training efficiency and scales better with an increasing number of actions than penalizing the agent for selecting irrelevant actions. Huo al. [4] have extended [15] to off-policy RL and have observed similar empirical results.
Action masking for discrete action spaces can be categorized by the purpose of the state-dependent relevant action set. Often, the set is obtained by removing irrelevant actions based on task knowledge 14
[15]. While the relevant action sets are usually manually engineered, recent study [46] takes data-driven approach and identifies redundant actions based on similarity metrics. Another common interpretation of the
15
21
30 42), e.g., executing a harvesting action before goods are produced
relevant action set that it only includes safe actions [10 18
23
38 41). These works typically
use the system dynamics to verify the safety of actions. A safe action avoids defined unsafe areas or complies with logic formulas. In our experiments, the relevant action sets are either state-dependent safe action sets global relevant action set modeling power supply constraints.
For continuous action spaces, there is work on utilizing action masking with multidimensional intervals (hereafter only referred to intervals) as relevant action sets (19.. In particular, the proposed continuous action masking represents relevant action sets by intervals that reflect safety
constraints and employs straightforward re- re-normalizatin to map the action space to the relevant action set. In this paper, we generalize to more expressive relevant action sets and demonstrate the applicability to four benchmark environments.
3 Preliminaries
As the basis for our derivations of the three masking methods, we provide concise overview of RI. with policy gradients. Further, we define the considered system dynamics as well as the set representations used in this work.

---

3.1 Reinforcement learning with policy gradients
Markov decision process is a tuple (S, A,T,, consisting of the following elements: the observable and continuous state set C T the action set A C R., the state- transtrraniiionn distribution T('laa,)) s), which is stationary and describes the transition probability to the next state s' E C Rn from the current state S when executing action
A, the reward r S A IR, and the discount factor for future rewards [37]. The goal of RL is to learn a parameterized policy mo (a)s) that maximizes the expected reward max, Ex Li=0 7 az).
For policy gradient algorithms, learning the optimal policy (a(a)s is achieved by updating its parameters / using the policy gradient (36) Thm. 2]
VJ((T)) = E. (Vg log (x (a. (a, s)]
(1)
where An (a, s) is the advantage function, which represents the expected improvement in reward by taking action a in state s compared to the average action taken in that state according to the policy (a(ss)) An estimation of the advantage Axo (a, s) is usually provided by neural network.
3.2 System model and set representations
We consider general continuous-time systems of the form
= [(s a, w),
(2)
where E W C Ra" denotes a disturbance. The input is piece-wise constant with sampling interval At. We assume S, A. and W to convex.
Zonotopes are a convex set representation well-suited for representing the relevant action set, due to the efficiency of computing their Minkowski summ and linear maps. zonotope Z IRN with center E IR N, generator matrix G E IR N and scaling factors 8 E IR. is defined as
Z == G3 || s 1 (c. G)
Additionally, let us denote that G(..)) returns the i-th column vector of the generator matrix G. The Minkowski addition Z1 @ Z2 of two zonotopes Z1. Z2 and the linear map M M21 of zonotope Z are given by [2 Eq. 2.1]
2 22 (a (a1 c2,
,2,,, G G2])2,
(4a) (4b)
M2, (Mc, MG1)2.
Continuous action masking
To apply action masking, a relevant action set (s) C A has to be available, which constrains the action space based on task knowledge. Let us denote the state-dependent relevant action set as AT(s) A. From on, we omit the dependency on the state to simplify notation. For our continuous action masking methods, we specifically require the following two assumptions:
Assumption 1. The relevant action set A' is convex and its center and boundary points are com- putable.
The policy Tg : S x A + R, of the agent is represented by a parameterized probability distribution a no(als).
Assumption 2.
Common convex set representations that fulfill Assumption [ are polytopes zonotopes. Our continuous action masking methods transform the policy To (als). for which the parameters 0 usually specify neural network, into the relevant policy Th S A" - R2 through functional h (II x P(A)) + II". Here, II is the space of all policies. II is the space of all relevant policies, and P(A) is the power set of all A". The transformation defined as
(5) and thereby ensures that a" E AA always holds. Please note that policy gradient methods only require the ability to sample from and to compute the gradient of the policy distribution. Therefore, explicit closed-form expressions for 7 (a" [s] and are not necessary.
a" ~' <a"""aa"" s) h (a
In the following subsections, we introduce and evaluate three masking methods: generator mask, ray mask, and distributional mask, as shown in Fig. [ We derive the effect of each masking approach on the gradient of the objective function for stochastic policy gradient methods.
3

---

Ar
relevant action set
A
latent action set relevant action
AA'
03
G (
a"
((,
center of A"
(a(aass)
G
generator mat. of A
G(.1)
To (a(ss
relevant policy
A
A
A
A
A'
QA
a
(a) Ray mask
(b) Generator mask
(c) Distributional mask
Figure I: Illustration of masking methods action space A with a hexagon-shapel relevant action set A"" The ray mask radially maps the actions towards the center of the relevant action set. The generator mask employs the latent action space A', which is the generator space of the zonotope modeling the relevant action set. The distributional mask augments the policy probability density function so that it is zero outside the relevant action set.
4.1
Ray mask
The ray mask maps the action set A to A by scaling each action a alongside a ray from the center of A" to the boundary of A, shown in Figure Ia Specifically, the relevant policy my results from mapping the action a, sampled from n (a|s), using the function ØR A + A":
A (a) (a-c).
a" gn (a) c+
Aa
Here, (a) and (a) denote the distances of the action a to the boundaries of the relevant action set and action space, respectively, measured from the center the relevant action set e the direction of a. Note that computing A (a) for a zonotope requires solving a convex optimization problem, as specified Appendix A..4 Yet, the ray mask applicable for all convex sets, for which we can compute the center and boundary points. Since 9R (a) is bijective (see Appendix A. for detailed proof), we can apply a change variables [5] Eq. 1.27] to compute the relevant policy
"o(a" s) To (oR (a"))s) det (d
da 9R (a')
where 9R (a") is the inverse of gn- In general, there is closed form of the distribution ny available. However, for stochastic policy gradient-based RI, we only require to sample from zÿ and compute its policy gradient. Samples from (a"ss))) a ~ (a]s))) followed by computing a" = IR (a). The policy gradient is derived next. Proposition 1. Policy gradient for the ray mask.
created by sampling from the original policy
The policy gradient of =a](alls) for the ray mask is VeJ (TT(a"(s) = Exg Vo log (c(ss))) (a", where Axy (a", s) is the advantage function associated with (a(asss)
(8)
Proof. The determinant in is independent of 0, i.e.,
d da 9R (a')) = O,
Vødet
(9)
which simplifies the score function of (s(aals) to
V log TV (a*ls) = Ve log (alss).
(10)
Combining (10) and the general form of the policy gradient in (T) for (a"))))) results in
8

---

4.2 Generator mask
Zonotopes can be interpreted as the map of a hypercube the generator space to lower-dimensionll space (see Section 3.2). The generator mask is based on exploiting this interpretation by letting the RL agent select actions in the hypereube the generator space. Since the size of the output layer of the policy network cannot change during the learning process, we fix the dimension of the generator space. The generator mask requires the following assumption:
Assumption The relevant action set (s) is represented by a zonotope (c(s), G(s)) 2. with G E NxP and c. RRN, and state cate-invariant number of generators P.
Note that in practice, Assumption 3 can often be trivially fulfilled by choosing sufficiently many generators, and the number of generators P is usually the output dimension of the parametrized policy. The domain of the policy To (a(a|s) is the hypercube A' = [-1, 1J which can be interpreted latent action space, and the domain of the relevant policy oz (s) is subset of the action space AA S A (see Figure lb).
To derive the policy gradient of the generator mask, we assume:
Assumption 4. To (a)s)) is a parametrized normal distribution ((;; Ee).
Proposition 2. The relevant policy of the generator mask is
(a)ss = NNa; Gjo + GEoG").
(11)
Proof. The generator mask gG A' + A" is: a" ga c + Ga,
(12)
which is a linear function. Therefore, the proof directly follows from the linear transformation of multivariate normal distributions [Thm. 3.3.3) 40).
Note that the ray mask and generator mask are mathematically equivalent to the approach in [19 the relevant action set is constrained to intervals. Since gG derive the gradient through a change of variables, as for the ray mask.
gG(a) (a) is not bijective in general, we cannot
Proposition 3.
The policy gradient for Th (7(("s)) as defined in III with respect to μi and Ee is
V no log (a (s) G. (GEaG*)) 5 (a" - c GGuo),
(13)
Vx, log T = [ - (GE CT (GE -c- (c-oo
(14)
(a cc' Gpo)) (GI/"")
Proof. The proposition is proven in Appendix A.2
Note that for the special case where G-1 exists. i.e., G is square and non-singular, the expressions in (13) and (1) simplify to Vu log To (a]s),, and Vv., log me (a|s), respectively, when Assumption holds (see Proposition 5 Appendix A.2). While the generator matrix will commonly not be square, as usually >> N, there are cases where N is valid choice, e.g., if there a linear dependency between the action dimensions for the 2D Quadrotor dynamics (see 40
4.3 Distributional mask
The intuition behind the distributional mask comes from discrete action masking, where the probability for irrelevant actions is set to 0 [15]. For continuous action spaces, aim to achieve the same by
ensuring that actions are only sampled from the relevant action set A",, by setting the density values
of the relevant policy distribution (a]ss))) to zero for actions outside the relevant action set (see Figure [T]. For the one-dimensional case, this can be expressed by the truncated distribution [7] In higher dimensions, the resulting policy distribution is
δ(a, T) L no dā
<a(ss)
(15)
5

---

where ő(a, s) is the indicator function
a 8) {1 a E A", otherwise.
(16)
Since there is no closed form of this distribution, we employ Markov chain Monte Carlo sampling to sample actions from the policy. More specifically, we utilize the random direction hit-and-run algorithm: a geometric random walk that allows sampling from a non-negative, integrable function f FRR + R4,, while constraining the samples to a bounded set (44). The algorithm iteratively chooses a random direction from the current point, computes the one-dimensional. truncated proba- bility density of f along this direction, and samples new point from this density. The approach is particularly effective for high dideemsioiinll spaces where other sampling methods might struggle with convergence efficiency. For the distributional mask, f is the policy Te (a(ss)) and A" is the set. As suggested by (22). we execute N iterations before accepting the sample. To estimate the integral in (15), we use numerical integration with cubature [11], which is method to approximate the definite integral of a function I TR - IR over a multidimensional geometric set.
Proposition The policy gradient for the distributional mask is
V log (a"lss)) V log (a(a)s) V log [ To(āls)dā.
(17)
Proof. Equation (17) is obtained by calculating the gradient of the logarithm of (5).. The indicator function ((a,s) not continuous and differentiable, which would necessitate the use of the sub-
gradient for learning. However, since a' E A' always holds, the gradient has to be computed for the continuous part of (a only, and thus Φ(a, s) can be omitted.
Since the gradient of the numeric integral fA- To (ā djā is intractable for zonotopes, we treat the integral as constant in practice, and use Vo log Tp (a"1s) (s) x Vo [ogsss (alss) We discuss potential improvements in Section 5.3
Numerical experiments
We compare the three continuous action masking methods in four different environments: The simple and intuitive Seeker Reach-Avoid environment, the 2D Quadrotor environment to demonstrate the generalization from the continuous action masking approach in [19), and the 3D Quadrotor and Mujoco Walker2D environment to show the applicability to action spaces of higher dimension. Because the derivation of the relevant action set is not trivial in practice, we selected four environments for which we could compute intuitive relevant action sets. In particular, for the Seeker and Quadrotor environments, the relevant action set is safe action set since it is computed so that the agent does not collide (Seeker) or leave a control invariant set (2D and 3D Quadrotor). For the Walker2D environment, the relevant action set is state-independent and models a power supply constraint for the actuators.
For the experiments, we extend the stable-baseline3 29 implementation of proximal policy opti- mization (PPO) (33] by our masking methods. PPO selected because it is widely used algorithm in the field of RL and fulfills both Assumptions 2 and by default. Apart from the masking agents, we also train a baseline agent with standard PPO that uses the action space A and a replacement agent, for which action outside of A" is replaced by uniformly sampled action from A" (see [19] for details). The replacement agent is an appropriate comparison to the masking agents since only relevant actions are executed while the replacement is implemented which is usually casier than an implementation as part of the policy as for the masking methods. We conduct hyperparameter optimization with 50 trials for each masking method and environment. The resulting hyperparameters are reported in Appendix A.9 All experiments are run on a machine with Intel(R) Xeon(R) Platinum 8380 2.30 GHz processor and TB RAM.
part of the environment,
5.1 Environments
We briefly introduce the three environments and their corresponding relevant action sets A". Parame- ters and dynamics for the environments are detailed in Appendix A.5

---

s
A
Sar
A action space
A
Ar
relevant action set
S
state space
relevant state set S 5 O
O
obstacle
agent position goal position
The Seeker Reach-Avoia environment with state and action space. The agent (black) has to reach the goal (gray) while avoiding the obstacle (red). The center of the action space is illustrated by a cross and the relevant action set A' for the current state is shown in green. The state set reachable at the next time step, by the relevant action set, is Sac.
Figure 2:
5.1.1 Seeker Reach-Avoid
This episodic environment features an agent navigating 2D space, tasked with reaching a goal while avoiding circular obstacle (see Figure 2) It is explicitly designed to provide intuitive relevant action set AA.. The system defined as in Section 3.2 The dynamics for the position of the agent = [Sg. sy] and the action a -a ay] is $ = a, and there are no disturbances.
The environment is characterized by the position of the agent, the goal position s". the obstacle position and the obstacle radius ra- These values are pseudo-randomly sampled at the heginning of each episode, with the constraints that the goal cannot be inside the obstacle and the obstacle blocks the direct path between the initial position of the agent and the goal. The reward for each time step is
100
if goal reached,
r(a, s)
- 100
if collision occurred, otherwise.
(18)
-1 || 8
We compute the relevant action set .4" so that all actions that cause a collision with the obstacle or the boundary are excluded (see Appendix A.3.3).
5.1.2 2D Quadrotor
The 2D Quadrotor environment models a stabilization task and employs an action space where the two action dimensions are coupled, i.e., rotational movement originating from differences between the action values and vertical movement is proportional to the sum of the action values. The relevant action is computed based on the system dynamics and a relevant state set (see Appendix, Eq. 33 The reward function is defined as
r(a, s) exp (-I1s
0.01
Q1 1imi
02
02.min
III)
(19)
a1 range
02,range
where is the stabilization goal state, a = [a a2] the two-dimensional action, aj.min the lower bound for the actions in dimension i, and range is the absolute difference between the lower and upper bound for the actions dimension
5.1.3 3D Quadrotor
The third environment models stabilization task for quadrotor defined in (16). The quadrotor has four action dimensions. We use the same reward (see (19) and the same calculation approach for relevant action set (see Appendix, Eq. (33 as for the 2D Quadrotor.
5.1.4 Mujoco Walker2D
The relevant action sets of the two Quadrotor and the Seeker environments are sets that only contain safe actions for the current state. Computing safe action sets requires considerable domain knowledge
7

---

Baseline
Replacement
Ray Mask
Generator Mask
Distributioaal Mask
2T> Quadrotor
3D Quadrotor
-0.6
-0.2
Reward
0.4
Reward
-0.8
0.6
0.8
-1
0
4
8
12
16
20
Steps (in 10")
Steps (in 10*)
Seeker Reaih
Walker21)
2,000
25
1,500
Reward
0
Reward
1,000
50
500
- 100
0
125
Steps (in 10a)
Steps (in 10
Figure 3: Average reward curves for benchmarks with transparent bootstrapped 95% confidence interval.
and for our case solving an optimization problems (see Appendix A.3 However, action masking is not restricted to safe relevant action sets, which we aim to demonstrate on the Mujoco Walker2D environment (39) We extend the environment with termination criterion, which ends an episode, when the the action violates the constraint ||a||2 S Op. This can be viewed as constraining the cumulative power output on all joints battery with a maximum power output that should not be exceeded in practice. Accordingly, we define the relevant action set as static set
maximum value ap. We motivate this constraint by having
A
faa [a/l|a/l2 S ap).
(20)
We under-approximate this relevant action with zonotope that consists 36 generators.
5.2 Results
The reported training results are based on ten random seeds for each configuration. Figure shows the mean reward and the bootstrapped 95% confidence intervals [27] for the four environments and Table depicts the mean and standard deviation of the episode return during deployment. First, we present the results for the Seeker and Quadrotor environments which the relevant action set is state-dependent and only includes safe actions. Then, we detail the results for the Walker2D environment with a static relevant action set.
For the Seeker and Quadotor environments, the baseline, i.c., PPO, converges significantly slower or all (see Figure Additionally, the initial rewards when using action masking are significantly higher than for the baseline, indicating that the exploration is indeed constrained to relevant task- fulfilling actions. During deployment (see Table T). the baseline episode returns are significantly lower than for action masking, while the three masking methods perform similarly. More specifically. for the Seeker environment, the relative volume of the relevant action set compared to the global action space is average 70%, and all action masking methods converge significantly faster to high rewards than the baseline. The generator mask and the distributional mask achieve the highest final reward. Further, action replacement performs better than the baseline but significantly worse than

---

Table 1: Mean and standard deviation of episode return for ten runs per trained model.
Seeker
2D Quad.
3D Quad.
Walker2D
Baseline
-71.03 + 19.67
-0.80 = 0.15
-1.00 + 0.00
1.19 = 0.06
Replacement
-60.21 + 19.98
-0.25 = 0.03
-0.68 0.68:00088
234.93 + 129.5
Ray
-20.45 + 16.39
-0.26 0.05
-0.63 + 0.03
1941.82 + 992.83
Generator
18.60 + 22.02
-0.25 0.02
0.68+0.00
1443.62 + 702.7
Distributional
- -13.66 + 19.97
-0.23 0.02
-0.66 + 0.02
masking. For the 2D Quadrotor, the relative volume is on average 28%, which is much smaller than for the Seeker environment. In the 2D Quadrotor environment, the ray mask, generator mask, and action replacement achieve the highest reward. While the baseline converges significantly slower, the final average reward similar to the one of the distributional mask. Please note that the confidence interval for the baseline is significantly larger, because three of the ten runs do not converge and, on average, exhibit reward of one throughout training. Additionally, we observed that if we constrain the relevant action set to intervals for this environment, our optimization problem in 33 often renders infeasible, because the maximal relevant action set cannot be well approximated by an interval. Thus, the masking method proposed in [19) is not suitable for the 2D Quadrotor task. The results on the 3D Quadrotor are similar to those on the 2D Quadrotor; again, the generator mask converges the fastest but yields final reward similar to that of the ray mask and action replacement. The relative volume of the relevant action set to the global action space is on average 25%. Notably, in this environment, the baseline does not learn meaningful policy. Based on these three environments with dependent relevant action sets, it seems that action masking performs better than action replacement when the relative volume is not too small.
The training results of the Walker2D experiment are shown in Figure 3 While the generator and ray mask both learn a performant policy, the ray mask outperforms by significant margin. The lower performance of the generator mask is likely due to the high-dimensional generator space with 36 dimensions. This is supported by initial experiments with 12 generators (i.e., a more conservative under-approximmatin of the L2-norm) where the generator mask performed better compared to the ray mask. Replacement performs significantly worse than the two masking approaches, and the PPO baseline does not learn a meaningful policy since the environment is frequently reset due to violations of the power constraint in (20) The frequent terminations occur since in six action dimensions the relative volume of the unit ball compared to the unit box is & 8%. i.c., more than 92% of actions are violating the power constraint. To compare masking to learning bascline, we also evaluated standard PPO without constraints, which performs slightly better than ray masking but almost always uses actions outside A" (see Appendix|A.. A.7p. The deployment results in Table reflect similar results as the training; the ray mask achieves better rewards than the generator mask, followed by replacement, and the baseline performs the worst. We excluded the distributional mask for the Walker2D, since its computation time is approximately 170 times slower than the baseline, compared to a 1.6 increase for the generator, 2.7 for the ray mask, and 2.5 for action replacement (see Table 3 The severely increased computational cost for the distributional mask arises from the geometric random walks, which scale cubically with action dimensions.
5.3 Discussion and limitations
Our experimental results indicate that continuous action masking with zonotopes can improve both the sample efficiency and the final policy of PPO. While the sample efficiency is higher in our experiments, computing the relevant action set adds computational load as shown by the increased computation times (see Appendix A.6). Thus, in practice, tight relevant action set might require more computation time than the additional samples for standard RL algorithms. Yet, if the relevant action set provides guarantees, e.g., is a set of verified safe actions, this increased computation time is often acceptable. Additionally, the computational effort for the masks differs. Given relevant action zonotope, the generator mask adds a matrix-vector multiplication, which scales quadratically with action dimensions, the ray mask is dominated by the computation of the boundary points, which scales polynomially with action dimensions (20), and the distributional mask scales cubically with
the dimension of the action space due to the mixing time of the hit-and-rum algorithm 22). Note that

---

for the Walker2D environment with six action dimensions, the computation time for the hit-and-rum algorithm is so high that the distributional mask evaluation was infeasible.
The ray mask and generator mask are hased on functions OR and 9G that map to relevant actions. There are two different approaches of incorporating these functions into RL algorithms. One is to apply the mapping as part of the environment on an action that is sampled by standard RL policy. The second option, which we use, is to consider the masking as part of the policy, which creates the relevant policy as in ( This has three main benefits over integrating masking as part of the environment. First, the actions passed to the environment are better interpretable, e.g. for the generator mask, adding the masking mapping function to the environment leads to a policy that samples actions from the generator space, which commonly does not have an intuitive interpretation. Second, more formulations of the functional h are possible, c.g., the distributional mask, and, third, the mapping function can be included in the gradient calculation. For mathematically sound masking as part of the policy, the correct gradient needs to be derived for each RI. algorithm, and the standard RL algorithms need to be adapted accordingly. However, the empirical benefit could be minor. Thus, future work should investigate the significance of the correct gradient on a variety of tasks. Note that we showed in Proposition | that for the ray mask, the PPO gradient with respect to the original policy and relevant policy is the same. Thus, for the ray mask, it does not matter if it is viewed as part of the policy or the environment.
PPO is common RL algorithm. However, off-policy algorithms such as Twin Delayed DDPG (TD3) [9] and Soft Actor-Critic (SAC) [12] are frequently employed as well. The ray mask and generator mask are conceptually applicable for deterministic policies as used in TD3. Yet, the implications on the gradient must be derived for each RI. algorithm and are subject to future work.
For the distributional mask, treating the integral in (15) as constant with respect to @ is a substantial simplification, which might be an explanation for the slightly worse convergence of the distributional
mask since this introduces off-policy bias. To address this in future work, one could approximate the integral with a neural network, which has the advantage that it is easily differentiable.
We focus this work on the integration of convex relevant action set into RL and assume that an appropriate relevant action set can be obtained. While convex sets are a sigmificant generalization from previous work [19). they might not be sufficient for some applications, c.g., tasks where relevant action sets are disjoint. Thus, future work could include investigating hybrid RI. approaches (25) to increase the applicability to multiple convex sets or non-convex sets, such as constrained polynomial zonotopes [17]. Further, obtaining the relevant action set can be a major challenge practice, in particular when the relevant action set is different for each state. Such high state-dependency is likely when the notion of relevance is safety, while for other definitions of action relevance, the relevant action set might be easy to pre-compute, e.g., excluding high steering angles at high velocities. Additionally, there might be an optimal precision of the relevant action set due to two opposing mechanisms. On the one hand, the larger the relevant action set is with respect to the action space, the smaller the sample efficiency gain from action masking might get. On the other hand, a tight relevant action set might require significant computation time to obtain. Thus, future work should investigate efficient methods to obtain sufficiently tight relevant action sets.
6 Conclusion
We propose action masking methods for continuous action spaces that focus the exploration on the relevant part of the action set. In particular, we extend previous work on continuous action masking from using intervals as relevant action sets to using convex sets. To this end, we have introduced three masking methods and have derived their implications on the gradient of PPO. We empirically evaluated our methods on four benchmarks and observed that the generator mask and ray mask perform best. If the relevant action set can be described by zonotope with fixed generator dimensions and the policy follows a normal distribution, the generator mask is straightforward to implement. If the assumptions for the generator mask cannot be fulfilled, the ray mask is recommended based on our experiments. Because of subpar performance and longer computation time, the distributional mask needs to be further improved. Future work should also investigate broad range of benchmarks to identify the applicability and limits of continuous action masking with convex sets more clearly.
10

---

Acknowledgments and Disclosure of Funding
We thank Matthias Killer for conducting preliminary experiments. We gratefully acknowledge that this project was funded by the Deutsche Forschunssemminnchaft (DFG, German Research Foundation) SFB 1608 501798663, AL 1185/9-1, AL 1185/33-1, and the Bavarian Research Foundation project STROM (Energy Sector coupling and microgrids).
References
[1] Takuya Akiba et al. "Optuna: Next-Generation Hyperparameter Optimization Framework". In: 25th ACM SIGKDD Int. Conf. on Knowledge Discovery & Data Mining. 2019, pp. 2623- 2631.
[2] Matthias Althoff. "Reachability Analysis and its Application to the Safety Assessment of Autonomous Cars". PhD thesis. Technische Universität München, 2010.
[3] Matthias Althoff and Goran Frehse. "Combining zonotopes and support functions for efficient reachability analysis of linear systems". In: IEEE Conf. on Decision and Control (CDC). 2016,
pp. 7439-7446.
[4] Dimitris Bertsimas and John N. Tsitsiklis. Introduction to linear optimization. Athena Scien- tific, 1997.
[5] Christopher M. Bishop. Pattern recognition and machine learning. Information science and statistics. 2006.
[6] Stephen Boyd et al. "A tutorial on geometric programming". In: Optimization and engineering
8 (2007), pp. 67-127.
[7] J. Burkardt. The Truncated Normal Distribution. Department of Scientific Computing Website, Florida State University, Tallahassee. Online resource. 2018.
[8] Xutao Feng et al. "Autonomous Decision Making with Reinforcement Learning in Multi-UAV Air Combat", In: IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC). 2023, pp. 2874- 2879.
[9] Scott Fujimoto, Herke Van Hoof, and David Meger. "Addressing Function Approximation Error in Actor-Critic Methods". In: Proc. of the Int. Conf. on Machine Learning (ICML). 2018, pp. 2587-2601.
[10] Nathan Fulton and André Platzer. "Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Leariin".. In: Proc. of the AAAI Conf. Artificial Intelligence (AAAI). 2018, pp. 6485-6492.
[11] Alan Genz and Ronald Cools. "An adaptive numerical cubature algorithm for simplices". In: ACM Trans. Math. Softw. 29.3 (2003), pp. 297-308.
[12] Tuomas Haarnoja et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor". : In: Proc. of the Int. Conf. on Machine Learning (ICML). 2018, pp. 1861-1870.
[13] Dong Han et al. "A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipu- lation". In: Sensors 23.7 (2023).
[14] Yueqi Hou al. "Exploring the Use of Invalid Action Masking in Reinforcement Learning: Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games". In: Applied Sciences 13.14 (2023).
[15] Shengyi Huang and Santiago Ontañón. "A Closer Look at Invalid Action Masking in Policy Gradient Algorithms". : In: Int. Florida Artificial Intelligence Research Society Conf. Proc. (FLAIRS) 35 (2022).
[16] Shahab Kaynama et al. "Scalable Lafety-Preserringg Robust Control Synthesis for Continuous- Time Linear Systems". In: IEEE Trans. on Automatic Control 60.1 1 (2015), pp- 3065-3070.
[17] Niklas Kochdumper and Matthias Althoff. "Constrained polynomial zonotopes". In: Acta
Informatica 60.3 (2023), pp. 279-316.
[18] Hanna Krasowski, Xiao Wang, and Matthias Althoff. "Safe Reinforcement Learning for Autonomous Lane Changing Using Set-Based Prediction".. In: Proc. of the IEEE Int. Intelligent Transportation Systems Conf. (ITSC). 2020, pp. 1-7.
[19] Hanna Krasowski et al. "Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking". In: Trans. on Machine Learning Research (2023).

---

[20] Adrian Kulmburg and Matthias Althoff. "On the co-NP- Complloepteessssss of the Zonotope Con- tainment Problem". In: European Journal of Control 62 (2021), pp. 84-91.
[21] Lingyu Liang et al. "Enhancement of Distribution Network Resilience: A Multi-Buffer Invalid- Action-Mask Double Q-Network Approach for Distribution Network Restoration". In: 3rd Int. Conf. on New Energy and Power Engineering (ICNEPE). 2023, pp. 1055-1060.
[22] László Lovász and Santosh Vempala. -Hit-and-Rur from a Corner". In: SIAM Journal on Computing 35.4 (2006), pp. 985-1005.
[23] Branka Mirchevska et al. "High-level Decision Making for Safe and Reasonable Autonomous Lane Changing using Reinforcement Learning". In: Proc. of the IEEE Int. Intelligent Truns- portation Systems Conf. (ITSC). 2018. pp. 2156-2162.
[24] Ian M. Mitchell, Jacob Budzis, and Andriy Bolyachevets. Invariant, Viability and Discriminat- ing Kernel -nder-Approximaaion via Zonotope Scaling. 2019. arXiv: 1901 01006
[25] Michael Neunert et al. centinuoous-dsccres reinforcement learning for hybrid control in robotics". In: Proc. of the Conference on Robot Learning. 2020, pp. 735-751.
[26] Long Ouyang et al. "Training language models to follow instructions with human feedback". In: Advances in Neural Information Processing Systems. Ed. by S. Koyejo et al. Vol. 35. Curran Associates, Inc., 2022, pp. 27730-2773 27744..
[27]
Andrew Patterson et al. Empirical Design in Reinforcement Learning. 2023, arXiv: 2304 01315
[28)
André Platzer and Edmund M. Clarke. "The Image Computation Problem in Hybrid Systems Model Checking". In: Hybrid Systems: Computation and Control. 2007.
[29]
Antonin Raffin et al. Babble-Baselines:: Reliable Reinforcement Learning Implementations".
In: Journal of Machine Learning Research 22.268 (2021), pp. 1-8.
[30] Thomas Rudolf et al. "Fuzzy Action-Masked Reinforcement Learning Behavior Planning for Highly Automated Driving". In: Int. Conf. on Control, Automation and Robotics (ICCAR). 2022, pp. 264-270.
[31] Sadra Sadraddini and Russ Tedrake. "Linear encodings for polytope containment problems".
In: IEEE Conf. Decision and Control (CDC). 2019, pp. 4367-4372.
[32] Lukas Schäfer, Felix Gruber, and Matthias Althoff. "Scalable Computation of Robust Control Invariant Sets of Nonlinear Systems". In: IEEE Trans. on Automatic Control 69.2 (2024),
pp. 755-770.
[33]
John Schulman et al. Proximal Policy Optimization Algorithms. 2017. arXiv: 1707. , 06347
[34] Ashish Kumar Shakya, Gopinatha Pillai, and Sohom Chakrabarty. "Reinforcement learning algorithms: A brief survey". In: Expert Systems with Applications 231 (2023).
[35]
Kun Shao al. A Survey of Deep Reinforcement Learning in Video Games. 2019, arXiv:
1912. 10944
[36] Richard S Sutton et al. "Policy Gradient Methods for Reinforcement Learning with Function Approximation". In: Advances in Neural Information Processing Systems. Ed. by S. Solla, T. Leen, and K. Müller. Vol. 12. MIT Press, 1999.
[37] Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed.
MIT Press, 2018
[38] Daniel Tabas and Baosen Zhang. "Computationally Efficient Safe Reinforcement Learning for Power Systems". In: Proc. of the American Control Conf. (ACC). 2022, pp. 3303-3310.
[39] Emanuel Todorov, Tom Erez, and Yuval Tassa. "MuJoCo: A physics engine for control". In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems. 2012, pp. 5026-5033.
[40]
Y. L. Tong. The Multivariate Normal Distribution. Springer Series in Statistics. 1990.
[41] G Varricchione et al. "Pure-past action masking". In: Proc. of the AAAI Conf. on Artificial Intelligence (AAAI). Vol. 38. 19. 2024, pp. 21646-21655.
[42] Yang Xiaofei et al. "Global path planning algorithm based on double DQN for multi-tasks amphibious unmanned surface vehicle". In: Ocean Engineering 266 (2022).
[43] Deheng Ye al. "Mastering Complex Control in MOBA Games with Deep Reinforcement Learning". In: Proc. of the AAAI Conf. on Artificial Intelligence (AAAJ) 34.04 (2020), pp. 6672- 6679.
12

---

[44] Zelda B. Zabinsky and Robert L. Smith. "Hit-and- Run Methods", In: Encyclopedia of Opera- tions Research and Management Science. Ed. by Saul Gass and Michael C. Fu, Boston, MA: Springer US, 2013, pp. 721-729.
[45] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Rematt-RReal Transfer in Deep Re- inforcement Learning for Robotics: a Survey". In: IEEE Symposium Series on Computational Intelligence (SSCI). 2020, pp. 737-744.
[46] Dianyu Zhong. Yiqin Yang, and Qianchuan Zhao. "No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning". In: Proc. the AAAI Conf. on Artificial Intelligence (AAAI). Vol. 38. 15. 2024, pp. 17078-17088
13

---

A Appendix
A.1 Proof of bijectivity of the ray mapping function.
Lemma 1. The mapping function g : A A' the the ray mask g(a) c+
Aar(a)
(a c) is bijective. Aa(a)
Proof. We show that g(a) is bijective by showing that the function is both injective and surjective. For any convex set A" with center c, we can construct a ray from through any a' E AA" as (t) c yt, where y Ta-ella and t E [0, maaz] Since A S A, this also holds Va E A. By construction, any two distinct rays only intersect in c. For all points on a given ray, the scaling factors Aa and A are constants, allowing rewrite g(a) for all (t))) as the linear function:
e-c
AA Xa
(r) = c
(r,(t)
- c).
= g(a1) # g(a2), consider the To show that g(a) injective, i.e. Van, 02 E A, if a1 # 02 following two cases for a1 # 02
Case 1: If aj and 02 are on the same ray, then, di (61) and a2 rr with & + lg. Since (t)) is linear and thereby monotonic, it follows that (41)) # 9 (12) and consequently g(ai) A g(a2).
Case 2: Otherwise, a1 and ag are on different rays and a1 # 02 # c, so g * g(22) follows directly the rays only intersect in c.
Thus, g(a) is injective.
For showing g(a) to be surjective, it is sufficient to show that Va" E A", there exists an a E A, for which ((a) = a".
Consider the ray (t) passing through a". We know that g (r, g(c) = e, and g (r,((maz)) is the boundary point of A" from in the direction d. Moreover, a" lies on the line segment between e and y (r-, max)) Since g (t), is linear and continuous, according to the intermediate value theorem, =t E [0, tmax]. for which g (r- (t*)) a" for any a" A" along the ray rr (t). As can construct such a ray through any point in A', we have shown that for every a" E A", there exists an E A such that g(a) a", thus proving surjectivity.
A.2 Policy gradient for the generator mask
The log probability density function of the relevant policy for the generator mask is - 1-"""""-((""--GGiii
1
1
log (s))ss 2
log (2m) log [GE
(21)
2
We can derive the gradient w.r.t μi as
V No log (a())))))) = v va c- Gpo)² -CCGGu)"(GS)CC"")
(a"
c- (d"= G11a)
2
2 (-2(GE, - c GPo)) Vao (a" - cc - G140)
1
(22)
=
(G
Gl18),
and w.r.t Egg as
V Σg
log Th (a"ls)
1
Vx, log GE
Vr, -c- cc- ("= -
2
( -V GT
(23)
2
1
+ (G ) c- GGo)) - -C-G - GT
2
(GI, Ge (GE c- Guo)(a" - (GE
We can state the following for the special case when the inverse of G exists.
14

---

Proposition 5. IfG G is invertible, Vo logog lo" = V log To(ls) holds for the generator mask.
Proof. If G invertible, we can further simplify
no log ap(aa [s) G² G G (a' Gpo)
= Eg (G - - Gpo)
(24)
Ig (a - μb) = V pa lolss,
and 23 to
1
V log (a()ss))
(Ep) E - - Gpo)(a" - GHu) ( G
2
(25)
1
(Iz²) Eg - ma) - mO) Em = Vxi log (a(()))
2
by using a = --- = (- c)) and thus proving the statement.
A.3
Computation of the relevant action set
A.3.1 General case
Given an initial state so, an input trajectory a(-), and disturbance trajectory w(-)) we denote the solution of (2) at time (so, a(.). w()).. Assuming sampled controller with piecewise-constan input a, we define the reachable set of (2 after one time step At given a set of initial states Sy and
set of possible inputs Á C as
Ra A) (Ear(so, a, w()) Emo E So. 3a E Á, Vt: Bw, E W].
(26)
The reachable set over the time interval [0, At] is defined as
R. Ri (So, A)
U
(S. A).
(27)
rE(O,Ad]
For many system classes it is impossible to compute the reachable set exactly 288 which why we generally compute overapproximations R(-) 2 R-)..
To guarantee constraint satisfaction over an infinite time horizon, we use a relevant state set S In always find an
this work, choose S" be robust control invariant in the sense that input which guarantees that the reachable set at the next time step is contained in S and that all state constraints are satisfied in the time interval in between (32
Ea E A: RSr(S" .) S S", (So.s. .a) S.
(28)
We compute the relevant action set the largest set of inputs that allows us to keep the system the relevant state in the next time step. We define parameterized action set A"))), where p R=" is a parameter vector. The relevant action set is then computed with the optimal program
max
Voi (Ar()))
subject
A² CA
(29)
(S. S S" (S. (p) S,
where Voi(-) is proxy function for the volume in the sense that maximization of Vol is
suboptimally maximizing the volume. In the following, we provide detailed formulation (2) as an exponential cone program.
15

---

A.3.2 Exponential cone program
We consider the discrete-time linearization of our system
8k-1 Ask + Bag + w/:
(30)
where w/ E (sx) additionally contains linearization errors and the enclosure of all possible trajectory curvatures between the two discrete time steps (2) Furthermore, we assume S, A, W, and S" to be zonotopes.
With regard to the input set parameterization, we consider template zonotope with a predefined template generator matrix G. We use the vector P E R of generator scaling factors to scale the generator matrix. The parameterized template zonotope is then given by
A" (p) {c, & diog(p))
(31)
where p- == |e p In the following, we denote the center and generator matrix of specific zonotope Z by c² and G2, respectively. zonotope 21 is contained in zonotope Z2 there exist Te R x xp E pêa, such that [3 Corollary 4]
(32)
|1, 1.
Based the linearized system dynamics in (30), the definitions from 4 and using R as shorthand for (-, we have G [AGso BG GW] and CR. = A Be cW]. Since we already consider trajectory curvature in the disturbance, we only need guarantee that the relevant input set is contained in the feasible input set and that the reachable set of the next time step is contained in the relevant state set. Using the containment condition from (32). define the auxiliary variables 1RR , 1'4", and 3-" and solve
max P
Vol (AA)
subject to
=0 G - G =0=
の R -65- gR=
[TV 3R) 11x0 $1
(33)
G - GdT =0
A- -AAA" =0
[r BAT 1lx0 1
where 0 are zero matrices of appropriate dimensions. By vectorizing R and A and resolving the infinity norm constraints [4, Sec. 1.3], we can obtain a formulation with purely linear constraints. Since computing the exact volume of zonotope is computationally expensive, approximate it by computing the geometric mean of the parameterization vector:
A+
voi(('')))
I
Pt
(34)
With the cost function from 34
the problem
333 an exponential cone program, which is convex
and can be efficiently computed 6
Sec. 2.5].
A.3.3 Seeker Reach-Avoid environment
Because of the dynamics of the Seeker Reach-Avoid environment, we can simplify the relevant action set computation to the following optimization problem:
max
(p))
subject to
AT(p) CA
SA. = A
(35)
SA. C 10,
St 0o = 0,
16

---

where Sa is the reachable set of the agent in the next time step, the set O represents the obstacle, and the box [-10, 10 is the outer boundary of the state space. The constraints represent three intuitive geometric constraints. First, containment of the relevant action set in the action set (usually, the interval [-1, [-1, second, the containment of the relevant state set of the next time step in the environment boundaries, and the last constraint enforces that there no collision with the obstacle Lemma
possible. To enforce these constraints for zonotopes,
the support function [
0z(1)=1fcc Σ Gaab
(36)
where is the vector in the direction of interest. For the first constraint, we need to ensure that the support function for the basis vectors ε1, and e2 (and there negative versions) are less than or equal to 1, i.e.,
paa(ee) PA- (e2)
GA (1.
1 H 1
G
(37)
GA (2,6)
From (36) it is apparent that using the negative basis vectors -61 and c2, just flips the sign of A in
(37.. The constraints for S C [--10, 10]² can be constructed similarly. For simplicity, we express
the clement-wise inequality in 37 using the -norm, and write the optimization problem as
max
(Ai (p))
subject to
+
Σ G <
- Ar
+ Σ ( 1
(38)
||~ + + Σ GGGallle I 10
- A + Σ
G Golllo I 10
n (c +8) + Σ In G S b.
The last constraint represents an ander-approximattion S no 1 enforcing the containment of A" in the halfspace {a|n 2 < b} through the support function p.A- (n). The directional vector o-s where o is the center of the obstacle, and the offset b = n (o n)r where is the radius of the obstacle. This ensures that the intersection between the reachable set 8 and the obstacle O will be empty since the halfspace is constructed tangential to the obstacle at the intersection point of the line from the agent to the center of the obstacle.
To-=|l
A.4 Computation of zonotope boundary points
The boundary point p E R zonotope (c, G) C R. in a direction d e R& starting from point & E & obtained by solving the linear program
min RER subject
a
x + ad c Gy |1 1
(39)
and computing p + ad 20
A.5 Parameters and dynamics for environments
We provide the action space bounds, the state space bounds, and the generator template matrix G for the Seeker and Quadrotor environments in Table 2
17

---

Table Parameters for important sets of environments.
Parameter
Seeker
2D Quadrotor
3D Quadrotor
Lower bound actions Upper bound actions
[-1,-1]
[6.83, 6.83]
[-9.81, -055 -0.5]
[]
[8.59, 8.59]
[2.38, 0.5,
0.3, -0.8, ...
3-3,
Lower bound states
[-10,-10]
-1,-1/12, -x/2]
T/4 -T/4, -3,
[1.7, 2.0, 0.8,
[3. 3, 3,3,,
Upper bound states
[10, 10]
1.0, a/12, m/2] 1 1
1/4, m/4, nt, 3,31
Template matrix G
1 1
0
I1
-1 0
1 -1
The system dynamics the 2D Quadrotor is:
0
(a1 + 02 02)k lkkk sin
9 + (a] + a2 cos(O)
W1
(40)
U22
0
do0 d,0 + no 01 + (2)
0
where the state is ar, z, ④, 2, 0.00 the action is [ai, u2)" and the disturbance is
[wi, w2]" The dynamics are adapted from 24 Eq. 43] so that the actions are two independent thrusts. Additionally, the parameters used in our experiments g =9.81 ms - 2, =1 1/kg, do =70, d1 =17, no =55, and W [-0.08, 0.00 x [0.08, 0.08].
m
The 3D Quadrotor is modeled in twelve-dimensions state space with state
[zc, y, z, d. U, 2, 0 o. ei f. à, O) and diur-dimensional action space with action = (ai a2 0.lT.
The system dynamics are [ē, y, 2, 99.1 9.810, a1, Ó, ò, ú, 02 43, a.)" [16]
For the Walker2D environment, we use the standard parameters of the gymnasium implementation
The relevant action set is a zonotope mnder-approximation of the relevant action set stated in (20) with 36 generators and ap
A.6 Computational time for training
Note that the increased computation time of the masking approaches compared to the baseline is mainly due to the computation of the relevant action sets. Since relevant action set is a zonotope, the generator mask does not require expensive additional computations. The increased runtime for the ray mask mainly originates from the computation the boundary points (see Appendix (A.4). For the distributional mask. the increased runtime is mostly caused by the Markov chain Monte Carlo sampling of the action.
Table 3: Mean and standard deviation of runtime in hours for training runs on the utilized machine.
Seeker
2D Quadrotor
3D Quadrotor
Walker2D
Baseline
0.045 0.001
1.049 + 0.197
0.493 + 0.077
0.350
0.007
Ray mask
0.865 + 0.013
3.140 + 0.097
2.031 + 0.258
0.962 = 0.043
Generator mask
0.699 + 0.016
2.533 + 0.021
1.528
0.010
0.557 = 0.021
Distributional mask
1.620 + 0.017
4.083
+ 0.080
3.130 + 0.045
~ 60
Replacement
0.823 0.015
2.782 + 0.048
1.765
0.033
0.880 = 0.039
18

---

Baseline
Standard PPO
Replacement
Ray Mask
Generatur Mask
Walker2D
2,300
2,000
Reward
1,500
1,000
500
0
Steps 10%)
Figure 4: Average reward curves for Walker2D with transparent bootstrapped 95% confidence interval including standard PPO as additional baseline.
A.7 Quantitative results for Walker2D environment compared to standard PPO
we remove the power constraint that resets the environment whenever an action outside of A" is selected, the standard PPO is able to learn policy that performs slightly better than the ray mask towards the end of training while converging slightly slower (see Figure However, during deployment, only 0.05% of the actions from this policy from within A". Yet, the reward that standard PPO achieves during deployment is on average 2177.57 with standard deviation 1238.42. This is slightly higher than the deployment results for the ray mask (see Table
A.8 Qualitative results for the Seeker environment
presents ten example trajectories for trained agent using each of the five approaches under consideration. Notably, the continuous action masking agents solve the task by safely reaching the goal while the PPO baseline still collides with the obstacle. The replacement approach is ineffective, with the agent frequently failing reach the goal. In fact, for the ten displayed rollouts, no replacement agent reaches the goal. This discrepancy with respect to the training performance where the replacement agent reaches the goal frequently is due to using the policy in deterministic setting for deployment. The simple dynamics of the Seeker environment make it possible to directly visualize the relevant action set A" each time step along the trajectory, which is depicted in the lower half of Figure 5 For the generator mask, the least amount of A" is plotted, which indicates that the generator mask agent reaches the goal most efficiently, i.e., the lowest amount of steps are required.
A.9 Hyperparameters for learning algorithms
We specify the hyperparameters for the three masking approaches, replacement and baseline in the Seeker (Table 4. the 2D Quadrotor (Table and the 3D Quadrotor environment (Table These obtained through hyperparameter optimization with 50 trials using the tree-structured Parzen estimator of Optuna with the default parameters I
19

---

Baseline PPO
Ray
Generator
Distributional
Replacement
Figure 5: Qualitative deployment results for ten initial states one goal-obstacle configuration for the Seeker environment. The top half shows ten trajectories with randomly sampled starting states. The lower half depicts the relevant action set (green polygon) for each time step along one trajectory.
Table 4: PPO hyperparameters for the Seeker environment.
Parameter
Baseline
Ray
Generator
Distributional
Replacement
Learning rate
5.43E 5
8.25E
3.45E
4
3.85E
1.92E -
Discount factor
0.98
0.98
0.98
0.98
0.98
Steps per update Optimization epochs
32
256
2084
32
128
16
Minibatch size Max gradient
128
256
128
0.9
0.9
0.9
0.9
0.9
Entropy coefficient
4.71E - 5
1.66E -
6.61E -
3.33E
1.83E -
Initial log std dev
-1.183
-0.010
-0.255
-1.213
-1.064
Value function coeff.
0.5
0.5
0.5
0.5
0.5
Clipping range
0.1
0.1
0.1
0.1
0.1
GAE X
0.9
0.9
0.9
0.9
0.9
Activation function
ReLU
ReLU
ReLU
ReLU
ReLU
Hidden layers
2
2
2
2
Neurons per layer
32
32
32
Table 5: PPO hyperparameters for the 2D Quadrotor environment.
Parameter
Baseline
Ray
Generator
Distributional
Replacement
Learning rate
1.24E
7.92E
4.34E
3.94E
1.13E
Discount factor
0.99
0.99
0.99
0.99
0.99
Steps per update
256
1024
1024
1024
512
Optimization epochs
32
8
8
8
Minibatch size
128
128
128
128
Max gradient norm
0.9
0.9
0.9
0.9
0.9
Entropy coefficient
8.9E -
5.65E -
5.08E 2
5.99E
5.42E -
Initial log std dev
-0.437
-0.784
-1.251
-1.217
-1.019
Value function coeff.
0.5
0.5
0.5
0.5
0.5
Clipping range
0.1
0.1
0.1
0.1
0.1
GAE X
0.95
0.95
0.95
0.95
0.95
Activation function
ReLU
ReLU
ReLU
ReLU
ReLU
Hidden layers
2
Neurons per layer
256
256
256
256
256
20

---

Table 6: PPO hyperparameters for the 3D Quadrotor environment.
Parameter
Baseline
Ray
Generator
Distributional
Replacement
Learning rate
2.38E
1.0SE
9.24E
5
7.88E
6.25E
Discount factor
0.98
0.98
0.98
0.98
0.98
Steps per update
32
128
128
64
128
Optimization epochs
16
4
Minibatch size
16
32
16
64
64
Max gradient
0.9
0.9
0.9
0.9
0.9
Entropy coefficient
5.85E 5
1.14E -
3.41E -
2.75E -
1.88E -
Initial log std dev
-3.609
-1.793
-1.363
-1.80
-1.582
Value function coeff.
0.5
0.5
0.5
0.5
0.5
Clipping range
0.1
0.1
0.1
0.1
0.1
GAE X
0.9
0.9
0.9
0.9
0.9
Activation function
ReLU
ReLU
ReLU
ReLU
ReLU
Hidden layers
2
2
2
2
Neurons per layer
32
Table 7: PPO hyperparameters for the Walker2D environment.
Parameter
Baseline
Replacement
Ray mask
Generator mask
Learning rate
6.992E 5
4.700E 3 3
2.607E 4
1.719E
Discount factor
0.99
0.99
0.99
0.99
Steps per update Minibatch size
2048
2048
2048
2048
128
64
16
32
Max gradient
0.603
0.336
0.156
0.152
Entropy coefficient
6.559F 7
5.960F 6
4.992F. 6
7.488F. -
Clipping range
0.165
0.131
0.102
0.192
GAE
0.970
0.944
0.919
0.957
Value function coefficient
0.259
0.330
0.181
0.500
Activation function
ReLU
ReLU
ReLU
ReLU
Hidden layers
2
2
Neurons per layer
64
64
64
64
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that focusing state-dependent relevant actions can improve training efficiency and effectiveness of reinforcement learning significantly, which we demonstrate with four experiments in Section [.2 We formally introduce three action masking methods, which can be used with convex relevant action sets, and derive their implications on the gradient in Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We highlighted the assumptions of the proposed action masking methods in Section 4 Additionally, we discuss limitations our experimental results and the
applicability of our methods in Section 5.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattio, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
22

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For all three methods, we assume convex relevant action sets and stochastic policy. Although, as mentioned in Section [53] the ray mask and generator mask could be adapted non-stochastic policies. For the generator mask, we additionally assume the set representation to be zonotope. We propose the adapted policy gradient for each method in Propositions
| and and provide proofs for all three.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the necessary details to implement the three different masking
approaches in Sections [.1 4.2 and 4.3 We provide information needed to re-implement the environments in Sections 5.1 and A.5 The computation of the relevant action is specified in A.3] Further, the hyperparameters used for the training runs are reported in A.9 Any remaining ambiguity can be resolved by the code attached to this submission.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution
specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
23

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We attached the code for the experiments including a detailed readme in the supplementary material to this submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
Sublic/guides Stte///oueuesioioooooooooooooeoo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the utilized method for hyperparameter tuning in Section 5 and specify the used hyperparameters, and optimizer in Appendix A.9
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
24

---

Answer: [Yes]
Justification: We train each configuration with ten random seeds and report the training results with 95% bootstrapped confidence intervals in transparent color in Figure 3 For the reported deployment results and computational runtime (see Table and respectively). we report the mean and Isiima standard deviation.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We reported the compute resources utilized in Section The average com- putation times are reported in Section A.6 Note that the compute resource used for the experiments is shared resource where also other researcher ran their experiments at the same time. Thus, runtime evaluations will only be partially reproducible.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httttt::///tutttt htttp::///nurips.. https///cc/....c//ccccciiieeeee.
Answer: [Yes]
Justification: Our experiments do not involve human subjects or datasets. Regarding safety and security, our approaches can be directly used to prevent agents from performing unsafe actions. We cannot identify any potentially harmful consequences of our algorithms and work in general.
25

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper introduces method to exploit task knowledge via relevant action sets to achieve more efficient and effective reinforcement learning. Given the limited scope of our experiments, a detailed discussion on broader societal impact would very speculative. However, we believe our approach has the potential to expand the use of reinforcement learning applications where training opportunities are limited but task knowledge easily transferable to relevant action sets and where enforcing hard constraints is important.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not work with, or release any datasets, and our models have no risk for potential misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
26

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: As mentioned in Section 5| we use the stable-baseline3 implementation of PPO. Furthermore, we clearly state the origin of the used Quadrotor environments in Sections
5.1.2 and 5.1.3
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi whwodeetehteee.eooeeo com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
27

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "OBvL7ywbPj", "reviewer": "Reviewer_8567", "summary": "This paper proposes mathematical formulations for continuous action masking in reinforcement learning, to incorporate domain-knowledge in the form of state-specific sets of relevant actions. It introduces 3 functional forms to extract relevant actions from the original action space, and consider its effect on the policy gradient. The policy gradient does not change much, and the paper shows that the forms compare similarly and better than learning an agent without any knowledge of the continuous action mask at all.", "questions": "Listed in weaknesses.", "limitations": "The authors discuss several limitations of the work, but the important ones of applicability and baselines need to be addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem of action masking in continuous action space is an underexplored one, but could have major impact in efficacy of agents and incorporating domain-specific knowledge.\n- The proposed continuous action masking could potentially be useful for safety demarcations.\n- The paper provides mathematical frameworks to formulate continuous action masking and also derive their (minimal) effect on the policy gradient.\n- The paper is mostly well-written and explains the mathematical derivations quite well. Quick note that Section 2.1 and 2.2 could be made more integrated, currently they seem completely disconnected.", "weaknesses": "## 1. General applicability of this paper's ideas\nObtaining a state-specific relevant action set can be really hard. The paper, however, makes contradictory statements about this:\n- L5: \"little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions.\"\n- L284-285: \"assume that an appropriate relevant action set can be obtained. Yet, obtaining this can be a major challenge in practice.\"\n\nFrom the experiments on the 3 environments, it already seems like defining the relevant action set requires a lot of domain knowledge about the state space features and the dynamics function.\n\nAs of now, there does not seem to be any way the ideas in this paper could be useful for any practical domain.\n- Can the authors provide some concrete examples of how one can obtain such relevant action sets for problems of practical interest and scale?\n- Can the authors provide any results on a commonly used continuous action space RL benchmark?\n\n## 2. Gains not coming from the policy gradient, but only because of constraining the action space\nThe paper's proposed formulation is interesting because it uses continuous action masking as part of the learning policy and informs the policy gradient update about the continuous action mask. However, when we look at the resultant policy gradients for each mask in Eq. 10, Line173, and Line199, it seems that the policy gradient simply reduces to $\\lambda_\\theta log \\pi_\\theta(a | s)$ for all cases.\n\nSo, the effective change in implementation is just how the action to take in the environment is model: $a^r = g(a)$. But, this **doesn't utilize continuous action masking to improve the policy learning objective** in any meaningful way. Is my understanding correct in this?\n\nAnother observation that validates the claim that policy learning is not influenced much is seen from the results and L248-249. The initial rewards themselves are significantly higher, which means that the action mask just reduces the space of exploration of the agent so much that, as long as it takes a valid action, it would get a high reward.\n\n## 3. Simpler baselines for continuous action masking\nContinuing from the above point, if all that needs to be done is to compare different formulations of g(a), there is a much simpler alternative perspective:\n- Action-Masking as part of environment: Simply, apply the action mask as part of the environment, without changing the PPO objective at all. So, there is an action-masking filter before executing an agent's action in the environment, and ignore if the action is invalid.\n- Sampling-augmented action-masking: Keep sampling actions from $\\pi_\\theta$ until you find a valid action that can pass through the known continuous action-masking map.\n\nThe current PPO baseline is very weak, and does not utilize action-masking at all. It seems most of the learning in PPO is going into the effort of learning the action mask. To really justify this paper's proposed action masking schemes are useful, they must compare against other forms of naive action masking, including the two listed above. This perspective on considering the action mask as part of the environment is also much more generally applicable and does not require any change to the policy gradient update.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful and critical comments which helped us to strengthen our arguments for the utility of action masking. We address your questions below.\n\n## 1. General applicability of this paper's ideas\nThank you for pointing this out. We agree that the two statements appear contradictory at first glance. However, \"can\" removes the contradiction in our view. In line 5, “can be” indicates that in some cases, minimal task knowledge might be enough to identify relevant actions. Using \"can be obtained\" in lines 284-285 highlights that while it is possible to obtain a relevant action set, it may also be quite challenging in many practical situations. Thus, our seemingly contradictory statements highlight the spectrum of difficulties to obtaining a relevant action set, on which we elaborate further in the summary rebuttal (A2). We will highlight this spectrum better in the revised version of this paper.\n\nAs we mention in the paper, action masking can be especially useful for safety-critical applications, where relevance means collision-avoidance. Regarding your first point, we provided such concrete examples for the application of autonomous driving in A2 of the summary rebuttal. To address the second point, we conducted an additional experiment for the MuJoCo Walker2D task, which we justify and evaluate in A3 of the summary rebuttal.\n\n## 2. Gains not coming from the policy gradient, but only because of constraining the action space\nThanks for your question. We defined action masking as transforming the unmasked policy $a \\sim \\pi_\\theta(a | s)$ to the masked policy $a^r \\sim \\pi_\\theta^r(a^r | s)$, for which $a^r \\in \\mathcal{A}^r$ always holds. While most stochastic policy gradient algorithms utilize normal distributions with well-known policy gradients $\\nabla_\\theta \\log \\pi$, masked policies generally do not follow standard normal distributions. Consequently, we derived gradients for these masked distributions to provide a mathematically sound method. It's important to note that we do not claim these adapted gradients necessarily improve algorithm performance in general.\n\nNevertheless, it is not true that all masked policy gradients always reduce to the unmasked policy gradient $\\nabla_\\theta \\log \\pi_\\theta(a | s)$. This only applies to the ray approach, and the proof for it is not trivial, making it also an important contribution. For the generator mask, the policy gradient remains the same, only if the generator matrix is invertible (line 173), which occurs solely when the zonotope has $2^N$ vertices. The distributional mask yields a substantially different gradient (see equation (17)). In practice, we approximate it with the original gradient, since the integral over $\\mathcal{A}^r$ is intractable. We acknowledge this simplification in Sec. 4.3, and suggest potential approximations for future work.\n\nOur primary intention was not to claim that the derived gradients universally improve RL algorithm performance. Rather, we emphasize that deriving correct gradients for modified distributions is crucial for establishing a mathematically sound method, thus forming an essential part of our contribution. \n\nRegarding your second observation, we agree that most of the learning of vanilla PPO goes into the effort of learning to select relevant actions. However, we see this as a justification for the utility of action masking because it allows us to encode task knowledge directly into the policy and, thereby reducing the exploration space. Nevertheless, we do not intend to claim that action masking is inherently better than standard PPO, which we will make sure to clarify in our revised paper. Yet, our experiments do show that utilizing relevant action sets with action masking can drastically improve the convergence speed, albeit at a higher computational cost.\n\n## 3. Simpler baselines for continuous action masking\nThank you for proposing two additional baselines for comparison. However, we do think that our implementation and experiments already cover both of them.\n\nThe ray mask is a version of the first proposed baseline. As derived for proposition 1, the ray mask does not affect the gradient of the policy distribution (because $g(a)$ is bijective), which makes it mathematically equivalent to being defined as part of the environment. We further address the proposal of generally viewing action masking as part of the environment in A1 of the summary rebuttal.\n\nThe second proposed baseline is mathematically equivalent to our distributional mask, since it only modifies the sampling procedure to use rejection sampling instead of geometric random walks. We initially used rejection sampling, but quickly found that it carries a significant downside, which makes it not applicable to RL. Consider the case in two dimensions, where $\\mathcal{A}^r$ is a small set centered at $\\left[ -0.5, -0.5 \\right]^T$. If the policy $\\pi_\\theta(a | s)$ defines a normal distribution with mean at $\\left[ 1.0, 1.0 \\right]^T$ and small variance, the likelihood of a sample $a \\sim \\pi_\\theta(a | s)$ being inside $\\mathcal{A}^r$ is almost zero. This can cause the algorithm to get stuck at a sampling step, which we observed in practice. The issue even aggravates in higher dimensions due to the curse of dimensionality. \n\nNevertheless, we acknowledge that the PPO baseline may be considered relatively weak. We add a comparison to another common approach for adapting the actions of RL agents; action replacement [14]. This method substitutes actions outside the relevant action set with randomly sampled actions from within it. We depict the comparison in Fig. 1 of the rebuttal PDF. In the experiment, masking performs as good as action replacement in the 2D Quadrotor task, better in the 3D Quadrotor, and significantly better in the Seeker environment. These result suggests that action masking is a competitive approach with respect to a baseline that also explores the relevant action set only.\n\n[14] Krasowski et al. 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MN8IK925Uk", "reviewer": "Reviewer_PubC", "summary": "This paper discusses methods for action masking in continuous action spaces to improve convergence stability and sample efficiency in reinforcement learning. The paper introduces three methods for action masking with convex relevant action sets, proves their convergence, and experimentally verifies their effects.", "questions": "-Is assumption 1 (relevant action set is convex) reasonable in most cases? I can imagine disjoint action subsets being relevant in many cases- for example, a self-driving car that needs to steer either left or right to avoid collision, but not go straight ahead or backwards.\n\n-I'm not sure it's actually necessary to compute the policy gradient across the action mask (with the exception of the distributional mask). Once an action is sampled from the policy, the mapping to the relevant action set can simply be treated as part of the environment/transition function which the policy can learn to manipulate without gradient flow. Does this simplify things or am I missing something? This would also permit arbitrary nonconvex relevant sets, I believe.\n\n-For the gradient of the distributional mask in proposition 4, isn't this affected by off-policyness due to the constrained sampling of actions from the policy distribution? For example, if most of the policy distribution probability mass lies outside the relevant set (e.g. in the event of transfer learning to a different task with a new relevant set) the actions sampled will not follow the policy distribution closely and thus \\pi_{\\theta}(a|s) will not be an accurate probability of sampling action a at state s. As noted above, this seems like a big issue that should be noted or addressed, unless I'm missing something that corrects for the off-policyness.\n\n-Small quibble: The goal in figure 2 looks black to me on two different monitors, perhaps using a lighter gray would make it more distinct from the agent position?\n\n-It would probably be reasonable to move the environment definitions for the two quadrotor tasks to the appendix to save space in the main paper, FWIW. I'm not sure the abbreviated version that's present provides all that much context over the text descriptions of the tasks.\n\n-It's not critical to have, and I realize it's a difficult thing to derive a relevant set for by nature, but having an experiment on an environment with 10+ action dimensions would be a nice addition to demonstrate that these masking approach can scale to higher dimensional action spaces tractably. I'd also appreciate some comments on compute cost scaling with action dimension count in the limitations or conclusions sections, if possible, since it seems like compute cost is likely to increase with the dimensionality of the action space.", "limitations": "This paper does an excellent job of making clear its limitations and scope.\n\nI don't see any potential negative societal impacts from this work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper is excellently written, defines a clear and well-motivated goal, and describes three intuitive and theoretically grounded methods to achieve that goal within well-defined and clearly stated limitations.\n\nTo my knowledge these approaches are novel (though the distributional mask I suspect has been used as a one-off solution in prior work as it is conceptually very simple), and their definition and analysis are nontrivial.", "weaknesses": "This paper is pretty solid overall, and I have few major complaints.\n\nThe one significant issue I see is that I think the distributional mask algorithm is off-policy by nature, meaning it's use with on-policy methods like PPO is biased and will cause performance loss or divergence. This may explain the observed underperformance of this masking method in two of the three experimental tasks, and while the algorithm can clearly converge in some cases it seems like a major issue with that particular mask in the context of PPO (off-policy algorithms could use it without issue, but those are left to future work here) that should be noted, or the mask omitted from this paper and left to future off-policy methods.\n\nBeyond that, the experimental evaluation is relatively simple (though I think it is sufficient to validate these algorithms), and more challenging tasks would be useful to demonstrate the limitations of these masking methods. That said, the paper makes it clear that defining a suitable convex relevant action set is a manual process and can be challenging (this is okay as a limitation), so it is understandable why such stress tests are not performed. If there was a way to increase difficulty without major manual action set definition work it would strengthen the evaluation to include it.\n\nI have a few other minor issues and questions noted below, but overall this is a paper that is clear in its goals and describes methods that achieve them, validated to a reasonable standard of theoretical and experimental evidence. There's more that could be done on this topic, but the contribution of this paper is significant on its own, so I'm inclined to recommend acceptance (particularly if something is done to address my concern about the distributional mask above).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and for recognizing the novelty and theoretical grounding of our proposed methods. In the following, we respond to the weaknesses (W1 and W2) stated and questions (Q1 - Q6) raised.\n\n# Weaknesses\n\n## W1. Distributional mask being off-policy by nature\nThank you for sharing your reading of our distributional mask approach. Could you reiterate on why you would refer to the distributional mask approach as off-policy? We would refer to it as on-policy as we are directly sampling from the relevant policy $\\pi_\\theta^r(a^r|s)$, which is normalized through the integral $\\int_{\\mathcal{A}^r} \\pi_\\theta(a | s) \\mathrm{d} a$, and derive the corresponding gradient $\\nabla_\\theta \\log \\pi_\\theta^r(a^r | s)$. However, our implementation approximates the gradient by treating the integral as a constant, which introduces some off-policyness. However, such off-policyness is also introduced by most stochastic on-policy RL implementations (including PPO) through the clipping of samples from the normal distribution.\n\n## W2. Simplicity of experiments\nThank you for your assessment. The main goal of our experiments is to provide initial empirical evidence for continuous action masking and compare the three masking approaches. To strengthen our evaluation, we will add a Walker2D Mujoco environment and an additional baseline to the revised paper as described in A3. of the summary rebuttal and 3. in rebuttal for reviewer 8567. \n\n# Questions\n\n## Q1. Convex set assumption\nThanks for your question. \nNote that convex sets are an important generalization of current practice that allows for interval sets only. \nWe believe that assuming relevant action sets are convex is reasonable since exclusion of extreme actions and dependencies between action dimensions can be described by them. Additionally, one can underapproximate a non-convex relevant set with a convex relevant set at the cost of excluding some relevant actions [https://arxiv.org/abs/1909.01778].\nFor disjoint convex sets, our continuous action masking could be extended with a discrete action that represents a decision variable switching between the disjoint subsets. To find an optimal policy, hybrid RL algorithms could be used [https://arxiv.org/abs/2001.00449]. Yet, this major extension of continuous action masking is subject to future work, and we will add these considerations in Sec. 4.3.\n\n## Q2. Mapping as part of the environment\nThank you for your question. Formulating action masking as part of the policy leads to more intuitive action spaces, potential integration of the masking map in the gradient, and more possible masking approaches (see summary rebuttal, A1). Note that implementing the ray mask on the policy or environment side is mathematically identical due to the unchanged gradient. To assess the empirical relevance of our formulation, we added another baseline where as part of the environment dynamics irrelevant actions are replaced with relevant ones. Please refer to 3. in rebuttal for reviewer 8567 for more details. \n\n## Q3. Gradient of the distributional mask\nYou are correct in your assessment that $\\pi_\\theta(a | s)$ is not an accurate probability of sampling action $a \\in \\mathcal{A}^r$ at state $s$. To address this issue, we normalize the constraint distribution with the integral in equation (15) and use the probability of the resulting distribution $\\pi_\\theta^r(a | s)$ for sampling and in the objective function.  With this formulation, the entire probability mass of the resulting distribution $\\pi_\\theta^r(a | s)$ lies within $\\mathcal{A}^r$, and the distribution accurately reflects the actual likelihood of sampling action $a^r$.\n\n## Q4. Goal color\nThanks for the hint. We will change the color in the revised paper.\n\n## Q5. Dynamics to the appendix\nThanks for the suggestion. We will extend Sec. 4 with the additional experiments as described in our summary rebuttal and move the details of the dynamics to the appendix.\n\n## Q6. Experiment with higher action dimensions\nThank you for your suggestion to extend our evaluation. To provide empirical insights for higher dimensional action spaces, we will add the Walker2D Mujoco environment as described in the summary rebuttal (see A3). Although the Walker2D has less than ten action dimensions, it has the highest action dimensionality of the four environments with six dimensions. \n\nThe compute costs are mainly affected by the calculation of the relevant action set in each state and by the additional operations for the mapping function $\\mathcal{F}$ of the specific masking approach. \nThe former depends strongly on the notion of relevance. For our experiments with collision avoidance as notion of relevance, we compute the relevant action sets at each state with an exponential cone program, which scales polynomially with system dimensions (i.e., state and action) assuming a suitable interior-point solver [Boyd, 04, Convex Optimization]. \nThe second aspect important for the compute costs is the specific masking approach.\nFor the ray mask, the cost is dominated by computing the boundary points, which is a linear program for zonotopes with polynomial complexity in the state dimension [Kulmburg, 21, On the co-NP-Completeness of the Zonotope Containment Problem]. For the generator mask, the matrix multiplication of $G$ with $a^l$ is the dominating operation. For the distributional mask, sampling with the random-direction hit-and-run algorithm introduces computational cost. With zonotopes, each step involves solving two linear programs to determine the boundary points. As mentioned in line 190 the mixing time, or the number of steps before accepting a sample, is set to $N^3$, where $N$ is the action dimension.\n\nWe will clarify the computational cost scaling in Sec. 4.3 and extend our runtime table in Appendix A.6 with the Walker2D environment, for which the PPO baseline and the generator mask training run at the same speed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iWoCR7wlWj", "reviewer": "Reviewer_Ke6H", "summary": "The paper addresses challenges in RL with continuous action spaces, typically defined as interval sets. These spaces often lead to inefficient exploration due to irrelevant actions. The authors propose three continuous action masking methods to focus learning on relevant actions based on current state, improving predictability and suitability for safety-critical applications. They analyze the implications on policy gradients and evaluate performance using PPO across three control tasks. Results show higher final rewards and faster convergence compared to baseline methods without action masking.", "questions": "See weaknesses.", "limitations": "n/a", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n- The paper presents a unique perspective on action spaces by utilizing the relevance of action utility in tasks to improve performance. Conventional methods are limited to discrete domains (tasks) so applying their methods to continuous environments was interesting to see.\n\n**Significance**\n- The proposed approach has practical implications, especially in complex environments where distinguishing between relevant and irrelevant actions is crucial. Regardless of the coverage of baseline, their methods significantly outperform it establishing the state of the art performance.", "weaknesses": "**Reinforcement Learning with Policy Gradients (Section 2.1)**\n- L84: \"r →: S × A\" appears incorrect.\n  \n**Continuous Action Masking (Section 3)**\n- Assumption 1: Clarify the definition of action relevance.\n  \n**Ray Mask (Section 3.1)**\n- L131: Need proof that g(a) is bijective.\n\n\n**Generator Mask (Section 3.2)**\n- Why is A(s) suddenly state-dependent? Provide motivation and further description.\n- In Proposition 2's proof, the matrix multiplication seems infeasible due to mismatched dimensions (C is N x 1 and Ga results in P x 1).\n\n\n**Experiment (Section 4)**\n- Justify the rationale behind the design choices for action relevance in each environment.\n- Compare the chosen action relevance approach to other relevant action settings.\n  \n**Results (Section 4.2)**\n- Why compare to a standard PPO baseline and not to prior relevant works?\n- Include qualitative results to validate the proposed methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable feedback on our manuscript. We are particularly grateful for the acknowledgment of our manuscript's originality and significance.\nIn the following, we outline how we incorporated your feedback and clarify open questions. \n\n## Improvements to the mathematical formulation\nThanks for pointing out notation improvements. We reworked the preliminaries and methodology sections to improve notational precision. Specifically, we corrected the following points:\n - $r \\rightarrow: \\mathcal{S} \\times \\mathcal{A}$ to $r : \\mathcal{S} \\times \\mathcal{A} \\rightarrow \\mathbb{R}$ in [L84].\n - in the ray mask approach: $g(a) = c + \\frac{\\lambda_{\\mathcal{A}}(a)}{\\lambda_{\\mathcal{A}^r}(a)} (a - c)$ to $g(a) = c + \\frac{\\lambda_{\\mathcal{A}^r}(a)}{\\lambda_{\\mathcal{A}}(a)} (a - c)$ in [Eq. 6, L126]\n - in the generator approach\n    - $g : \\mathcal{A} \\rightarrow \\mathcal{A}^r$ to $g : \\mathcal{A}^l \\rightarrow \\mathcal{A}^r$ in [L163]\n    - $G \\in \\mathbb{R}^{P \\times N}$ to $G \\in \\mathbb{R}^{N \\times P}$ in [L154]. \n    - Now we have $G \\in \\mathbb{R}^{N \\times P}$, so the dimensions in [Eq. 12, L164] fit correctly.\n \n## Clarification of the relevant action space\n\nThank you for highlighting that the notion of action relevance was not properly defined. We will added a formal definition of the relevant action set to the preliminaries: \n  > *We further assume that we can compute a state-dependent relevant action set $\\mathcal{A}^r(s) \\subseteq \\mathcal{A}$, which potentially reduces the action space, based on task knowledge.*\n\nTo introduce our notion of relevant actions, we are adding to the first paragraph of the introduction\n > *Irrelevant actions are actions that are either physically impossible, forbidden due to some formal specification, or evidently counterproductive for solving the task.*\n\nAdditionally, we will clarify that only the relevant action set $\\mathcal{A}^r(s)$ is state-dependent, the action sets $\\mathcal{A}$ and $\\mathcal{A}^l$ are not. \nWe would like to refer the reviewer to the summary rebuttal A.2 **Computing relevant action sets** for a more in-depth discussion on this topic.\n\n## Proof of bijectivity of $g(a)$ in the ray mask\n\n**Lemma**: The mapping function $g: \\mathcal{A} \\rightarrow \\mathcal{A}^r$ of the ray mask $g(a) = c + \\frac{\\lambda_{\\mathcal{A}^r}(a)}{\\lambda_{\\mathcal{A}}(a)} (a - c)$ is bijective.\n\n*Proof*: We prove that $g(a)$ is bijective by showing that it is both injective and surjective.\n\nFor any convex set $\\mathcal{A}^r$ with center $c$, we can construct a ray from $c$ through any $a^r \\in \\mathcal{A}^r$ as $r_d(t) = c + d t$, where $d = \\frac{a-c}{\\Vert a-c \\Vert_2}$, and $t \\in \\left[ 0, t_{max} \\right]$ limits the ray to $\\mathcal{A}$. Since $\\mathcal{A}^r \\subseteq \\mathcal{A}$, this also holds $\\forall a \\in \\mathcal{A}$. By construction, any two distinct rays only intersect in $c$. For all points on a given ray, the scaling factors $\\lambda_\\mathcal{A}$ and $\\lambda_{\\mathcal{A}^r}$ are constants, allowing us to rewrite $g(a)$ for all $a=r_d(t)$ as the linear function:\n$$\ng \\left( r_d(t) \\right) = c + \\frac{\\lambda_{\\mathcal{A}^r}}{\\lambda_{\\mathcal{A}}} \\left( r_d(t) - c \\right).\n$$\nTo show that $g(a)$ is injective, i.e. $\\forall a_1, a_2 \\in \\mathcal{A}$, if $a_1 \\neq a_2 \\implies g(a_1) \\neq g(a_2)$, consider the following two cases for $a_1 \\neq a_2$.\n\nCase 1: If $a_1$ and $a_2$ are on the same ray, then, $a_1 = r_d(t_1)$ and $a_2 = r_d(t_2)$ with $t_1 \\neq t_2$. Since $g \\left( r_d(t) \\right)$ is linear and thereby monotonic, it follows that $g \\left( r_d(t_1) \\right) \\neq g \\left( r_d(t_2) \\right)$ and consequently $g(a_1) \\neq g(a_2)$. \n\n\nCase 2: Otherwise, $a_1$ and $a_2$ are on different rays and $a_1 \\neq a_2 \\neq c$, so $g(a_1) \\neq g(a_2)$ follows directly as the rays only intersect in $c$.\n\nThus, $g(a)$ is injective.\n\nFor showing $g(a)$ to be surjective, it is sufficient to show that $\\forall a^r \\in \\mathcal{A}^r$, there exists an $a \\in \\mathcal{A}$, for which $g(a) = a^r$. \n\nConsider the ray $r_d(t)$ passing through $a^r$. We know that $g \\left( r_d(0) \\right) = g(c) = c$, and $g \\left( r_d(t_{max}) \\right)$ is the boundary point of $\\mathcal{A}^r$ from $c$ in the direction $d$. Moreover, $a^r$ lies on the line segment between $c$ and $g \\left( r_d(t_{\\max}) \\right)$.\nSince $g \\left( r_d(t) \\right)$ is linear and continuous, according to the intermediate value theorem, $\\exists t^* \\in \\left[ 0, t_{\\max} \\right]$, for which $g \\left( r_d(t^*) \\right) = a^r$ for any $a^r \\in \\mathcal{A}^r$ along the ray $r_d(t)$. \nAs we can construct such a ray through any point in $\\mathcal{A}^r$, we have shown that for every $a^r \\in \\mathcal{A}^r$, there exists an $a \\in \\mathcal{A}$ such that $g(a) = a^r$, thus proving surjectivity.\n\n## Additional experiments required\nBased on your recommendations, we added further evaluations as discussed in A3. of the summary rebuttal. \nAdditionally, we will add an illustration of typical rollouts on the Seeker environment as interpretable qualitative result (see rebuttal PDF, Fig. 4).\n\n## Rationale behind the design choices for action relevance in our environments\nThank you for your question regarding the design choices of our relevant action space.\nA key advantage of our masking approaches is that they ensure that only relevant actions are executed and therefore can be used for safety-critical tasks. There are other notions of relevance, which we explore in the summary rebuttal A2.\nHere, we choose experiments where the relevant action set is a safe action set. In particular, a relevant action is one that avoids collision with an unsafe region. For the Seeker environment, the unsafe region is defined by an obstacle (see Fig. 2). For the quadrotor stabilization experiments, we compute a control-invariant set, which acts as relevant state set $\\mathcal{S}^r$. \n\nWe will clarify in 4.1.1 and 4.1.2 the motivation and computation of the relevant action set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mQ5kL4bUVr", "reviewer": "Reviewer_T9Vb", "summary": "Learn a state specific mask for actions. Rather than simply a state specific interval, extend the action mask to different convex set representations. Then, derive a policy gradient for each of these masking schemes.  The masking schemes are ray masks, hypercube transform mask and distributional masks. Applies action masking to seeker and quadrotor tasks and shows that this action masking improves performance.", "questions": "It is not clear why related work is not in a separate section, rather than subsection. There does not appear to be a special connection to the introduction.\n\nIt isn't obvious that if G is square and non-singular, that this does not restrict the space of possible relevant action sets, since this would ensure that the hypercube space had an invertible, i.e. one to one, mapping between itself and the action distribution. It seems like many to one would be preferred if the space of the zonotope's hypercube was higher dimension than the action set.", "limitations": "While noted in the limitations, deriving a method for identifying not only the policy gradient, but a policy from a learned value function is highly relevant for RL, and it is not clear how these restricted action spaces can be applied to Q value calculations, though it seems reasonable to assume it is possible.\n\nSome suggestions in the main work of how the distributional or generator action space restriction could be defined as a function of the dynamics could be relevant since it seems like these functions have to be hand-engineered, and it is not obvious how to do that in domains where the dynamics are less well defined.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed action masking covers a wide range of possible action mask definitions.\n\nThe derived policy gradients are relatively straightforward given the definitions of the action boundaries.\n\nThe derivations appear to be sound when applied empirically.", "weaknesses": "It is not clear how easy it is to recover the action masking criteria, especially under the more complex generator or distributional schemes, and it seems like this would be rare\n\nThe experiments are not particularly convincing because they all follow similar control tasks, but it also seems like these are the only tasks for which the action mask could be easily defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable feedback on our manuscript, and for highlighting the broad applicability of our proposed action masking approach. We would like to address your concerns and questions in the following.\n\n# Weaknesses\n## Action masking criteria\nThank you for sharing your assessment. There can be different notions of relevance, which has implications for required task knowledge. A relevant action set could encode a formal specification, guarantee the stability/safety of a dynamic system, or enforce system constraints involving interdependent actions (see summary rebuttal A2 for more details). For an explicit relevant action set, one can determine if it is convex. If so, our continuous action masking can be applied. If not, the relevant action set could be underapproximated with a convex set (see rebuttal for reviewer PubC, Q1 for more details). We will clarify the notion of relevance in the introduction and extend Sec. 4.3 with more details on application limitations originating from the assumptions.\n\n## Unconvincing experiments\nWe appreciate your suggestion to include a different type of task with a more intuitive relevant action set. We will extend our empirical evaluation with the Walker2D MuJoCo environment. Here, the Walker2D is interpreted as a battery-driven robot and the relevant action set is based on the total power constraint of the battery. Such a constraint encodes the interdependence of multiple action dimensions and cannot easily be enforced with vanilla RL. Please refer to the summary rebuttal for more details on the new experiment (see A3) and more examples for relevant action sets (see A2).\n\n\n# Questions\n## Related work as separate section\nThank you for this remark. We agree and will move the related work subsection to a dedicated section.\n\n## Clarification on the effect of G being square and non-singular\nThank you for this question. A generator matrix $G$ that is square and non-singular does restrict the possibilities for relevant action sets, e.g., in two dimensions only rectangles and parallelotopes can be represented but not hexagons. We state in the paper that for this special case of an invertible matrix $G$, the policy gradient for the generator mask does not change (see Proposition 5). You correctly note that it is likely that the generator dimensions $P$ are larger than the action dimensions $N$. Yet, there are cases where relevant action sets with $N=P$ are a valid choice. For example, if the relevant action sets are state-dependent interval sets as for the 3D Quadrotor or if there is a linear dependency between the action dimensions. The latter is the case for the 2D Quadrotor where the force at the left and right rotors should be similar to avoid flipping over. Thus, a parallelotope shape would be a valid choice. Note that existing continuous action masking on interval sets would only poorly cover parallelotope relevant action sets as visualized in the rebuttal PDF, Fig. 2. We will reflect this answer in the paper by clarifying the implications of different choices of $G$ in Sec. 3.2.\n\n\n# Limitations\n\n## Action masking for value-based RL\nThank you for your comment. To the best of our knowledge, there is no popular deep RL algorithm for Q-learning in continuous action spaces that is not an actor-critic. For discrete action spaces, previous action masking work has demonstrated its effectiveness [4,6,11,22] and continuous relevant action sets could be discretized to make these approaches applicable.\nIn our limitations, we comment on TD3 and SAC, which both learn a policy and for which future work is required to determine the theoretical and empirical effects of action masking on the policy.\nNevertheless, we see that our statement in Line 149 could be misleading:\n\n> Since the size of the output layer of the policy or Q-function network cannot change during the learning process\n\nWe will change it to:\n\n> Since the size of the output layer of the policy network cannot change during the learning process\n\n## Obtaining relevant action sets and relevance of action masking\nThanks for your comment. We see that our paper lacked a more explicit specification of the notion of action relevance and a clarification of the practical relevance and benefits of the formulation as part of the policy. We will add this in the revised paper and would like to refer you to A1. and A2. of the summary rebuttal for a detailed answer. \n\n## References:\n\n[4] Feng et al. 2023\n\n[6] Fulton et al. 2018 \n\n[11] Huang et al. 2022 \n\n[22] Rudolf et al. 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

Thank you for your thoughtful comments and questions. We address general points below.

## A1. Relevance of continuous action masking as part of the policy

Action masking enforces task knowledge by focusing learning on relevant actions, thereby increasing sample efficiency and reducing the need for reward tuning - two common practical problems in RL. The existing literature mostly regards discrete action spaces and shows masking is highly effective [4,6,11,22]. However, real-world systems often operate in continuous action spaces, for which only interval action masks have been explored [14]. We develop three methods that enable action masking for arbitrary convex sets. With our experiments, we demonstrate that continuous action masking can increase convergence speed and performance significantly.

While reviewers 8567 and PubC suggest to define action masking as part of the environment, we argue for incorporating it into the policy distribution, as done for discrete action spaces [11]. This has three main advantages. First, this formulation is more intuitive since the relevant actions $a^r$ stay interpretable in the original action space $\mathcal{A}$. E.g., for the generator mask, adding the mapping function $g(a)$ to the environment would result in an action space with a dimension for each generator of the relevant action zonotope. An action would be the generator factors, for which the real-world meaning is not intuitive. Second, our formulation allows to incorporate information about the modification into the gradient. This is relevant for the generator and distributional mask where the relevant policy gradient is different to the original policy gradient (see Proposition 3 and 4). Since computing relevant action sets can be costly, it is desirable to use this information in the backward pass as well. Third, action masking as part of the policy can be used for more formulations. Specifically, the distributional mask cannot be moved to the environment since it directly modifies the sampling of the policy distribution. We will clarify these advantages in Sec. 3 of the revised paper.

## A2. Computing relevant action sets

We assume that a set of relevant actions is computable. A relevant action set reduces the action space, based on task knowledge. This is a flexible definition that allows for different levels of complexity and required task knowledge. Thus, action relevance is a spectrum and the specific definition is a design choice. For our experiments, we equate relevance with guaranteed collision-avoidance since this is an important feature for safety-critical tasks that standard RL does not achieve [14]. For this highly interpretable and strict notion of relevance, the necessary level of task knowledge (e.g., system dynamics and unsafe sets) and computational cost is high. Yet, this can be a reasonable effort for gaining safety guarantees. 

However, there are many other notions of relevance with substantially different levels of required task knowledge and compute. Let us illustrate this for autonomous driving. Large steering angles at high velocity potentially destabilize a vehicle, i.e., can be seen as irrelevant actions. Relevant action sets that restrict steering angles depending on the velocity are trivial to derive. Another notion of relevance is compliance with traffic rules [R1], e.g., only turning right in a right-turn lane, or not accelerating in front of a red light. Here, a medium amount of task knowledge (e.g., road map) is required and the relevant action sets for a single rule are still straightforward to compute (e.g., only allowing steering that leads to a right turn). Similar to our experiments, one can also define relevance as collision avoidance for driving. Due to a highly dynamic environment with other participants, this leads to high required task knowledge. Yet, this is not infeasible as demonstrated by [R2]. Note that there is also recent work on obtaining relevant action sets in a data-driven manner [R3].

Nevertheless, we agree that action relevance and resulting implications for computational costs are not described sufficiently, hence we will clarify this in the revised paper.

## A3. Additional experiments

While we believe that our experiments provide sufficient initial evidence validating our theoretical contributions to action masking, we agree with the reviewers that an evaluation of our continuous action masking approaches on a more diverse set of tasks would further strengthen our claims. Therefore, we are adding experiments on the Walker2D environments (https://gymnasium.farama.org/environments/mujoco/walker2d/). Our concept here is to define the relevant action space as all actions for which $||a||_1 \leq \alpha_p$. This can be viewed as a maximal power output constraint $\alpha_p$.

The results of the experiment are depicted in Fig. 3 of the rebuttal PDF. We observe that the generator and ray mask both learn a performant policy and that the generator mask slightly outperforms standard PPO. Further, encoding the relevant action space as a termination condition prohibits standard PPO to learn. Note that distributional mask is excluded, since it's slow computation time failed to produce results in the available time. In conclusion, this small experiment further highlights the practical utility of action masking.

In addition, we add a comparison to an common approach that adapts irrelevant actions in the environment; action replacement [14]. Our experiment shows that action masking performs better or as good as replacement depending on the environment (see Fig. 1 of rebuttal PDF).

## References:

[4] Feng et al. 2023

[6] Fulton et al. 2018 

[11] Huang et al. 2022 

[14] Krasowski et al. 2023  

[22] Rudolf et al. 2022 

Additional:

[R1] Mehdipour et al. 2023. "Formal methods to comply..."

[R2] Wang et al. 2023. "Safe Reinforcement Learning for Automated Vehicles..."

[R3] Theile et al. 2024 "Learning to Generate..."

Meta-review: The paper proposes three continuous action masking methods to focus reinforcement learning on state-specific relevant actions, enhancing training efficiency, effectiveness, and safety. Although there is consensus about the overall positive evaluation of this paper, the reviewers suggest that the paper would benefit from (a) additional experiments supporting the need for the gradient derivations. (b) improved clarity of presentation. Apart from this, the reviews indicate that evaluations on more complex domains would strengthen the work.
Decision: Accept (poster)