Title: Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Policy Gradient

Action Masking

Robotics

Continuous Actions
Abstract: Continuous action spaces in reinforcement learning (RL) are commonly defined as multidimensional intervals. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning because the typically large global action space leads to frequent exploration of irrelevant actions. Yet, little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions. Focusing learning on these relevant actions can significantly improve training efficiency and effectiveness. In this paper, we propose to focus learning on the set of relevant actions and introduce three continuous action masking methods for exactly mapping the action space to the state-dependent set of relevant actions. Thus, our methods ensure that only relevant actions are executed, enhancing the predictability of the RL agent and enabling its use in safety-critical applications. We further derive the implications of the proposed methods on the policy gradient. Using proximal policy optimization ( PPO), we evaluate our methods on four control tasks, where the relevant action set is computed based on the system dynamics and a relevant state set. Our experiments show that the three action masking methods achieve higher final rewards and converge faster than the baseline without action masking.

Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking

Abstract

Continuous action spaces in reinforcement learning (RL) are commonly defined as multidimensional intervals. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning because the typically large global action space leads to frequent exploration of irrelevant actions. Yet, little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions. Focusing learning on these relevant actions can significantly improve training efficiency and effectiveness. In this paper, we propose to focus learning on the set of relevant actions and introduce three continuous action masking methods for exactly mapping the action space to the state-dependent set of relevant actions. Thus, our methods ensure that only relevant actions are executed, enhancing the predictability of the RL agent and enabling its use in safety-critical applications. We further derive the implications of the proposed methods on the policy gradient. Using proximal policy optimization (PPO), we evaluate our methods on four control tasks, where the relevant action set is computed based on the system dynamics and a relevant state set. Our experiments show that the three action masking methods achieve higher final rewards and converge faster than the baseline without action masking.

1 Introduction

Reinforcement learning (RL) can solve complex tasks in areas such as robotics [13], games [35], and large language models [26]. Yet, training RL agents is often sample-inefficient due to frequent exploration of actions, which are irrelevant to learning a good policy. Irrelevant actions are actions that are either physically impossible, forbidden due to some formal specification, or evidently counterproductive for solving the task. Since the global action space is typically large in relation to the relevant actions in each state, exploring these actions frequently can introduce unnecessary costs, lead to slow convergence, or even prevent the agent from learning a suitable policy.

Action masking mitigates this problem by constraining the exploration to the set of relevant statespecific actions, which can be obtained based on task knowledge. For example, when there is no opponent within reach in video games, attack actions are masked from the action space [43]. Leveraging task knowledge through action masking usually leads to faster convergence and also improves the predictability of the RL agent, especially when the set of relevant actions has a specific

notion, such as being the set of safe actions. For instance, if the set of relevant actions is a set of verified safe actions, action masking can be used to provide safety guarantees [10, 19].

When the relevant action set is easy to compute, action masking usually benefits RL by improving the sample efficiency and is the quasi-standard for discrete action spaces [34, 46], e.g., in motion planning [8, 18, 23, 30, 42], games [14, 15, 43], and power systems [21, 38]. However, real-world systems operate in continuous space and discretizing it might prevent learning optimal policies. Furthermore, simulation of real-world systems is computationally expensive, and developing RL agents for them often requires additional real-world training [45]. Thus, sample efficiency is particularly valuable for these applications.

In this work, we propose three action masking methods for continuous action spaces. They can employ convex set representations, e.g., polytopes or zonotopes, for the relevant action set. Our action masking methods generalize previous work in [19], which is constrained to intervals as relevant action sets. This extends the applicability of continuous action masking to expressive convex relevant action sets, which is especially useful when action dimensions are coupled, e.g., for a thrust-controlled quadrotor. To integrate the relevant action set into RL, we introduce three methods: the generator mask , which exploits the generator representation of a zonotope, the ray mask , which projects an action into the relevant action set based on radial directions, and the distributional mask , which truncates the policy distribution to the relevant action set. In summary, our main contributions are:

We introduce continuous action masking based on convex sets representing the statedependent relevant action sets;

We present three methods to utilize the relevant action sets and derive their integration in the backward pass of RL with stochastic policies;

Weevaluate our approach on four benchmark environments that demonstrate the applicability of our continuous action masking approaches.

2 Related literature

Action masking has been mainly applied to discrete action spaces [8, 10, 14, 15, 19, 18, 21, 23, 30, 38, 41, 42, 43, 46]. Huang et al. [15] derive implications on policy gradient RL and evaluate their theoretical findings on real-time strategy games. They show that masking actions leads to higher training efficiency and scales better with an increasing number of actions than penalizing the agent for selecting irrelevant actions. Huo et al. [14] have extended [15] to off-policy RL and have observed similar empirical results.

Action masking for discrete action spaces can be categorized by the purpose of the state-dependent relevant action set. Often, the set is obtained by removing irrelevant actions based on task knowledge [8, 14, 15, 21, 30, 42], e.g., executing a harvesting action before goods are produced [15]. While the relevant action sets are usually manually engineered, a recent study [46] takes a data-driven approach and identifies redundant actions based on similarity metrics. Another common interpretation of the relevant action set is that it only includes safe actions [10, 18, 23, 38, 41]. These works typically use the system dynamics to verify the safety of actions. A safe action avoids defined unsafe areas or complies with logic formulas. In our experiments, the relevant action sets are either state-dependent safe action sets or a global relevant action set modeling power supply constraints.

For continuous action spaces, there is work on utilizing action masking with multidimensional intervals (hereafter only referred to as intervals) as relevant action sets [19]. In particular, the proposed continuous action masking represents relevant action sets by intervals that reflect safety constraints and employs straightforward re-normalization to map the action space to the relevant action set. In this paper, we generalize to more expressive relevant action sets and demonstrate the applicability to four benchmark environments.

3 Preliminaries

As the basis for our derivations of the three masking methods, we provide a concise overview of RL with policy gradients. Further, we define the considered system dynamics as well as the set representations used in this work.

3.1 Reinforcement learning with policy gradients

A Markov decision process is a tuple ( S , A , T, r, γ ) consisting of the following elements: the observable and continuous state set S ⊂ R n S , the action set A ⊂ R N , the state-transition distribution T ( s ′ | a, s ) , which is stationary and describes the transition probability to the next state s ′ ∈ S ⊂ R n S from the current state s ∈ S when executing action a ∈ A , the reward r : S × A → R , and the discount factor γ for future rewards [37]. The goal of RL is to learn a parameterized policy π θ ( a | s ) that maximizes the expected reward max θ E π θ ∑ ∞ t =0 γ t r ( s t , a t ) .

For policy gradient algorithms, learning the optimal policy π ∗ θ ( a | s ) is achieved by updating its parameters θ using the policy gradient [36, Thm. 2]

where A π θ ( a, s ) is the advantage function, which represents the expected improvement in reward by taking action a in state s compared to the average action taken in that state according to the policy π θ ( a | s ) . An estimation of the advantage A π θ ( a, s ) is usually provided by a neural network.

3.2 System model and set representations

We consider general continuous-time systems of the form

where w ∈ W ⊂ R n W denotes a disturbance. The input is piece-wise constant with a sampling interval ∆ t . We assume S , A , and W to be convex.

Zonotopes are a convex set representation well-suited for representing the relevant action set, due to the efficiency of computing their Minkowski sums and linear maps. A zonotope Z ⊂ R N with center c ∈ R N , generator matrix G ∈ R N × P , and scaling factors β ∈ R P is defined as

Additionally, let us denote that G ( · ,i ) returns the i -th column vector of the generator matrix G . The Minkowski addition Z 1 ⊕Z 2 of two zonotopes Z 1 , Z 2 and the linear map M Z 1 of a zonotope Z 1 are given by [2, Eq. 2.1]

4 Continuous action masking

To apply action masking, a relevant action set A r ( s ) ⊆ A has to be available, which constrains the action space A based on task knowledge. Let us denote the state-dependent relevant action set as A r ( s ) ⊆ A . From now on, we omit the dependency on the state to simplify notation. For our continuous action masking methods, we specifically require the following two assumptions:

Assumption 1. The relevant action set A r is convex and its center and boundary points are computable.

Assumption 2. The policy π θ : S × A → R + of the agent is represented by a parameterized probability distribution a ∼ π θ ( a | s ) .

Common convex set representations that fulfill Assumption 1 are polytopes or zonotopes. Our continuous action masking methods transform the policy π θ ( a | s ) , for which the parameters θ usually specify a neural network, into the relevant policy π r θ : S × A r → R + through a functional h : (Π ×P ( A )) → Π r . Here, Π is the space of all policies, Π r is the space of all relevant policies, and P ( A ) is the power set of all A r . The transformation is defined as

and thereby ensures that a r ∈ A r always holds. Please note that policy gradient methods only require the ability to sample from and to compute the gradient of the policy distribution. Therefore, explicit closed-form expressions for π r θ ( a r | s ) and h are not necessary.

In the following subsections, we introduce and evaluate three masking methods: generator mask, ray mask, and distributional mask, as shown in Fig. 1. We derive the effect of each masking approach on the gradient of the objective function for stochastic policy gradient methods.

A

r

A

c

a

a

r

a

a

r

a

a

r

A

r

relevant action set

A

l

latent action set

a

r

relevant action

c

center of

A

r

G

generator mat. of

A

r

π

r

θ

relevant policy

(a) Ray mask

A

r

A

G

(

·

,

3)

a

3

G

(

·

,

2)

a

2

G

(

·

,

1)

a

1

A

l

c

a

r

(b) Generator mask

A

r

A

π

θ

(

a

|

s

)

π

r

θ

(

a

r

|

s

)

a

r

(c) Distributional mask

4.1 Ray mask

The ray mask maps the action set A to A r by scaling each action a alongside a ray from the center of A r to the boundary of A , as shown in Figure 1a. Specifically, the relevant policy π r θ results from mapping the action a , sampled from π θ ( a | s ) , using the function g R : A → A r :

Here, λ A ( a ) and λ A r ( a ) denote the distances of the action a to the boundaries of the relevant action set and action space, respectively, measured from the center of the relevant action set c in the direction of a . Note that computing λ A r ( a ) for a zonotope requires solving a convex optimization problem, as specified in Appendix A.4. Yet, the ray mask is applicable for all convex sets, for which we can compute the center and boundary points. Since g R ( a ) is bijective (see Appendix A.1 for a detailed proof), we can apply a change of variables [5, Eq. 1.27] to compute the relevant policy

where g -1 R ( a r ) = a is the inverse of g R . In general, there is no closed form of the distribution π r θ available. However, for stochastic policy gradient-based RL, we only require to sample from π r θ and compute its policy gradient. Samples from π r θ ( a r | s ) are created by sampling from the original policy a ∼ π θ ( a | s ) followed by computing a r = g R ( a ) . The policy gradient is derived next.

Proposition 1. Policy gradient for the ray mask.

The policy gradient of π r θ ( a r | s ) for the ray mask is

where A π r θ ( a r , s ) is the advantage function associated with π r θ ( a r | s ) .

Proof. The determinant in (7) is independent of θ , i.e.,

which simplifies the score function of π r θ ( a r | s ) to

Combining (10) and the general form of the policy gradient in (1) for π r θ ( a r | s ) results in (8).

4.2 Generator mask

Zonotopes can be interpreted as the map of a hypercube in the generator space to a lower-dimensional space (see Section 3.2). The generator mask is based on exploiting this interpretation by letting the RL agent select actions in the hypercube of the generator space. Since the size of the output layer of the policy network cannot change during the learning process, we fix the dimension of the generator space. The generator mask requires the following assumption:

Assumption 3. The relevant action set A r ( s ) is represented by a zonotope ⟨ c ( s ) , G ( s ) ⟩ Z , with G ∈ R N × P and c ∈ R N , and a state-invariant number of generators P .

Note that in practice, Assumption 3 can often be trivially fulfilled by choosing sufficiently many generators, and the number of generators P is usually the output dimension of the parametrized policy. The domain of the policy π θ ( a | s ) is the hypercube A l = [ -1 , 1] P , which can be interpreted as a latent action space, and the domain of the relevant policy π r θ ( a r | s ) is a subset of the action space A r ⊆ A (see Figure 1b).

To derive the policy gradient of the generator mask, we assume:

Assumption 4. π θ ( a | s ) is a parametrized normal distribution N ( a ; µ θ , Σ θ ) .

Proposition 2. The relevant policy of the generator mask is

Proof. The generator mask g G : A l →A r is:

which is a linear function. Therefore, the proof directly follows from the linear transformation of multivariate normal distributions [Thm. 3.3.3][40].

Note that the ray mask and generator mask are mathematically equivalent to the approach in [19] if the relevant action set is constrained to intervals. Since g G ( a ) is not bijective in general, we cannot derive the gradient through a change of variables, as for the ray mask.

Proposition 3. The policy gradient for π r θ ( a r | s ) as defined in (11) with respect to µ θ and Σ θ is

Proof. The proposition is proven in Appendix A.2.

Note that for the special case where G -1 exists, i.e., G is square and non-singular, the expressions in (13) and (14) simplify to ∇ µ θ log π θ ( a | s ) , and ∇ Σ θ log π θ ( a | s ) , respectively, when Assumption 4 holds (see Proposition 5 in Appendix A.2). While the generator matrix will commonly not be square, as usually P > N , there are cases where P = N is a valid choice, e.g., if there is a linear dependency between the action dimensions as for the 2D Quadrotor dynamics (see (40)).

4.3 Distributional mask

The intuition behind the distributional mask comes from discrete action masking, where the probability for irrelevant actions is set to 0 [15]. For continuous action spaces, we aim to achieve the same by ensuring that actions are only sampled from the relevant action set A r , by setting the density values of the relevant policy distribution π r θ ( a r | s ) to zero for actions outside the relevant action set (see Figure 1c). For the one-dimensional case, this can be expressed by the truncated distribution [7]. In higher dimensions, the resulting policy distribution is

where ϕ ( a, s ) is the indicator function

Since there is no closed form of this distribution, we employ Markov chain Monte Carlo sampling to sample actions from the policy. More specifically, we utilize the random direction hit-and-run algorithm: a geometric random walk that allows sampling from a non-negative, integrable function f : R N → R + , while constraining the samples to a bounded set [44]. The algorithm iteratively chooses a random direction from the current point, computes the one-dimensional, truncated probability density of f along this direction, and samples a new point from this density. The approach is particularly effective for high-dimensional spaces where other sampling methods might struggle with convergence or efficiency. For the distributional mask, f is the policy π θ ( a | s ) , and A r is the set. As suggested by [22], we execute N 3 iterations before accepting the sample. To estimate the integral in (15), we use numerical integration with cubature [11], which is a method to approximate the definite integral of a function l : R N → R over a multidimensional geometric set.

Proposition 4. The policy gradient for the distributional mask is

Proof. Equation (17) is obtained by calculating the gradient of the logarithm of (15). The indicator function ϕ ( a, s ) is not continuous and differentiable, which would necessitate the use of the subgradient for learning. However, since a r ∈ A r always holds, the gradient has to be computed for the continuous part of π r θ ( a r | s ) only, and thus ϕ ( a, s ) can be omitted.

Since the gradient of the numeric integral ∫ A r π θ (˜ a | s )d˜ a is intractable for zonotopes, we treat the integral as a constant in practice, and use ∇ θ log π r θ ( a r | s ) ≈ ∇ θ log π θ ( a | s ) . We discuss potential improvements in Section 5.3.

5 Numerical experiments

We compare the three continuous action masking methods in four different environments: The simple and intuitive Seeker Reach-Avoid environment, the 2D Quadrotor environment to demonstrate the generalization from the continuous action masking approach in [19], and the 3D Quadrotor and Mujoco Walker2D environment to show the applicability to action spaces of higher dimension. Because the derivation of the relevant action set is not trivial in practice, we selected four environments for which we could compute intuitive relevant action sets. In particular, for the Seeker and Quadrotor environments, the relevant action set is a safe action set since it is computed so that the agent does not collide (Seeker) or leave a control invariant set (2D and 3D Quadrotor). For the Walker2D environment, the relevant action set is state-independent and models a power supply constraint for the actuators.

For the experiments, we extend the stable-baseline3 [29] implementation of proximal policy optimization (PPO) [33] by our masking methods. PPO is selected because it is a widely used algorithm in the field of RL and fulfills both Assumptions 2 and 4 by default. Apart from the masking agents, we also train a baseline agent with standard PPO that uses the action space A and a replacement agent, for which an action outside of A r is replaced by a uniformly sampled action from A r (see [19] for details). The replacement agent is an appropriate comparison to the masking agents since only relevant actions are executed while the replacement is implemented as part of the environment, which is usually easier than an implementation as part of the policy as for the masking methods. We conduct a hyperparameter optimization with 50 trials for each masking method and environment. The resulting hyperparameters are reported in Appendix A.9. All experiments are run on a machine with a Intel(R) Xeon(R) Platinum 8380 2.30 GHz processor and 2 TB RAM.

5.1 Environments

We briefly introduce the three environments and their corresponding relevant action sets A r . Parameters and dynamics for the environments are detailed in Appendix A.5.

S

s

∗

O

S

∆

t

s

A

A

r

A action space A r relevant action set S state space S r relevant state set = S \ O O obstacle s agent position s ∗ goal position

5.1.1 Seeker Reach-Avoid

This episodic environment features an agent navigating a 2D space, tasked with reaching a goal while avoiding a circular obstacle (see Figure 2). It is explicitly designed to provide an intuitive relevant action set A r . The system is defined as in Section 3.2: The dynamics for the position of the agent s = [ s x , s y ] T and the action a = [ a x , a y ] T is ˙ s = a , and there are no disturbances.

The environment is characterized by the position of the agent, the goal position s ∗ , the obstacle position o , and the obstacle radius r o . These values are pseudo-randomly sampled at the beginning of each episode, with the constraints that the goal cannot be inside the obstacle and the obstacle blocks the direct path between the initial position of the agent and the goal. The reward for each time step is

We compute the relevant action set A r so that all actions that cause a collision with the obstacle or the boundary are excluded (see Appendix A.3.3).

5.1.2 2D Quadrotor

The 2D Quadrotor environment models a stabilization task and employs an action space where the two action dimensions are coupled, i.e., rotational movement is originating from differences between the action values and vertical movement is proportional to the sum of the action values. The relevant action set is computed based on the system dynamics and a relevant state set (see Appendix, Eq. (33)). The reward function is defined as

where s ∗ = 0 is the stabilization goal state, a = [ a 1 , a 2 ] is the two-dimensional action, a i , min is the lower bound for the actions in dimension i , and a i , range is the absolute difference between the lower and upper bound for the actions in dimension i .

5.1.3 3D Quadrotor

The third environment models a stabilization task for a quadrotor defined in [16]. The quadrotor has four action dimensions. We use the same reward (see (19)) and the same calculation approach for the relevant action set (see Appendix, Eq. (33)) as for the 2D Quadrotor.

5.1.4 Mujoco Walker2D

The relevant action sets of the two Quadrotor and the Seeker environments are sets that only contain safe actions for the current state. Computing safe action sets requires considerable domain knowledge

-

Baseline

Replacement

Generator Mask

Distributional Mask

0

4

8

12

16

20

-

1

-

0

.

8

-

0

.

6

-

0

.

4

-

0

.

2

Steps (in

10

4

)

Reward

2D Quadrotor

0

1

2

3

4

5

-

1

-

0

.

8

-

0

.

6

Steps (in

10

4

)

Reward

3D Quadrotor

0

5

10

15

20

-

125

-

100

-

75

-

50

-

25

0

25

50

Steps (in

10

4

)

Reward

0

1

2

3

0

500

1

,

000

1

,

500

2

,

000

Steps (in

10

6

)

Reward

-

-

-

-

and for our case solving an optimization problems (see Appendix A.3). However, action masking is not restricted to safe relevant action sets, which we aim to demonstrate on the Mujoco Walker2D environment [39]. We extend the environment with a termination criterion, which ends an episode, when the the action violates the constraint ∥ a ∥ 2 ≤ α p . This can be viewed as constraining the cumulative power output on all joints to a maximum value α p . We motivate this constraint by having a battery with a maximum power output that should not be exceeded in practice. Accordingly, we define the relevant action set as the static set

We under-approximate this relevant action set with a zonotope that consists of 36 generators.

5.2 Results

The reported training results are based on ten random seeds for each configuration. Figure 3 shows the mean reward and the bootstrapped 95% confidence intervals [27] for the four environments and Table 1 depicts the mean and standard deviation of the episode return during deployment. First, we present the results for the Seeker and Quadrotor environments for which the relevant action set is state-dependent and only includes safe actions. Then, we detail the results for the Walker2D environment with a static relevant action set.

For the Seeker and Quadotor environments, the baseline, i.e., PPO, converges significantly slower or not at all (see Figure 3). Additionally, the initial rewards when using action masking are significantly higher than for the baseline, indicating that the exploration is indeed constrained to relevant taskfulfilling actions. During deployment (see Table 1), the baseline episode returns are significantly lower than for action masking, while the three masking methods perform similarly. More specifically, for the Seeker environment, the relative volume of the relevant action set compared to the global action space is on average 70% , and all action masking methods converge significantly faster to high rewards than the baseline. The generator mask and the distributional mask achieve the highest final reward. Further, action replacement performs better than the baseline but significantly worse than

masking. For the 2D Quadrotor, the relative volume is on average 28% , which is much smaller than for the Seeker environment. In the 2D Quadrotor environment, the ray mask, generator mask, and action replacement achieve the highest reward. While the baseline converges significantly slower, the final average reward is similar to the one of the distributional mask. Please note that the confidence interval for the baseline is significantly larger, because three of the ten runs do not converge and, on average, exhibit a reward of one throughout training. Additionally, we observed that if we constrain the relevant action set to intervals for this environment, our optimization problem in (33) often renders infeasible, because the maximal relevant action set cannot be well approximated by an interval. Thus, the masking method proposed in [19] is not suitable for the 2D Quadrotor task. The results on the 3D Quadrotor are similar to those on the 2D Quadrotor; again, the generator mask converges the fastest but yields a final reward similar to that of the ray mask and action replacement. The relative volume of the relevant action set to the global action space is on average 25% . Notably, in this environment, the baseline does not learn a meaningful policy. Based on these three environments with state-dependent relevant action sets, it seems that action masking performs better than action replacement when the relative volume is not too small.

The training results of the Walker2D experiment are shown in Figure 3. While the generator and ray mask both learn a performant policy, the ray mask outperforms by a significant margin. The lower performance of the generator mask is likely due to the high-dimensional generator space with 36 dimensions. This is supported by initial experiments with 12 generators (i.e., a more conservative under-approximation of the L 2 -norm) where the generator mask performed better compared to the ray mask. Replacement performs significantly worse than the two masking approaches, and the PPO baseline does not learn a meaningful policy since the environment is frequently reset due to violations of the power constraint in (20). The frequent terminations occur since in six action dimensions the relative volume of the unit ball compared to the unit box is ≈ 8% , i.e., more than 92% of actions are violating the power constraint. To compare masking to a learning baseline, we also evaluated standard PPO without constraints, which performs slightly better than ray masking but almost always uses actions outside A r (see Appendix A.7). The deployment results in Table 1 reflect similar results as the training; the ray mask achieves better rewards than the generator mask, followed by replacement, and the baseline performs the worst. We excluded the distributional mask for the Walker2D, since its computation time is approximately 170 times slower than the baseline, compared to a 1 . 6 increase for the generator, 2 . 7 for the ray mask, and 2 . 5 for action replacement (see Table 3). The severely increased computational cost for the distributional mask arises from the geometric random walks, which scale cubically with action dimensions.

5.3 Discussion and limitations

Our experimental results indicate that continuous action masking with zonotopes can improve both the sample efficiency and the final policy of PPO. While the sample efficiency is higher in our experiments, computing the relevant action set adds computational load as shown by the increased computation times (see Appendix A.6). Thus, in practice, a tight relevant action set might require more computation time than the additional samples for standard RL algorithms. Yet, if the relevant action set provides guarantees, e.g., is a set of verified safe actions, this increased computation time is often acceptable. Additionally, the computational effort for the masks differs. Given a relevant action zonotope, the generator mask adds a matrix-vector multiplication, which scales quadratically with action dimensions, the ray mask is dominated by the computation of the boundary points, which scales polynomially with action dimensions [20], and the distributional mask scales cubically with the dimension of the action space due to the mixing time of the hit-and-run algorithm [22]. Note that

for the Walker2D environment with six action dimensions, the computation time for the hit-and-run algorithm is so high that the distributional mask evaluation was infeasible.

The ray mask and generator mask are based on functions g R and g G that map to relevant actions. There are two different approaches of incorporating these functions into RL algorithms. One is to apply the mapping as part of the environment on an action that is sampled by a standard RL policy. The second option, which we use, is to consider the masking as part of the policy, which creates the relevant policy as in (5). This has three main benefits over integrating masking as part of the environment. First, the actions passed to the environment are better interpretable, e.g., for the generator mask, adding the masking mapping function to the environment leads to a policy that samples actions from the generator space, which commonly does not have an intuitive interpretation. Second, more formulations of the functional h are possible, e.g., the distributional mask, and, third, the mapping function can be included in the gradient calculation. For mathematically sound masking as part of the policy, the correct gradient needs to be derived for each RL algorithm, and the standard RL algorithms need to be adapted accordingly. However, the empirical benefit could be minor. Thus, future work should investigate the significance of the correct gradient on a variety of tasks. Note that we showed in Proposition 1 that for the ray mask, the PPO gradient with respect to the original policy and relevant policy is the same. Thus, for the ray mask, it does not matter if it is viewed as part of the policy or the environment.

PPO is a common RL algorithm. However, off-policy algorithms such as Twin Delayed DDPG (TD3) [9] and Soft Actor-Critic (SAC) [12] are frequently employed as well. The ray mask and generator mask are conceptually applicable for deterministic policies as used in TD3. Yet, the implications on the gradient must be derived for each RL algorithm and are subject to future work. For the distributional mask, treating the integral in (15) as constant with respect to θ is a substantial simplification, which might be an explanation for the slightly worse convergence of the distributional mask since this introduces an off-policy bias. To address this in future work, one could approximate the integral with a neural network, which has the advantage that it is easily differentiable.

We focus this work on the integration of a convex relevant action set into RL and assume that an appropriate relevant action set can be obtained. While convex sets are a significant generalization from previous work [19], they might not be sufficient for some applications, e.g., tasks where relevant action sets are disjoint. Thus, future work could include investigating hybrid RL approaches [25] to increase the applicability to multiple convex sets or non-convex sets, such as constrained polynomial zonotopes [17]. Further, obtaining the relevant action set can be a major challenge in practice, in particular when the relevant action set is different for each state. Such a high state-dependency is likely when the notion of relevance is safety, while for other definitions of action relevance, the relevant action set might be easy to pre-compute, e.g., excluding high steering angles at high velocities. Additionally, there might be an optimal precision of the relevant action set due to two opposing mechanisms. On the one hand, the larger the relevant action set is with respect to the action space, the smaller the sample efficiency gain from action masking might get. On the other hand, a tight relevant action set might require significant computation time to obtain. Thus, future work should investigate efficient methods to obtain sufficiently tight relevant action sets.

6 Conclusion

We propose action masking methods for continuous action spaces that focus the exploration on the relevant part of the action set. In particular, we extend previous work on continuous action masking from using intervals as relevant action sets to using convex sets. To this end, we have introduced three masking methods and have derived their implications on the gradient of PPO. We empirically evaluated our methods on four benchmarks and observed that the generator mask and ray mask perform best. If the relevant action set can be described by a zonotope with fixed generator dimensions and the policy follows a normal distribution, the generator mask is straightforward to implement. If the assumptions for the generator mask cannot be fulfilled, the ray mask is recommended based on our experiments. Because of subpar performance and longer computation time, the distributional mask needs to be further improved. Future work should also investigate a broad range of benchmarks to identify the applicability and limits of continuous action masking with convex sets more clearly.

Acknowledgments and Disclosure of Funding

We thank Matthias Killer for conducting preliminary experiments. We gratefully acknowledge that this project was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - SFB 1608 - 501798263, AL 1185/9-1, AL 1185/33-1, and the Bavarian Research Foundation project STROM (Energy - Sector coupling and microgrids).

References

Takuya Akiba et al. 'Optuna: A Next-Generation Hyperparameter Optimization Framework'. In: 25th ACM SIGKDD Int. Conf. on Knowledge Discovery & Data Mining . 2019, pp. 26232631.

Matthias Althoff. 'Reachability Analysis and its Application to the Safety Assessment of Autonomous Cars'. PhD thesis. Technische Universität München, 2010.

Matthias Althoff and Goran Frehse. 'Combining zonotopes and support functions for efficient reachability analysis of linear systems'. In: IEEE Conf. on Decision and Control (CDC) . 2016, pp. 7439-7446.

Dimitris Bertsimas and John N. Tsitsiklis. Introduction to linear optimization . Athena Scientific, 1997.

Christopher M. Bishop. Pattern recognition and machine learning . Information science and statistics. 2006.

Stephen Boyd et al. 'A tutorial on geometric programming'. In: Optimization and engineering 8 (2007), pp. 67-127.

J. Burkardt. The Truncated Normal Distribution . Department of Scientific Computing Website, Florida State University, Tallahassee. Online resource. 2018.

Xutao Feng et al. 'Autonomous Decision Making with Reinforcement Learning in Multi-UAV Air Combat'. In: IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC) . 2023, pp. 28742879.

Scott Fujimoto, Herke Van Hoof, and David Meger. 'Addressing Function Approximation Error in Actor-Critic Methods'. In: Proc. of the Int. Conf. on Machine Learning (ICML) . 2018, pp. 2587-2601.

Nathan Fulton and André Platzer. 'Safe Reinforcement Learning via Formal Methods: Toward Safe Control Through Proof and Learning'. In: Proc. of the AAAI Conf. on Artificial Intelligence (AAAI) . 2018, pp. 6485-6492.

Alan Genz and Ronald Cools. 'An adaptive numerical cubature algorithm for simplices'. In: ACM Trans. Math. Softw. 29.3 (2003), pp. 297-308.

Tuomas Haarnoja et al. 'Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor'. In: Proc. of the Int. Conf. on Machine Learning (ICML) . 2018, pp. 1861-1870.

Dong Han et al. 'A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation'. In: Sensors 23.7 (2023).

Yueqi Hou et al. 'Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games'. In: Applied Sciences 13.14 (2023).

Shengyi Huang and Santiago Ontañón. 'A Closer Look at Invalid Action Masking in Policy Gradient Algorithms'. In: Int. Florida Artificial Intelligence Research Society Conf. Proc. (FLAIRS) 35 (2022).

Shahab Kaynama et al. 'Scalable Safety-Preserving Robust Control Synthesis for ContinuousTime Linear Systems'. In: IEEE Trans. on Automatic Control 60.11 (2015), pp. 3065-3070.

Niklas Kochdumper and Matthias Althoff. 'Constrained polynomial zonotopes'. In: Acta Informatica 60.3 (2023), pp. 279-316.

Hanna Krasowski, Xiao Wang, and Matthias Althoff. 'Safe Reinforcement Learning for Autonomous Lane Changing Using Set-Based Prediction'. In: Proc. of the IEEE Int. Intelligent Transportation Systems Conf. (ITSC) . 2020, pp. 1-7.

Hanna Krasowski et al. 'Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking'. In: Trans. on Machine Learning Research (2023).

Adrian Kulmburg and Matthias Althoff. 'On the co-NP-Completeness of the Zonotope Containment Problem'. In: European Journal of Control 62 (2021), pp. 84-91.

Lingyu Liang et al. 'Enhancement of Distribution Network Resilience: A Multi-Buffer InvalidAction-Mask Double Q-Network Approach for Distribution Network Restoration'. In: 3rd Int. Conf. on New Energy and Power Engineering (ICNEPE) . 2023, pp. 1055-1060.

László Lovász and Santosh Vempala. 'Hit-and-Run from a Corner'. In: SIAM Journal on Computing 35.4 (2006), pp. 985-1005.

Branka Mirchevska et al. 'High-level Decision Making for Safe and Reasonable Autonomous Lane Changing using Reinforcement Learning'. In: Proc. of the IEEE Int. Intelligent Transportation Systems Conf. (ITSC) . 2018, pp. 2156-2162.

Ian M. Mitchell, Jacob Budzis, and Andriy Bolyachevets. Invariant, Viability and Discriminating Kernel Under-Approximation via Zonotope Scaling . 2019. arXiv: 1901.01006 .

Michael Neunert et al. 'Continuous-discrete reinforcement learning for hybrid control in robotics'. In: Proc. of the Conference on Robot Learning . 2020, pp. 735-751.

Long Ouyang et al. 'Training language models to follow instructions with human feedback'. In: Advances in Neural Information Processing Systems . Ed. by S. Koyejo et al. V ol. 35. Curran Associates, Inc., 2022, pp. 27730-27744.

Andrew Patterson et al. Empirical Design in Reinforcement Learning . 2023. arXiv: 2304. 01315 .

André Platzer and Edmund M. Clarke. 'The Image Computation Problem in Hybrid Systems Model Checking'. In: Hybrid Systems: Computation and Control . 2007.

Antonin Raffin et al. 'Stable-Baselines3: Reliable Reinforcement Learning Implementations'. In: Journal of Machine Learning Research 22.268 (2021), pp. 1-8.

Thomas Rudolf et al. 'Fuzzy Action-Masked Reinforcement Learning Behavior Planning for Highly Automated Driving'. In: Int. Conf. on Control, Automation and Robotics (ICCAR) . 2022, pp. 264-270.

Sadra Sadraddini and Russ Tedrake. 'Linear encodings for polytope containment problems'. In: IEEE Conf. on Decision and Control (CDC) . 2019, pp. 4367-4372.

Lukas Schäfer, Felix Gruber, and Matthias Althoff. 'Scalable Computation of Robust Control Invariant Sets of Nonlinear Systems'. In: IEEE Trans. on Automatic Control 69.2 (2024), pp. 755-770.

John Schulman et al. Proximal Policy Optimization Algorithms . 2017. arXiv: 1707.06347 .

Ashish Kumar Shakya, Gopinatha Pillai, and Sohom Chakrabarty. 'Reinforcement learning algorithms: A brief survey'. In: Expert Systems with Applications 231 (2023).

Kun Shao et al. A Survey of Deep Reinforcement Learning in Video Games . 2019. arXiv: 1912.10944 .

Richard S Sutton et al. 'Policy Gradient Methods for Reinforcement Learning with Function Approximation'. In: Advances in Neural Information Processing Systems . Ed. by S. Solla, T. Leen, and K. Müller. Vol. 12. MIT Press, 1999.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . 2nd ed. MIT Press, 2018.

Daniel Tabas and Baosen Zhang. 'Computationally Efficient Safe Reinforcement Learning for Power Systems'. In: Proc. of the American Control Conf. (ACC) . 2022, pp. 3303-3310.

Emanuel Todorov, Tom Erez, and Yuval Tassa. 'MuJoCo: A physics engine for model-based control'. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems . 2012, pp. 5026-5033.

Y. L. Tong. The Multivariate Normal Distribution . Springer Series in Statistics. 1990.

G Varricchione et al. 'Pure-past action masking'. In: Proc. of the AAAI Conf. on Artificial Intelligence (AAAI) . Vol. 38. 19. 2024, pp. 21646-21655.

Yang Xiaofei et al. 'Global path planning algorithm based on double DQN for multi-tasks amphibious unmanned surface vehicle'. In: Ocean Engineering 266 (2022).

Deheng Ye et al. 'Mastering Complex Control in MOBA Games with Deep Reinforcement Learning'. In: Proc. of the AAAI Conf. on Artificial Intelligence (AAAI) 34.04 (2020), pp. 66726679.

Zelda B. Zabinsky and Robert L. Smith. 'Hit-and-Run Methods'. In: Encyclopedia of Operations Research and Management Science . Ed. by Saul I. Gass and Michael C. Fu. Boston, MA: Springer US, 2013, pp. 721-729.

Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. 'Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey'. In: IEEE Symposium Series on Computational Intelligence (SSCI) . 2020, pp. 737-744.

Dianyu Zhong, Yiqin Yang, and Qianchuan Zhao. 'No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning'. In: Proc. of the AAAI Conf. on Artificial Intelligence (AAAI) . Vol. 38. 15. 2024, pp. 17078-17086.

A Appendix

A.1 Proof of bijectivity of the ray mapping function.

Lemma 1. The mapping function g : A → A r of the ray mask g ( a ) = c + λ A r ( a ) λ A ( a ) ( a -c ) is bijective.

Proof. We show that g ( a ) is bijective by showing that the function is both injective and surjective.

For any convex set A r with center c , we can construct a ray from c through any a r ∈ A r as r γ ( t ) = c + γt , where γ = a -c ∥ a -c ∥ 2 , and t ∈ [0 , t max ] . Since A r ⊆ A , this also holds ∀ a ∈ A . By construction, any two distinct rays only intersect in c . For all points on a given ray, the scaling factors λ A and λ A r are constants, allowing us to rewrite g ( a ) for all a = r γ ( t ) as the linear function:

̸

To show that g ( a ) is injective, i.e. ∀ a 1 , a 2 ∈ A , if a 1 = a 2 = ⇒ g ( a 1 ) = g ( a 2 ) , consider the following two cases for a 1 = a 2 .

̸

̸

Case 1: If a 1 and a 2 are on the same ray, then, a 1 = r γ ( t 1 ) and a 2 = r γ ( t 2 ) with t 1 = t 2 . Since g ( r γ ( t )) is linear and thereby monotonic, it follows that g ( r γ ( t 1 )) = g ( r γ ( t 2 )) and consequently g ( a 1 ) = g ( a 2 ) .

̸

̸

̸

̸

Case 2: Otherwise, a 1 and a 2 are on different rays and a 1 = a 2 = c , so g ( a 1 ) = g ( a 2 ) follows directly as the rays only intersect in c .

Thus, g ( a ) is injective.

For showing g ( a ) to be surjective, it is sufficient to show that ∀ a r ∈ A r , there exists an a ∈ A , for which g ( a ) = a r .

Consider the ray r γ ( t ) passing through a r . We know that g ( r γ (0)) = g ( c ) = c , and g ( r γ ( t max )) is the boundary point of A r from c in the direction d . Moreover, a r lies on the line segment between c and g ( r γ ( t max )) . Since g ( r γ ( t )) is linear and continuous, according to the intermediate value theorem, ∃ t ∗ ∈ [0 , t max ] , for which g ( r γ ( t ∗ )) = a r for any a r ∈ A r along the ray r γ ( t ) . As we can construct such a ray through any point in A r , we have shown that for every a r ∈ A r , there exists an a ∈ A such that g ( a ) = a r , thus proving surjectivity.

A.2 Policy gradient for the generator mask

The log probability density function of the relevant policy for the generator mask is

We can derive the gradient w.r.t µ θ as

and w.r.t Σ θ as

We can state the following for the special case when the inverse of G exists.

̸

̸

Proposition 5. If G is invertible, ∇ θ log π r θ ( a r | s ) = ∇ θ log π θ ( a | s ) holds for the generator mask.

Proof. If G is invertible, we can further simplify (22) to

and (23) to

by using a = g -1 ( a r ) = G -1 ( a r -c ) , and thus proving the statement.

A.3 Computation of the relevant action set

A.3.1 General case

Given an initial state s 0 , an input trajectory a ( · ) , and a disturbance trajectory w ( · ) , we denote the solution of (2) at time t as ξ t ( s 0 , a ( · ) , w ( · ) ) . Assuming a sampled controller with piecewise-constant input a , we define the reachable set of (2) after one time step ∆ t given a set of initial states S 0 and a set of possible inputs ˜ A ⊆ A as

The reachable set over the time interval [0 , ∆ t ] is defined as

For many system classes it is impossible to compute the reachable set exactly [28], which is why we generally compute overapproximations R ( · ) ⊇ R e ( · ) .

To guarantee constraint satisfaction over an infinite time horizon, we use a relevant state set S r . In this work, we choose S r to be a robust control invariant set in the sense that we can always find an input which guarantees that the reachable set at the next time step is contained in S r and that all state constraints are satisfied in the time interval in between [32]:

We compute the relevant action set as the largest set of inputs that allows us to keep the system in the relevant state set in the next time step. We define a parameterized action set A r ( p ) , where p ∈ R n p is a parameter vector. The relevant action set is then computed with the optimal program

where ˜ Vol ( · ) is a proxy function for the volume in the sense that a maximization of ˜ Vol ( · ) is suboptimally maximizing the volume. In the following, we provide a detailed formulation of (29) as an exponential cone program.

A.3.2 Exponential cone program

We consider the discrete-time linearization of our system

where w ′ k ∈ W ′ ( s k ) additionally contains linearization errors and the enclosure of all possible trajectory curvatures between the two discrete time steps [2]. Furthermore, we assume S , A , W , and S r to be zonotopes.

With regard to the input set parameterization, we consider a template zonotope with a predefined template generator matrix ˜ G . We use the vector ˜ p ∈ R P > 0 of generator scaling factors to scale the generator matrix. The parameterized template zonotope is then given by

where p = [ c ˜ p ] ⊤ . In the following, we denote the center and generator matrix of a specific zonotope Z by c Z and G Z , respectively. A zonotope Z 1 is contained in a zonotope Z 2 if there exist Γ ∈ R P Z 2 × P Z 1 , β ∈ P Z 2 , such that [31, Corollary 4]

Based on the linearized system dynamics in (30), the definitions from (4), and using R as a shorthand for R ∆ t ( · ) , we have G R = [ AG S 0 BG A r G W ] and c R = [ Ac S 0 Bc A r c W ] . Since we already consider trajectory curvature in the disturbance, we only need to guarantee that the relevant input set is contained in the feasible input set and that the reachable set of the next time step is contained in the relevant state set. Using the containment condition from (32), we define the auxiliary variables Γ R , β R , Γ A r , and β A r and solve

where 0 are zero matrices of appropriate dimensions. By vectorizing Γ R and Γ A r and resolving the infinity norm constraints [4, Sec. 1.3], we can obtain a formulation with purely linear constraints. Since computing the exact volume of a zonotope is computationally expensive, we approximate it by computing the geometric mean of the parameterization vector:

With the cost function from (34), the problem in (33) is an exponential cone program, which is convex and can be efficiently computed [6, Sec. 2.5].

A.3.3 Seeker Reach-Avoid environment

Because of the dynamics of the Seeker Reach-Avoid environment, we can simplify the relevant action set computation to the following optimization problem:

where S ∆ t is the reachable set of the agent in the next time step, the set O represents the obstacle, and the box [ -10 , 10] 2 is the outer boundary of the state space. The constraints represent three intuitive geometric constraints. First, containment of the relevant action set in the action set (usually, the interval [ -1 , 1] 2 , second, the containment of the relevant state set of the next time step in the environment boundaries, and the last constraint enforces that there is no collision with the obstacle possible. To enforce these constraints for zonotopes, we use the support function [3, Lemma 1]

where l is the vector in the direction of interest. For the first constraint, we need to ensure that the support function for the basis vectors e 1 , and e 2 (and there negative versions) are less than or equal to 1 , i.e.,

From (36) it is apparent that using the negative basis vectors -e 1 and -e 2 , just flips the sign of c A r in (37). The constraints for S r ∆ t ⊆ [ -10 , 10] 2 can be constructed similarly. For simplicity, we express the element-wise inequality in (37) using the L ∞ -norm, and write the optimization problem as

The last constraint represents an under-approximation of S r ∆ t ∩ O = ∅ enforcing the containment of A r in the halfspace { x | n T x ≤ b } through the support function ρ A r ( n ) . The directional vector n = o -s ∥ o -s ∥ where o is the center of the obstacle, and the offset b = n T ( o -n ) r where r is the radius of the obstacle. This ensures that the intersection between the reachable set S r ∆ t and the obstacle O will be empty since the halfspace is constructed tangential to the obstacle at the intersection point of the line from the agent to the center of the obstacle.

A.4 Computation of zonotope boundary points

The boundary point p ∈ R N on a zonotope ⟨ c, G ⟩ Z ⊂ R N in a direction d ∈ R N starting from a point x ∈ R N is obtained by solving the linear program

and computing p = x + αd [20].

A.5 Parameters and dynamics for environments

We provide the action space bounds, the state space bounds, and the generator template matrix ˜ G for the Seeker and Quadrotor environments in Table 2.

The system dynamics of the 2D Quadrotor is:

where the state is s = [ x, z, ˙ x, ˙ z, θ, ˙ θ ] T , the action is a = [ a 1 , a 2 ] T , and the disturbance is w =

,

[ w 1 , w 2 ] T . The dynamics are adapted from [24, Eq. 43] so that the actions are two independent thrusts. Additionally, the parameters used in our experiments are g =9 . 81 m s -2 , k =11 / kg , d 0 =70 d 1 =17 , n 0 =55 , and W = [ -0 . 08 , -0 . 08] × [0 . 08 , 0 . 08] .

The 3D Quadrotor is modeled in a twelve-dimensional state space with state s = [ x, y, z, ˙ x, ˙ y, ˙ z, θ, ϕ, ψ, ˙ θ, ˙ ϕ, ˙ ψ ] T and four-dimensional action space with action a = [ a 1 , a 2 , a 3 , a 4 ] T . The system dynamics are ˙ s = [ ˙ x, ˙ y, ˙ z, -9 . 81 ϕ, 9 . 81 θ, a 1 , ˙ θ, ˙ ϕ, ˙ ψ, a 2 , a 3 , a 4 ] T [16].

For the Walker2D environment, we use the standard parameters of the gymnasium implementation 2 . The relevant action set is a zonotope under-approximation of the relevant action set stated in (20) with 36 generators and α P = 1 .

A.6 Computational time for training

Note that the increased computation time of the masking approaches compared to the baseline is mainly due to the computation of the relevant action sets. Since the relevant action set is a zonotope, the generator mask does not require expensive additional computations. The increased runtime for the ray mask mainly originates from the computation of the boundary points (see Appendix A.4). For the distributional mask, the increased runtime is mostly caused by the Markov chain Monte Carlo sampling of the action.

-

Baseline

Standard PPO

Replacement

Generator Mask

0

1

2

3

0

500

1

,

000

1

,

500

2

,

000

2

,

500

Steps (in

10

6

)

Reward

-

-

-

-

A.7 Quantitative results for Walker2D environment compared to standard PPO

If we remove the power constraint that resets the environment whenever an action outside of A r is selected, the standard PPO is able to learn a policy that performs slightly better than the ray mask towards the end of training while converging slightly slower (see Figure 4). However, during deployment, only 0 . 05% of the actions from this policy are from within A r . Yet, the reward that standard PPO achieves during deployment is on average 2177 . 57 with standard deviation 1238 . 42 . This is slightly higher than the deployment results for the ray mask (see Table 1).

A.8 Qualitative results for the Seeker environment

Figure 5 presents ten example trajectories for a trained agent using each of the five approaches under consideration. Notably, the continuous action masking agents solve the task by safely reaching the goal while the PPO baseline still collides with the obstacle. The replacement approach is ineffective, with the agent frequently failing to reach the goal. In fact, for the ten displayed rollouts, no replacement agent reaches the goal. This discrepancy with respect to the training performance where the replacement agent reaches the goal frequently is due to using the policy in a deterministic setting for deployment. The simple dynamics of the Seeker environment make it possible to directly visualize the relevant action set A r at each time step along the trajectory, which is depicted in the lower half of Figure 5. For the generator mask, the least amount of A r is plotted, which indicates that the generator mask agent reaches the goal most efficiently, i.e., the lowest amount of steps are required.

A.9 Hyperparameters for learning algorithms

We specify the hyperparameters for the three masking approaches, replacement and baseline in the Seeker (Table 4), the 2D Quadrotor (Table 5), and the 3D Quadrotor environment (Table 6). These were obtained through hyperparameter optimization with 50 trials using the tree-structured Parzen estimator of Optuna with the default parameters [1].

Ray

Generator

Distributional

Replacement

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim that focusing on state-dependent relevant actions can improve training efficiency and effectiveness of reinforcement learning significantly, which we demonstrate with four experiments in Section 5.2. We formally introduce three action masking methods, which can be used with convex relevant action sets, and derive their implications on the gradient in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We highlighted the assumptions of the proposed action masking methods in Section 4. Additionally, we discuss limitations of our experimental results and the applicability of our methods in Section 5.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For all three methods, we assume convex relevant action sets and a stochastic policy. Although, as mentioned in Section 5.3, the ray mask and generator mask could be adapted to non-stochastic policies. For the generator mask, we additionally assume the set representation to be a zonotope. We propose the adapted policy gradient for each method in Propositions 1, 3, and 4, and provide proofs for all three.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the necessary details to implement the three different masking approaches in Sections 4.1, 4.2, and 4.3. We provide information needed to re-implement the environments in Sections 5.1 and A.5. The computation of the relevant action set is specified in A.3. Further, the hyperparameters used for the training runs are reported in A.9. Any remaining ambiguity can be resolved by the code attached to this submission.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attached the code for the experiments including a detailed readme in the supplementary material to this submission.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the utilized method for hyperparameter tuning in Section 5, and specify the used hyperparameters, and optimizer in Appendix A.9.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We train each configuration with ten random seeds and report the training results with 95% bootstrapped confidence intervals in transparent color in Figure 3. For the reported deployment results and computational runtime (see Table 1 and 3, respectively), we report the mean and 1-sigma standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We reported the compute resources utilized in Section 5. The average computation times are reported in Section A.6. Note that the compute resource used for the experiments is a shared resource where also other researcher ran their experiments at the same time. Thus, runtime evaluations will only be partially reproducible.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our experiments do not involve human subjects or datasets. Regarding safety and security, our approaches can be directly used to prevent agents from performing unsafe actions. We cannot identify any potentially harmful consequences of our algorithms and work in general.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper introduces a method to exploit task knowledge via relevant action sets to achieve more efficient and effective reinforcement learning. Given the limited scope of our experiments, a detailed discussion on broader societal impact would be very speculative. However, we believe our approach has the potential to expand the use of reinforcement learning in applications where training opportunities are limited but task knowledge is easily transferable to relevant action sets and where enforcing hard constraints is important.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not work with, or release any datasets, and our models have no risk for potential misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: As mentioned in Section 5, we use the stable-baseline3 implementation of PPO. Furthermore, we clearly state the origin of the used Quadrotor environments in Sections 5.1.2 and 5.1.3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLearn a state specific mask for actions. Rather than simply a state specific interval, extend the action mask to different convex set representations. Then, derive a policy gradient for each of these masking schemes.  The masking schemes are ray masks, hypercube transform mask and distributional masks. Applies action masking to seeker and quadrotor tasks and shows that this action masking improves performance.\n\nSTRENGTHS:\nThe proposed action masking covers a wide range of possible action mask definitions.\n\nThe derived policy gradients are relatively straightforward given the definitions of the action boundaries.\n\nThe derivations appear to be sound when applied empirically.\n\nWEAKNESSES:\nIt is not clear how easy it is to recover the action masking criteria, especially under the more complex generator or distributional schemes, and it seems like this would be rare\n\nThe experiments are not particularly convincing because they all follow similar control tasks, but it also seems like these are the only tasks for which the action mask could be easily defined.\n\nQUESTIONS:\nIt is not clear why related work is not in a separate section, rather than subsection. There does not appear to be a special connection to the introduction.\n\nIt isn't obvious that if G is square and non-singular, that this does not restrict the space of possible relevant action sets, since this would ensure that the hypercube space had an invertible, i.e. one to one, mapping between itself and the action distribution. It seems like many to one would be preferred if the space of the zonotope's hypercube was higher dimension than the action set.\n\nLIMITATIONS:\nWhile noted in the limitations, deriving a method for identifying not only the policy gradient, but a policy from a learned value function is highly relevant for RL, and it is not clear how these restricted action spaces can be applied to Q value calculations, though it seems reasonable to assume it is possible.\n\nSome suggestions in the main work of how the distributional or generator action space restriction could be defined as a function of the dynamics could be relevant since it seems like these functions have to be hand-engineered, and it is not obvious how to do that in domains where the dynamics are less well defined.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses challenges in RL with continuous action spaces, typically defined as interval sets. These spaces often lead to inefficient exploration due to irrelevant actions. The authors propose three continuous action masking methods to focus learning on relevant actions based on current state, improving predictability and suitability for safety-critical applications. They analyze the implications on policy gradients and evaluate performance using PPO across three control tasks. Results show higher final rewards and faster convergence compared to baseline methods without action masking.\n\nSTRENGTHS:\n**Originality**\n- The paper presents a unique perspective on action spaces by utilizing the relevance of action utility in tasks to improve performance. Conventional methods are limited to discrete domains (tasks) so applying their methods to continuous environments was interesting to see.\n\n**Significance**\n- The proposed approach has practical implications, especially in complex environments where distinguishing between relevant and irrelevant actions is crucial. Regardless of the coverage of baseline, their methods significantly outperform it establishing the state of the art performance.\n\nWEAKNESSES:\n**Reinforcement Learning with Policy Gradients (Section 2.1)**\n- L84: \"r →: S × A\" appears incorrect.\n  \n**Continuous Action Masking (Section 3)**\n- Assumption 1: Clarify the definition of action relevance.\n  \n**Ray Mask (Section 3.1)**\n- L131: Need proof that g(a) is bijective.\n\n\n**Generator Mask (Section 3.2)**\n- Why is A(s) suddenly state-dependent? Provide motivation and further description.\n- In Proposition 2's proof, the matrix multiplication seems infeasible due to mismatched dimensions (C is N x 1 and Ga results in P x 1).\n\n\n**Experiment (Section 4)**\n- Justify the rationale behind the design choices for action relevance in each environment.\n- Compare the chosen action relevance approach to other relevant action settings.\n  \n**Results (Section 4.2)**\n- Why compare to a standard PPO baseline and not to prior relevant works?\n- Include qualitative results to validate the proposed methods.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses methods for action masking in continuous action spaces to improve convergence stability and sample efficiency in reinforcement learning. The paper introduces three methods for action masking with convex relevant action sets, proves their convergence, and experimentally verifies their effects.\n\nSTRENGTHS:\nThis paper is excellently written, defines a clear and well-motivated goal, and describes three intuitive and theoretically grounded methods to achieve that goal within well-defined and clearly stated limitations.\n\nTo my knowledge these approaches are novel (though the distributional mask I suspect has been used as a one-off solution in prior work as it is conceptually very simple), and their definition and analysis are nontrivial.\n\nWEAKNESSES:\nThis paper is pretty solid overall, and I have few major complaints.\n\nThe one significant issue I see is that I think the distributional mask algorithm is off-policy by nature, meaning it's use with on-policy methods like PPO is biased and will cause performance loss or divergence. This may explain the observed underperformance of this masking method in two of the three experimental tasks, and while the algorithm can clearly converge in some cases it seems like a major issue with that particular mask in the context of PPO (off-policy algorithms could use it without issue, but those are left to future work here) that should be noted, or the mask omitted from this paper and left to future off-policy methods.\n\nBeyond that, the experimental evaluation is relatively simple (though I think it is sufficient to validate these algorithms), and more challenging tasks would be useful to demonstrate the limitations of these masking methods. That said, the paper makes it clear that defining a suitable convex relevant action set is a manual process and can be challenging (this is okay as a limitation), so it is understandable why such stress tests are not performed. If there was a way to increase difficulty without major manual action set definition work it would strengthen the evaluation to include it.\n\nI have a few other minor issues and questions noted below, but overall this is a paper that is clear in its goals and describes methods that achieve them, validated to a reasonable standard of theoretical and experimental evidence. There's more that could be done on this topic, but the contribution of this paper is significant on its own, so I'm inclined to recommend acceptance (particularly if something is done to address my concern about the distributional mask above).\n\nQUESTIONS:\n-Is assumption 1 (relevant action set is convex) reasonable in most cases? I can imagine disjoint action subsets being relevant in many cases- for example, a self-driving car that needs to steer either left or right to avoid collision, but not go straight ahead or backwards.\n\n-I'm not sure it's actually necessary to compute the policy gradient across the action mask (with the exception of the distributional mask). Once an action is sampled from the policy, the mapping to the relevant action set can simply be treated as part of the environment/transition function which the policy can learn to manipulate without gradient flow. Does this simplify things or am I missing something? This would also permit arbitrary nonconvex relevant sets, I believe.\n\n-For the gradient of the distributional mask in proposition 4, isn't this affected by off-policyness due to the constrained sampling of actions from the policy distribution? For example, if most of the policy distribution probability mass lies outside the relevant set (e.g. in the event of transfer learning to a different task with a new relevant set) the actions sampled will not follow the policy distribution closely and thus \\pi_{\\theta}(a|s) will not be an accurate probability of sampling action a at state s. As noted above, this seems like a big issue that should be noted or addressed, unless I'm missing something that corrects for the off-policyness.\n\n-Small quibble: The goal in figure 2 looks black to me on two different monitors, perhaps using a lighter gray would make it more distinct from the agent position?\n\n-It would probably be reasonable to move the environment definitions for the two quadrotor tasks to the appendix to save space in the main paper, FWIW. I'm not sure the abbreviated version that's present provides all that much context over the text descriptions of the tasks.\n\n-It's not critical to have, and I realize it's a difficult thing to derive a relevant set for by nature, but having an experiment on an environment with 10+ action dimensions would be a nice addition to demonstrate that these masking approach can scale to higher dimensional action spaces tractably. I'd also appreciate some comments on compute cost scaling with action dimension count in the limitations or conclusions sections, if possible, since it seems like compute cost is likely to increase with the dimensionality of the action space.\n\nLIMITATIONS:\nThis paper does an excellent job of making clear its limitations and scope.\n\nI don't see any potential negative societal impacts from this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes mathematical formulations for continuous action masking in reinforcement learning, to incorporate domain-knowledge in the form of state-specific sets of relevant actions. It introduces 3 functional forms to extract relevant actions from the original action space, and consider its effect on the policy gradient. The policy gradient does not change much, and the paper shows that the forms compare similarly and better than learning an agent without any knowledge of the continuous action mask at all.\n\nSTRENGTHS:\n- The problem of action masking in continuous action space is an underexplored one, but could have major impact in efficacy of agents and incorporating domain-specific knowledge.\n- The proposed continuous action masking could potentially be useful for safety demarcations.\n- The paper provides mathematical frameworks to formulate continuous action masking and also derive their (minimal) effect on the policy gradient.\n- The paper is mostly well-written and explains the mathematical derivations quite well. Quick note that Section 2.1 and 2.2 could be made more integrated, currently they seem completely disconnected.\n\nWEAKNESSES:\n## 1. General applicability of this paper's ideas\nObtaining a state-specific relevant action set can be really hard. The paper, however, makes contradictory statements about this:\n- L5: \"little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions.\"\n- L284-285: \"assume that an appropriate relevant action set can be obtained. Yet, obtaining this can be a major challenge in practice.\"\n\nFrom the experiments on the 3 environments, it already seems like defining the relevant action set requires a lot of domain knowledge about the state space features and the dynamics function.\n\nAs of now, there does not seem to be any way the ideas in this paper could be useful for any practical domain.\n- Can the authors provide some concrete examples of how one can obtain such relevant action sets for problems of practical interest and scale?\n- Can the authors provide any results on a commonly used continuous action space RL benchmark?\n\n## 2. Gains not coming from the policy gradient, but only because of constraining the action space\nThe paper's proposed formulation is interesting because it uses continuous action masking as part of the learning policy and informs the policy gradient update about the continuous action mask. However, when we look at the resultant policy gradients for each mask in Eq. 10, Line173, and Line199, it seems that the policy gradient simply reduces to $\\lambda_\\theta log \\pi_\\theta(a | s)$ for all cases.\n\nSo, the effective change in implementation is just how the action to take in the environment is model: $a^r = g(a)$. But, this **doesn't utilize continuous action masking to improve the policy learning objective** in any meaningful way. Is my understanding correct in this?\n\nAnother observation that validates the claim that policy learning is not influenced much is seen from the results and L248-249. The initial rewards themselves are significantly higher, which means that the action mask just reduces the space of exploration of the agent so much that, as long as it takes a valid action, it would get a high reward.\n\n## 3. Simpler baselines for continuous action masking\nContinuing from the above point, if all that needs to be done is to compare different formulations of g(a), there is a much simpler alternative perspective:\n- Action-Masking as part of environment: Simply, apply the action mask as part of the environment, without changing the PPO objective at all. So, there is an action-masking filter before executing an agent's action in the environment, and ignore if the action is invalid.\n- Sampling-augmented action-masking: Keep sampling actions from $\\pi_\\theta$ until you find a valid action that can pass through the known continuous action-masking map.\n\nThe current PPO baseline is very weak, and does not utilize action-masking at all. It seems most of the learning in PPO is going into the effort of learning the action mask. To really justify this paper's proposed action masking schemes are useful, they must compare against other forms of naive action masking, including the two listed above. This perspective on considering the action mask as part of the environment is also much more generally applicable and does not require any change to the policy gradient update.\n\nQUESTIONS:\nListed in weaknesses.\n\nLIMITATIONS:\nThe authors discuss several limitations of the work, but the important ones of applicability and baselines need to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}