Where to Intervene: Action Selection in Deep Reinforcement Learning
Authors: Wenbo Zhang
Hengrui Cai
Venue: Accepted by TMLR ()

Abstract: Deep reinforcement learning (RL) has gained widespread adoption in recent years but faces significant challenges, particularly in unknown and complex environments. Among these, high-dimensional action selection stands out as a critical problem. Existing works often require a sophisticated prior design to eliminate redundancy in the action space, relying heavily on domain expert experience or involving high computational complexity, which limits their generalizability across different RL tasks. In this paper, we address these challenges by proposing a general data-driven action selection approach with model-free and computationally friendly properties. Our method not only selects minimal sufficient actions but also controls the false discovery rate via knockoff sampling. More importantly, we seamlessly integrate the action selection into deep RL methods during online training. Empirical experiments validate the established theoretical guarantees, demonstrating that our method surpasses various alternative techniques in terms of both performance in variable selection and overall achieved rewards.

Published in Transactions on Machine Learning Research (7/2025)
Where to Intervene: Action Selection in Deep Reinforcement Learning
Wenbo Zhang Deportment of Statistics University of California, Irvine
wenba13@uci.com edu
Hongrui Cai
hengre I druci. edu
Department of Statistics University of Colifornia, Irvine
Reviewed on OpenReview: https: openreview net/ net/forum?
forum? ? id= D3w.9Xkkuy
Abstract
Dcep reinforcement learning (RL) has gained widespread adoption in recent years but faces significant challenges, particularly in unknown and complex environments. Among these, high-dimennsoonll action selection stands out as a critical problem. Existing works often require sophisticated prior design to eliminate redundancy in the action space, relying heavily domain expert experience involving high computational complexity, which limits their generalizability across different RT tasks. In this paper, we address these challenges by proposing general data-driven action selection approach with model-free and computationally friendly properties. Our method not only selects minimall sufficient actions but also controls the false discovery rate via knockoff sampling. More importantly, we scamlessly integrate the action selection into deep RL methods during online training. Empirical experiments validate the established theoretical guarantees, demonstrating that method surpasses various alternative techniques in terms of both performance in variable selection and overall achieved rewards.
1 Introduction
Recent advances in deep reinforcement learning (RL) have attracted significant attention, with applications spanning numerous fields such as robotics, games, healthcare, and finance (Kober el al. 2013 Kaiser al., 2019: Kolm & Ritter 2020 Yu al. [022]). Despite their ability handle sequential
pecision-making. the practical utility of RL methods in real-world scenarios is often limited, especially in dealing with the
high-dimensional action spaces (Sunehag et al. 2015 Kaiser et al. 2019 Sakryukin et al. 2020 Xiao et al. 2020). Wigh-dimensional action spaces are prevalent in "black box" " systems, characterized by overloaded actionable variables that are often abundant and redundant. Examples include precision medicine, where numerous combinations of treatments and dosages are possible (see e.g., JJhnson et al. 2016 Liu et al.) 2017: Cai et al., 2023a]; neuroscience, which involves various stimulation points and intensities (see e.g.,
Gershman et al. 2009): and robotics, particularly in muscle-driven robot control, where coordination of numerous muscles is required (see e.g., |chumacher et al., 2023).. Nevertheless, these high-dimensional action spaces often contain many actions that either ineffective have negligible impact states and rewards. Training RL models on the entire action space can result in substantial inefficiencies in both computation and data collection.
To handle high dimensionality, a promising approach to employ automatic dimension reduction techniques to seloct only the essential minimum action set necessary for effectively learning the environment and optimizing the policy based on the subspace. Having such minimal yet sufficient action space can significantly enhance learning efficiency, agents thoroughly explore a concise set of actions (Zahavy al.

---

Published in Transactions on Machine Learning Research (7/2025)
Zhou et al. 2024). Moreover, smaller action space can reduce computational complexity, notable benefit in deep RI, where neural networks are used for function approximation (Sun et al. 2011 Sadamoto et al. 2020). In practical scenarios, eliminating superfluous actions saves the cost of extensive mcasurement equipment and thus allows a more comprehensive exploration of available actions. Yet, existing works oflen require a sophisticated prior design to eliminate redundancy in the action space (e.g., [ynnnevee al.] 2019 Jiang et al. 2019: Farquhar et al.) 2020 Luo et al., 2023). ) relying heavily on domain expert experience or involving high computational complexity, limiting their generalizability across different RL tasks.
2018
Kanervisto al. 2020 Jain et al., 2020
In this paper, we propose general data-driven action selection approach to identify the minimum sufficient actions in the high-dimensional action space. To handle the complex environments often seen in deep RL, we develop a novel variable selection approach called knockoff sampling (KS) for online RL, with theoretical guarantees of false discovery rate control, inspired by the model-free knockoff method (Candes et al.) 2018). The effectiveness of this action selection method is demonstrated in Fig. D A proximal policy optimization (PPO) algorithm (Schulman al. 2017) enhanced with variable selection outperforms its counterpart without selection and achieves performance comparable to that of PPO trained with the pre-known true minimal sufficient action. To remain computationally friendly, we design an adaptive strategy with simple mask operation that seamlessly integrates this action selection method into deep RL methods during online training. Our method does not. reduce the dimensionality of the action space itself but rather focuses on identifying redundant actions and mitigating their influence during policy training to enhance sample efficiency and significantly accelerate learning with implementation and interpretation advantages.
Our main contributions are fourfold:
Conceptually, this work pioneers exploring high-dimensional action selection in online RL. We formally define the sufficient action set as encompassing all influential actions and the minimal sufficient action set as containing the smallest number of actions necessary for effective decision-making.
Methodologically, our method bypasses the common challenge of crealing accurule knockoff features in fodel-free knockoffs. We use the established distribution of actions from the current policy network in online RL to resample action values, producing exact knockoff features.
Algorithm-wise, to flexibly inlegrate arbilrary variable selection inlo deep RL and eliminate the need to initialize a new RL model after the selection, we design binary hard mask approach based on the indices of selected actions. This efficiently neutralizes the influence of non-chosen actions.
Theoretically, to address the issues of highly dependent dala in online RL, we couple our KS method with sample splitting and majority vote; under commonly imposed conditions, we theoretically show our method consistently identifies the minimal sufficient action set with false discovery rate control.
1.1 Related Works
Deep reinforcement learning has made significant breakthroughs in complex sequential decision-making
across various tasks (Mnih et al. 2013 Silver et al. 2016 Schulman et al. 2017 Haarnoja et al.) 2018 Cai
et al. 2021). Yet, several considerable obstacles exist when dealing with high-dimensions spaces using deep
RL. In terms of high dimensional state space, the state abstraction (Misra et al. 2020 Pavse & Hanna 2023
has been studied to learn a mapping from the original state space to a much smaller abstract space to preserve the original Markov decision process. Yel, these methods, such as bisimulation can be computationally expensive and challenging when the state space very large or has complex dynamics (Ruan et al. 2015
Tied to our topic, is hard to utilize such abstraction-bused methods to implement transformed actions. This redirects us to variable selection & Brunskkill 2017). Recently, Hao et al. (2021) combined LASSO with fitted Q-iteration to reduce states; following this context, Ma et al. (2023) employs the knockoff method for state selection but with discrete
the redundant state space (see e.g., Kroon & Whiteson| 2009 Guo
action spaces. However, all these works focus on the high dimensional state space in offline data, while our method aims to extract sufficient and necessary actions during online learning.
For RL with the high-dimensional action space, especially for continuous actions, some studies (Synnaeve et al. 2019 Farquhar et al. 2020) transformed the continuous control problem into the combinatorial action
2

---

Published in Transactions on Machine Learning Research (7/2025)
5000
4500
3500
3000
2500
PPO with The Actions
PPO with KS
1500
with
0.0
0.2
0.4
0.6
0.8
1.0
Total Enviroment Stops (6o4)
Figure I: Average rewards under three proximal policy optimization (PPO) methods in a synthetic environment with 54 actions (among which only 4 actions influence states and rewards). The green line refers to the PPO trained based on the true influential actions, the red line refers to the PPO with the estimated minimal sufficient actions by the proposed variable selection (KS), the black line represents the PPO with the entire redundant action space, and the dashed line is the optimal reward. The red line outperforms the black line, indicating the effectiveness of the variable selection step.
problem, by discretizing large action spaces into smaller subspaces. However, this transformation can lead to significant loss of precision and hence produce suboptimal solutions (Lee et al.) 2018 Tan et al. 2019
ct al. 2023) Other works (sec e.g., Jiang et al. 2019: Luo et al. 2023) focused on musele control tasks and used architectures reducing the action dimensionality before deploying RL methods. One recent study by |Schumacher al. (2023) combined differential extrinsic plasticity with RL to control nigh-dimensional large systems. Yet, all these works require specialized data collection, known joint ranges of actions, forced dynamics, or desired bchaviors of policies, before implementing RL. In contrast, our method is entirely dala-driven wilhoul prior knowledge of environments and thus can be generalized to lasks beyond muscle control. Some studies (Zahavy et al. 2018 Zhong et al.) 2024) have also explored eliminating actions; however, their approaches are limited to discrete action spaces and cither require explicit climination signals provided by the environment or fit an inverse dynamics model of the environment.
Variable selection, also known as feature selection, critical process to choose the most relevant variables representing the target outcome of interest, enhancing both model performance and interpretation. Over the past few decades, many well- methods have been established, ranging from classical LASSO, Fisher
score, and kernel dimension reduction (Tibshirani 1996 Gu et al. 2012: Chen et al.] 2017 Zhou et al.
2021), towards deep learning (Liang et al. 2018 Balin et al. 2019 Lee et al. 2021 Cai et al. 2023b Zhang ct al. 2023). Yet, these works either suffer from madell-aseed construints or lack theoretical guarantees. The model-X knockoff method proposed by Candes et al. (0118) aims to achieve both goals via a general variable selection framework for black-box algorithms with guarantecs of false discovery rate control. Due to its model-agnostic nature, the knockoff method has been extended to complement. a wide range of variable selection approaches (Sesia et al. 2017 Ma et al.) 2021 Liu et al. 2022). The main price central challenge within the knockoff method lies in the generation of faithful knockoff features. Existing techniques either use model-specific methods (see e.g., [eesia et al.) 2017 Liu & Zheng 2018) that assume the underlying covariate
distribution, or model-free approaches (see e.g., Jordon et al. 2018 Romano et al. 2020) that utilize deep generative models to obtain knockoffss without further assumptions on feature distribution. Owing to the blessing of online RL, our method bypasscs this challenge through the known joint distribution of actions
represented by the ongoing policy network, and thus can easily resample the action values create exact knockoff features.
2
Problem Setup
2.1 Notations
Consider a Markov Decision Process (MDP) characterized by the tuple (S.A.p,r,))) in which both the state space S and the action space A are continuous. The state transition probability, denoted p: S S A
3

---

Published in Transactions on Machine Learning Research (7/2025)
[0, oo), is an unknown probability density function that determines the likelihood of transitioning to a next state 8:+1 E S, given the current state 8, E 8 and the action a, E A. The environment provides a reward, bounded within [Tmin: Fmax], for each transition, expressed as r S x A + [Fmin, Tmax] The discount factor, represented by y E (0, 1), inffuences the weighting of future rewards. We denote a generic tuple consisting of the current stale, action, reward, and subsequent. state as AS, R₁, S1+1). The Markovian property of MDP is that given the current state S, and action A,, the current Re and the next state St+1 are conditionally independent of the past trajectory history. Consider A, E IR? where p is very large indicating a high dimensional action space. We utilize (sx and (s,, to denote the state and state-action marginal distributions, respectively, of the trajectory distribution generated by a policy ( st). The notation ((r) is used to represent the expected discounted reward under this policy: J(z) = E, (e a.)pp [ (st, ay]]- The goal of RL is to maximize the expected sum of discounted rewards above. This can be exlended to a more general maximum entropy objective with the expected entropy of the policy over Px (5c).
2.2 Minimal Sufficient Action Set in Online RL
To address the high-dinensional action space, we propose to utilize variable selection instead of representatin for practical usefulness. To achieve this goal, we first formally define the minimal sufficient action set. Denote the subvector of A, indexed by components in G as A with an index set G C (1. p}. Let G (l....,p) G) be the complement of G.
Definition 2.1. (Sufficient Action Set) We say G is the sufficient action (index) set in an MDP if
R₁ Ar, St, Ar,G, S 1 Ag,as St, Ar, for all t 120 0.
The sufficient action set can be scen sufficient conditional set to achieve past and future independence. The sulficient action set may not be unique.
Definition 2.2. (Minimal Sufficient Action Set) We say G is the minimal sufficient action set in an MDP if it has the smallest cardinality among all sufficient action sets.
Unlike the sufficient action set, there is only one unique minimal sufficient action set to achieve conditional independence there are no identical action variables in the environment. We also call G the redundant set when G is the minimal sufficient action (index) set. Here, to achieve such minimal sufficient action sct, one should also require the states S, to be the sufficient states, so there is no useless state (Ma et al., 2023) introduce related redundant actions that. possibly lead to ineffective exploration loss of generality, redundant actions in a higl-dimensional action space. Our goal is to identify the minimal sufficient action set for online deep reinforcement learning to improve exploration.
data inefficiency. Without assune sufficient states throughout this paper and focus eliminating the influence of
2.3 Preliminary: Knockoff Variable Selection
Without making additional assumptions on the dependence among variables, in this work, we utilize the model-X knockoffs (Candes et al. 2018) for flexible variable selection, which ensures finite-sample control of the false discovery rate (FDR). We first briefly review the model-X knockoff's (Candes et al. (2018) in the supervised regression setting with independent samples, which will be leveraged later as the base variable selector of our proposed method for dependent data in the online RL setting.
Suppose we have i.i.d. samples from a population, each of the form (X,Y), where X (X1, ................................ Xp) R and the outcome Y E R. We further denote Y as n-dimensional response vector and X as an x p matrix of covariates by aggregating n samples. The variable selection problem stems from the fact that, in many real-world scenarios, the outcome variable Y is influenced by only a small subset of the predictors X. Formally, the goal is to identify a subset of indices G C {l,...,p]. with |G| p, such that the conditional distribution Fyix depends only on the variables (Xjbea; and Y is conditionally independent of the remaining variables given this subset. That is,
Y 1. X3 (Xi)ked for all j & G.
4

---

Published in Transactions on Machine Learning Research (7/2025)
The variable selection is looking for the Markov blanket G, i.e., the "smallest" subset G such that conditionally on {X)] Y is independent of all other variables. To ensure the uniqueness of relevant variables in the Markov blanket, pairwise independence is introduced as follows.
Definition 2.3. A variable X, is said to be 'null' if and only if Y is independent of X, conditionally on the other variables X-, = (X,,,...pp)) MXj}, otherwise said to be 'nonnull' or relevant. The set of null variables is denoted by Ho C p).
For a selected subset G of the covariates constructed from data and some pro-spocified level E (0, 1), the false discovery rate (FDR) and modified FDR (mFDR) associated with & are formally defined G and mFDR == Bio@nol/LLL/ Hol/(L/q+ [ā]]], respectively, where H denotes the
FDR HolGnHHlmmaae cardinality of a set. Here, FDR is the expected proportion of falsely selected variables among the selected set, and mFDR offers less conservative measurement by adjusting the denominator. Knockoff variable selection aims to discover as many relevant (conditionally dependent) variables as possible while keeping the FDR under control.
Towards this goal, the model-X knockoff generates an n x p matrix x= (.,, have the similar propertics as the collected covariates. This matrix is constructed by the joint distribution of X and satisfies:
.00....,,,, Fp) as knockoff features that
XLYIX and (X, swap(()) 4 (X,X),
(1)
for each subset Q within the set {I,..., .P). where swap([?) indicates the operation of swapping such that for each j E S2 the j-th and (j + p)-t columns are interchanged. Here, the swap is performed between corresponding coordinates of X and X, and we must ensure that the knockoffs are constructed such that the joint distribution remains invariant under coordinate-wist swaps. The notation _ signifies equality distribution. After obtaining knockoff features, let D = {X, X, Y} denote an augmented dataset and we can calculate the feature importance scores Z, and 2, for each variable z, and its corresponding knockoff 2; based on any regression or machine learning methods like lasso or random forest. Define the function f RR2 - R v. Set
as an anti-symmetric function, meaning that f(u, v) - [[( u) for all u, E R2, e.g., f(u, v) knockoff statistics W = (WI..... Wr) where W2 f(z,, Z,) in such a way that higher values of W, indicate stronger evidence of the significance of æ, being influential covariate. The j-th variable selected its corresponding W, is least certain threshold Ta when the target FDR level a. Then the set of chosen variables can be represented as I {j: : W 2 Ta), where
# { p
W
< W
-T} 2T] saj.
Fa min
T T>
{ E
p
The knockoff procedure controls the mFDR at pre-specified level ensuring that the approximated expected proportion of false positives among the selected variables does not exceed a. A more conservative variant, known as Knockoffs+t, can be used to provide guaranteed control of the FDR. More details can be found in (Candes et al. 2018).
3 Online Deep RL with Variable Selection
To identify the minimal sufficient action set in online deep RL, integrate the action selection into RL to find truly influential actions during the training process. Its advantages are manifold. Firstly, its model-agnostic nature ensures compatibility across various RL architectures and algorithms. Morcover, its data-driven characteristic allows for straightforward application across diverse scenarios, thereby increasing practical utility. Crucially, the action selection boosts the explainability and reliability of RL systems by clearly delincating actions that contribute to model performance. In the following, we first introduce an action- selected exploration strategy for online deep RL in Section 3.1 followed by the model-free free knockoff-sanie saappiiin method for action selection in Section 3.2
5

---

Published in Transactions on Machine Learning Research (7/2025)
3.1 Action-Seleciee Exploration Algorithm
We propose innovalive action-selected exploration for deep RL. Suppose at predefined time step ( t= Tes; a set of actions G is identified from the buffered data, where the cardinality of G (IGl) is d, with d S p indicting a size of selected actions. critical challenge ariscs in leveraging the insights gained from action selection for updating the deep RL models. The conventional approach of constructing an entirely new model based on the selected actions is not only time-consuming but also inefficient, particularly in dynamic, non-stationary environments where the requisite action sets are subject to frequent changes. Although our study primarily focuses stationary environments, the inefficiency of model reinitialization post-selection remains notable concern.
To searnlessly and efficiently integrate action selection results into deep RL, we propose to mask the non- selected actions and remove their influence once a hard mask constructed, and thus is flezible to integrate with arbitrary variable selection method. Specifically, in continuous control tasks, deep RL algorithms utilize a policy network Ty to sample a certain action a given current state s, namely a ~ To (- [-] Here, we use the Gaussian policy as an illustrative example, but it can be flexibly generalized to other distributions. Assume the policy network is parameterized by multivariate Gaussian with the diagonal covariance matrix as:
a N~N (s). diag (a (s))²)
where i and a are parameterized functions to output mean and standard deviations. Each time we obtain an action from the policy network. The updates of the policy network To and the action-value function Qo usually involve sampled actions a and log me (a, st), which is the log density of sampled actions. Our strategy is to use a binary mask to set them a certain constant value during the forward pass, and it will block the gradient when doing backpropagation and also remove influence when fitting function. Given selected action set G, we focus on integrating this selection into the model components Qa(a, s) and z s). To facilitate this, we define a selection vector m (m....., mp) {0, 1)", where m, if i E G and otherwise. This vector enables the application of selection mask to both the Qo and mo as follows.
For Qs we use the hard mask to remove the influence of non-selected actions during Q function fitting,
QTe (a. = Qp(m 3 a)s),
where C is the element-wise product. The adoption of action selection can reduce bias in the Q function fitting when sufficient action is correctly identified. It can also reduce variance by decreasing the complexity of the hypothesis space of the Q function.
For ne, considering the necessity of updating the policy network via policy gradient, we integrate a hard mask into the logarithm of the policy probability. The modified log probability formulated as
log Th ( (a s) = m T s). Te(ap s)),
(4)
where is the dot product. This masking of the log probability helps mitigate the likelihood of encountering extremely high entropy values, thereby facilitating more stable and efficient training process. We demonstrate the integration of action selection into deep RL as detailed in Algorithm| [
Remark 3.1. Here, we focus the case where actions parameterized as diagonal Gaussian which are conditionally independent given states. However, our method can be casily extended to the correlated actions, with details provided in Appendix[1 [T]
Remark 3.2. In scenarios where the algorithm exclusively employs the state-value function Vs(s), the use of the mask operation is unnecessary. Our empirical studies suggest that, even without masking, the model maintains robust performance. This implics that updates the policy network may hold greater significance than those to the critic in certain contexts.
3.2 Smockoff-Samplinn for Action Selection
Despite the large volume of variable selection (VS) methods (sce e.g., Tibshirani 1996 Gu al. 2012
Chen et al. 2017 Liang al.] 2018 Balm et al. 2019 Tee et. al. 2021 these works eit either suffer from
6

---

Published in Transactions on Machine Learning Research (7/2025)
Algorithm Action-Selected Exploration in Reinforcement Learning
Require: FDR rate majority voting ratio r, max steps T, variable selection step Tu.
Begin: Initialize the selection set = (,, policy value function parameter Ø, augmented replay buffer D
while steps smaller than do
Sample a, و (- (s))
Sample knockoff copy a, ~ my 8t)
S1+1 Env Sa)
D < DU (8t, a,, ar; Tt, S
if t == Tu. then
Utilize a variable selection algorithm (optional: Knockoff-Sampling in Algorithm 2 on to obtain
the estimated minimal sufficient, action set G Generate a mask m based on G to prune RL networks based on equation end
3 and equation 4
it's time to update then update & and based on the specific RT algorithm used end if end while
Algorithm 2 Knockoff-Sampling Variable Selection
Require: FDR rate a, majority voting ratio I, data buffer D = {(s,, ar; åz r', 1
Split D into non-overlapping sets {D*} and let Yt = (r', St+1) as the response vector
for = 1,... K do
for i-th dimension in {yr) do
Apply machine learning algorithm to all (st, ar, Yc) E Dx to construct feature importance statistics Zj, and Zy for the j-th action and its knockoff copy, respectively, for each E [p].
end for
for each j [p] do
Set Z, = max, Z000 7, max, Z,,0 and Wy (2
end for
Utilize the threshold Ta defined in equation and get G2 = ij E [p] : W, 2 Ta}
end for
return G= {3 jE (I. Ek= (j Gz) 2 KT}
model-based constrwints or lack theoretical guaranlees. The traditional VS often identifies unimportant actions, leading to high false discovery rate and further causing performance degeneration, as shown in Fig. 2 To provide a general action selection approach for deep RL. with false discovery rate control, we propose a novel knockoff-sanne -oooooifsaammiigggg (KS) method that handles dependent data the online setting with a model-agnostic nature as follows.
Suppose now have a data buffer with the size M, collected from N trajectories where each trajectory has length T, for j= J.....N N....N and Σ T, M. Each time obtain action from the policy network, we also resample knockoff copy conditional on the same state ä, no (- st), and append it to the buffer. (S,,A,, A., Rr, S.41.. Note that steps within each trajectory are
The transition tuples thus is redefined temporally dependent. To address the issues of highly dependent data in online RL, couple our method with sample splitting and majority vote following Ma et al. (202). The proposed KS method consists of three steps as summarized in Algorithm 2 1. Sample Splitting; 2. Shockolf-SSmplinn Variable Selection; 3.
Majority Vote. We detail each step below.
1. Sample Splitting: We first split all transition tuples (S,, A,, A,, R, into K non-overlapping sub-datasets. This process results a segmentation of the dataset D into distinct subsets Dj for k E [K]. We combine response variables and denote Y, (R,, Se+1) to simplify the notation, based on the target.
7

---

Published in Transactions on Machine Learning Research (7/2025)
Ant-v4 PPO
Ant-vé SAC
PPG Trus Actions
1400
PVD Tue Act
600
PPQ V5
PPG KS
1200
P9D KS
PPD -VS
1000
400
Axcrigg Rocum
200
600
200
-200
200
3.0
0.4
0.8
1.0
0.0
0.4
0.8
3.0
(led)
Steps (1ef))
Figure 2: Learning curves in the Ant-v4 environment reveal that the Knockoff Sampling (KS) method outperforms the traditional Variable Sclection (VS) method. When implemented with either the Proximal Policy Optimization (PPO) or Soft-Actor-Critte (SAC) algorithm, KS achieves performance comparable to that of the true actions by automatically set ting optimal thresholds to filter out redundant. actions, whereas VS often selects useless actions, leading to high false discovery rate.
outcomes in Definition 2.1 Here, each sequence (St, Ac, Y,) is assigned to D& if L mod K k 1. Subsequent to this division, any two sequences located within the same subset Dj either originate from the same trajectory with a temporal separation of no less than K stem from different trajectories. If the system adheres to 3-mixing conditions (Bradley 2005), then a careful selection (Berbee 1987h can allow us to assert that transition sequences within each subset Dr are approximately independent.
2. Stockoff-SSmpllee
Variable Selection: set using the model-X knockoffs as the base selector. Unlike the knockoff method detailed in Section 2.3 that either construets knockoff features based on second-order machines estimates the full distribution, directly sample knockoff copy of actions from the policy network, i.c., ā no (-1 bypass the common challenge of creuling accurale knockoff features in model-free knockofJs. We ically validate that the sampled knockoffis in online RL meet the swapping property equation
For each data subset DR, we select a minimal sufficient action
ss).. This helps to
[ in Section 4 For every single dimension i of the outcome vector Y, = (R4, Si+1), we use a general machine learning mothod (e.g., LASSO, random forest, neural networks) to provide variable importance scores Z3,, and Zj,x for the j-th dimension of actions and its knockoff copy, respectively. By the maximum score Zj .max; Zi and
Zj max; Zj,,, the selected action set GR is then obtained following the same procedure in Section 2.3 after computing knockoff statistics W.
3. Majority Vote:
To combine the results on the whole K folds, we calculate the frequency of subsets where the j-th action is chosen, i.e., Pj = Ek= T(j E G and establish the ultimate selection of ions G {j p Т}, with I' being predetermined cutoff between 0 and 1.
4 Theoretical Results
Without loss of generality, we assume that the data buffer DD consists of N i.i.d. finite-horizon trajectories, each of length T, which can be summarized as NT transition tuples. We first define two properties to establish theoretical results.
Definition 4.1. (Flip Sign Property for Augmented Data) Knockoff statistics W = (W,,... Wr)) satisties property on the augmented data matrix D& [Ax, AR. Sp. Y&] if for any j e [p] and c [p],
([As, Ar] Se. Ys). W. ([As, Ar] Se, Ys),
if
W: Ar) swap(()) Ss, Yx)
otherwise,
where A A E / denote the matrices of the actions and their knockoffs, S (NT/K))xd denote the matrice of state, Y& (NNTKK)xd+1 denotes the response matrix, and [AR, As] (k]aaa)()) is obtained by swapping all j-th columns in A À for j E Ω.
8

---

Published in Transactions on Machine Learning Research (7/2025)
The above flip sign property is common property that needs to be satisfied in knockoff-type methods. We show that our method automatically satisfies this property in Lemma 1.3 of the Appendix.
We now define the f-mixing coefficient, which quantifies the strength of dependence between observations separated by a time lag in a stochastic process.
Definition 4.2. (6-mixing Coefficient, Bradley 2005 ). For a sequence of random variables {Xx), define its β-mixing coefficient as
3(1) == sup 3 (F"h no1 Fimm mEZ
Using this definition, we now introduce the concept of exponential B-mixing, which describes processes where the dependence decays at exponential rate.
Definition 4.3. (Stationarity and Exponential 3-Mixing) The process {(S,, A,, Re) Re)3r2 is stationary and exponentially 3-mixing if its ß-mixing coelficient at time lag k is of the order p for some 0 < <p<
This exponential f-mixing condition has been assumed in the RL literature (see e.g., Antos et al. 2008 Dai
et al. 2018) to derive the theoretical results for the dependent data. Such a condition quantifies the decay in dependence as the future moves farther from the past to achieve the dependence of the future on the past. Based on the above definitions, we establish the following false discovery cont results of our method.
Theorem 4.4. Set the number of sample splits k= ko log(NT) for some knn > llog where p is defined in Definition 4.3 Assume that the following assumption hold: the process {(S,, A., Ro))e2 is stationary and exponentially 3-miring.
Then GR obtained by Algorithm with the stondard knockoffs controls the modified FDR (mFDR),
where the constant c == - 1 11 0.
The proof can be mainly divided into two parts. Firstly, we show that valid mFDR control can be achieved when data are independent. Then, for dependent data satisfying the 3-mixing condition, the upper bound can be relaxed account for the cost of dependence, which vanishes as the number of samples in D, approaches infinity. Finally, combination of the two would provide the final upper bound on mFDR control. The detailed proof is in Appendix [ This thcorem provides theoretical guarantee for controlling the modified false discovery rate by our proposed method at pre-specified level a with small order term that goes to zero as the sample size goes to infinity. In reinforcement learning, this ensures the identification of a minimal sufficient set of actions with a controlled error tolerance, which is essential for efficient learning. Our method selects and only utilizes those relevant actions, avoiding irrelevant ones that could dograde policy performance and lead to unnecessary costs.
5 Experiments
Experiment Setup We aim to answer whether the variable selection is helpful for deep RL training when the action dimension is high and redundant. We conduct experiments on standard locomotion tasks in MuJoCo (Tooroo et al. 2012) and treatment allocation tasks calibrated from electronic health records (EHR), the MIMIC-III dataset (Johnson et al. 2016). The environment details are in Table C.1 of the Appendix. Here, we focus on two representative actor-critic algorithms, Proximal Policy Optimization (PPO) (Schulman et al. 2017) and Soft-Actor-Crits (SAC) (Haarnoja et. al., 2018). We adopt the implementation from Open Al Spinning up Framework (Achiam| 2018). For SAC, the implementation involves fitting both Qo and Te, and we use a mask on both components. For PPO, it fits Vo and лo hence we only combine the mask with go- Tables B. and B.2 summarize the hyperparameters we used. We set the FDR rate a = 0.1 and voting ratio T= 0.5 in all settings. All the experiments are conducted on the server with 4x NVIDIA RTX A6000 GPU.
Semi-synthetic MuJoCo Environments We chose three tasks: Ant, HalfChectah, and Hopper. To increase the dimension of action space, we artificially add extra p actions to the raw action space and consider two scenarios, p= 20 and 50. For each setting, we run experiments over 2 x 103 and 106 steps for SAC and
9

---

Published in Transactions on Machine Learning Research (7/2025)
Adb in Edtr 20 Actions Eoggriing State!
PO of with Extra 10 AcAcco Dog miny Exate
Hospur vd Extra 20 Actioms Doggrning Statel
PPC Aclaen
2004
2000
200
1000
-00
(a) PPO Aut 20 (Initial)
(b) PPO Cheetah 20 (Initial)
(c) PPO Hopper 20 (Initial)
oning
Begirning
DDeg
1i1. Pir entins
1580
(10 Tia
2000
2000
300
300
(d) PPO Ant 50 (Tnitial)
(e) PPO Cheetah p 50 (Initial)
(() PPO Hopper p 50 (Initial)
Figure 3: Learning curves for PPO in the MujoCo environments with different approaches during the initial stage. In all experiments, our knockoff sampling (KS) method not only performs comparably to the true actions but also consistently delivers higher rewards than using all actions.
HPp Tackiiuaa
2100
FDD
Al/ction
Thes Arlen
Tiee
(a) PPO Ant p 50 (Middle)
(b) PPO Cheetah == 50 (Middle)
(c) PPO Hopper == 50 (Middle)
Figure 4: Learning curves for PPO in the MujoCo environments during the middle stage, where the red line indicates the time point we utilize the proposed KS. After identifying the essential action set, the policy can be more efficient and achieve higher rewards than continuing training on all actions.
PPO, respectively, averaged over 10 training runs. The running steps for SAC and PPO are set adaptively to obtain better exploration for each method and save computation costs, as the main goal is to show how action selection can improve sample efficiency rather than compare these two methods. For each evaluation point, run 10 test trajectories and average their reward the average return. Besides RL algorithm performance, also evaluate variable selection performance in terms of True Positive Rate (TPR), False Positive Rate (FPR), and FDR.
Action Selection in the Initial Stage of Training We utilize action selection in the beginning stage of the training. For both methods, we utilize the first 4000 samples for variable selection and then use the selection results to build a hard mask for action in deep RL. models. We compare our knockoff sampling (KS) method with the baseline of selecting all actions (All) to evaluate the impact of integrating masking mechanism with selection strategy in deep RL. We also provide the experimental results with only ground- ttruhthttthhhoo actions selected (True) as reference. To reduce the computational complexity, we choose LASSO [Tibshiran] 1996) as our base variable selection algorithm for KS. Here, selecting all actions (All) and ground-truth actions (True) are the cases where RL models are trained on the whole action space and minimal sufficient action space, respectively. Hence, the model corresponding to mound-truthe action has smaller parameters than all other methods because its initialization is basod on the minimal sufficient action set. The results are shown in Fig. for PPO, Fig. C.1 for SAC, and Table D for all numerical details. Due to space constraints, we mainly present the PPO figures in the main text. In all cases, we find that KS-guided models putporform
10

---

Published in Transactions on Machine Learning Research (7/2025)
Table Results on the PPO and SAC for three Mujoco tasks: Ant, HalfCheetah, and Hopper. Action selection is utilized the beginning stage RI training. The final reward the performance evaluation for the agent after training. The best pest-performing results between KS and All are highlighted in bold.
Ant
Env
RL Algo.
p
Selection
TPR (t)
FDR (4)
FPR (4)
Reward (t)
0
True
1.00
0.0
0.00
567.77
KS
1.00
0.01
0.01
507.90
PPO
20
All
1.00
0.71
1.00
202.65
KS
1.00
0.00
0.00
572.39
50
All
1.00
0.86
1.00
151.66
Ant
0
True
1.00
0.00
0.00
817.95
KS
1.00
0.01
0.01
937. 74
SAC
20
All
1.00
0.71
1.00
12.61
KS
1.00
0.00
0.00
731.73
50
All
1.00
0.86
1.00
-28..04
0
True
1.00
0.0
0.00
2130.55
KS
1.00
0.01
0.01
2237.00 .08
PPO
20
All
1.00
0.77
1.00
1356.46
KS
1.00
0.00
0.00
1932.27
50
All
1.00
0.89
1.00
619.67
HalfCheetah
0
True
1.00
0.00
0.00
6640.05
KS
1.00
0.00
0.00
6607.55
SAC
20
All
1.00
0.77
1.00
5631.20
KS
1.00
0.00
0.00
6873.95
50
All
1.00
0.89
1.00
4748.24
0
True
1.00
0.0
0.00
1736.65
KS
1.00
0.00
0.00
1540.82
PPO
20
All
1.00
0.87
1.00
1205.12 12
KS
1.00
0.00
0.00
1710.82
50
All
1.00
0.94
1.00
703.08
Hopper
0
True
1.00
0.00
0.00
2511.00
KS
1.00
0.00
0.00
2165.81
SAC
20
All
1.00
0.87
1.00
398.75
KS
1.00
0.00
0.00
2424.09
50
All
1.00
0.94
1.00
137.67
those trained on the whole action space in terms of average return and have much lower FDR and FPR, with larger improvement gains as p increases. This empirically validates our theory of FDR control with the proposed KS method, demonstrating that action selection can enhance learning efficiency during the initial stages of RT training where action space high and redundant.
Action Selection in the Middle Stage of Training To show whether action selection can be used in the middle stage of training to remedy the inefficiency brought by exploring the whole action space, we conduct experiments where in the first half of the training steps the models are trained on the whole action space, and in the middle of the stage, we utilize action selection and build hard masks for them and then continue training for the rest of the steps. We compare our KS method with selecting all actions (All) and ground-truth actions (True) similarly. The results in Fig. 4 and Fig. (.2 a notable pattern: agents initially struggle to learn effectively, but mid-stage variable selection significantly improves their performance, with models trained on the correct actions. This demonstrates the effectiveness of mid-stage variable selection in enhancing learning outcomes.
11

---

Published in Transactions on Machine Learning Research (7/2025)
P70 155
14
PTO leticc
PRO ALACToNG
PIO gade
Arwnge Retum
12
1.
10
15
0.6
0.8
1.0
Treatment Index
Total Stops 12040
(a) Learning curves of treatment allocation environments.
(b) Treatment Selection frequency of KS method
Figure 5: Results for PPO in the treatment allocation environments during the initial stage using different approaches. The learning curves of various methods are shown on the left, while the selection frequencies from our method are presented on the right, with red bars indicating the essential treatments. Our approach, KS, demonstrates improved performance over time compared to using all actions. In contrast, latent exploration- based methods such as Lattice and gSDE exhibit degraded performance.
Treatment Allocation for Sepsis Patients We evaluate our method using PPO and utilize the first 1000 samples for action selection during the initial stage of training. In addition to our proposed KS method and the baseline that uses all actions, we include two latent exploration- based baselines: Lattice (Chiappa let al. (024) and gSDE (Raffin et al. 2022). While both Lattice and gSDE operate over the full action space, they incorporate temporally correlated Gaussian noise into the training process, where the noise variance is learned from latent. representations. Experiments are conducted
2 10a time steps, with results averaged across 5 independent runs. At each evaluation point, generate 5 test trajectories and report the average return. The results are presented in Fig. [[a] and Fig. 5(b) Our method consistently achieves more stable and superior performance compared to all other approaches in Fig. 5(a) As shown in Fig. 5(b] KS effectively identifies treatments directly relevant to sepsis management, such as vaso_dose and iv input, which are closely tied to key physiological indicators. Importantly, KS avoids selecting those non-essential treatments beta_blocker
like beta_ blocker and diuretic, which may influence patient dynamics but have no direct effect on the SOFA score, making them less relevant for optimizing sepsis-specific outcomes. In contrast, Lattice and gSDE exhibit slower convergence and, in some cases, degraded performance, potentially due to verr-xxploratio. We also observe that these methods are sensitive to the inilialization of the log standard deviation and the scaling of latent representations, which can lead to unstable learning dynamics. In comparison, our method requires less parameter tuning and demonstrates greater robustness across diffcrent environments. These findings highlight the potential of KS for enabling more targeted, interpretable, and efficient treatment strategies in real-world medical decision-making applications.
Action Selection is Fast and Lightweight With just few thousand data points and lightweight machine learning al like random forest LASSO, the whole action selection process outlined in Algorithm completes in under 20 seconds seconds-including including knockoff threshold determination. This is significantly faster and less computationally intensive than the RL training part. Even when incorporating more sophisticated feature selection methods, the additional computational overhead remains negligible compared to the time required for RL training. Moreover, for the RT. agent's deep neural network, only a few lightweight masking parameters are introduced, which have minimal effects on both training and inference speed. Yet, these in turn substantially enhance policy optimization.
Additional Experiments We conduct additional experiments to visualize the action distribution during training, both with and without masking. The results indicate that masking promotes more focused and potentially more effective learning process. Furthermore, we increase the PPO step size from 106 to 4 x 106, demonstrating that our method achieves both high efficiency and improved performance, Additionally, we investigate whether network capacity plays critical role in addressing high-dimensional action problems. However, we find that merely increasing network capacity does not necessarily simplify the learning process. Detailed results can be found in Appendix D
12

---

Published in Transactions on Machine Learning Research (7/2025)
6 Conclusion, Limitation, and Future Work
In this work, address the high-dimensional action selection problem in online RL. We formally define the objective of action selection by identifying minimal sufficient action set. We innovate by integrating a knockoff-sampling variable selection into broadly applicable deep RI. algorithms. Empirical evaluations in syuthotic robotics and treatment allocation environments demonstrate the enhanced efficacy of our approach. Yet, a notable constraint of our method its singular application during the training phase, coupled with the potential risk of overlooking essential actions with weak signals. Inadequate action selection could degrade the agent's performance. Intriguing future research includes extending our methodology to incorporate multiple and adaptive selection stages. This adaptation could counterbalance initial omissions in action selection. Additionally, formulating effective termination criterion for this process represents another compelling research direction.
Acknowledgements
The authors thank the Action Editor and anonymous reviewers for their constructive and insightful feedback. This work was supported by the National Science Foundation under grant DMS-CDS&FFMMS No. 2401271.
References
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residuull minimization based fitted policy iteration and single sample path. Machine Learning, 71:89-129, 2008.
Muhammed Fatih Balm, Abubakar Abid, and Jamos Zou. Conerete autoencodors: Differentiable feature selection and reconstruction. In International conference on machine learning, pp. 444 453. PMLR, 2019.
Rina Foygel Barber and Emmanuel Candès. Controlling the false discovery rate via knockoffs. 2015.
Henry Berbee. Convergence rates in the strong law for bounded mixing sequences. Probability theory and related fields, 14(2):255-2700 1987.
Richard C Bradley. Basic properties of strong mixing conditions. a survey and some open questions. 2005.
Hengrui Cai, Chengehun Shi, Rui Song, and Wenbin Lu. Deep jump learning for off-policy evaluation in continuous treatment settings. Advances in Neural Information Processing Systems, 34:15285 15300, 2021.
Hengrui Cai, Chengchun Shi, Rui Song, and Wenbin Lu. Jump interval-learning for individualized decision making with continuous treatments. Journal of Machine Learning Research, 24(140):1 2023a.
Hengrui Cai, Yixin Wang, Michael Jordan, and Rui Song. On learning necessary and sufficient causal graphs. Advances in Neural Information Processing Systems, 36:42148 42160, 2023b.
Emmanuel Candes, Yingying Fan, Lucas Janson, and Jinchi Lv. Panning for mode: m for high dimensional controlled variable selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(3):551-577, 2018.
Jianbo Chen, Mitchell Stern, Martin Wainwright, and Michael JJodan. Kernel feature selection via conditional covariance minimization. Advances in Neural Information Processing Systems, 30, 2017.
Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Huang, and Alexander Mathis. Latent exploration for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao Hc, Zhen Liu, Jianshu Chen, and Le Song. Sbood: Convergent reinforcement learning with nonlinear function approximation. In International conference on machine learning, pp. 1125-1134 PMI.R, 2018.
13

---

Published in Transactions on Machine Learning Research (7/2025)
Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnaeve. Growing action spaces. In International Conference on Machine Learning, pp. 3040-3051. PMLR, 2020.
Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep neural networks for estimation and inference. Econometrica, 89(1):181-213, 2021.
Samuel J Gershman, Bijan Pesaran, and Nathaniel D Daw. Human reinforcement learning subdivides structured action spaces by learning effector-specific values. Journal of Neuroscience, 23(43)):3622-13331, 2009.
Quanquan Gu, Zhenhui Li, and Jiawei Han. Generalized fisher score for feature selection, arXiv preprint arXiv:1202.3725, 2012.
Zhaohan Damiel Guo and Emma Brunskill. Sample efficient feature selection for factored mdps. arXiv preprint arXiv:1703.03454, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018.
Botao Hao, Yaqi Duan, Tor Lattimore, Csaba Szepesvári, and Mengdi Wang. Sparse feature selection makes batch reinforcement learning more sample efficient. In International Conference on Machine Learning, pp. 4063 4073. PMLR, 2021.
Vishal Jain, William Fedus, Hugo Larochelle, Doina Precup, and Marc G Bellemare. Algorithmic improvements for deep reinforcement learning applied to interactive fiction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3328-4336, 2020.
Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and C Karen Liu. Synthesis of biologically realistic human motion using joint torque actuation. ACM Transactions On Graphics (TOG), 38(4):1-12, 2019.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1 9, 2016.
James Jordon, Jinsung Yoon, and Mihaela van der Schaar. Knockoffgan: Generating knockoffs for feature selection using generative adversarial networks. In International conference on learning representations, 2018.
Lukasz Kaiser, Mohammad Babacizadch, Piotr Milos, Blazej Osinski, Roy H Camphell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint vcXiv:1903.00377 2019.
Anssi Kanervisto, Christian Scheller, and Ville Hautamāki. Action space shaping in deep reinforcement learning. In 2020 IEEE conference on games (CoG), pp. 479 486. IEEE, 2020.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238 1274, 2013.
Petter N Kolm and Gordon Ritter. Modorn perspectives on reinforcement learning in finance. Modern Perspectives on Reinforcement Learning in Finance (September 6, 2019). The Journal of Machine Learning Finance, 1(1), 2020.
Mark Kroon and Shimon Whiteson. Automatic feature selection for model-based reinforcement learning in factored mdps. In 2009 International Conference on Machine Learning and Applications, pp. 324-330. IEEE, 2009.
Changhee Lee, Fergus Imric, and Mihacla van der Schaar. Self-supervision enhanced feature selection with correlated gates. Tn International Conference on Learning Representations, 2021.
14

---

Published in Transactions on Machine Learning Research (7/2025)
Kyowoon Lee, Sol-A Kim, Jacsik Choi, and Scong- Whan Lee. Deep reinforcement learning in continuous action spaces: case study in the game of simulated curling. In Internat conference on machine learning, pp. 2937 2946. PMLR, 2018
Yuhan Li, Wenzhuo Zhou, and Ruoqing Zhu. Quasi-optimal reinforcement learning with continuous actions. arXiv preprint 0rX1v:2301.08940, 2023.
Faming Liang, Qizhai Li, and Lei Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Stalistical Association, 113(523):955 972, 2018.
Jingyuan Liu, Ao Sun, and Yuan Ke. A generalized knockoff procedure for fdr control in structural change detection. Journal of Econometrics, 2022.
Ying Liu and Cheng Zheng. Auto-encoding knockoff generator for fdr controlled variable selection. arXiv preprint 1iti:18099.1666. 2018.
Ying Liu, Brent Logan, Ning Liu, Zhiyuan Xu, Jian Tang, and Yangzhi Wang. Deep reinforcement learning for dynamic treatment regimes medical registry data. In 2017 IEEE international conference on healthcare informaties (TCHT), pp. 380-385. IEEE, 2017.
Shuzhen Luo, Ghaith Androwis, Sergei Adamovich, Erick Nunez, Hao Su, and Xianlian Zhou, Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning. Journal of neuroengineering and rehabilitation, 20(1):1 19, 2023.
Shiyang Ma, James Dalgleish, Justin Lee, Chen Wang, Linxi Liu, Richard Gill, Joseph D Buxbaum, Wendy K Chung, Hugnes Aschard, Edwin K Silverman, et al. Powerful gene-based based testing by integrating long-range chromatin interactions and knockoff genotypes. Proceedings of the National Academy of Sciences, 118(47): e2105191118, 2021.
Tao Ma, Hengrui Cai, Zhengling Qi, Chengchun Shi, and Eric B Laber. Sequential knockoffs for variable selection in reinforcement learning, arXiv preprint ar arXiv:2303. 14281, 2023.
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning, pp. 6961 19716 PMLR, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.508) 2013.
Brahma S Pavse and Josiah P Hanna. Scaling marginalized importance sampling high-dimensional state- spaces via state abstraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 2023.
pp.
Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement learning. Tn Conference on Robot Learning, pp. 1634 1644. PMLR, 2022.
Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Deep knockoffs. Journal of the American Statistical Association, 115(5)2):1861 1872 2020.
Sherry Ruan, Gheorghe Comanici, Prakash Panangaden, and Doina Procup. Representation discovery for mdps using bisimulation metrics. Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
Tomonori Sadamoto, Aranya Chakrabortty, and Jun-ichi Imura. Fast online reinforcement learning control using state-space dimensionality reduction. IEEE Transactions on Control of Network Systems, 8(1): 342 353, 2020.
Andrey Sakryukin, Chedy Raissi, and Mohan Kankanhalli. Inferring dqn structure for high-dimensional continuous control. In International Conference Machine Learning, pp. 8408 8416. PMI.R, 2020.
15

---

Published in Transactions on Machine Learning Research (7/2025)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint Seti:17777003340 2017.
Pierre Schumacher, Daniel Haeufle, Dieter Büchler, Syn Schmitt, and Georg Martius. DEP-RL: Embodied
exploration for reinforcement learning in overactuated and musculoskeletal systems. In The Eleventh International Conference on Learning Representations, 2023. URL htttge///ppaareviieeeneee//eeeee wwr.errreiiee.nnt/ooouurtdd
x=x__D3oTG
Matteo Sesia, Chiara Sabatti, and Emmanuel Candès. Gene hunting with knockoffs for hidden markov models. arXiv preprint mrXiv:1706.04677, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844455 2016.
Xueqing Sun, Tao Mao, Laura Ray, Dongqing Shi, and Jerald Kralik. Hierarchical state-abstracted and socially augmented q-lcarning for reducing complexity in agent-based lcarning. Journal of Control Theory and Applications, 9:440 450, 2011.
Peter Sunchag, Richard Evans, Gabriel Dulac-Amold, Yori Zwols, Daniel Visentin, and Ben Coppin. Decp reinforcement learning with atlention for slate markov decision processes with high-dimensional states and actions. arXiv preprint 111:1151,.01144 2015.
Gabriel Synnaeve, Jonas Gehring, Zeming Lin, Daniel Haziza, Nicolas Usunier, Danielle Rothermel, Vegard Mella, Da Ju, Nicolas Carion, Laura Gustafson, et al. Growing up together: Structured exploration for large action spaces. 2019.
Huachun Tan, Hailong Zhang, Jiankun Peng, Zhuxi Jiang, and Yuankai Wu. Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space. Energy Conversion and Management, 195:548 060.8856600000 2019.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, (8(1):267-288) 1996.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference intelligent robots and systems, pp. 5026 5033. IEEE, 2012.
Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, and Radha Poovendran. Fresh: Interactive reward shaping in high-dimensional state spaces using human feedback. arXiv preprint arXiv:2001.0677 2020.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 551:):1,36, 2021.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. Advances in neural information processing systems, 31, 2018
Wenbo Zhang, Tong Wu, Yunlong Wang, Yong Cai, and Hengrui Cai. Towards trustworthy explanation: On cansal rationalization. In International Conference on Machine Learning, pp. 41715-41736. PMLR, 2023.
Dianyu Zhong, Yiqin Yang, and Qianchuan Zhao. No prior mask: Eliminate redundant action for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17078 17086, 2024.
Wenzhuo Zhou, Ruoqing Zhu, and Donglin Zeng. A parsimonious personalized doso-finding model via dimension reduction. Biometrika, 108(3):643 659, 2021.
Wenzhuo Zhou, Ruoqing Zhu, and Annie Qu. Estimating optimal infinite horizon dynamic treatment regimes via pt-learning. Journal of the American Statistical Association, 119(545):625 638, 2024.
16

---

Published in Transactions on Machine Learning Research (7/2025)
A Comparison with Closely Related Work
Ma et al. 2023) adopted a two-stage framework, performing variable selection offline before applying reinforcement learning. This design is ill-suited for online settings, where dynamic environments require frequent updates to the action set. Moreover, their approach relies on strong data- dattagenerating assumptions that restrict its applicability in realistic scenarios. In contrast, our method integrates action selection into the online RL process via a learned masking mechanism, allowing real-time adaptation to environmental shifts. This not only removes the dependency on stringent assumptions but also supports seamless integration with deep RL algorithms, such as PPO and SAC, which were not addressed in their work.
Zahavy et al. (2018) introduced an action elimination method that relies on explicit elimination signals from the environment to identify invalid actions. While effective certain settings, this approach depends on domain-specifie feedback that may not always be available. Our method avoids this reliance by implicitly iden- tifying essential actions through learned representations, enabling broader applicability without handcrafted elimination signals external supervision.
Zhong et al. 2024] proposed method that constructs an action mask using an inverse dynamics model trained prior to policy learning. However, fitting such a model is computationally demanding in high-dimensional state spaces and completely decoupled from the learning process, making it less responsive to evolving environments. Additionally, extending their approach or Zahavy et al. (2018) continuous action spaces typically requires discretization, which introduces challenges such as loss of precision and the need to carefully tune bin sizes. In contrast, method operates directly in the continuous action space, preserving precision and scalability while enabling efficient, end-to-end learning.
B Implementation Details
We adopt the implementation from Open AT Spinning up Framework (Achiam 2018 Tables B.1 and B.2 show the hyperparameters for the RL algorithms we used in our experiments. We the FDR rate a 0.1 and voting ratio 0.5 for knockofT method in all settings. All the experiments are conducted in the server with 4x NVIDIA RIX A6000 GPU.
Table B.1: PPO Hyperparameters
Paramoter
Mujoco
EHR
optimizer
Adam
Adam
learning rate π
3.0 10-
10-
learning rate V
1.0 10-3
10-3
learning rate schedule
constant
constant
discount (7)
0.99
0.99
number of hidden layers (all networks)
2
number of hidden units per layer
664, 32]
[64, 32]
number of samples per minibatch
256
100
number of steps per rollout
1000
100
non-linearity
ReLU
ReLU
gSDE
initial log o
Full std matrix
Yes
Yes
Lattice
initial log o
Full std matrix
Yes (0.001,1)
Yes (0.001,1)
Std clip
17

---

Published in Transactions on Machine Learning Research (7/2025)
Table B.2: SAC Hyperparameters
Parameter
Mujoco
optimizer
Adam
learning rate π
3.0 10
learning rate Q
3.0 10
learning rate schedule discount (7)
constant
0.9
replay buffer size
10
number of hidden layers (all networks)
2
number of hidden units per layer number of samples per minibatch non-linearity
[256, 256]
256
ReLU
entropy coefficient (a)
0.2
warm-up steps
1.0 10
Table C.1: Summary of Environments
Env
Dimension of Action
Dimension of State
Ant
8
27
HalfCheetah
6
17
Hopper
3
11
EHR
20
46
C
More Experimental Results and Analyses
We list the dimension of action and state in terms of the environments we used in Table C1
C.1 MuJoCo
The results for the initial stage are shown in Fig. + and Table 1. For different environments, action selection difficulty varies, and Hopper is the easiest one where the proposed KS method can correctly select the minimal sufficient action set. Also, the patterns of the results for PPO and SAC are similar. One observation is that KS can select all the sufficient actions with all TPRs oqual to 1. It is shown that KS is efficient in selecting only the minimal sufficient action set in almost all scenarios, which also empirically validates our theory of FDR control under the proposed method.
C.2 Treatment Allocation for Sepsis Patients
We utilize the MIMIC-111 Clinical Database to construct our environment for Sepsis patients. We filter and clean the data to collect 250, 000 data points from the database.
State Space. The observed state 8 E 1R46 encompasses a broad spectrum of clinical and laboratory variables for assessing patient health and outcomes in modical setting, such as demographic information (gender, age), physiological metrics (weight, vital signs such heart rate, blood pressure, respiratory rale, oxygen saturation, temperature), and neurological status. The transition is modeled by s; fe at-1) + e, where fo is a fitted long short-term memory (LSTM) and is the random noise to represent the perturbation.
Action Space. The action a, E R includes treatments such as vasopressors and intravenous fluids, which are commonly administered in sepsis management to stabilize blood pressure and maintain fluid balance, in total 4 actions. Additionally, we incorporate other modications, beta-blockers, and diuretics, which althoughh not primarily intended for sepsis-mit may be prescribed manage comorbid conditions like hypertension or fluid overload that often coexist with complicate sepsis. These treatments considered less essential
18

---

Published in Transactions on Machine Learning Research (7/2025)
AdE in Extro Actions Doogg raing Stare)
16 FiFhhecaa-- will with Extra 20 Aciins (80g reing States
Hiippor wi Extra 20 Actions (oog raing States
2025
710
2010
200
1014
5000
(a) SAC Ant p = 20
(b) SAC HalfCheetah p
(c) SAC Hopper p 20
Adi se Extra 50 Actions (bogieai Stare)
44 #rDhoerann wn with Extra 50 Ac (Bog reing Stare)
Haipper wit th Extra 50 Actions Hhogiraing States
1100-
34C--00
510
Jallons
-005
800-
5000
(d) SAC Ant p 50
(e) SAC HalfCheetah
= 50
(f) SAC Hopper p 50
actions Plegirning
11 Extra been Stare)
detions (hogirning State)
Altara BPC Actane
3000
PPO Alticaan
1000
700
200
1824
Wea Civicrese
Cret
(g) PPO Ant p 20
(h) PPO HalfCheetah p 20
(i) PPO Hopper p 20
3500
900
Tuak Alsira
PR2 Tua
2004
2000
1804
305
(j) PPO Ant p =
(k) PPO HalfCheetah p
PPO Hopper p 50
Figure C.1: Results of SAC and PPO when using different variable selection approaches during the initial stage.
19

---

Published in Transactions on Machine Learning Research (7/2025)
Perter Uniil B-1
2002
VURCE SPRELINS
1350
S/L- Atsora dil
Eos Sariess
2000
600
1300
(a) SAC Ant p = 20
(b) SAC HalfCheetah = 20
(c) SAC Hopper p 20
14:0
2004
XM. Sal-NDO
1300
-500
600
1205
3004
1000
53 Tra Craiorneee de13
Tea www.rermeerrseee (rd1
05
State dhit
(d) SAC Aut p 50
(c) SAC HalfChectah 50
(f) SAC Hopper p 50
Thpp Actone
Fost acciony
1800
1300
05
59 (g) PPO Ant p 20
95
(h) PPO HalfChectah p 20
(i) PPO Hopper p 20
O
1800
(i) PPO Ant p =
(k) PPO HalfCheetah p =
(1) PPO Hopper p == 50
Figure C.2: Learning curves in the MujoCo enviromments with our method during the middle stage, where the red line indicates the time we utilize KS. After identifying a less redundant action set, the policy can be more efficient and achieve higher rewards than continuing training on all actions.
or potentially redundant in the context of sepsis management. To further increase the complexity of the environment, we augment the action space with treatments and medications that have minimal relevance to sepsis, thereby making the decision-making task more challenging.
Reward Function. The overall reward at each timestep is composed of two components, represented Ti =r + rrt, where re denotes the SOFA-based scpsis reward and r captures a health-conditioned bonus. The sepsis reward rf reflects changes in the patient's SOFA score, assigning +1.0 if the score decreases (indicating clinical improvement), and -1.0 if the score increases (indicating deterioration). In addition to this primary reward signal, we introduce healtt--ooniiioned bonus rf which provides supplementary guidance based on the patient's overall physiological state s, and the administered treatment a, This component allows the reward function to capture nuanced aspects of patient health beyond SOFA score dynamics.
20

---

Published in Transactions on Machine Learning Research (7/2025)
Action Dimension
Action Dimension
Action Dimansion
Action Dimension
Densay
Denaity
Decsuity
DenGtt
1.4
44
Action Values
Action Values
Action Values
Action Values
Density
Density
Demitty
Density
Action Values
Action Values
Action Values
Action Valuea
Intial Training
Middle of tralninng without KS
Middle of training with KS
Figure D.1: The distributions of actions in 3 stages: initial training, middle of training without KS, and middle of training with KS. It can be seen that with KS, the actions have slightly less variance than other methods, which could be a positive indicator of a more focused and potentially more effective learning process.
high Extra Actions 18 Bggginning
HailCheetah Extra Actions (Beginning
high Extra Actions State! AlAikwo
4000
5.000
2006
2000
1000
www.Beedroeeett.oooo Sirge
Texil Reoieemeee Strget 1804
(a) Learning Curves
(b) Learning Curves
(c) Learning Curves
Figure D.2: Learning curves in the MujoCo environments with 4e6 steps.
Termination of an episode is achieved based on the patient's 's mortality rate reaching the minimum (SOFA Score being 0) or the patient's mortality rate reaching the maximum (SOFA score being 24). We also observe that Weight-kg and momulatedbbalame have minimal influence on Sepsis; therefore, we treat them non-ossential state variables and exclude them when constructing the response ye for variable selection.
21

---

Published in Transactions on Machine Learning Research (7/2025)
Table C.2: List of state and action variables with their corresponding indices in the Treatment Allocation Environment for Sepsis Patients. Here, red indicates essential treatments, and light red represents non- essential treatments and ineffective treatments.
State Index
State Name
Action Index
Action Name
gender age
vaso dose
1
1
vaso dose
2
clixhauser
2
dose
3
admission
3
iv input
4
Weight kg
4
beta blocker
5
GCS
5
diuretic
6
HR
6
antihistamino
7
SysBP
proton pump inhibitor
8
MeanBP
8
statin
9
DiaBP
9
metformin
10
RR
10
calcium _aaanne__haee__eeee blocker
11
SpO2
11
antidepressant
12
Temp C
12
antipsychotic
13
FiO2
13
antacid
14
Potassium
14
levothyroxine
15
Sodium
15
NSAID
16
Chloride
16
laxative
17
Glucose
17
multivitamin
18
BUN
18
topical_ointment ointment
19
Creatinine
19
congh_suppressant
20
Magnesium
20
homeopathic remedy
21
Calcium
21
herbal susplemm_ppemeettt
22
Tonised Ca
22
eye_ _re_dross
23
CO2 mEqL
23
muscle anasleerreaxxnnt relaxant
24
SGOT
24
antihypertensive class B sleep aid nasal decongestant
25
SGPT
25
26
Total bili
26
27
Albumin
27
acne treatment antiemetic vitamin supplement
28
Hb
28
29
WBC count
29
30
Platelets count
31
PTT
32
PT
33
INR
34
Arterial pH
35
paO2
36
paCO2
37
Arterial BE
38
Arterial lactate
39
HС03
40
mechvent
41
Shock Index
42
PaO2 FiO2
43
cumulated balance balancee
44
SOFA
45
SIRS
22

---

Published in Transactions on Machine Learning Research (7/2025)
Ant-rrigh (leginning
(Bleginning
Hopper-high
(Beeg nning
Aetiasi betwork
Network 68"10
RG. Ninwert 66713
Art Kimwork
3000
KS tetwork 64122
A.tits Mitwalk G4112
0.00
Accarti Intwant e 2001200
2000
ABl Act Nitomet 23
AAllaaoo Metwercc 250-123
430
3020
1300
1003
-230
920
5.5
Teall inviement Stepe
Taral Eistrament Stegs Lotte
Toaal Endiromnn Steae (Len))
(a) Learning Curves
(b) Learning Curves
(c) Learning Curves
Figure D.3: Learning curves in the MujoCo environments with different network sizes.
D Supporting Analyses
We conducted additional experiments regarding the distribution of actions that were sampled over the training period. The new results are summarized in Figure D.1 Based on the results together with existing figures in the main text, we can conclude that there exist changes regarding the distribution of actions that are sampled during different periods of training, which could be a positive indicator of more focused and potentially more effective learning process.
Since we only use steps for PPO training, which might be able to converge in the end, we further add the steps to 4c6. The new results are summarized in Figure D.2 Based on the results together with cxisting figures in the main text, we can conclude that the benefit of the proposed framework is both sample efficiency and performance.
We also conducted additional experiments by increasing the network size. The new results under MujoCo with different network sizes are summarized in Figure D.3 where we can conclude that the network capacity has certain influence the performance of All Action. However, simply increasing the network capacity unnecessarily makes learning easier. In addition, the proposed method consistently performs better than All Action, no matter how we increase the network size, which indicates that the performance difference results from the redundancy of the action space. In addition, we admit there are other factors that affect how the agent learns with a larger action dimension, such as regularization techniques, albeit with limited influence compared with the redundancy of the action space.
E Dimensionality Reduction and Estimation Bias
A common concern in feature or action space reduction is the potential introduction of bias due to the restriction of the hypothesis space, particularly when informative variables are inadvertently excluded. In our setting, however, the redundant actions are conditionally independent of both the reward and the next state, and thus can be regarded as nuisance variables that do not contribute meaningful information to the decision-making process. When these actions are correctly identified and excluded, the resulting reduction in dimensionality does not increase bias in principle. In fact, including irrelevant or weakly related features may introduce additional bias and variance by complicating the model-fitting process. Moreover, from a thcoretical perspective, reducing the input dimensionality is beneficial for both bias and variance. According to established results in deep learning theory (Farrell et al. 2021), the error bound for value or policy estimation using multilayer perceptrons is on the order of O(n M/(++)) ). where denotes input dimensionality, n is the sample size, and B is the Hölder smoothness paramcter. Thus, selecting minimal set of sufficient. actions and removing redundant ones leads to improved sample efficiency and more accurate value function approximation.
23

---

Published in Transactions on Machine Learning Research (7/2025)
F
Extend to Correlated Actions
Now, assume the policy network is parameterized by multivariate Gaussian with covariance matrix
N (s(ss){(ss)
where i and are parameterized functions to output the mean and covariance matrix.
Since the actions are correlated, we couldn't mask the individual log iTe (a; s). To solve this problem, we can transform the non-selected actions be conditional independent first.
Given selected action set G, define selection vector m (m1,-.., mp) E {0, 1J,, where m, 1 if E õ and 0 otherwise. Then mask the covariance matrix as follows:
(ss)
if m, 1, and m, = 1
(s)
m = 0, or m. = 0
The masking only changes non-selected actions be independent and removes their influence on selected actions, which still keeps the covariance structure of selected actions. Then can easily mask the log density of non-selected actions.
G Machine Learning Algorithm for Calculating Importance Scores
The feature importance score Z, for the i-th outcome (the reward or the next state) and the j-th action is computed by fitting this outcome based on all inputs using a machine learning method. Specifically, this process fits a predictive model f(x) to estimate target variable y, where y corresponds to either the reward the next state, the feature vector comprises the current state, action, and knockoff copy, and the function f can be the function classes of LASSO, random forests, neural networks. The importance score of a specific variable corresponds to its estimated coefficient in LASSO, its feature importance score in random forests, or its weight/ /gaddient in neural networks. Separate models are constructed for each target outcome as detailed in Algorithm 2
The only requirement for the machine learning method is that it satisfies fairness constraint, ensuring that exchanging an original feature with its knockoff counterpart results solely in the corresponding exchange of the model's 's importance score for those features. This condition is typically met by standard tabular machine learning algorithms.
H Extension to Non-stationary Environments
In the context of our paper, stationarity refers the policy being fixed and the transition dynamics and reward function remaining unchanged. Hence, in the common batch update setting, the data collected between policy updates can be treated as stationary. Our method can accommodate different forms of non-stationarity.
When non-stationarity arises solely due to policy updates-whit the environment's dynamics and rewards remain unchanged- our method can effectively handle this scenario by conducting knockoff variable selection within cach batch. This approach boosts policy optimization by ensuring that action masking informed by relevant, stable data. To enhance robustness, we can also extend our selection procedure to operate every k batch, leading to more stable and adaptive masking throughout training.
In cases where non-stationarity stems from changes in the transition dynamics or reward function-whide is rare in simulated environments like MuJoCo but plausible in real-world scenarios tte problem becomes more challenging. Nonetheless, our method remains effective under the realistic assumption piecewise stationarity. By integrating change point detoction, we can segment the data into stationary intervals and apply feature selection within each segment. This ensures that the selected action subset remains relevant. and informative, thereby preserving and even enhancing policy optimization.
24

---

Published in Transactions on Machine Learning Research (7/2025)
Technical Proofs
1.1 Preliminary Results
Before we prove Theorem 1, we first provide a preliminary lemma of our procedure that can enable the flip-sign property of W-statistics. This property can be used to prove Theorem 1 when observations are independent. Now we focus on one data split Dz and assume the data are independent.
Lemma I.1. A and Sy are a action and a state matrix. For any subsel C (T,...,P), and Ag obatined by resampling, we have
(AAR, An) swap(s) Sa) - ([As, Sk),
where swap(S?) represents swapping the j-th entry of A and As for all j E Q.
The proof of Lemma L.1 is based on the property of constructed variables where As and Á have the same marginal distribution and the whole joint distribution is symmetrical terms of Aş and Å.
In the following, we show that the exchangeability holds jointly on actions, states, and rewards when swapping null variables.
Lemma I.2. Let Ho S (1.....p} be the indices of the null variables, for any subset Q C Ho
(IAs, Aul swap(() St. Yn) d ([As, An) St. Yu),
where Y& is response including the next state and reward.
Based on the exchangeability proved in Lemma Candès et al. (2018). We just need to extend the derivation by conditioning on S₁ and show equivalence by swapping action variables in Ω one by one. We omit further details of the proof. Lemma I.3.
Proof:
directly utilize the proof of Lemma 3.2 in
Ar], W: (IAs, ARk,,AA)saaappi Sk, Yx) = W; ([An,As] Sr. Yι)
[-,, +1, otherwise
iiER
Proof: We require the method for constructing W to satisfy a fairness requirement so that swapping two variables would have the only effect of swapping corresponding feature importance scores. The fairness constraint is satisfied with many general machine learning algorithuns, like LASSO and random forest. Once the fairness constraint satisfied, W will be anti-symmetric, and the equality above automatically holds.
Lemma L4. Assume the flip-coin property in Lemma L.S is satisfied, on dato Dk. the selection G3 obtained from applying knockoff method in Algorithm 2 controls modified FDR (mFDR), e.g.
mFDR (Gx)= aa
For statistics W calculated, we denote W Wswap(() be the W-statistics computed after the swap w.r.t. Ω C (1....,pl p}. Now consider a sign vector e E { independent of W = [W...., W where c; 1 for all non-null null state variables and IP (e; = 1) 1/2 are independent for all null state variables. Then for such denote Ω = {i :4 = =1,, which a subset of Hy by the assumption (and recall that Hy is the collection of null variables). By Lemma T.3 we know
Proof:
(W1 EI. Wp <p) W sWa)()))
For convenience, we also use to denole measurable mapping function from a data set to its W-statistics, i.., on [sk, šy, a ykl.
W h (Aa, St, Sk,Y Yz).
25

---

Published in Transactions on Machine Learning Research (7/2025)
Then we get:
W swap([)) = h ([As, An] Amaappp()) ,k, vx)
d h (|Ar, Sk. Ya) W,
where the second equality (in distribution) is due to Lemma T.2 and h is measurable. The rest of the proof will be the same as that for Theorems and 2 in Barber & Candès 2015
1.2
Proof of Theorem 4.4
Using Lemma 1.4
we can show that the data points in D₃ are independent, then mFDR can be controlled.
Now we want to weaken the independence assumption to stationarity and exponential 3-mixing assumption
in 4.3 Based on Lemma [.4 the following proof is essentially the same as Theorem in Ma el al. (023). We
will omit those steps for brevity.
26


Reviews:
[{"review_id": "jOXeORuU0u", "reviewer": "Reviewer_wPAB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This is an import challenge in RL to develop practical techniques that can efficiency handle high-dimensional actions.\n- The paper is very well-written and clear.\n- The paper makes multiple contributions, including formulating sufficient and minimial sufficient actions sets and a knockoff sampling techniques for online setting and continuous action spaces, which is practical.\n- It’s nice to see experiments on both (though semi-synthetic) MuJoCo tasks and on clinical data. It’s interesting that even in some cases, the KS approach nearly matches PPO with true actions.\n\nWeaknesses\n- In experiments, high-dimensional action space is artificially created by adding extra actions, which may not capture the complexities of real scenarios.\n- MoJuCo experiments only compare KS with True Actions and All actions, and comparison with other methods are not presented. For example, how does a simple low-rank approximation of actions perform? What about the approach of Zahavy et al. 2018 by discreteizing the action space? Additionally, are there any action limitation benefits to bonus-based exploration methods that leverage function approximation such as RND?\n- Theoretical contribution is limited and appear to be straightforward extension of Ma et al. 2023. It would be good if the authors clarify differences with Ma et al. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on RL in the setting of high-dimension actions, where the straightforward application of existing exploration methods requires significant computation. The existing works reduce the action space through eliminating action space redundancy using domain expertise or require action spaces to be discrete and suffer from high computational cost. The paper proposes a new data-driven strategy for reducing the number of action selections through knockoff sampling. The paper provides theoretical guarantees and empirical validation of the approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the comments and questions in the weakness above. Additional questions:\n\n- Could you please explain the challenges involved in extending the knockoff method of Ma et al. 2023 from offline and high-dimensional state space setting to online and high-dimensional action space setting, from both theoretical analysis and algorithm design standpoints? \n- Compared to the works Zahavy et al. 2018 and Zhong et al. 2024, could you please explain how your approach is superior or different, considering the naive extension of these works by discretizing the action space?\n- Is it possible to extend the definition of minimal action set to only hold approximately? In that case, how are the theoretical results impacted?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JUqZFEr100", "reviewer": "Reviewer_1D57", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Handling high dimensional action spaces is a less studied but potentially important direction. I can see one example is the treatment allocation environment given by the authors where the agent is required to shrink the enormous amount of possible treatment combinations to a meaningful subset. The paper is well-organized in detailing the relevant techniques borrowed from related literature and how these tools should be properly handled when dealing with non-iid reinforcement learning data. \n\nWhile the paper is interesting, I believe there is room for making the paper more convincing. My suggestions are mainly the following two points:\n - **methodologically**, Eq. 4 looks quite restricting by transforming the log-likelihood into a binary vector.  The authors' claim that \n> This masking of the log probability helps mitigate the likelihood of encountering extremely high entropy values, thereby facilitating a more stable and efficient training process. \n\nis not convincing and seems to bias towards only the good side. When we take the gradient of log-likelihood for updating the policy, the information carried by $\\nabla \\log \\pi(a|s)$ is lost by this binary vector. Careful expositions are needed here to clearly state why the authors did not use $\\odot$ like Eq. 3 but rather dot product. More importantly, the authors should explain, or at least conjecture why the current design leads to good performance. On this matter, perhaps the authors can link it to the sparse filtering log-likelihood update, see refs [1,2]. Specifically, we can consider the following more general policy learning objective $\\mathbb{E}_{s,a \\sim \\mathcal{D}}[ w(s,a) \\log \\pi(a|s) ] $.\nSparse filtering log-likelihood update designates $w(s,a)$ to be 0 for certain actions (in this context, the redundant actions ). \n\n - **empirically**, I think the paper would benefit greatly from emphasizing more on the simulated treatment environment that runs on the MIMIC-III dataset. Artificial action dimensions for MuJoCo serve as a proof-of-concept example and are less interesting. I would encourage the authors to first show results on this environment, and provide more details including how states, actions and rewards are designed; how high dimensional action spaces emerge naturally from the combination of possible treatments; and learning curves.\n\nReferences: \\\n[1] Offline RL with no ood actions: in-sample learning via implicit value regularization\\\n[2] Offline RL via Tsallis Regularization", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced a new framework for RL with high dimensional action spaces by combining several techniques from other research areas like model-X knockoff and sample splitting. The paper is in general well-written and sufficiently motivated, with experimental results backing the proposed action selection method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the my two points above.", "broader_impact_concerns": "not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vUAbqXC2fS", "reviewer": "Reviewer_3oS8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: The use of variable selection for identifying a minimal\nset of actions is interesting, and in particular, the application of\nknockoff sampling for this purpose appears to be novel and non-trivial.\nThe empirical results of the proposed method are reasonably good,\nparticularly I found the results of Figure 4 to be interesting.\n\n**Weaknesses**: There are a few major weaknesses, in my opinion, which I\nlist below. Further details are given in the Requested Changes.\n\n1.  The background on variable selection and knockoff sampling is\n    extremely difficult to follow—I could not make any sense of it.\n    Clarity of this section absolutely must be improved, since it plays\n    a large role in the paper. One consequence, besides not having an\n    intuitive understanding of the proposed algorithm, is that Theorem\n    4.3 (the main theorem in the paper) is extemely difficult to\n    comprehend and appreciate.\n2.  The motivation of the problem was fairly weak, in my opinion. It was\n    not clear to me until the algorithmic section that the goal was not\n    to reduce the action space for the sake of, say, enabling a\n    tractable class of policies. Indeed, from the perspective of the\n    neural net architecture, the size of the action space doesn't\n    change. This paper focuses entirely on the problem of identifying\n    redundant actions to effectively improve sample efficiency. While I\n    understand that this can lead to significantly accelerated training,\n    I don't really know of any domains where there are so many redundant\n    action features (the synthetic MuJoCo domains don't count). The\n    authors highlight medical domains, but this is not discussed much in\n    the experimental sections, and few concrete examples are given.\n3.  Similarly to the previous point, the experiments could have been\n    more convincing. While I liked Figure 4 as it shows basically how\n    artificially adding redundant actions makes MuJoCo more difficult, I\n    would have loved to see this figure for a domain that is more real.\n    Are there every enough redundant actions that you can see a dramatic\n    change like this? Where do these redundant actions arise from? Why\n    couldn't we see a similar plot, for example, in the sepsis treatment\n    domain?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the issues resulting from large action spaces with\nredundant components in reinforcement learning. The authors argue\nnaively applying RL methods in these settings leads to needless increase\nin sample complexity, and they propose methods, using tools such as\n*knockoff sampling* from the variable selection literature, for\nautomatically discovering these redundant action components so they can\nbe pruned. Results show that their method successfully identifies\nredundant action components, leading to sample complexity rivalling\nequivalent systems with a minimum viable action space.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.  There is no discussion about what \"false discovery rate\" (FDR)\n    means. I don't know what this means, so I am having a hard time\n    following the paper.\n2.  Section 2.3 requires much more detail. For example,\n    1.  What is a \"response vector\"? Are these essentially labels for a\n        supervised learning problem? Perhaps this would simply be\n        cleared up again with a description of FDR / the setting you\n        consider in this section.\n    2.  Equation (1) says $(\\pmb{X},\n                \\widetilde{\\pmb{X}})_{\\mathrm{swap}(\\Omega)}\\overset{d}{=}(\\pmb{X},\n                \\widetilde{\\pmb{X}})$, but in your definitions of these\n        variables/operations, it is entirely unclear to me where there\n        is any randomness.\n    3.  How do you calculate feature importance scores?\n    4.  It says \"set $W_j = f(Z_j, \\widetilde{Z}_j)$ in such a way that\n        higher values of $W_j$ indicate stronger evidence of $\\pmb{x}_j$\n        being influential covariate\". Isn't this entirely dictated by\n        the choice of $f$? How should one choose $f$? How do you know\n        chich covariates are likely to be influential so as to design\n        $f$ to satisfy this condition?\n    5.  Again, it would be very helpful if the final result (e.g.\n        equation (2)) could be contextualized. What is accomplished with\n        this filtering rule?\n3.  Under equation (3), it says \"action selection reduces the\n    dimensionality of the input action space, there by reducing bias in\n    $Q$ function fitting\". This warrants some more discussion,\n    intuitively I would expect the opposite, since you are effectively\n    restricting the hypothesis class over $Q$ functions to a smaller one\n    (more bias).\n4.  In equation (4), essentially you're suggesting to stop policy\n    gradients through action dimensions that are redundant. Why is this\n    the correct strategy? How does this allow the policy to ignore those\n    dimensions (e.g., perform \"action selection\")? Why don't you\n    additionally act according to $\\mathbf{m}\\odot\\mathbf{a}$ instead of\n    $\\mathbf{a}$ sampled under the policy (or alternatively, is there any way to concretely use the learned mask to enforce that the policy acting in a manner consistent with the unmasked actions only)?\n5.  In definition 4.1, there appears to be a missing word or something\n    at the beginning of the definition. There appears to be a sentence\n    starting with \"property on the augmented data matrix \\[…\\]\".\n6.  In Theorem 4.3, what is \"modified FDR\"? What is $\\alpha$?\n7.  There is a notable lack of discussion regarding Theorem 4.3.\n    Probably exacerbated by the difficulty I had following the\n    background section, it is very hard to intuit the consequences of\n    this theorem. It should be clearly explained how this bound applies\n    to RL / policies, and why we want systems under which this upper\n    bound is minimized.\n8.  In the \"Semi-synthetic MuJoCo Environments\", what does it mean to\n    \"add extra $p$ actions to the raw action space\"? What are these\n    action dimensions? Are they just ignored by the simulator?", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)