Title: Toward Causal-Aware RL: State-Wise Action-Refined Temporal Difference
Venue: NeurIPS 2023
Accepted: no
Keywords: Action Space Refinery
Abstract: Although it is well known that exploration plays a key role in Reinforcement Learning (RL), prevailing exploration strategies for continuous control tasks in RL are mainly based on naive isotropic Gaussian noise regardless of the causality relationship between action space and the task and consider all dimensions of actions equally important. In this work, we propose to conduct interventions on the primal action space to discover the causal relationship between the action space and the task reward. We propose the method of State-Wise Action Refined (SWAR), which addresses the issue of action space redundancy and promote causality discovery in RL. We formulate causality discovery in RL tasks as a state-dependent action space selection problem and propose two practical algorithms as solutions. The first approach, TD-SWAR, detects task-related actions during temporal difference learning, while the second approach, Dyn-SWAR, reveals important actions through dynamic model prediction. Empirically, both methods provide approaches to understand the decisions made by RL agents and improve learning efficiency in action-redundant tasks. 

Abstract

Although it is well known that exploration plays a key role in Reinforcement Learning (RL), prevailing exploration strategies for continuous control tasks in RL are mainly based on naive isotropic Gaussian noise regardless of the causality relationship between action space and the task and consider all dimensions of actions equally important. In this work, we propose to conduct interventions on the primal action space to discover the causal relationship between the action space and the task reward. We propose the method of State-Wise Action Refined (SWAR), which addresses the issue of action space redundancy and promote causality discovery in RL. We formulate causality discovery in RL tasks as a statedependent action space selection problem and propose two practical algorithms as solutions. The first approach, TD-SWAR, detects task-related actions during temporal difference learning, while the second approach, Dyn-SWAR, reveals important actions through dynamic model prediction. Empirically, both methods provide approaches to understand the decisions made by RL agents and improve learning efficiency in action-redundant tasks.

1 Introduction

Although model-free RL has achieved great success in various challenging tasks and outperforms experts in most cases [21, 26, 17, 34, 4], the design of action space always requires elaboration. For example, in the game StarCraftII, hundreds of units can be selected and controlled to perform various actions. To tackle the difficulty in exploration caused by the extremely large action and state space, hierarchical action space design and imitation learning are used [27, 34] to reduce the exploration space. Both of those approaches require expert knowledge of the task. On the other hand, even in the context of imitation learning where expert data is assumed to be accessible, causal confusion will still hinder the performance of an agent [8]. Those defects motivate us to explore the causality-awareness of an agent that permits an agent to discover the causal relationship for the environment and select useful dimensions of action space during policy learning in pursuance of improved learning efficiency. Another motivating example is the in-hand manipulation tasks [2]: robotics equipped with touch sensors outperforms the policies learned without sensors by a clear margin in hand-in manipulation tasks [20], showing the importance of causality discovery between actions and feedbacks in RL. A similar example can be found in human learning: knowing nothing about how to control the finger joints flexibly may not hinder a baby learns to walk, and a baby has not learned how to walk can still learn to use forks and spoons skillfully, inspiring us to believe that the challenge for exploration can be greatly eased after the causality between action space and the given task is learned.

In this work, the recent advance of instance-wise feature selection technique [38] is improved to be 34 more suitable in large-scale state-wise action selection tasks and adapted to the time-series causal 35 discovery setting to select state-conditioned action space in RL with redundant action space. With the 36

CriticNetwork

Element-Wise

Product

→0

0

TD-Error

Action

a2

a

Selection

a3

Probability

Mask

P1

0

Back-Propagation

a3

SelectorNetwork

P2

1

0

Loss

P3

1

Policy Gradient

Difference

State

ad

S

Back-Propagation

Pd

0

BaselineNetwork

TD-Error

(s,a,r,s') ～ Replay Buffer

a'～Policy

r +·Baseline(s',a')

proposed method, the agent learns to perform intervention, discover the true structural causal model 37

(SCM) and select task-related actions for a given task, remarkably reduces the burden of exploration 38

and obtains on-par learning efficiency as well as asymptotic performance compared with agents 39

trained in the oracle settings where the action spaces are pruned according to given tasks manually. 40

2 Preliminary 41

Markov Decision Processes RL tasks can be formally defined as Markov Decision Processes 42

(MDPs), where an agent interacts with the environment and learns to make decision at every timestep. 43

Formally, we consider the deterministic MDP with a fixed horizon H ∈ N + denoted by a tuple 44

r : S × A ↦→ R denotes the reward function; γ ∈ (0 , 1] is the discount factor indicating importance 46

( S , A , H, r, γ , T , ρ 0 ) , where S and A are the |S| -dimensional state and |A| -dimensional action space; 45

of present returns compared with long-term returns; T : S × A ↦→ S denotes the transition dynamics; 47

ρ 0 is the initial state distribution. 48

We use Π to represent the stationary deterministic policy class, i.e., Π = { π : S ↦→ A} . The 49

problem: max π ∈ Π E τ ∼ ρ 0 , π , T [ ∑ H t =1 γ t r t ] where the expectation is taken over the trajectory τ = 51 ( s 1 , a 1 , r 1 , . . . , s H , a H , r H ) generated by policy π under the environment T , starting from s 0 ∼ ρ 0 . 52

learning objective of an RL algorithm is to find π ∗ ∈ Π as the solution of the following optimization 50

INVASE INVASE is proposed by [38] to perform instance-wise feature selection to reduce over53 fitting in predictive models. The learning objective is to minimize the KL-Divergence of the full54 conditional distribution and the minimal-selected-features-only conditional distribution of the out55 come, i.e., min F L , with 56

where F : X → { 0 , 1 } d is a feature selection function and | F ( x ) | 0 denotes the cardinality ( l 0 57 norm) of selected features, i.e., the number of 1 's in F ( x ) . 1 d is the dimension of input features. 58

59

60

61

62

63

64

65

66

67

68

x ( F ( x )) = F ( x ) /circledot x denotes the element-wise product of x and generated mask m = F ( x ) . Ideally, the optimal selection function F should be able to minimize the two terms in Equation (1) simultaneously.

INVASE applies the Actor-Critic framework in the optimization of F through sampling, where f θ ( ·| x ) , parameterized by a neural network θ 2 , is used as a stochastic actor. Two predictive networks C φ ( · ) , B ψ ( · ) are considered as the critic and the baseline network used for variance reduction [36] and trained with the Cross-Entropy loss to produce return signal L , based on which f θ ( ·| x ) can be optimized through policy gradient:

3 Proposed Method

The objective of this work is to carry out state-wise action selection in RL through intervention, 69 and thereby enhance the learning efficiency with a pruned task-related action space after finding the 70 correct causal model. Section 3.1 starts with the formalization of the action space refinery objective 71 in RL tasks under the framework of causal discovery. Section 3.2 introduces SWAR, which improves 72 the scalability of INV ASE in high dimensional variable selection tasks. We integrate SWAR with 73 deterministic policy gradient methods [25] in Section 3.3 to perform state-wise action space pruning, 74 resulting in two practical causality-aware RL algorithms. 75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

3.1 Temporal Difference Objective with Structural Causal Models

In modern RL algorithms, the most general approach is based on the Actor-Critic framework [15], where the critic Q w ( s, a ) approximates the return of given state-action pair ( s, a ) and guides the Actor to maximize the approximated return at state s . The Critic is optimized to reduce Temporal Difference (TD) error [29], defined as

where B = ( s i , a i , r i , s ′ i ) i =1 , 2 ,... is the replay buffer used for off-policy learning [17, 10, 12, 28], and a ′ i = π ( s ′ i ) is the predicted action for state s ′ i . In practice, the calculations of Q w ( s ′ i , a ′ i ) are usually based on another set of slowly updated target networks for stability [10, 12]. Henceforth, TD-learning can be roughly simplified as regression with notion y i = r i + γ Q w ( s ′ i , a ′ i ) :

Assume there are only M < L actions are related to a specific task among the L -dimensional actions a i = a (1) i , ..., a ( L ) i , i.e., Q w ( · , · ) is function of s i , a (1) i , ..., a ( M ) i . Learning with the primal redundant action space will lead to around L + |S| M + |S| times sample complexity [9, 39]. Therefore, we are motivated to improve the learning efficiency of Q by pruning those task-irrelevant action dimensions a ( M +1) i , ..., a ( L ) i by finding an action selection function G , satisfying

Such a problem can be addressed from the perspective of causal discovery. Formally, we can use 91 the Structural Causal Models (SCMs) to represent the underlying causal structure of a sequential 92 decision making process, as shown in Figure 2. Under this language, we use the notion of causal 93 actions to denote a (1 ,...,M ) i , and nuisance actions for other dimension of actions. In our work, we use 94 IC-INVASE for causal discovery. Ideally, the action selection function G should be able to distinguish 95 between nuisance action dimensions and the causal ones that has causal relation with either dynamics 96 or reward mechanism. We present in the next section our causal discovery algorithms. 97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

a

a3

a4

I

a5

1

a6

a

1

a3

a4

a5

1

a6

Dyn

R

3.2 Iterative Curriculum INVASE (IC-INVASE)

Instead of directly applying INVASE to solve Equation (6). We first propose two improvements to make the vanilla INVASE more suitable for large-scale variable selection tasks as the action dimension in RL might be extremely large [34]. Specifically, the first improvement, based on curriculum learning, is introduced to tackle the exploration difficulty when λ in Equation (1) is large, where INVASE tends to converge to poor sub-optimal solutions and prune all variables including the useful ones [38]. The second improvement is based on the iterative structure of variable selection tasks: the feature selection operator G can be applied multiple times to conduct hierarchical feature selection without introducing extra computation expenses.

3.2.1 Curriculum Learning For High Dimensional Variable Selection

The work of [3] first introduces Curriculum Learning to mimic human learning by gradually learn more complex concepts or handle more difficult tasks. Effectiveness of the method has been demonstrated in various set-ups [3, 19, 7, 35, 37]. In general, it should be easier to select M useful variables out of L input variables when M is larger. The most trivial case is to select all L variables, with an identical mapping x ( G ( x )) = G ( x ) /circledot x = x . Formally, we have

Proof. By the definition of the [ · ] 1 N 2 -N 1 operator, || G ( x ) | 0 -N 2 | = 0 is minimized. On the other hand, starting from N 1 = M , minimizing D KL ( p ( Y | X = x ) || p ( Y | X ( G ( x )) = x ( G ( x )) )) requires all the M outcome-related variables being selected by G N 1 . Therefore, G N 2 also minimizes the KL-divergence by the independent assumption of the other L -M variables with the outcomes.

The proposition indicates the difficulty of selecting N useful out of L variables decreases monotonically as N ≥ M increase from M,M +1 , ..., L . In this work, two classes of practical curriculum are designed: 1. curriculum on the l 0 penalty coefficient, and 2. curriculum on the proportion of variables to be selected.

Curriculum on l 0 Penalty Coefficient In this curriculum design, the penalty coefficient λ in Equation (1) is increased from 0 to a pre-determined number (e.g., 1 . 0 ). Increasing the value of λ will lead to a larger penalty on the number of variables selected by the feature selector. Experiments in [38] has shown a large λ always lead to a trivial selector that does not select any variable.

Curriculum on the Proportion of Selected Features In this curriculum design, the proportion of variables to be selected, denoted by p r , is adjusted from the default setting 0 to a decreasing number

134

135

136

137

138

139

140

141

142

143

144

145

146

Algorithm 1 TD3 with TD-SWAR

Initialize critic networks C φ 1 , C φ 2 , baseline networks B ψ 1 , B ψ 2 and actor network π ν , IC-INVASE selector network G θ

Initialize target networks φ ′ 1 ← φ 1 , φ ′ 2 ← φ 2 , ψ ′ 1 ← ψ 1 , ψ ′ 2 ← ψ 2 , ν ′ ← ν Initialize replay buffer B

is sampled from a clipped Gaussian.

Interact with environment and store transition tuple ( s, a, r, s ′ ) in B Sample mini-batch of transitions { ( s, a, r, s ′ ) } from B Calculate perturbed next action by ˜ a ← π ν ′ ( s ′ ) + /epsilon1 , /epsilon1 Select actions with target selector network

Calculate target critic value y c and baseline value y b :

i Update critics and baselines with selected actions:

Update IC-INVASE selector network by the policy gradient, with learning rate η 1 :

losses in the previous step.

Update ν by the deterministic policy gradient, with learning rate η 2 :

)

a

=

π

(

s

)

ν

π

ν

ν ← ν -η 2 ∇ a C φ 1 ( s, a | ν ∇ Update target networks, with τ ∈ (0 , 1) :

end for

from a pre-determined value (e.g., 0 . 5 ) to 0 . i.e., the l 0 penalty term λ | G ( x ) | 0 in Equation (1) is revised to be λ || G ( x ) | 0 -d · p r | , where d is the dimension of input x . When the proportion is set to be p r = 0 . 5 , the selector will be penalized whenever less or more than half of all variables are selected. Such a curriculum design forces the feature selector to learn to select less but increasingly more important variables gradually.

Thus, we get the learning objective of curriculum-INVASE:

where λ increases from 0 to some value and p r decreases from a value in [0 , 1) to 0 .

3.2.2 Iterative Variable Selection

The second improvement proposed in this work is based on the iterative structure of variable selection tasks. Specifically, the G ( x ) mapping x ∈ X to { 0 , 1 } d is an iterative operator, which can be applied for multiple times to perform coarse-to-fine variable selection. Although in practice we follow [38] to apply an element-wise product in producing x ( G ( x )) : x ( G ( x )) = G ( x ) /circledot x ∈ X . In more general cases, the i-th element of x ( G ( x )) i is

where ∗ can be an arbitrary identifiable indicator that represents the variable is not selected. 147

On the other hand, once the outputs G ( x ) of the selector have been recorded, ∗ can be replaced by 148 any label-independent variable G ( x ) /circledot z , where z ∼ p z ( · ) is outcome-independent. Then x ( G ( x )) 149 can be regarded as a new sample and be fed into the variable selector, resulting in a hierarchical 150

(

s

)

variable selection process: 151

where z ∼ p z ( · ) , and ⊕ is the element-wise sum operator. Moreover, if the distribution of irrelevant variable p x ( · ) is known, applying the variable selection operator obtained from Equation (7) for multiple times with p z ( · ) d = p x ( · ) has the meaning of hierarchical variable selection: after each operation, the most obvious 1 -p r irrelevant variables are discarded. e.g., when p r = 0 . 5 , ideally top50% , 25% , 12 . 5% most important variables will be selected after the first three selection operations. In this work, a coarse approximation is utilized by selecting z to be z = 0 for simplicity. 3

Combining those two improvements lead to an Iterative Curriculum version of INVASE (IC-INVASE) that addresses the exploration difficulty in high-dimensional variable selection tasks. Curriculum learning helps IC-INVASE to achieve better asymptotic performance, i.e., achieve higher True Positive Rate (TPR) and lower False Discovery Rate (FDR), while iterative application of the selection operator contributes to higher learning efficiency: selectors models with different level of TPR/FDR can be generated on-the-fly.

3.3 State-Wise Action Refinery with IC-INVASE

3.3.1 Temporal Difference State-Wise Action Refinery

With the techniques introduced in the previous section, higher dimensional variable selection tasks can be better solved, therefore we are ready to use IC-INV ASE to solve Equation (6). The resulting algorithm is called Temporal Difference State-Wise Action Refinery (TD-SWAR).

In this work, TD3 [10] is used as the basic algorithm we build TD-SWAR up on. In addition to the policy network π ν , double critic networks C φ 1 , C φ 2 and their corresponding target networks used in vanilla TD3, TD-SWAR includes an action selector model G θ and two baseline networks B ψ 1 , B ψ 2 following [38] to reduce the variance in policy gradient learning. Pseudo-code for the proposed algorithm is shown in Algorithm 1. And the block diagram in Figure 1 illustrates how different modules in TD-SWAR updates their parameters.

3.3.2 Static Approximation: Model-Based Action Selection

While IC-INVASE can be formally integrated with temporal difference learning, the learning stability is not guaranteed. Different from general regression tasks where the label for every instance is fixed across training, in temporal difference learning, the regression target is closely related to the present critic function C φ , the policy π ν that generates the transition tuples used for training, and the selector model of IC-INVASE itself. In this section, a static approach is proposed to approximately solve the challenge of instability in TD-SWAR 4 .

Other than applying the IC-INVASE algorithm to solve Equation (6), another way of leveraging IC-INVASE in action space pruning is to combine it with the model-based methods [11, 16, 13, 14], where a dynamic model P : S × A ↦→ S is learned through regression:

Although the task of precise model-based prediction is in general challenging [24], in this work, we 185 only adopt model-based prediction in action selection, and the target is action discovery other than 186 precise prediction. As the dynamic models are always static across learning, such an approach can be 187 much more stable than TD-SWAR. We name this method as Dyn-SWAR and present the pseudo-code 188 in Algorithm 2, where we infuse IC-INVASE to Equation (10) and get the learning objective: 189

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

190

191

192

193

194

195

196

197

198

199

200

201

202

203

Algorithm 2 TD3 with Dyn-SWAR

+10

+10

+10

+10

(a) 4Rew.-Maze

(b) Pendulum

(c) Walker2d

(d) LunarLander

(e) BipedalWalker

4 Experiment

In this section, we apply our proposed methodologies to five continuous control RL tasks characterized by redundant action spaces, wherein our methods facilitate causality-aware RL. We also present a quantitative comparison between IC-INVASE and the standard INVASE on synthetic datasets in Appendix B, which serves to underscore the enhanced scalability of our approach.

In the present set of experiments, we employed five RL environments (Figure 5), detailed in Table 1 5 . The symbol |S| designates the dimension of the state space for each task, while |A| signifies the dimension of the action space relevant to the task, and |A red. | represents the dimension of the redundant action space incorporated into each task. These surplus dimensions of actions don't impact state transitions or reward calculations, but it is essential for an agent to identify these redundant dimensions for efficient learning.

We assessed both TD-SWAR, which combines IC-INVASE with temporal difference learning, and its static counterpart, Dyn-SWAR, which employs IC-INVASE in dynamics prediction. The results are benchmarked against two base conditions: the Oracle , where redundant action dimensions are

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

(a) FourRewardMaze

(b) Pendulum

(c) Walker2d

(d) LunarLander

(e) BipedalWalker

manually removed; and TD3 , which is the standard TD3 algorithm devoid of any explicit action redundancy reduction.

In our experimental findings, we observed that Dyn-SWAR's deployment demonstrates superior efficiency with respect to both sample complexity and computational cost. In contrast, TD-SWAR requires a persistent update of all parameters for the IC-INV ASE selector to maintain congruence with the real-time policy and value networks, given the fluctuating regression label over time. However, the Dyn-SWAR selector necessitates a significantly reduced data set for training, specifically between 10,000 and 25,000 timesteps of environmental interaction. This attribute can seamlessly integrate with the warm-up technique utilized in TD3 [10]. Namely, the Dyn-SWAR selector could be trained with warm-up transition tuples gathered during the random exploration phase, and then remain static throughout the subsequent learning process. Compared to traditional RL configurations that generally require millions of environmental interactions, the training of Dyn-SWAR incurs only a minuscule computational cost.

These findings are illustrated in Figure 4. Across all environments, agent learning with IC-INV ASE in both TD- and Dyn- methods exceeds the performance of the standard TD3 baseline. Dyn-SWAR achieves a learning efficiency that is on par with oracle benchmarks. However, the performance of TD-SWAR in tasks of higher dimensions (Walker2d-v2 and BipedalWalker-v3) indicates significant potential for enhancement. Accordingly, future work should prioritize enhancing the stability and scalability of instance-wise variable selection within temporal difference learning.

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

5 Related Work

Instance-Wise Feature Selection While traditional feature selection method like LASSO [31] aims at finding globally important features across the whole dataset, instance-wise feature selection try to discover the feature-label dependency on a case-by-case basis. L2X [5] performs instancewise feature selection through mutual information maximization with the technique of Gumbel softmax. L2X requires pre-determined hyper-parameter k to indicate how many features should be selected for each instance, which limits its performance while the number of label-relevant features varies across instances. In this work, we build our instance-wise action selection model on top of INVASE [38], where policy gradient is applied to replace the Gumbel softmax trick and the size of chosen features per instance is more flexible. [32] considers instance-wise feature selection problems in time-series setting, and build generative models to capture counterfactual effects in time series data. Their work enables evaluation of the importance of features over time, which is crucial in the context of healthcare. [18] formally defines different types of feature redundancy and leverages mutual information maximization in instance-wise feature group discovery and introduces theoretical guidance to find the optimal number of different groups.

Our work is distinguished from previous works for instance-wise feature selection in two aspects. First, while previous works focus on static scenarios like classification and regression, this work focus on temporal difference learning where there is no static label. Second, the scalability of previous methods in variable selection is challenged as there might exist hundreds of redundant actions in the context of RL.

Dimension Reduction in RL In the context of RL, attention models [33] have been applied to interpret the behaviors of learned policies. [30] proposes to perceive the state information through a self-attention bottleneck in vision-based RL tasks, which concentrates on the state space redundancy reduction with image inputs. The work of [22] also applies the attention mechanism to learn taskrelevant information. The proposed method achieves state-of-the-art performance on Atari games with image input while being more understandable with top-down attention models.

Different from those papers, this work considers relatively tight state representations (vector input), and focuses on the task-irrelevant action reduction. We aim at finding the task-related actions and improving the learning efficiency without wasting samples in learning the task-irrelevant dimensions of actions. Our work is most closely related to AE-DQN [39] in that we both consider the problem of redundant action elimination. AE-DQN tackles action space redundancy with an action-elimination network that eliminates sub-optimal actions. Yet its discussion is limited in the discrete settings. In contrast, our work focuses on action elimination in continuous control tasks.

6 Conclusion and Future Work

In this study, we address the issue of pruning the action space in action redundant RL tasks. We employ the recent advancements in instance-wise feature selection technology (INV ASE), incorporating both curriculum learning and iterative processes, to aim for improved scalability and efficiency. This leads to the creation of the IC-INV ASE method, which is then adapted to the RL environment where we introduce two novel algorithms, TD-SWAR and Dyn-SWAR, to implement causality-conscious RL. The former algorithm directly addresses the issue of action redundancy in temporal difference learning, whereas the latter algorithm leverages model-based prediction to capture dynamic causality. Experimental evidence from a range of tasks underscores the importance of causality-awareness for RL agents to achieve efficient learning in action-redundant settings.

As for future research, the iterative characteristic of this method could be further investigated to apply ensemble methods in variable selection. Additionally, the design of a more appropriate curriculum could enhance the fusion of multiple curricula. From the RL perspective, the stability of TD-SWAR could be further optimized to enhance sample efficiency. The design of the curriculum could potentially offer benefits. For instance, an agent might initially learn to identify actions of general importance before concentrating on discerning state-dependent crucial actions. Furthermore, the selection process can be extended to include both the state space and action space, allowing for efficient temporal difference learning that is mindful of the causal relationships among states, actions, and the task at hand. Additionally, model-based prediction could be broadened to anticipate future returns.

References 276

Martín Abadi, Paul Barham, Jianmin Chen, et al. Tensorflow: A system for large-scale machine 277 learning. In OSDI , 2016. 278

OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, et al. Learning dexterous 279 in-hand manipulation. IJRR , 2020. 280

Yoshua Bengio, Jérôme Louradour, et al. Curriculum learning. In ICML , 2009. 281

Christopher Berner, Greg Brockman, Brooke Chan, et al. Dota 2 with large scale deep reinforce282 ment learning. arXiv:1912.06680 , 2019. 283

Jianbo Chen, Le Song, et al. Learning to explain: An information-theoretic perspective on model 284 interpretation. arXiv:1802.07814 , 2018. 285

François Chollet et al. Keras. https://github.com/fchollet/keras , 2015. 286

Wojciech Marian Czarnecki, Siddhant M Jayakumar, Max Jaderberg, et al. Mix&match-agent 287 curricula for reinforcement learning. arXiv:1806.01780 , 2018. 288

Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. In 289 NeurIPS , pages 11698-11709, 2019. 290

Eyal Even-Dar, Shie M., et al. Action elimination and stopping conditions for the multi-armed 291 bandit and reinforcement learning problems. JMLR , 2006. 292

Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error 293 in actor-critic methods. arXiv:1802.09477 , 2018. 294

David Ha and Jürgen Schmidhuber. World models. arXiv:1803.10122 , 2018. 295

Tuomas Haarnoja, Aurick Zhou, et al. Soft actor-critic: Off-policy maximum entropy deep 296 reinforcement learning with a stochastic actor. arXiv:1801.01290 , 2018. 297

Danijar Hafner, Timothy Lillicrap, Jimmy Ba, et al. Dream to control: Learning behaviors by 298 latent imagination. arXiv:1912.01603 , 2019. 299

Michael Janner, Justin Fu, Marvin Zhang, et al. When to trust your model: Model-based policy 300 optimization. In NeurIPS , 2019. 301

Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In NeurIPS , 2000. 302

Eric Langlois, Shunshi Zhang, Guodong Zhang, et al. Benchmarking model-based reinforcement 303 learning. arXiv:1907.02057 , 2019. 304

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, et al. Continuous control with deep 305 reinforcement learning. arXiv:1509.02971 , 2015. 306

Aria Masoomi, Chieh Wu, Tingting Zhao, et al. Instance-wise feature grouping. NeurIPS , 2020. 307

Tambet Matiisen, Avital Oliver, et al. Teacher-student curriculum learning. TNNLS , 2019. 308

Andrew Melnik, Luca Lach, Matthias Plappert, et al. Tactile sensing and deep reinforcement 309 learning for in-hand manipulation tasks. 310

311

312

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep reinforcement learning. Nature , 2015.

Alexander Mott, Daniel Zoran, et al. Towards interpretable reinforcement learning using 313 attention augmented agents. In NeurIPS , 2019. 314

Adam Paszke, Sam Gross, Soumith Chintala, et al. Automatic differentiation in pytorch. 2017. 315

Archit Sharma, Shixiang Gu, Sergey Levine, et al. Dynamics-aware unsupervised discovery of 316 skills. arXiv:1907.01657 , 2019. 317

David Silver, Guy Lever, et al. Deterministic policy gradient algorithms. In ICML , 2014. 318

David Silver, Aja Huang, Chris J Maddison, et al. Mastering the game of go with deep neural 319 networks and tree search. nature , 2016. 320

Peng Sun, Xinghai Sun, Lei Han, et al. Tstarbots: Defeating the cheating level builtin ai in 321 starcraft ii in the full game. arXiv:1809.07193 , 2018. 322

Hao Sun, Ziping Xu, Yuhang Song, Meng Fang, Jiechao Xiong, Bo Dai, and Bolei Zhou. 323 Zeroth-order supervised policy improvement. arXiv preprint arXiv:2006.06600 , 2020. 324

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 1998. 325

326

327

Yujin Tang, Duong Nguyen, and David Ha. Neuroevolution of self-interpretable agents. arXiv:2003.08165 , 2020.

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

Robert Tibshirani. Regression shrinkage and selection via the lasso. JRSS , 1996.

Sana Tonekaboni, S. Joshi, et al. What went wrong and when? instance-wise feature importance for time-series black-box models. NeurIPS , 2020.

Ashish Vaswani, Noam Shazeer, et al. Attention is all you need. In NeurIPS , 2017.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 2019.

Daphna Weinshall, Gad Cohen, et al. Curriculum learning by transfer learning: Theory and experiments with deep networks. arXiv:1802.03796 , 2018.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3-4):229-256, 1992.

Benfeng Xu, L. Zhang, et al. Curriculum learning for natural language understanding. In ACL , 2020.

Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. In ICLR , 2018.

Tom Zahavy, Matan Haroush, et al. Learn what not to learn: Action elimination with deep reinforcement learning. In NeurIPS , 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a method called State-Wise Action Refined to explore the causal relationship between actions and task rewards in reinforcement learning. They address the issue of action space redundancy and propose interventions on the primal action space as a means to discover causality. The authors present two practical algorithms, TD-SWAR and Dyn-SWAR, which respectively identify task-related actions during temporal difference learning and uncover important actions through dynamic model prediction. These approaches not only provide insights into the decision-making process of RL agents but also enhance learning efficiency in tasks with redundant actions.\n\nSTRENGTHS:\nThe authors proposed a method called State-Wise Action Refined to explore the causal relationship between actions and task rewards in reinforcement learning, which is significant and interesting.\n\nWEAKNESSES:\nThe technical description of the proposed approach is complicated, and the main concern lies in the fact that the main technical contribution is not clear.\n\nThe paper contains some grammar mistakes and lacks sentence flow, making it difficult to understand the main contribution of the paper. For example, lines 34-36.\n\nThe proposed two practical algorithms, TD-SWAR and Dyn-SWAR, do not have sufficient experimental analysis support.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nThe technical description of the proposed approach is intricate, and the primary concern is the lack of clarity regarding its main technical contribution. The paper contains grammar mistakes and lacks sentence flow, making it difficult to understand the central contribution. For instance, lines 34-36 are particularly confusing. Additionally, the practical algorithms proposed, TD-SWAR and Dyn-SWAR, suffer from insufficient experimental analysis support,", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a strategy for action space reduction, based on causality-inspired modeling. It employs a method of action space selection, using collected data to identify the specific dimensions of actions that are meaningful to the task, as well as to improve the efficiency of exploration in reinforcement learning. The empirical evaluation confirms the effectiveness of the proposed module. Overall, the method is straightforward and can be embedded into most existing model-free and model-based RL frameworks. However, there are some details of the method and specific applications of causality, along with theoretical guarantees, that need clarification from the authors. Therefore, in the initial review, I give it a borderline score. If the authors provide the corresponding discussion and explanations, I will increase the score in subsequent reviews.\n\nSTRENGTHS:\n[**About Motivation and General Idea**] Even though there have been related studies, research on action reduction is very meaningful for Reinforcement Learning (RL), as it can improve sample efficiency and potentially even learning efficiency. The proposed approach of integrating causality to identify which actions have a causal relationship with the task reward provides an interpretable action reduction method.\n\n[**About presentation**] The overall presentation is clear and easy to follow. The pipeline figure (Fig. 1) helps the readers to understand the steps. \n\n[**About experiments**] Although there are similar baseline methods for comparison, overall, the experimental results validate the effectiveness of the method proposed in the paper, and the experimental design is fairly comprehensive. I will provide some additional suggestions regarding the experiments in a later section.\n\nWEAKNESSES:\nNote: Most of the weaknesses listed below are more like questions, discussion points, or suggestions, rather than outright flaws. Any clarifications provided by the authors would be very welcome and appreciated. Additionally, I don't have any personal interest in the related papers I suggested for reference. Given the limited time for rebuttal, it is not necessary to fully supplement experiments for comparison with these papers. Some clarifications and discussions on these methods would be greatly appreciated.\n\n[**Regarding Intervention**] The authors mentioned that the causal discovery here is based on intervention, but there is a distinction between action and intervention. The authors could consider providing a clearer definition of intervention in this context, how interventions are conducted, and whether the target of each intervention is known, etc.\n\n[**Regarding the Causality**] Although the entire pipeline is well designed and its effectiveness can be verified empirically, more explanation is needed on how to connect it with causality. From the method, it's not clear how to use causal properties for modeling or how to directly learn the causal relationships among states, actions, and rewards. The authors could consider providing some clarifications, either regarding theoretical guarantees (on the causal level) or presenting learned causal graphs, to check whether they empirically match the causal relationships between actions and state rewards in the environment. Here are some references: identifiability proof with action involved system: [1-2], and causal RL with learned causal graphs [3-4]. \n\n\n\n[**Aout related works**]\n\nRegarding the discovery of causal RL and how to use causal structure to simplify the action space in RL, there are some related works. The authors could try to discuss them in the main text, possibly adding a subsection to introduce these works.\n\n[**About Experiments**]\n\n- For the experimental section, is it possible to visualize the learned sparse graph structure and compare it with the physical structure in real scenarios to see if they are similar?\n\n- The current comparison does not involve related causal RL methods. It would be beneficial to discuss these, especially the one presented in [5], which uses a learned causal structure to find the minimal space—a very similar approach.\n\n[**About Presentation**]\n\n- In Figure 2, the graphical model is presented, but the title lacks some explanations, such as which dimensions have a causal relationship with the dynamics and reward, and what the blue and pink colors represent.\n\n-  On line 195, does \"figure 5\" refer to \"figure 3\"? I noticed that figure 5 is in the appendix, and its content is similar to that of figure 3.\n\n- In my opinion, it would be beneficial to first introduce the overall framework of action reduction and causal discovery, then provide specifics about its implementation within Reinforcement Learning, in conjunction with both model-free and model-based RL. This approach could also simplify algorithms 1 and 2, merging them into one algorithmic pipeline, detailing the implementations under different RL frameworks.\n\n**References**\n\n[1] Lachapelle, Sébastien, et al. \"Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA.\" Conference on Causal Learning and Reasoning. PMLR, 2022.\n\n[2] Yao, Weiran, Guangyi Chen, and Kun Zhang. \"Temporally disentangled representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 26492-26503.\n\n[3] Ding, Wenhao, et al. \"Generalizing goal-conditioned reinforcement learning with variational causal reasoning.\" Advances in Neural Information Processing Systems 35 (2022): 26532-26548.\n\n[4] Wang, Zizhao, et al. \"Causal dynamics learning for task-independent state abstraction.\" arXiv preprint arXiv:2206.13452 (2022).\n\n[5] Huang, Biwei, et al. \"Action-sufficient state representation learning for control with structural constraints.\" International Conference on Machine Learning. PMLR, 2022.\n\nQUESTIONS:\nMost of the questions have been consolidated in the previous section, but there is still one remaining: for the optimization objective in eq.7, how can we theoretically and empirically ensure the learned structure is an identifiable causal structure? The authors could provide some insights, possibly in conjunction with some theoretical results from [1].\n\n**References**\n\n[1] Lachapelle, Sébastien, et al. \"Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA.\" Conference on Causal Learning and Reasoning. PMLR, 2022.\n\nLIMITATIONS:\nThe authors briefly describe some limitations in the paper. I believe the main limitation might lie in how to theoretically link with the causality claimed in the article. From an empirical perspective, the work is relatively complete. Future work could consider making the entire framework scalable to more complex scenarios, such as MARL, etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an intervention-based method for understanding causality in Reinforcement Learning, aiming to enhance learning efficiency in tasks with redundant actions. The authors present two novel algorithms, Temporal Difference State-Wise Action Refined (TD-SWAR) and Dynamic State-Wise Action Refined (Dyn-SWAR). TD-SWAR identifies task-related actions during temporal difference learning, while Dyn-SWAR uncovers important actions through dynamic model prediction. The paper expands upon the recent advancements in instance-wise feature selection technology (INVASE), demonstrating how these improvements allow for more effective selection of task-related actions. Experiments underscore the effectiveness of causality-aware RL agents in action-redundant settings.\n\nSTRENGTHS:\n1. The paper tackles the underexplored topic of causality-awareness in RL, utilizing interventions on the action space to ascertain task-related actions. cal underpinnings.\n2. The authors have made a clear effort to explain their work, providing intuitive examples such as the in-hand manipulation tasks and human learning analogies. The logical flow of the paper seems coherent.\n3. The paper tackles a relevant issue in RL - action space redundancy and the importance of understanding causal relationships for learning efficiency. The findings have the potential to influence future RL research, particularly on exploration strategies and action space selection.\n\nWEAKNESSES:\n1. The action space of experiments environments are all very small. Given the known scaling limitation of causal discovery, it's important to discuss the capability of the proposed model as most of the real world RL environments are complex.\n2. The paper could benefit from more comprehensive evaluation. They only compared with TD3 while there are many more advanced RL algorithms that have better performance on the experimented environments. Therefore it's not convincing that pruning action space is beneficial even in the simple environment.\n3. The connection between the proposed methods and real-world application scenarios seems somewhat abstract. The authors could better explain how their proposed methods can help solve practical problems.\n4. It's better to provide code to enhance reproducibility.\n5. Formatting issue in line 6.\n\nQUESTIONS:\n1. It would be helpful to compare with state-of-the-art RL algorithms.\n2. Could you please provide a discussion on the problem complexity (e.g., the number of variables) that the proposed method can handle? And discuss any limitations or potential pitfalls in their methods? This could include discussion on how robust their methods are to different types of environments or tasks. It's important to know whether these methods could be generalized across various RL applications.\n3. It would be helpful if the authors could clarify how the proposed methods could be applied to real-world problems, and possibly provide case studies or scenarios demonstrating this.\n\nLIMITATIONS:\nThe authors didn't address limitations nor broader societal impact, but I didn't see any ethical concerns.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows a Q-learning-based RL algorithm that jointly performs feature selection to detect action variables that are not relevant to dynamics or rewards. The feature variable selection algorithm adopts INVASE, and the experiment confirms the idea of five artificially created extended environments.\n\n----\n\nDuring the rebuttal phase, we found that there are several major technical flaws around the causality. \nReflecting on such issues, I adjusted the score to the criteria.\n\nSTRENGTHS:\nThe strength of the proposed approach is a simple extension of Q-learning-like algorithms that jointly perform action variable selection that improves the overall learning performance when there is a large number of redundant actions.\n\n* originality: The idea of combining feature selection with INVASE during TD learning is novel\n* correctness: The proposed idea looks correct\n* clarity: The algorithm was presented in detail\n* significance: This method may work well when there are large number of redundant actions\n\nWEAKNESSES:\nAlthough the title and Figure 2 mention “causal” and SCM, this paper is not relevant to causal discovery, or SCM is not relevant in the context. It can be better said as a kind of “feature selection.”\nI am not sure how realistic it is to assume the nuisance actions and inflate action space to 100 by adding redundant actions in the experiment. \n\n* Originality: The idea is original, and it could be improved if it considers settings common in causal discovery\n* correctness: I believe the overall framework is correct, and I don't see a particular weakness\n* clarity: This paper is not relevant to causal/SCM, which confused me by giving the impression that it performs causal discovery while TD-learning.\n* significance: The experiment setting is unrealistic, and it limits assessing the significance\n\nQUESTIONS:\n1 In Proposition 1, How to ensure that a larger set of actions always contains the smaller ones?\nIn the proof, it is by definition of the operator, but it needs more explanation.\n\n2 Is it possible to extend the proposed work to discrete action space?\n\nLIMITATIONS:\nI think this paper is not relevant for discussing limitations.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-07"}