Select before Act: Spatially Decoupled Action Repetition for Continuous Control
Authors: Buqing Nie
Yangqing Fu
Yue Gao
Venue: ICLR 2025 Poster (2025)
Keywords: Reinforcement Learning
Action Repetition
Primary area: reinforcement learning
TL;DR: A novel reinforcement learning method introduces an act-or-repeat mechanism for individual action dimensions, enhancing sample efficiency and final performance with reduced action fluctuation.

Abstract: Reinforcement Learning (RL) has achieved remarkable success in various continuous control tasks, such as robot manipulation and locomotion.
Different to mainstream RL which makes decisions at individual steps, recent studies have incorporated action repetition into RL, achieving enhanced action persistence with improved sample efficiency and superior performance.
However, existing methods treat all action dimensions as a whole during repetition, ignoring variations among them.
This constraint leads to inflexibility in decisions, which reduces policy agility with inferior effectiveness. 
In this work, we propose a novel repetition framework called SDAR, which implements Spatially Decoupled Action Repetition through performing closed-loop act-or-repeat selection for each action dimension individually.
SDAR achieves more flexible repetition strategies, leading to an improved balance between action persistence and diversity.
Compared to existing repetition frameworks, SDAR is more sample efficient with higher policy performance and reduced action fluctuation.
Experiments are conducted on various continuous control scenarios, 
demonstrating the effectiveness of spatially decoupled repetition design proposed in this work.

Published as a conference paper at ICLR 2025
SELECT BEFORE ACT: SPATIALLY DECOUPLED AC- TION REPETITION FOR CONTINUOUS CONTROL
Buging Nie', Yangqing Fu', Yue Ga
"MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University Shanghai Innovation Institute, Shanghai, P.R. China (niebuqing, frank mrank79110,
ABSTRACT
Reinforcement Learning (RL) has achieved remarkable success in various con- tinuous control tasks, such as robot manipulation and locomotion. Different to mainstream RL which makes decisions individual steps, recent studies have in- corporated action repetition into RL, achieving enhanced action persistence with improved sample efficiency and superior performance. However, existing meth- whole during repetition, ignoring variations
ods treat all action dimensions among them. This constraint leads to inflexibility in decisions, which reduces policy agility with inferior effectiveness. In this work, we propose novel rep- etition framework called SDAR, which implements Spatially Decoupled Action Repetition through performing closed- loop act-or-repeat selection for each action dimension individually. SDAR achieves more flexible repetition strategies, lead- ing to improved balance between action persistence and diversity. Compared existing repetition frameworks, SDAR is more sample-efficient with higher pol- icy performance and reduced action fluctuation. Experiments are conducted on various continuous control scenarios, demonstrating the effectiveness of spatially decoupled repetition design proposed in this work.
INTRODUCTION
Recently, Deep Reinforcement Learning (DRL) (Sutton & Barto 2018 has achieved remarkable success in continuous control domains, such as robot manipulation (Gu et al. (017). locomo- tion (Lee et al. 2020b Zhang al. 2024), and autonomous driving (Kiran al. 202T. Al-
though conventional Reinforcement Learning (RL) algorithms have demonstrated sigmificant po- tential across various applications, they typically make decisions at individual time steps. neglecting higher-level decision-making mechanisms and the temporal consistency of action sequences (Silver al. 2014 Schulman al., [017 Haarnoja et al. 2018). This leads to inefficient exploration during training. challenging credit assignment tasks over long horizons, and poor sample efficiency (Dab-
ney et al. 2021 Yu et al. 2021 Biedenkapp et al. 202 Zhang et al. 2022 Lee et al. 2024).
One mainstream solution is Hierarchical RL (HRL) based on temporal abstraction (Sutton et al. 1999 Precup 2000), where the high-level policy decomposes the task into simpler subgoals, and low-level policies designed to solve isolated subtasks (Vezhnevets et al.) 2017 Pateria et al. 2021]. However, most HRL methods are task specific, which requires expert knowledge and hand- craft design given different applications, such pre-defining options in the option framework (Ba- con et al.) 2017] Nachum et al.] 2018 Zhang et af.) 2021). simple yet effective strategy for low-level policies involves executing an action repeatedly for a number of steps, which has been actively explored and proven to be effective in various applications (Lakshminarayanan et al. 2017] Sharma al. 2017 Dabney al. 2021 Lee al. 2024). Action repetition methods improve ac- fion persistence through act-or- selections, making exploration trajectories more temporally consistent, leading higher training efficiency and reduced fluctuations (Chen al. 202T Yu al.)
2021 Biedenkapp et al. 2021).
"Corresponding author.

---

Published as a conference paper at ICLR 2025
However, illustrated in Fig. exist- ing action-repetition approaches treat all action dimensions whole during act- or-repeat selection, disregarding differ- ences among them. This constrains all action dimensions to repeat previous ac- tions make new decisions simultane- ously at each step. This design reduces the effectiveness of action repetition, be- cause different dimensions may be re- quired to select repetition at different time steps, which is quite common in continu- ous control tasks (Kalyankrrishnan et al. 202I]. For example, robotic systems typ- ically consist of multiple controllers oper- ating at different frequencies due their system specification, corresponding to dif- ferent repetition schema for each actua- tor (Raman et al., 2017: Lee et al. 2020a)
Previous action
New action
4-hh dimension
Previous Methods
SDAR Algorithm (Ours)
(3)
Repeat Act
Decoupled repethions: more flexible
Difference between repetition strategies of previous methods (left) and SDAR (right), Our method achieves more flexible strategy through the spatially decoupled repetition design.
Figure 1:
To address this issue, propose Spatially Decoupled Action Repetition (SDAR), new flexible action repetition framework for continuous control tasks. SDAR conducts closed-loop repetition for each action dimension individually, which is composed of stages: selection and action. During selection, the agent selects whether the previous actions should be repeated for each dimension. Af- terwards, the agent generates new decisions for dimensions where act chosen in the previous stage. As shown in Fig. [ different to previous methods, act-or-repeal selections in SDAR are performed a decoupled manner, leading to more flexible repetition strategies. Compared to exsiting repe- tition frameworks. SDAR achieves improved balance between action persistence and diversity, resulting in improved efficiency and superior performance with reduced action fluctuation.
The main contributions of this work summarized as follows:
We propose a novel action repetition framework called SDAR for continuous control tasks. SDAR implements Spatially Decoupled Action Repetition through performing closed-loop act- or-repeat selection for each action dimension individually.
Compared to previous repetition methods, SDAR offers more flexible repetition strategies, leading to improved balance between action persistence and diversity. This results in higher sample efficiency with superior performance and reduced action fluctuation simultaneously.
Experiments conducted various continuous control tasks, demonstrating the superior performance of our method on training efficiency and final performance. This demonstrates the effectiveness of the spatially decoupled framework for action repetition.
RELATED WORK
2.1
TEMPORAL ABSTRACTION
Temporal Abstraction is proposed in the semi-MDP formulation (Sutton et al. 1999 Precup 2000 and commonly implemented based on the options framework (Stolle & Precup 2002 Bacon al. 2017 Harutyunyan et al. 2018). Each option describes a low-level policy and is defined as (Z, ग, 33) where I denotes the admissible states for the option initialization, is the policy that the option fol- lows, and 3 determines when the option is terminated. High-level policies are trained to solve tasks utilizing temporally extended actions provided in the options, rather than one-step actions without action persistence. Plenty of Hierarchical RL. methods are proposed based on temporal abstraction, achieving faster exploration and higher sample efficiency in various sequential decision tasks (Lin) al. 2021 Yang et al. 2021). Some works are proposed to learn to design options through various techniques, including discovering state connectedness (Chaganty et al. 2012). replay buffer analy- sis (Eysenbach al. 2011). and learning termination criteria (Vezhnevets et al. 2016 Harutyunyan cet al. 2009)). However, designing options is still challenging task, which requires prior knowledge
polipiess
and handcraft tunning (Pateria al. 2021 Yu et al. 2021
Lee al. 2024).
2

---

Published as a conference paper at ICLR 2025
2.2 ACTION REPETITION
One simple option strategy is repeating a primitive action for a number of steps. which is simi- lar to the frame-skipping utilized RL for video games (Bellemare al. 2013; Braylan et al.
2013). Recently, action repetition has been actively researched and widely adopted in RL, which can achieve deeper exploration (Dabney al. 202T), improve sample efficiency by reducing control granularity (Biedenkapp et al., (021), and reduce action oscillations (Chen et al. (021). Existing repetition works are classified as two categories: open-loop and closed-loop manners. Open-loop methods force the agent to repeat actions for predicted number of steps without opportunity of early terminations, such as DAR (Lakshminarayanan et al. (017). FiGAR (Sharma et al., 2017).
TempoRL (Biedenkapp et al., 2021), and UTE (Lee al. 2024). In contrast, closed-loop methods conduct act-or-repeat binary decision to decide the previous action should repeated, such as PIC (Chen et al. 2021) and TAAC (Yu et al. 2021). Compared to open-loop methods, closed-loop methods examine whether to repeat based on the current state, which is more flexible and improves performance in emergency situations. In this work, we propose a new closed-loop method con- duct act act-or-repeat selections for each actuator individually given current state, which is more flexible and achieves higher action persistence with sufficient action diversity.
3 PRELIMINARIES
3.1 PROBLEM FORMULATION
In this work, we focus on model-free RL with continuous action space. The interaction process RL is formulated Markov Decision Process (MDP), denoted as tuple M = (S, A. P, R, 7). where the state space, is the action space, P(s' s, a) is the transition probability of the environment, R S x A + IR denotes the reward function, and y [0, 1) denotes the discount factor. The agent takes actions according to its policy, i.c. a ~ (-1s). Our objective is to find a policy π that maximizes the expected discounted return, i.e. n²" arg max, Ba EE 7 R(st, de)]-
3.2 MODEL-FREE RL FOR CONTINUOUS CONTROL
In order to conduct policy evaluation, define state-action value function Q(st, az) R(s,, a:) + ylE,,, Em_,7' R(sr, ax) as the discounted return starting from 5t, given that a, is taken and then π is followed. The value function Vss:) Ear~a [Q(s,, ax)] denotes the discounted return starting from 8; following æ. Typically, both Q and V functions are modeled as neural networks, which are optimized using the Mean Square Error (MSE) loss, with target values obtained based on the Bellman equation. Take Soft Actor Critic (SAC) (Haarnoja et al. 2018) as an example, the policy is trained through the entropy augmented objective:
n' f- arg max LE a~a [Q (s,a) log (als)],
(1)
where denotes the replay buffer, a is the temperature parameter, and Q represents accumulated discounted reward augmented by the entropy.
4 METHODOLOGG
4.1 TWO-STAGE POLICY: SELECTION AND ACTION
As described in Fig. 2 the decision process of SDAR is composed of two stages. (1) Selection: choose which action dimensions to change previous decisions utilizing the selection policy 3. (2) Action: generate new actions for action dimensions that choose act in the previous stage utilizing action policy π. Both two stages are described as follows.
4.1.1 STAGE 1: SELECTION POLICY
As illustrated the gray region of Fig. 2 given current state and the action a at the previous step. conduct act-or-repeat selection for each action dimension individually. Formally, B(b|s, a-)
3

---

Published as a conference paper at ICLR 2025
Stage
Stage 2: Action
Action mask
alate
Gaussian
sample
sample
MILP
R
?
MLP
Fad
mixx
selection policy 8(b) a,
repettion echema &
action policy (ā
oulput action
previous
Repeat (0)
Act (1)
Action mask
New action
adion
Figure 2: The two-stage decision process of SDAR algorithm. In the first stage (gray region), the selection policy makes act-or-repeat decision for each action dimension. determining whether the previous action a (yellow blocks) should be repeated. In the second stage (blue region), the action policy æ generates new actions (green blocks) for dimensions that choose act in the first stage.
[0, 1] 11, where 33 denotes the probability of choosing act in the i-th action dimension. Afterwards, the repetition schema {0, 13 at the current state is obtained by sampling the Bernoulli distribution, i.e. b, B a where b; and b; indicate repetition and action of the i-th action dimension correspondingly.
4.1.2 STAGE 2: ACTION POLICY
As shown in the blue region of Fig. 2 the agent generates new actions for dimensions that choose based on the current state s, the previous action and the repetition schema Firstly, we reduce redundant input dimensions through introducing the following Mix operation:
a mix Mix(b, a ,) =(1 @ + +
(2)
where @ denotes element- wise hadamard product and denotes the action mask, which is a constant mask vector filled with meaningless values. In this work, 2 for 1 < i is |A] is utilized with A 1]1 This operation eliminates the information of previous actions where act was chosen in the previous stage. This information redundant and may disrupt the training process of Afterwards, compute á utilizing action policy
dimensions
Mmean Jstd MLP (s, mix), à Umean + sttd n, (O,II),
where jmean and astd are the mean and standard derivation of the action distribution predicted by action policy æ. The reparameterization trick utilized ensure that this process remains differ- entiable (Haarnoja et al. 2018). Finally, we obtain the output action a through the Mix operation formulated as follows:
a Mix(b, a _a) b) b
This procedure guarantees that the final decision replicates the same actions as the previous action a dimensions where repetition selected, i.e. (a a (1-b) = 0. The detailed proof is straightforward and can be found in Appendix A
Above all, the decision of the whole two-stage policy aلl can be described as follows:
all (a|s, a-) Σ B(bls a) =(āls, a .b) (Mix (b, a (b)aa,)) - a) då.
(5)
4.2 POLICY EVALUATION
In this work, we employ two identical Q functions to evaluate the performance of -lll, Each Q is trained utilizing Mean Square Error (MSE) loss with targets obtained based on the Bellman operator formulated as follows:
min E2 D QQ (8,a) -TQQQQ (s, with TQ(s, R(s,a) BRa, (8')
(6)
4

---

Published as a conference paper at ICLR 2025
Algorithm Spatially Decoupled Action Repetition (SDAR) Algorithm
Initialize: Selection policy 3, action policy n, buffer crities Q112) Quarg, (1.2)-
2: for each environment step & do
3: if new episode starts then
4: b = [1,1,..... |||
5: else
6: b ~~ B(-1st, 0t-1)
7:
end if
8:
Obtain d through action policy, i.c Eq. (2-(4)
9:
Execute a, and observe 8t+1 with reward Tt-
10:
Store transition (St, at-1; 01,1,,0,,7 8t4,,, de) into buffer D
11:
Sample a batch of data {(s, a a, s,,, ,d) from D
Compute a' for state s' using and n.
13:
Update Q1122) through Eq. if mod policy -delay then
15:
Update H through Eq. 7
Update 8 through Eq. (8) Eq. 9
17:
Update ag and through Eq. (00)
18:
Update target networks Qtarg, (1,2) based on soft updates.
19:
end if
20:
end for
where D the replay buffer. T in this work is similar to that in standard off-policy RL algo- rithms (Silver et al. 2014 Lillicrap 2015). which is demonstrated to converge to the optimal Q after sufficient iterations. In addition, we formulate two Q functions based on neural networks, and train using clipped double- learning to alleviate the overestimation problem(Fujmoto et al. 2018). More details are given in Algorithm
4.3 POLICY IMPROVEMENT
We propose to optimize the policy 33 and through maximizing the objective [ (a)) formulated as follows, where 0 and 0= are learnable parameters of 3 and æ correspondingly.
J(0 07) (e Ben B, Bron
(s, -ag log 8 (b),,a
log "T (á (a)s,a ;))
(7)
make docisions on samples
values
entropy
casopy-bused based exploration
where entropy terms E - log ((b]s,a7) and | log π (a|s, aa,ab b)] are utilized to encourage ex- ploration during training. which is widely utilized in prior works (Haarnoja et al. 2018 Yu al. 2021). 0g and a, are temperature parameters to adjust exploration strategy, both of which are tuned automatically during training. As illustrated in Sec. [.1 and Sec. 4.2] the computation process of and Q functions are both differentiable. Therefore, the action policy can be directly optimized using gradient descent with V J.
In order to optimize selection policy 3, we can transform objective J into the following formulations:
max (s,a )~D Σ DEB
a ((bls.a")Baan [Q a) 0g log (bls,a a a- log æ (āls, a bb)] (8)
where {0, 11A1 is the action space for selection policy B. In contrast to Eq. 7 the new objective in Eq. (8) substitutes the expectation operator Es-g with the summation operator Eee by listing all possible situations b B. The objective is differentiable for 03, thus be used to optimize 3 through gradient descent directly.
However, Eq. is quite computationally expensive, because are required to compute the score for all b B, which needs to calculate Q and for [B] 2| times. Thus, this method is only practical for tasks with small action spaces, such as LunarLander with | 2. For tasks with large action spaces such Humanoid ([A) 17), the selection policy & be optimized by sampling

---

Published as a conference paper at ICLR 2025
several b B, which is formulated follows:
max LED Bwa (s. ag log Bota (bls, a-) Our log (āls, a b)]
3 (b|s, a Bold (b|s,ar) (
62
where Sold denotes the old selection policy before optimization. This objective finds the expectation value [+ [:] in Eq. 7 utilizing importance sampling. Compared to uniform sampling, sampling with Sold increases the probability of sampling b with high Q values during training, leading to
higher sampling efficiency and training stability. As illustrated in Eq. 99. we are only required to compute Q and и given several sampled from 3, thus is more computationally efficient and practical than Eq. (8) in applications.
In addition, temperature parameters ag and as are tuned automatically following the objective:
min log (ag) az
BØ,x [- log (at) (log B(bls, a Ho) logg ax (logn(āls,a" ,b) + H+)]] (10)
where Hg and H, are target entropies. During training, log ( and log og are updated automatically through solving Eq. (T0) with gradient descent, which is similar to prior works (Haarnoja et al. 2018). The detailed settings of the target entropies are described in Appendix [.I
The whole training process of SDAR is illustrated in Algorithm T which iteratively collects data, trains critics, and updates policies. As illustrated line 10, different to typical DRL algorithms, the previous action a;-1 is stored in the buffer D, which is utilized to improve H" and 6 described in line 15-16. Besides, the agent is forced to choose acl at the initial step of an episode, i.e. b = [I,-.. J114), because is absent at this state.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Tasks: In this work, we conduct experiments on multiple continuous control tasks, which are categorized into the following three types of scenarios. More details are given in Appendix B.2
Classic Control: Several control tasks with small observation and action spaces, including MountainCarContinnuss CainarLandeConttinon and SipedalWalkerr
(b) Locomotion: Locomotion tasks based on the MuJoCo (Todorov et al. 2012) simulation envi- ronment: Walker2d, Hopper, HalfCheetah, Humanoid, and Ant.
(c)
Manipulation tasks including Pusher, Reacher, and FetchReach.
Baseline Methods. In this experiment, we compare the performance of SDAR with the following
baseline methods: (a) SAC: vanilla Soft Actor Critic (Haarnoja et al.) 2018) algorithm: (b) N-Rep:
repeat policy actions for times, where is fixed number (Mnih et al. 2015); (c) TempoRL: repeat policy actions dynamically in an open-loop manner, using a skip-policy to predict repetition steps (Biedenkapp et al. 202T); (d) UTE: Uncertainty-awan Temporal Extension (Lee et al. 2024) enhances training efficiency through incorporating uncertainty estimation of repeated action values into the open-loop repetition framework. (e) TAAC: Temporally Abstract Actor-Critic (Yu et al.) 2021], which repeats actions dynamically in a closed-loop manner. TAAC determines whether to repeat the previous action at each step utilizing switch policy. More details of the baseline methods are illustrated in Appendix B.3
This study trains each method on various tasks using multiple random seeds over a range of 100K to 3M steps, depending on the complexity of the task. More settings including hyperparameters settings are described in Appendix B.I
5.2 EXPERIMENT RESULTS ON SAMPLE EFFICIENCY
The training curves are illustrated in Fig. 3. and the AUC scores are shown in Table I The AUC scores normalized into [0, 1] to evaluate the sample efficiency across different tasks, where 0.0

---

Published as a conference paper at ICLR 2025
lifeer
LMC
Taemscc
(a) MountainCar
(b) LunarLander
(c) SipedalWalkee
(d) Humanoid
Dark
0010 pied
30
(e) Ant
(I) Walker2d
(g) HalfCheetah
(h) Pusher
Figure 3: Learning curves of SDAR (red) in various tasks against baseline methods. Each method trained with least 10 random seeds. The lines denote the episode return, while shaded regions denote the standard error during training. As shown in the figures, our method generally achieves higher sample efficiency in various tasks compared to previous methods. More learning curves are given in Appendix. D
Table 1: Experiment results on AUC scores (area under the learning curves) in different types of environments, where higher AUC scores indicate higher sample efficiency of corresponding DRL algorithms. The average scores along with the standard errors are calculated across various tasks within the respective category. The results are normalized evaluate performance across different tasks, where 1. 1.0 denotes the best performance. The best results are boldfaced.
Normalized AUC Score
Env. Category
SAC
N-Rep
TempoRL
UTE
TAAC
SDAR
Classic Control
0.60
0.8910.01
0.66:10.02
0.5510.03
0.9243E--
1.0:0.0
Locomotion
0.78+
0.71+7E-3
0.35+0.02
0.43+0.01
0.80+0.02
1.0=0.0
Manipulation
0.91+4E-5
0.90+8E-4
0.77#0.02
0.7940.02
0.95+6E-3
1.0=0.0
Average 0.76:10.02
0.83+10.01
0.59+00.05
0.59+00.3
0.90+16E-3
1.0
denotes the performance of random policies without effective training, while 1.0 denotes the perfor- mance of the best method. See Appendix C for the detailed computation process the normaliza- tion. More experiment results given in the Appendix D
As illustrated in Fig. 3 our method SDAR (red lines) achieves higher sample efficiency than other baseline methods in various continuous control environments, corresponding to higher AUC scores shown in Table []] This demonstrate effectiveness of performing act-or-repeat selection for each action dimension individually. Such spatially decoupled repetition framework enhances action per- sistence while maintaining policy flexibility, leading to efficient exploration and training. In the following, we analyze the performance of each method individually:
Naive action repetition N-Rep achieves unstable improvement compared to vanilla SAC, but underperforms closed-loop repetition approaches generally. As described in Table T N-Rep achieves higher AUC scores than SAC in classic control tasks, such MountainCar shown in Fig. B(a] Naive repetition improves the persistence actions during exploration, accelerating the training progress effectively in simple tasks. Nevertheless, this approach lacks flexibility, which
7

---

Published as a conference paper at ICLR 2025
could negatively impact the performance in tasks requiring agile movements, such as HalfCheetah (Fig. ((g) in locomotion domains.
Open-loop methods (TempoRL and UTE) show minor improvements over SAC in classic control tasks, while performing worse performance in other domains. These approaches force the agent to repeat actions for a predicted number of steps, climinating the possibility of carly termination. This inflexibility makes such methods difficult to be utilized in tasks requiring agile movements, such as Humanoid shown in (dd)
The closed-loop method TAAC outperforms SAC and open-loop approaches generally in various classes of tasks. TAAC introduces switch policy check whether to repeat previous actions at each step, which solves the lack of termination mechanism in open-loop repetition methods. This leads more flexible repetition, which suitable for both simple tasks and locomotion tasks requiring agile motions, such as Ant shown in Fig. [(e)
However, TAAC treats all action dimensions as whole during act-or-repeat selection, which downgrades the effectiveness of action repetition. Take Humanoid and HalfCheetah shown in Fig. ((d) and Fig. ((g) respectivel as examples, TAAC (brown lines) demonstrates superior sam- ple efficiency during the initial training phase, specifically between 0 and 1M steps. However, in later stages, the agent's learning speed diminishes compared to SDAR, which even results in performance decline in the Humanoid task depicted in Fig. 3(d) This is because the Humamoid is composed of multiple actuators, where some actuators are required to select act for action di- versity, while others choosing repeat for action persistence at the same time. However, TAAC is constrained to repeat actions of all actuators simultaneously, leading to either inadequate repeti- tion for inefficient exploration, or excessive repetition to damage the policy performance.
Our method SDAR outperforms other methods in various tasks, including TAAC in locomotion tasks. SDAR decouples action dimensions during closed-loop act-or-repeat selection, which more flexible and suitable for various types of tasks. Taking the Humanoid task illustrated in Fig. ((d) as an instance, SDAR compromises marginal efficiency in the initial stages to achieve greater overall efficiency and stability throughout the entire training process. Above all, SDAR achieves a better balance between action persistence for efficient exploration, and action diversity for high policy performance.
5.3 POLICY PERFORMANCE AND FLUCTUATION
In this work, we assess the policies trained through each method on the following metrics: (1) Episode Return: the average cumulative reward acquired by the policy in an episode, assessing the policy's capability solve the task successfully. (2) Action Persistence Rate (APR) and (3) Action Fluctuation Rate (AFR) formulated as follows:
T
IAI
T 0-
APR 1
p == E
(0
(1) -0 22-1
AFR E
Σ T
T-I
where T denotes the trajectory length, dj (i) denotes the action at the 1-th step in the i-th dimension, and p denotes the average repetition probability of at" at each step. In this work, APR represents the average interval that two new decisions are made. APR evaluates action persistenc of the policy, where larger APR denotes more repetition times during interactions. AFR evaluates the mean amplitude of fluctuations between successive steps, where a smaller AFR indicates smoother actions with fewer fluctuations.
The results of cpisode return, APR, and AFR are shown in Table 2 As illustrated in the tables, although the baseline method such N-Rep achieves excellent results in both APR and AFR, it suffers a considerable reduction on policy effectiveness, corresponding to lower episode return com- pared SAC. This is unacceptable and makes such methods impractical for real-world applications. In contrast, our method SDAR outperforms baseline methods on episode return, while achieving a higher APR and lower AFR than the vanilla DRL, demonstrating the effectiveness of our method. In addition, we can also observe that:
Naive repetition method N-Rep achieves high APR and low AFR, at the expense of performance reduction episode returns. This is because the actions in N-Rep are repeated for frozen times,
8

---

Published as a conference paper at ICLR 2025
Table 2: Experiment results on Episode Return, APR (higher is better), and AFR (lower is better) in different tasks. The best results on episode returns are boldfaced. Our method SDAR outperforms baseline methods on episode returns generally, achieving excellent action persistence with fewer fluctuation than vanilla methods.
Episode Return (Mean + Standard Error)
Tasks
Action Persistence Rate (APR) Action Fluctuation Rate (AFR)
SAC
N-Rep
TempoRL
UTE
TAAC
SDAR (Ours)
275.9-16.83
1.2.
281.5
282.8 8.74
281.4=18.9
282.2:15.84
LunarLander
1.00 0.09
4.00 0.08
1,43 0.118
1,38 / 0.36
3.05 0.111
11.18 / 0.10
33051:36
47241163
2866-1977
298611836
5660=1944
602811406
Walker2d
1.00//..5
4.00 0.09
5.74 0.26
7,81 0.24
1.30 1.30/0.22
2.96 / 0.12
13122:1:2877
8378-11553
8065-11799
7917:1293
11148
15131
HalfChee.
1.00 / 0.68
2.00 0.47
2.10/
2.69 /
1.02 / 0.61
1.22 / 0.62
6184=717
5074#310
1022=397
2595=3334
7308=244
7483#2288
Humanoid
1.00 / 0.28
4.00 0.09
5.27 0.15
5.69 0.118
1,21 / 0.25
1.67 / 0.19
12.5=
-21.2=
48,2=2.08
441.3:55.66
30.5=
1.23:11.66
Pusher
1.00 0.002
4.00 0.019
1.15 0.032
/
1.75 /.033
1.69 0.0015
0.89-=0.07
0.81
0.59-+0.33
0.64
0.91+-0.10
1.00-10.001
Average
1.00 0.245
3.60 / 0.150
3.12 0.257
3.71 / 0.2276
1.66 / 0.244
3.75 0.208
where repeated actions may be unreasonable in some states. This inflexible repetition strategy cannot agilely adjusted during interaction, leading to performance reduction, especially in locomotion tasks such as HalfCheetah.
Open-loop methods achieve high APR, indicating lot of action repetitions during interaction, the expense of more fluctuations and superior episode return performance. instance, UTE achieves higher APR (3.71) than SAC (1.0), with large fluctuation and low episode return. This suggests that the increment of repetition times in an inffexible mamnr may be harmful to policy effectiveness and smoothness. In contrast, TAAC solves this problem, achieving a moderate AFR (1.66), with a comparable AFR (0.244) and high episode return (0.91).
Different to UTE and SDAR, although SDAR also selects a of repeat during interaction with a high APR (3.69), SDAR achieves excellent performance on fluctuation and episode returns simultaneously. This is because SDAR performs repetitions a more flexible manner. The de- coupling design is consistent with the requirement of continuous control tasks, thus can conduct repetition without harms to episode returns and fluctuation performance.
5.4 VISUALIZATION OF ACT-OR-REPEE -EREEAT SELECTION
In order analyze the difference between repetition be- haviors of SDAR and previous methods, we perform visu- alization of act-or-repeat selections LunarLander and Walker2d. The results are illustrated in Fig. 44a) and Fig. [bb] where dark blue and light blue blocks denote act and repeat respectively.
Table 3: Experiment results of each action dimensions in Walker2d.
APR Val.
TAAC
SDAR
Thigh (ap)
1.30
3.70
Leg (a1)
1.30
2.17
As shown in the figures, different dimensions require differ- ent decision frequencies, corresponding to actions being re- peated at distinct steps for each dimension. Take LunarLan- der in Fig. ((a for example, the lateral boosters (a, change actions frequently to adjust the rocket pose, while the main engine (og) adjusts decisions occasionally.
Foot (az)
1.30
3.44
Episode Ret.
5660
6028
AFR
0.22
0.12

---

Published as a conference paper at ICLR 2025
Act-orrepest Dedision 1SDAR)
Isteral booslers
Repeat
31
maann engine
Actoosoped Deckion (1AA))
21
Tineskp
LunarLander
Repeat
trigh
Decision (TAAC)
leg
fool
Walker2d
Figure 4: Visualization of act-or-repeat selections of SDAR and TAAC algorithms in LunarLander and Walker2d tasks. The z-axis denotes timesteps. and the y-axis denotes different action dimen- sions. The light blue blocks indicate repetition, while dark blue blocks represent act, i.e. change actions the corresponding action dimensions.
SDAR adjust action persistence for each dimension automatically through spatially decoupled framework, offering more flexible repetition strategies compared to TAAC. As described in Fig. 4bb) and Table 3 SDAR controls the walker with higher persistence for the leg joints, while lower per- sistence for thighs and feet. In contrast, all joints in TAAC are required to repeat simultaneously, resulting in same persistence for each joint with lower repetitions behaviors.
SDAR achieves high action persistence, while maintaining agility required by the task, leading to higher policy performance with fewer action fluctuations. As shown in Table SDAR obtains higher APR than TAAC in Walker2d, while achieving higher episode return with lower AFR, demonstrating the effectiveness of our framework in continuous control tasks.
CONCLUSION
In this work, we propose novel action repetition framework for continuous control tasks called SDAR, which implements closed-loop act-or-repeat selection for each action dimension individu- ally. Such a spatially decoupled design improves the ficxibility of repetition strategies, leading to improved balance between action persistence and diversity, while maintaining action agility for con- tinuous control tasks. Experiments are conducted in various task scenarios, where SDAR achieves higher sample efficiency, superior policy performance, and reduced action fluctuation, demonstrat- ing the effectiveness of our method.
This work provides insights into spatially decoupled framework for action repetition and temporal abstraction. In this study, the selection policy designed to decide act-or-repeal for each dimen- sion independently, without accounting for inter-dimensional correlations. For instance, actuators in humanoid robots can be divided into multiple categories, with each category typically requiring same decisions during act-or-repeat selections. This issue will be further researched in future works.
ACKNOWLEDGMENTE
This work supported by the National Natural Science Foundation of China (Grant No. 62373242 and No.92248303). Shanghai Municipal Science and Technology Major Project (Grant No. 2021 2021SHZDZX0102),
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
André Biedenkapp, Raghu Rajan, Frank Hutter, and Marius Lindauer. Temporl: Learning when to act. In International Conference on Machine Learning, pp. 914-924. PMLR, 2021.
Alex Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Mikkkulainen. Frame skip is a pow- erful parameter for learning to play atari. In Workshops at the twenty-ninth AAAI conference artificial intelligence, 2015.
Arun Tejasvi
Chaganty, Prateek Gaur, and Balaraman Ravindran. Learning in small world. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems- Volume 1. pp. 391-3977 2012.
Chen Chen, Hongyao Tang, Jianye Hao, Wulong Liu, and Zhaopeng Meng. Addressing action oscillations through learning policy inertia. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7020-7027, 2021.
Will Dabney, Georg Ostrovski, and Andre Barreto. Temporally-extended e-greedy explocxo- ration. In International Conference on Learning Representations, 2021. URL https://
openreview Beeenrevee.met/fooeeee www.///eoo/urmu fott:
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. Advances neural information processing systems, 32, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Shixiang Gu, Ethan Holly. Timothy Lillicrap, and Sergey Levine.
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Anna Harutyunyan, Peter Vrancx, Pierre-Luc Bacon, Doina Precup, and Ann Nowe. Learning with options that terminate off-policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Anna Harutyunyan, Will Dabney, Diana Borsa, Nicolas Heess, Remi Munos, and Doina Precup. The termination critic. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2019.
Matteo Hessel, Joseph Modayil, H. V. Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining im- provements in deep reinforcement learning. In AAAI Conference Artificial Intelligence, 2017.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mchta, and JoAGo GM AraAšjo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Shivaram Kalyanakrishnan, Siddharth Aravindan, Vishwajeet Bagdawat, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla. An analysis of srame-skipping in rein- forcement learning. arXiv preprint arxiv:2100.003/18. 2021.

---

Published as a conference paper at ICLR 2025
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, 23(6):4909-49222) 2021.
Aravind Lakshminarayanan, Sahil Sharma, and Balaraman Ravindran. Dynamic action repetition for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Jongmin Lee, Jyung-Jun Lee, and Kee-Eung Kim. Reinforcement learning for control with multiple frequencies. Advances in Neural Information Processing Systems, 33:3254-3266 2020a.
Joongkyu Lee. Seung Joon Park, Yunhao Tang, and Min-hwan Oh. Learning uncertainty- aware temporally-extended actions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13391-13399, 2024.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xeabc5986, 2020b.
TP Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint 100.00990, 2015.
Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-me: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint 122:::211 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. nature, 51 518(7540):529 533, 2015.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficies hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey, ACM Computing Surveys (CSUR), 14(5):1-35, 2021.
Matthias Plappert, Marcin Andrychowicz. Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging roboties environments and request for research. arXiv preprint arXiv:1802.09464 2018.
Doina Precup. Temporal abstraction in reinforcement learning. Amherst, 2000.
University of Massachusetts
Ritu Raman, Caroline Cvetkovic, and Rashid Bashir. A modular approach to the design, fabrication, and characterization of musele-powered biological machines. Nature protocols, 12(3):519-5333 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Sahil Sharma, Aravind S. Lakshminarayanan. and Balaraman Ravindran. Learning repeat: Fine grained action repetition for deep reinforcement learning. In International Conference on Learn- ing Representations, 2017. URL htt.ttttt///peeeeeeeeeee
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. Pmlr, 2014.
Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Abstraction, Re- formulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 24,, 2002 Proceedings 5, pp. 212-223, Springer, 2002.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
12

---

Published as a conference paper at ICLR 2025
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(112):181- 211. 1999.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1109/R00..2022.6881111
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Aga- piou, et al. Strategic attentive writer for learning macro-actions. Advances in neural information processing systems, 29, 2016.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pp. 3540-3549, PMLR, 2017.
Xintong Yang. Ze Ji, Jing Wu, Yu-Kun Lai, Changyun Wei, Guoliang Liu, and Rossitza Setchi. Hierarchical reinforcement learning with universal policies for multistep robotic manipulation. IEEE Transactions on Neural Networks and Learning Systems, 33(9):47774471, 2021.
Haonan Yu, Wei Xu, and Haichao Zhang. Taac: Temporally abstract actor-critic for continuous control. Advances in Neural Information Processing Systems, 34:29021-29033, 2021.
Haichao Zhang, Wei Xu, and Haonan Yu. Generative planning for temporally coordinated explo- ration in reinforcement learning. In International Conference Learning Representations, 2022. URL https::/
net/ / forum? id YZHES8wIdE
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic
options. In International Conference on Learning Representations, 2021. URL https:// openreview nwt. wwwtt/orr/uniiee F jomw
Yang Zhang, Buqing Nie. and Yue Gao. Robust locomotion policy with adaptive lipschitz constraint for legged robots. IEEE Robotics and Automation Letters, 2024.
13

---

Published as a conference paper at ICLR 2025
A PROOF OF THE STATEMENT IN SECTION 4.1
Statement. Given Vs E S,a E A,b E {0, 10.1) based on the two-stage policy described in
Sec. 4.1 the output action replicates the same actions a in repetition dimensions {ilba 0, A |||].
Proof. As described in Sec,
4.1 a (1 b) 2 a + +b a. Thus, for vb, a we can obtain that:
(a
O b) (11 o + +b00
+ (1 b)
= b (ā a- 1 b)
(1I)
bo (1 b) ② (a
0
For Forvi filb, = 0}. we have (a--a-)@ b)]; ( a000-a-000 0, where ( denotes the action
in the i-th dimension. Thus, replicates the same actions in dimensions choosing repetition.
B EXPERIMENT DETAILS
B. ALGORITIM SETTINGS
Table 4: prreee- settings for SDAR algorithm.
Parameter
Setting
Parameter
Setting
Learning rate (iT)
3 10-4
Learning rate (3)
3 10
Learning rate (Q)
1 x 10-3
Learning rate (a)
1 x 10 -3
Optimizer
Adam
Discount factor y
0.99
Batch size
256
Policy delay
2
Soft update r
0.005
Sample number (b)
10
The hyper-parameter settings ar shown in Table. In addition, need tune the target entropies Hs and Hz to improve the efficiency of the mntropy-basedl exploration described in Eq. (T).. In this work, utilize H, -|A], where | denotes the size of the action space, corresponding to the recommendation given in (Haarnoja et al. 2018). Besides, H.g A |A] log 2. where À is hyper-parameter tuned according to the tasks. In this work, we set E [0.4, 0.6). which achieves excellent performance sample efficiency and policy effectiveness.
As illustrated in Sec. 4.1 SDAR is composed of two policy networks and æ based on MLP. this work, design two MLPs with the same structures: (|S) + [A] + 256 256 + |A]) with ReLU activation functions.
SDAR policies for tasks with |A| < are trained through Eq. 8 such as Reacher and LunarLan- derContinnous. Policies in other environments such as HalfCheetah and Humanoid are optimized through Eq. (9) for lower computation costs.
B.2 EXPERIMENT TASKS
The experiment tasks are listed in Table 5 where the action spaces of all tasks are normalized as
[-1, 1] for convenience. All tasks constructed based Gymnasium (Plappert al.) 2018)). The FetchPickandPlac and FetchReach tasks are implemented by GyymnasiumRRobow 2018), where observation and desired goal concatenated the observation in this experiment.
(Plappert et al.)
14

---

Published as a conference paper at ICLR 2025
Table 5: Descriptions of experiment tasks in this work.
Caterogy
Task Name
Obervation Space
Action Space
MountainCarcontinuon
R²
[ 1, 1] 1, 1j²
Classic Control
LumarLandeerCotinuous
R
BipkdalWalker
2221
[ 1, 1,,
HalfCheetah
IR17
1, 1]6
Humanoid
R 376
-1,
Locomotion
Walker2d
F 17
1,
Hopper
R 11
1, 1
Ant
T
-1,
FetchReach
RR 13
- 1, 1]4
Manipulation
Pusher
IR 23
1, 1
Reacher
R 11
-1.1
B.3 BASELINES
(1) SAC (Haarnoja et al. 2018 is famous model-free RL in continuous control domains, which trains policies efficiently with entropy-based exploration strategies. In this work, we utilize SAC implementation and hyper-parameter settings proposed in CleanRL. (Huang et al. 2022
(2) N-Rep forces the agent repeat the actions output by the policy for times, where is hyper-parameter. In this work, this algorithm is implemented based on SAC, where n is tuned in [2, 5] to achieve the balance between persistence of action and diversity.
(3) TempoRL (Biedenkapp et al. 2021 repeats policy actions dynamically in an open-loop manner, using a skip-policy to predict when to make the next decision. In this work, this method is implemented based the official repository
(4) UTE (Lee et al. 2024) is improved based on TempoRL, which performs repetition while incor- porating the estimated uncertainty of the repeated action values. In this work, this method implemented based the official repository
(4) TAAC YY et al., 202I) is closed-loop repetition framework, which determines whether to repeat the previous action each step utilizing switch policy, this work, this experiment is conducted utilizing the official implemenatation" o
B.4 COMPUTATION RESOURCES
In this work, we conduct all experiment utilizing NVIDIA RTX 3090 GPU and Pytorch 2.1 with CUDA 12.2. The training time of our method compared to vanilla SAC is shown in Table
As shown in Table 6. our method requires less training times than vanilla SAC to achieve the same policy performance, because of the higher sample efficiency of SDAR, requiring less training steps in this experiment. In addition, there exist two policy networks 3 and π in the SDAR, both which need to be optimized through gradient descent. However, B and π be optimized in parallel during training, achieving an improved time efficiency, which will be researched in future works.
C NORMALIZATION OF PERFORMANCE
Different tasks can have vastly different reward scales, thus cannot be directly averaged. In order to depict the average episode return and AUC scores across various tasks, compute the normalized normalized score (n-score) of each method, which is widely utilized in prior works Hessel et al.
-Extension-/
15

---

Published as a conference paper at ICLR 2025
Table 6: The computational cost for the training of SAC and SDAR algorithms. Both two methods are trained to achieve the same performance as SAC with sufficient steps.
Steps Cost
Time Cost
Tasks
SAC
SDAR
SAC
SDAR
LunarLander
300K
150K
60min
36min
Bipede/
600K
300K
108min
63min
Walker2d
1.5M
700K
282min
167min
Humanoid
3.0M
1.5M
595min
365min
2017; ; Yu et al. (202T). Take normalized episode return example, given episode return Z, Z-Zo. where Zo and Zi denote the episode return of random
its n-score is calculated as Znorm 7
policies and vanilla DRL policies respectively. The n-score Znorm [0, 1], thus can be averaged across different tasks.
D ADDITIONAL TRAINING CURVES
Additional training curves of each method in additional tasks are illustrated Fig.
F20 part
120 2040
325
(a) Reacher
(b) Hopper
(c) FetchReack
Figure 5: Learning curves SDAR (red) in additional tasks against baseline methods.
16


Reviews:
[{"review_id": "SqGbyHAXRc", "reviewer": "Reviewer_VdNA", "summary": "This paper introduces Spatially Decoupled Action Repetition (SDAR), a framework that allows reinforcement learning (RL) agents to make separate act-or-repeat decisions for each action dimension. The method consists of a two-stage process: first, a selection policy determines if each dimension should repeat, followed by an action policy that generates new actions for those dimensions that chose not to repeat. Experiments across several continuous control tasks demonstrates the framework's performance and sample efficiency over baseline methods.", "questions": "* Can TAAC be regarded as an ablation of the proposed method by treating all action dimensions as a whole for repetition? How about its switch policy?\n* Would the proposed method be able to be applied to on-policy methods like PPO, where better sample efficiency would be more valuable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed method decouples the action dimensions and achieves more flexible and diverse repetition strategies, while maintaining high action persistence.\n* The paper provides a thorough comparison with multiple baseline methods, including SAC, TAAC, and UTE. This context clarifies the advantages of SDAR over both open-loop and closed-loop action repetition approaches.", "weaknesses": "* The method might be limited when the action dimension is high, which might cause computational overheads.\n* The novelty of this paper appears limited, as TAAC has employed a similar two-stage act-or-repeat mechanism; the primary contribution here seems to be the separation of action dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TAGnTOUwp7", "reviewer": "Reviewer_NAN2", "summary": "A version of state-conditioned action-repeat is proposed that also allows\nfor arbitrary subsets of the action dimensions to be repeated, while the rest are\ndirectly controlled by the policy. The paper describes in detail how to learn the\naction-repeat-selection policy to be learned.  To cope with the combinatorial complexity\nissue that arises for high-D action spaces, an importance-based sampling approach is further proposed.\nThe method is evaluated on a reasonably-sized set of continuous action problems, \nranging from low-D to high-D action spaces. The method is shown to be more sample efficient\nthan a reasonably-chosen set of model-free alternatives, including SAC and a variety\nof other action-repeat-learning baselines.  Overall, the proposed method is shown to excel.\nThe learned patterns of action-subset-repeats are also visualized, to help provide\na qualitative understanding of their usage over time by the learned policies.", "questions": "Questions:\n- how does experience replay sampling work when a- will be different?\n  Or is a- explicitly stored with each tuple in the replay buffer?\n- Could the learned \"synergies\" potentially be transferred when learning a new task?\n- Perhaps I missed it, but are the exploration rates for the repetition schema policy, and the regular policy, being annealed?  Might that make sense?\n- Ideally a method would also work when the \"repetition\" substructure could also be discovered even after the action-space was rotated, i.e., imagine all actions spanned all joints, to differing degrees, but the latent structure would still be there. Just a suggestion for something that would be fun (but arguably difficult) to pursue!\n\nMinor feedback:\n- At the end of the introduction, the second and third listed contributions probably belong together, as they\n  describe the same thing. I like the visualization as another minor contribution.\n- The notation might be easier to understand if the repetition schema is called a mask, m, rather than the current \"b\".\n- Is it possible to collapse the two policies to a single deterministic policy pi, after training? \n  This would be of important practical importance.\n- \"This demonstrates the necessity and effectiveness\":  necessity is perhaps too strong;\n  it is effective in terms of speeding learning, but arguably not \"necessary\"\n- Table 2: \"Avearage\"  (typo)\n- lines 479-480: \"rotors\" --> \"joints\"?\n- line 483:  \"Besides,\"  suggest to remove", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n- clean idea, good execution, clearly written\n- a \"relatively simple\" idea, entirely in the good sense, and so likely to have impact\n- will inspire follow-up work;  learning structure in action representations remains\n  very much an underexplored area, compared to learning structured or latent state representations", "weaknesses": "Weaknesses:\n- I suspect that one could find an example where there is a price to be paid for using this approach,\n  as opposed to it always being more sample efficient\n- it is always hard to see if equal time has been spent tuning all the comparison RL algos;\n  with the right setup and high-replay ratios, some of the RL algos can possibly see further improved sample efficiencies\n- it's clear why we should care about learning rate / sample efficiency, but not so clear why\n  we should actually care about action persistence of action fluctuation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X1b8aXZewu", "reviewer": "Reviewer_sXke", "summary": "The paper proposes a novel reinforcement learning algorithm, \"Spatially Decoupled Action Repetition\", which decouples policy and repetition. This philosophy is achieved by learning two policies, selection and action policies. The selection policy outputs which action dimension should be masked and the action policy proposes a new action. The proposed algorithm was evaluated on several control domains, such as classical control, locomotion, manipulation.", "questions": "* Would you be able to give us the intuition as to why the proposed method \"should\" work better than the baselines? Examples or toy-problems would be appreciated.\n* The performance is only 10% better than SAC - many other ensemble-based approaches, such as RED-Q or Dro-Q, might perform much better than the proposed method.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper demonstrates a novel algorithm that decouples repetition and control, which is an interesting idea.\n* The proposed algorithm performs well on a wide range of control domains.", "weaknesses": "I don't think I am fully convinced by the paper.\n1. Overall, I don't think I understand why the proposed decoupling is important. It is an interesting idea that extends the existing action dimension. But why? In continuous control, I believe many challenging problems mandate new actions at every single time step. Even many discrete action problems may require the same. \n2. In other words, what will be the problem class that must require decoupled action repetition? \n3. As a result, I currently think there can be other reasons that the proposed method outperforms the baselines, such as better hyperparameters or larger network sizes. Please note that I am not questioning the integrity of the proposed authors; there can be other reasons that we may not understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)