Action Space Design in Reinforcement Learning for Robot Motor Skills
Authors: Julian Eßer
Gabriel B. Margolis
Oliver Urbann
Sören Kerner
Pulkit Agrawal
Venue: CoRL 2024 (2024)
Keywords: Reinforcement Learning
Action Spaces
Sim-to-Real
TL;DR: We examine action space selection for robots trough the lense of exploration, expressivity, and timing.

Abstract: Practitioners often rely on intuition to select action spaces for learning. The choice can substantially impact final performance even when choosing among configuration-space representations such as joint position, velocity, and torque commands. We examine action space selection considering a wheeled-legged robot, a quadruped robot, and a simulated suite of locomotion, manipulation, and control tasks. 
We analyze the mechanisms by which action space can improve performance and conclude that the action space can influence learning performance substantially in a task-dependent way. Moreover, we find that much of the practical impact of action space selection on learning dynamics can be explained by improved policy initialization and behavior between timesteps.

Action Space Design in Reinforcement Learning for Robot Motor Skills
Julian Eßer"!, Gabriel B. Margolis 22 Oliver Urbann', Sören Kerner', Pulkit Agrawal?
Fraunhofer Institute for Material Flow and Logistics IML
Massachusetts Institute of Technology
Practitioners often rely on intuition to select action spaces for learn- ing. The choice can substantially impact final performance even when choosing among configuration-space representations such as joint position, velocity, and torque commands. We examine action space selection considering a wheeled- legged robot, a quadruped robot, and simulated suite of locomotion, manipula- tion, and control tasks. We analyze the mechanisms by which action space can improve performance and conclude that the action space can influence learning performance substantially
Abstract:
task-dependent way. Moreover, we find that much of the practical impact action space selection on learning dynamics can be ex- plained by improved policy initialization and behavior between timesteps.
Keywords: Reinforcement Learning, Action Spaces, Sim-to-Real
Introduction
Reinforcement learning (RL) is powerful tool for synthesizing robot motor skills. However, prac- titioners must carefully select the action space where learning occurs, decision often guided by intuition. For instance, a wheeled robot might be associated with wheel velocity action space, legged robot with joint positions, and manipulator with Cartesian space targets. For some well- studied tasks, the field has converged on common action spaces. For example, position control action spaces are widely adopted for learning legged locomotion [1, 2, 3J. However, to actuate the robot's motion, these position commands must be converted into torques by a feedback law, both to perform simulation and control the real robot. This raises several questions: What properties of position control make particularly useful for legged locomotion tasks instead of directly learning to act with torques? Is position control helpful for all types of robot tasks, or are there other, poten- tially more effective, action spaces for systems with different dynamics? How might other tasks that have not been extensively studied benefit from different action space design choices?
The historical motivations for considering joint position or velocity control instead of torque differ from their roles action spaces in reinforcement learning. Classical systems aim for precise regula- tion simplify and stabilize high-level control tasks. In contrast, RL aims to maximize cumulative rewards over time, which might involve deliberately altering position and velocity tracking errors to achieve the overall objective. The key point raised originally by Hwangbo et al. [1] is that the position target in RL is state-indexed and not time-indexed as typical in non-learning approaches. Still, a difference often arises in performance among action spaces due to their impact on the system dynamics and the learning dynamics. Just as PD controllers support high-bandwidth control beneath slower model- optimization [4, 5], learned policy might benefit from high-frequency low- level control loop, achieving better performance a lower frequency. On the other hand, the action space may also experience benefits in RL that are separate from those in model-based control. For example, the action space can alter the policy's exploratory behavior or architecture-relate biases.
"Denotes equal contribution. julian.cssere.a iml. fraunhofer de, gmwrgommit..dd
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Default Action Space
Task
Position
Delta Position
Torque
AllegroHand
Ant
Anymal
Anyrain
Mics
Ral1Belance
Cartpole
FrankaCabinet
Humanoid
(a) evoBOT
(b) Gol
ShadowHand
Figure I: Diverse action spaces in the OmnilsaacCymmnns suite (left) are present across multiple
robotic tasks. We aim to guide practitioners in selecting action space for new tasks by analyzing subset of the suite (highlighted) and the evoBOT and Gol platforms (right) as a case study.
Contributions
Through this lens, we examine the contribution of action space selection on the Unitree Gol quadruped, the evoBOT hybrid robot [6], and the OmmilsuacGymEnvs task suite [7]. We pose four novel questions that haven't been explored in prior work:
1. How does the choice of action space impact learning systems with distinct dynamics? We find that even for the same locomotion objective, two robots with distinct dynamics have opposite trends with regard to action space selection. Section 4.1
2. Is action space selection mostly about tuning exploration? We find that for some tasks, the performance gap between action spaces can be mostly recovered by intervening in the initial exploration behavior, while for others, this aspect is less influential. Section 4.2
3. Is action space selection mostly about expressive capacity? We find that despite a signifi- cant gap in reinforcement learning performance between action spaces, performant policies can be re-expressed all those spaces with nearly equal performance. Section 4.3
4. How important is the policy behavior between timesteps? We evaluate the impact of high- level and low-level control frequency across action space representations and find that the tuning of these parameters is coupled. Section 4.4
2 Related Work
Action Spaces Robots operate within a physics-defined action space, where for instance elec- tric actuators use current track torque via nigh-frequency control loop [4]. While reinforce- ment learning policies can directly output torques [8, 9]. many studies suggest alternative action spaces such joint position [10, 2, 3], joint velocity [1 6). or task-space setpoints [12], which transformed to torques through feedback laws. Research indicates that action space choices signifi- cantly affect robot learning in various contexts, including character animation [13, 14], manipulation [15, 16]. and flying robots [17]. While position control is often favored, some studies suggest joint velocity may be more effective for specific tasks. The OmnilsaneCyymEany suite [7) underscores the complexity of action space implementation, featuring balanced mix action types. Our work focuses on configuration-spate action spaces due their relevance in sim-to-real RL.
Inductive Bias through Action Space Design Action space selection serves as an inductive bias in reinforcement learning, influencing the policy's 's hypothesis space before learning starts [14]. This can also be viewed as form of environment shaping [18] or a method of policy initialization [19].
Temporal Aspects in Action Space Design Parameters like control frequency can impact learning as well; lower frequencies may simplify learning by shortening trajectories [20]. Additionally, gain choices for setpoint tracking influence performance and sim-to-real transfer [21, 22, 23]. Techniques like action filtering [24] and chunking [25] also enhance efficiency in long-horizon tasks.

---

A
RL
RL
/ttrl
Policy Network
Linoar
Ctrl
q
S
Policy Network
PD Controller
4
C2
+
MLP
a
C3
Accummaation
C
5,
Random Roloots
Randons Rooouuss
Cross-action space initlarizatinn loss
Figure 2: Action Space Illustrative Diagram. A. Depicts the classical decomposition of the pol- icy network and low-level controller. B. Shows our generalized parameterization of action spaces, where the low-level controller acts linear layer when the learning frequency matches the control frequency. C. Suggests viewing differences among action spaces terms of policy architecture and initialization, proposing pre-training action space replicate another's exploration.
3 Method
3.1 A Generalized Parameterization Action Space
In generalized form for RL, the transformation T from the policy output a, state s,, and historical information h, -==jo.r-11 aş into motor torque commands 7, be expressed
4 T( h,)
This formulation includes task-space action spaces as well as configuration-space action spaces. Here we focus on configuration space control where actions are expressed terms of the joint position q, velocity 4, and torque G. Table lists a set common action spaces that fall this category. If the command update and torque update operate the same frequency (i.e. 5Ctrl R combine them to express any of these action spaces linear mapping from the state and action to torque. This leads to the following general form for configuration- space action spaces, which is special case of the formulation above:
= (; h,), (a, = c1q, + c2q, + c3a, c/h,-
We enumerate action spaces represented by this parameterization Table 1. Each action space is simply a linear function of the state and action, independentlyly for each joint. Under this param- eterization, we can see some simple relationships among the action spaces that may be unintuitive. Delta position control with one-step integrator equivalent torque control with damping term. Delta velocity control with one-step integrator and torque control are also equivalent.
Figure illustrates how the parameters [c1,02, C2,C,,,,Cc C3, ca] may be interpreted as a final linear layer (.). of the policy network T(s,). The choice among action spaces corresponds to a choice manual initialization for these policy weights. This interpretation encourages us consider whether the benefit of the action space is architectural, i.c. allowing the policy to learn parameters of this form results in better performance, initializaation-ellate i.e. proper initialization of the network parameters seeds good performance.

---

Action Space
Command Update
Torque Update
lei
c4|
Position
des
f- a;
t- (p(gte qi kdq
Kp
Ka
Kp
0]
Delta Position (MS)
des
+ 4, 1-1
des
f- Kp Kp(gtes
qt) Kgd.
Kp
Ka
0
Kpl
Delta Position (OS)
des
a, 4
Kp kp(gfes
qi) kgq
(0
Ka
Kp
0]
Velocity
q a,
4 x Ka( 4
j0
Ka
0]
Delta Velocity (MS)
des
4- a, + des
4 +- (a(á 4,)
[0
K₂
0
Kg)
Delta Velocity (OS)
des
a, 4
G + Ka 4.)
0
Ka
0]
Torque
4 a,
00
0]
Table I: Seven special cases for the generalized parametrization of action spaces. The command update relates the policy output a, the desired state (q q or G)). The torque update relates the command to the desired state. MS = Multi-step, i.e. the action is applied as delta to the previous desired state (e.g. 9. which is equal the sum of all previous actions h, OS One-step, i.c. the action is applied as delta to the current state.
3.2 Temporal Aspects of Action Space Design
The generalized parameterization of action spaces indicates they are equivalent to linear func- tion of state and policy output (see Fig. 2). However, this does not account for policy behavior between timesteps when the linear function operates at a higher frequency than policy evaluation, which is common in real systems. particularly in tasks where torque control is less important. For instance, well-tuned low-level po- sition controller mitigate the need for accurate torque modeling. In force-sensitive applications like locomotion, successful methods achieve low torque sim-to-real gap through direct drives or maming--bssed system identification. For further analysis, we denote the following frequencies:
modules can help bridge the sim-to-real gap,
Physics Frequency Phhyy)): The frequency of the physics step simulation, ideally as high as possible, but balanced against computation speed. This frequency is crucial for simulating dynamics accurately while managing resource constraints in real systems.
Control Frequency (fCtrl); The frequency of torque updates (see Table 1). determines how fre- quently actions can change. A lower control frequency can limit the variability of action sequences but may enhance learning stability by shortening trajectories, thus simplifying credit attribution dur- ing training. We adopt the methodology from [20] to evaluate different control frequencies while adjusting the training parameters to keep the effective discount factor and batch size constant.
Learning Frequency (RRL): The frequency of command updates (see Table 1), which, when lower than the control frequency, allows the same action persist several timesteps. This can improve credit attribution and speed up learning by reducing the number of policy evaluations, even torque updates occur multiple times based on local state information.
3.3 Experimental Setup
Policy Training We utilize the Proximal Policy Optimization (PPO) [26] reinforcement learning algorithm, known for its effectiveness in sim-to-real motor control [1, 27]. Details on policy training and network architecture are provided the appendix. We train on five continuous control tasks, including three from the OnmilsaasGymEEns suite (BallBalance, Cartpole, FrankaCabinet) [7] and two custom locomotion tasks for evoBOT and Gol. The main focus is on evoBOT and Gol, where the reward combines velocity tracking and regularization terms. Training parameters and reward formulations adapted from previous works [3, 6). with all terms independent of action represen- tations for fair comparison. Simulation is conducted using NVIDIA Isaac Sim.

---

10
15
20
30
Gye
10
15
20
30
01-90
0.984900
000:0000
000:5000
00000000000000000000000000000000
005
00000000000000000000000000000000
0322000
C1-
01
6-- 3.0
0545000
0.21000
1,20:000
000000000000000
00000
0:000000
0.544000
0.362001
1.0
0.99000
0.5060000
065-001
0.01000
085+0.01
40
0.5
0.1160011
0633000
00110011
40
C3- 0.5
00440000
0.991001
000:300
0.711:012
03:000
047:0000
000:1002
00020000
0511000
C1-000
0.910001
0,81:000
000:006
02/10.05
<1-30
0.8956000
0512901
0.5:5001
DS5+00000
C1- 30
c2- 3.0
0.171001
0.374091
0:::0000
0.436009
41-30 30
C--
2.0
0,50100000
0.531000
0.5520011
05571001
C1- 20
02- 2.0
0.53:1001
0.64::000
060:0000
0.51:000
0.5110000
0.554000
0:57-0011
06220002
c -- 1.0
0.9610001
0.99:0011
059:0011
0 0.0000000
0:1100001
0.540001
00522001
0.0000001
<]=
C2-
0,97:001
027-002
0.0610.5
Q--20
<2-40
0U89500000
0500000
022+000
C1-
C2-00
0.96200000
0,76:1005
021:000
0,3050155
q1=20
Q2:30
05110001
0.53:900
0.50:000
0.500000
0.161000
0,342000
044:0006
0.0500004
<1-20 20
Ppm 2.0
0.5160000
0.5310011
0:5-2001
0.9900003
20
C-- 2.0
0.7740011
0.65:0011
024:0000
0.7610000
G- 1.0
05240.001
0.5:1001
0:58:001
06310000
20
C- 1.0
0.951000
10000001
0960000
053:0000
05240000
0:292001
002:0000
1003000
0,94:0002
028:005
<1-20 20
<2-00
0.3300000
0.9910002
0.59:2002
0.
<1- 20
C2-00
0.05:10011
0.71:0002
0:5::009
0:521:005
4p-10
Q- 3.0
0.5410.01
0,55:0000
0:55:000
0.57200011
C1-10 Q-330 3.0
0.0110000
0,32:0.17
01000018
0.0010022
0:3340.01
0.5420.00
0.54:0000
0.5910.01
<1=10
0.70::0.4
066-0005
0.6760001
<-
0.5260.00
0.420.01
0:5620022
0.6100000
<1-
C2- 10
0.551001
0.82:0.03
073:0002
0.624:00
41-10
Q-11
0.3300011
0.5710.00
0.0::000
0.010001
4p-10
Q-405
0.96:10:04
0,74:00.11
06420002
0.5110005
41-10
c2-0.0
0.55:60.00
0000:00:1
062::0.11
0.6320.022
<r-10
C2-0.0
0.7230002
0.62:0.01
0,57:0011
0.43:0006
0.7110.01
0.78:0000
0,44:001
0.9410000
0.210002
0.59:0.02
061-007
0.4510044
9-0
2-220
0.6:00.2
Q,40
Q--20
0.59:0002
0.4:0005
91=0
1-- 1.0
0.94200.11
0.9:10000
100000.00
0.9910.01
4140
22-1.0
0.56:10003
0.59:10.04
0.61:005
0.542:00
4-0
015
1.0060.000
0:99:0.01
1.0000000
0.972:003
C2-05 0.5
0.61:00000
0,53::0.03
0.50::001
0.4810012
0.90:60.01
1,00:0.00
0.99:0011
0.910007
0.5420002
0,40:0000
0.39
0.4360.55
(a) cvoBOT
(b) Gol
Figure Full sweep over the generalized action spaces with 100 different parameterizations (c4 0). We report the mean and standard error of the normalized return across three seeds for each table entry, each representing one unique combination of the generalized action space.
Robot Platforms To evaluate the choice of action spaces, we use two robots in our case study: The Unitree Gol and the evoBOT. The Unitree Go1 is quadrupedal rohot with 12 electric actuators standing 40 cm tall and weighing kg. The evoBOT is dynamically unstable robot based on the principle of compound inverted pendulum containing 8 electric actuators, standing 80 cm high (without the arms) and total weight of 50
Sim-to-Real To transfer the trained policies from simulation to the real world, we use standard domain randomization techniques. To model uncertainty in the robot's sensing and actuation, we add Gaussian noise to the sensor observations and implement a lag model to simulate processing delays. To make the robot robust to varied terrains and disturbances, we also randomize the ground friction randomization and randomly push the robot during training. The details of the domain randomization parameters and ranges for each task are given in Section 6.
4 Results
In this section, we evaluate our research questions through experiments in simulation. To verify that our environment is reflective of sim-to-real scenario, we also transferrd the highest- learned policies from Section 4.1 to the Gol and evoBOT robots in the real world.
4.1 How does the choice of action space impact learning?
We train policies for the Gol and evoBOT covering 100 sets of parameters C2, C3 in the general- ized action space parameterization (Section 3). All policies are trained in simulation. The results are reported in Fig. 3. Our aim in this experiment was not find the most optimal action space for each task, but to broadly characterize the trends in the action space parameters for two robots with differ- ent dynamics. To accomplish this, it wasn't 't important to use the most efficient search method (e.g. bayesian optimization) but instead cover wide variety of action spaces to expose trends. Therefore, we opted for a simple grid search. The ranges for the grid search were chosen to span the settings provided in SannisaacCyymEnv as the tuned default action spaces for similar robots.
First, we observe that the optimal action space can strongly depend on the learning task. Even for locomotion environments, the Gol and evoBOT have almost opposite trends in terms of how adjusting the action space parameters impacts their performance.
5

---

Position Control
Torque Control
Joint Position
Joint Velocity
evoBOT
Torqua
Joint Position
Joint Volocity
Go1
Tonque
Timeseries
Histogram
Timeseries
Histogram
opposing action space with pre-trained initialization
Figure 4: Random rollout behavior different action spaces. For each action space and robot, the left column plots the joint positions, velocities, and torques during the rollout of randomly initialized policy, while the right column plots the histogram of the same states. The different colored lines in each plot correspond to different joints of the robot.
Second, that the learning dynamics be very sensitive the choice of action space pa- rameters. For evoBOT, introducing nonzero e₁ corresponding to position control loop substantially weakens performance, does increasing 22 corresponding to the damping parameter. For Gol, the robot cannot achieve a high reward without a sufficiently high C1. The difference is due to differ- ence each robot's natural dynamics. The Gol benefits from a bias towards a nominal standing pose since learning gravity compensation is a key part of its task and the optimal strategy involves oscillating the joints around their nominal pose. The evoBOT on the other hand requires continuous rotations of the wheel locomote so the bias towards nominal position introduced by nonzero cl harm performance.
4.2 Is action space selection mostly about tuning initial exploration?
Different action spaces yield different initial exploratory behavior, illustrated in Fig. 4, which plots the joint position, velocity, and acceleration during random rollouts. The initial behavior is thought to be impactful for learning locomotion, the position control action space results in standing behavior under common network initializations while commanding torques directly will bias the robot to wriggle and fall. To experi- mentally evaluate the impact of the initialization, we propose an intervention to alter the initial explo- ration of the policy. We randomly initialize teacher policy nt 21.02--..44 and student policy zgū -2----43 The student policy trained using supervised learning imitate the randomly initialized teacher's torque outputs but with different internal action space: ming, [[[ (1.22..11134
tasks; for example, Hwangbo al. [1] observed that for quadruped
-1,22.93.4 (s)) T0y for T f1.02.033-4 (s) obtained from rolling the teacher in the training environment. This will incentivize the rollouts of the student operating in one action space match the initial exploratory behavior of another. The green highlighted blocks in Fig. 4 show that the random rollouts from the student policy adopt similar characteristics to each teacher.
To evaluate the impact of initial exploration on the dynamics learning, we evaluate four configura- tions in Fig. 5: Torque from Scratch, Torque with PD-like Initialization, Position from Scratch, and Position with Torque-like like Initialization, where X with Y Ylikee initialization means pretrain the X policy to imitate rollouts from a randomly initialized Y policy, then train the X policy with reinforce-

---

Gol
evoBOT
FrankeCabinel
SallBelance
Curliple
0.3
0.3
3.6
0.5
64
0.5
0.4
0.2
02
82
150
150
Tirestop (X201
Timedop 1X209)
Timestop (X207)
Timestes (x12)
Timestop (X301
PD from Scratch
Torque from Scrutch
Tbrque with PD-lkte altistion
ID with lonqueiilu Intialization
Figure 5: Initial exploratory behavior plays major role in the performance of different action spaces. For dashed lines, the policy underwent a pretraining stage where it was trained through imitation learning to match the initial random rollouts sampled from another action space. Mean and standard error of normalized returns are reported across three random seeds.
ment learning. For Gol, torque policy with position-like initialization boosts the final performance close to training with position-controlled policy. Conversely, for evoBOT where the torque action space more performant, pre-initializing PD controller with torque-like like initialization results in gain, although not full recovery of final performance. We also consider the FrankaCabinet, Ball- Balance, and Cartpole tasks from the OmnilsaacGymmiing suite. We find that for some tasks (Gol, evoBOT), the performance gap between action spaces can be partly or fully recovered by interven- ing in the initial exploration behavior, while for others, this aspect is less influential. Surprisingly, FrankaCabinet obtains the highest performance from torque policy with PD-like initialization, suggesting it may benefit from initially exploring like PD policy and later exploring like torque policy. This seems intuitively reasonable since the Franka can benefit from gravity compensation at the start derived from the bias of PD control and later explore different space of movements to open the drawer.
4.3 Is action space selection mostly about expressive capacity?
The complementary view of action spaces being tools for exploration would be that they have dif- ferent expressive capacities due to their different architecture. We find this is not true for policies that run at synchronized frequency, and we can train a policy with any of position, delta position, or torque action space to match high-performing policy trained in another action space using dis- tillation (see Table 2). This is naively true because neural networks are universal function approxi- mators; however, it illustrates that our comparison is devoid of factors that would make the policies impossible imitate, such as varied behavior between timesteps, which we evaluate in Section 4.4.
4.4 How important is the policy behavior between timesteps?
We examine the impact of temporal aspects on policy performance across different action spaces (see Fig. 6). Policies are trained for evoBOT and Gol with control frequencies ranging from 25 to 200 Hz, and their performance is evaluated based on total reward. All experiments maintain a fixed physics frequency of 200 Hz and implement the corresponding RL frequency via frame skips [1, 11,2,44 4, 8). Results indicate that control frequency significantly affects performance, with higher frequencies generally yielding better rewards in torque control mode for evoBOT, while lower frequencies also perform well for Gol.
Additionally, the behavior between physics substeps impacts final policy performance. We analyze scenarios where torques are recomputed at each physics substep (fCul = Phyys versus when they remain fixed until the next RL update (fCul fRL). Our findings suggest that updating torques at the physics level leads to superior performance for most action spaces and tasks, especially in position control, likely due to the effects of stiffness and damping in PD control.

---

Normalized Return
14.
evoBOT
Go1
Teacher
1.00 1.00++0.00
1.00
Position Student
1.00
1.00
Rotum
Volocity Student
1.00
1.03
Torque Student
1.00
0.99
Table 2:
Expressive capacity plays a negligible role in the performance of different action spaces. The best policy from the action space sweep serves the teacher, while student policies im- itate using specific action rep- resentations. Mean and standard error of normalized returns are re- ported across three random seeds.
C4.
Proquency
AL Prequancy
(a) evoBOT
(b) Gol
Figure 6: Temporal behavior of RL frequencies and torque updates significantly influences policy performance across action spaces. Performance is evaluated across three seeds, with mean and standard error of normalized returns.
5 Practical Guidelines for Action Space Selection
While our findings generally support that the choice of action space is specific to the robot and task, we distill some overall guidelines to assist practitioners when selecting action space for new tasks:
1.
When selecting an action space for learning, first consider the robot's dynamics and the types of movements you expect to be required. keeping the joints close a nominal pos- ture requires some torque, position gain may be helpful. the joints require continuous rotation, consider torque action space.
2. The choice of action space has a considerable impact on the initial exploratory behavior of the policy. You can gain intuition for this by visualizing the position, velocity, and torque of random rollouts. Combining the initial behavior of one action space with the representation of another is possible through an initial imitation stage.
3. In general, select the action space gains that provide the most advantageous learning dy- namics for your task. If you need to deploy the final policy using a different set of gains, you can use teacher-student learning to perform the conversion.
4. Policy behavior between timesteps can influence performance, but with mixed trends. It's generally preferable to run the policy at higher frequency and without any frame skip.
6 Limitations and Future Work
Although work identified trends in action space design across multiple tasks, these trends do not fully explain the differences in learning dynamics. For example, initializing the policy mimic the initial exploration of another action space improved performance in several tasks but did not yield identical learning curves, indicating that some factors remain unexplained. While we enhanced our mechanistic understanding of how action space selection affects learning, we could not leverage this understanding to stabilize training or introduce new initialization scheme that consistently improves performance across tasks. This underscores the challenge of creating robust training that less sensitive environment design, action spaces introduce strong, task-specific biases.
Future work could focus on establishing metrics to predict whether an action space suitable for new tasks before training, potentially accelcrating automated environment design, such as using LLMs [28]. Additionally,
important to note that our analysis was restricted the behavior of action spaces in combination with the PPO algorithm. While we believe our conclusions are useful since PPO is widely adopted for sim-to-real tasks, further research is needed explore how these results generalize to other RL algorithms.

---

Acknowledgments
This research has received funding from the Federal Ministry of Education and Research of Ger- many and the state of North Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence (LAMARR22B) and the Junior Research Group DynaFoRo (011S22074). This research was partly supported by Hyundai Motor Company, the MIT-IBM Watson AI Lab, and the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://aiif..rrg).
References
[1] J. Hwangho, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26))eau58872, 2019.
[2] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Boheez and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 10332, 2018.
[3] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via rein- forcement learning, 2022.
[4] G. Bledt, M. J. Powell, B. Katz, Di Carlo, P. M. Wensing, and S. Kim. Mit cheetah 3: Design and control of robust, dynamic quadruped robot. In 2018 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 2245-2252. IEEE, 2018.
[5] D. Kim, Di Carlo, B. Katz, Bledt, and S. Kim. Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control. arXiv preprint 2019.
[6] Klokowski, J. EBer, N. Gramse, B. Pschera, M. Plitt, Feldmeier, S. Bajpai, C. Jestel, N. Bach, O. Urbann, and S. Kerner. evobot design and learning-based control of two- wheeled compound inverted pendulum robot. In 2023 JEEE/RSI International Conference Intelligent Robots and Systems (IROS), pages 10425-10432, 2023.
[7] V. Makoviychuk, Wawrzyniak, Guo, M. Lu, Storey, M. Macklin, Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470 2021.
[8] P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 109,0915 2015.
[9] Schulman, Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arxiv preprint arXiv:/506.02233, 2015.
[10] J. Lee, J. Hwangbo, I.. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47):eabc5986, 2020.
[11] Y.H.. Wu, Z.-. Yu, C.-Y. Li, M.-J He, B. Hua, and Z.-M. Chen. Reinforcement learning in dual-arm trajectory planning for free-floating space robot. Aerospace Science and Technol- ogy, 98: 05565,, 2020.
[12] R. Martin-Martin, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pages 1010-1017. IEEE, 2019.
[13] X. B. Peng and M. Van De Panne. Learning locomotion skills using deeprl: Does the choice of action space matter? In Proceedings of the ACM SICGRAPH/Eungrapienn Symposium Computer Animation, pages 1-13, 2017.

---

[14] D. Reda, T. Tao, and M. van de Panne. Learning to locomote: Understanding how environment design matters for deep reinforcement learning. Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games, pages 1-10, 2020.
[15] P. Varin, I. Grossman, and S. Kuindersma. comparison of action spaces for learning manip- ulation tasks. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6015-6021. IEEE, 2019.
[16] E. Aljalbout, F. Frank, M. Karl, and van der Smagt. On the role of the action space in robot manipulation learning and sim-to-real transfer, 2023.
[17] E. Kaufmann, L. Bauersfeld, and D. Scaramuzza. A benchmark comparison of learned control policies for agile quadrotor flight. In 2022 International Conference on Robotics and Automa- tion (ICRA), pages 10504-10510. IEEE, 2022.
[18] D. Co-Reyes, Sanjeev, G. Berseth, A. Gupta, and Levine. Ecological reinforcement learning. arXiv preprini arXiv:2006. /2478, 2020.
[19] Eßer, N. Bach, C. Jestel, O. Urbann, and Kerner. Guided reinforcement learning: A review and evaluation for efficient and effective real-world robotics [survey]. IEEE Robotics & Automation Magazine, 30(2):67-855 85, 2023.
[20] S. Gangapurwala, L. Campanaro, and I. Havoutis. Learning low-frequency motion control for robust and dynamic robot locomotion. In 2023 IEEE International Conference Robotics and Automation (ICRA), pages 5085-5091. IEEE, 2023.
[21] Z. Xie, X. Da, M. Van de Panne, B. Babich, and A. Garg. Dynamics randomization revisited: A case study for quadrupedal locomotion. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 1955549611 IEEE, 2021.
[22] L. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning, arXiv preprint w:Xiv:2208.07860 2022.
[23] J. Siekmann, Y. Godse, A. Fern, and J. Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7309-7315. IEEF, 2021.
[24] S. Mysore, B. Mabsout, R. Mancuso, and K. Saenko. Regularizing action policies for smooth control with reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1810-1816. IEEE, 2021.
[25] T. Z. Zhao, Kumar, Levine, and C. Finn. Learning fine- tne-ggrined bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[26] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: ha0p::/77004444 2017.
[27] E. Kaufmann, L. Bauersfeld, A. Loquercio, M. Müller, V. Koltun, and D. Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):982-988) 2023.
[28] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310. 1e211..293111 12931, 2023.
[29] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. 91-100.PMLR, 2022.
[30] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. Conference on Robot Learning, 2022.
10

---

A Training Details
Ilyerpparaneter Value
timesteps per rollout
48
A minihatches per epoch
mini-batch
32768
discount parameter
0.95
leamiing rate adaptive [29)
threshold 0.008
# workers
# enviromments per worker
4096
Table 3: Hyperparameters used during training with PPO [26].
Hyperparameter
Value
Hyperparameter
Hidden dimensions
252,
Hidden layer dimensions [128,64] Activation function
Activation function
elu
elu
Table 4:
Neural network architecture for the Gol quadruped locomotion task.
Table 5: Neural network architecture for the
evoBOT locomotion task.
Term
Form
Coeflicient
Lin vel tracking (xy)
expp
1ay
virl //,,y
1.0
vel tracking (yaw)
exp
(00 2ogni /6mc)
0.5
Joint acceleration
1q.1
-2.5e-7
torque
1- 9-1
Delta pos
1q.
Lin vel (2)
v.
1.0
Ang vel pitch)
Jen, P
0.05
Power
-
Foot elearance
Eieee (h. (b. leat)
Af and Pocue (gard
20.0
Base height
80.0
Base orientation
lR,
-.2
Foot contact schedule (force)
Exsel! Camd expp (opppppfp pran (amp//aa) (ar)
0.2
Foot contact schedule (velocity)
Etool EtoolCSuud (Crul gime (p' Pastoal
grmde_ .0] expp - loppep// //,))
0.2
Raibert beuristic
-5.0
Table 6: Reward for quadruped locomotion (adapted from [30] to eliminate dependence action representation, e.g. action change penalty is replaced by other smoothness terms).
Term
Form
Coefficient
Lin vel tracking (K)
expl Nay / /aay) exp[-
vel tracking (yaw)
(c0,, (xp,-((0,,
or /2/ //a.))
0.5
acveleration
Della lonque
1s,
1-
le
Table 7: Reward for evoBOT locomotion (adapted from [6] for improved velocity tracking and eliminate dependence action representation, e.g. action change penalty replaced by other smoothness terms).

---

Term
Dim
Scale
Noise Sid
Scale
Velocity command
2.0
0.0
Gravity
3
1.0
0.01
Linear x velocity
2.0
0.0003
Velocity command
3
1.0
Angular ver-velocitt
0.25
0.0005
Joint positions
Pitch ungle
0.5
0.005
velocities
Joint positions
0.5
0.001
Clock variable
1.0
Joint velociries
0.05
0.01
Table 8: Observations for the Gol quadruped locomotion task.
Table 9: Observations for the evoBOT locomo- tion task.
Term
Min
Max
Term
Min
Max
Ground friction
0.4
2.0
Ground friction
0.5
1.0
Ground restitution
0.0
1.0
Ground restitution
0.0
1.0
Initial joint velocities
-.11
0.1
Initial joint velocities
-0.1
0.1
Random pushes (every 10 seconds, sn/s)
-0.5
0.5
Random pushes (every seconds, mus)
-0.5
0.5
Table 10: Domain randomization parameters for the Gol quadruped locomotion task.
Table 11: Domain randomization parameters for the evoBOT locomotion task.
12


Reviews:
[{"review_id": "hrIiuqXyZj", "reviewer": "Reviewer_FMNG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper assesses the role of action space design on the performance of robotic control policies trained using reinforcement learning. The work aims to understand if and how the choice of action space affects learning dynamics. Through a variety of simulated experiments, the work assesses the extent to which the action space affects initial exploration, a model’s expressive capacity, and the role of multi-timestep action formulations. The work concludes that the optimal action space is strongly dependent on the particular task, and that this can only be partially explained by differences in exploration.", "summary_of_recommendation": "This paper is on an interesting and relevant topic, with nicely designed experiments, but poor clarity as the paper stands. Additionally, the action space scope is somewhat narrow, ignoring a key choice between end effector / leg space and joint space control.  The main reasons for my recommendation are (a) clarity which could be addressed during rebuttal and (b) narrow action space scope. Given there is no algorithmic, dataset, or system design novelty and the experimental results are in simulation, I wish to see a deeper exploration of action spaces and more insight, for example by including in this work the metric proposed in the limitations section.  **Post-rebuttal update: The revisions provided during rebuttal substantially improved the paper. As a result I am revising my recommendation to weak accept**", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Related work \n* How does this work differ to prior work on action space design?\n\nMethod\n* Fig 2. h_t is undefined\n* How do c1, c2, c3, and c4 in fig 2 relate to [c1, c2, c3, c4] in table 1?\n\nExperimental setup\n* What are the tasks that are being evaluated? What is the maximum and minimum reward per task? \n* What are the policy training details - network architecture, number of training steps, number of training runs\n* Sim-to-real - please give specific details. What parameters were domain randomized, what were the ranges, how much noise was added to the observations, how much latency?\n\nResults\n* 4.1\n  * Line 139: What does a “full sweep” mean? I see that c1, c2 and c3 are varied but c4 appears constant. Is that the case and what was it’s value?\n  * Was training in simulation or the real world?\n  * What are your hypotheses about why there is such a difference between the Go1 and evoBOT embodiments?\n* Fig 3: What does “maximum reward mean”? What is the maximum possible score?  Is the reported score a mean value? Over how many reported training runs? What was the variance of these results (RL is known to exhibit high variance)?\n* Table 2: Is the reported score a mean value? Over how many training runs?\n* Fig 5: What do the different color lines represent?\n* Fig 6. Is there something wrong with the y axis of both charts? The value appears to be very low compared to Fig 3 and I assume this is the same task.\n* How does the supplementary video relate to the paper? There are no real robot experiments in the paper and the video is of an evoBot wheeling around a space\n\nMinor\n* Line 31: This sentence doesn’t make sense\n* Line 157: The teacher and student policy presumably have different action spaces, it would be helpful to make this explicit.\n* Line 160: There is no bottom row to the referred figure\n* Line 161: Should the be Fig 5 instead of Fig 4\n* Line 196: effect → affect", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ts6KyoumII", "reviewer": "Reviewer_15bq", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies the impact of action space in RL for robot control. It introduces a general reparameterization of the action space,  enabling the representation of common action space choices as a linear mapping from state and action to torque. Through experiments conducted in both simulated and real-world robots, the paper shows that action space design affects control performance in a task-dependent manner. Additionally, it demonstrates that performance difference is largely due to policy initialization and the interplay of control frequency and torque update frequency.", "summary_of_recommendation": "**Updated Post-Rebuttal**: The paper addresses an important topic, but the results are inconclusive despite that the authors have made some improvements during the rebuttal.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Table 2 caption mentions an “optimal teacher policy”: how was this policy obtained?\n- As shown in Fig. 1, none of the three simulation environments use position-based action space. What’s the rationale of picking these environments?\n- Line 160 refers to “The bottom row of Fig. 4”,  is this plot missing or incorrectly referenced?\n- Line 175, mentions “exploring like a PD policy and later exploring like a torque policy”.\n    - can you elaborate on what it means to be exploring like a PD policy, or exploring a torque policy?\n    - In PPO, exploration typically reduce gradually. Is there more evidence to support the claim that the observed benefits come from exploration?\n- In Sec. 4.4, Line 200 mentions performance increase from more frequent torque update. I wonder if this increase can be attributed to a more accurate physics simulation from the faster update, rather than the action space itself.\n- I’m having trouble understanding the x-axis label “Decimation” in Fig. 6. Why is it not simply labeled “frame skip”?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bccvfvzGBc", "reviewer": "Reviewer_29AA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper discusses the impact of action space selection on the performance of reinforcement learning for robot motor skills. Through a unified formulation, it explores the mechanisms by which action space influences performance, notably during the initial exploration phase and when using different control frequencies.", "summary_of_recommendation": "I recommend a weak rejection of the paper due to several issues that need to be addressed, including a lack of clarity on certain points and in the presentation of results. Overall, while the paper makes a good contribution by exploring the impact of action spaces, it needs further refinement before it can be considered for publication. **post-rebuttal update: I updated my score to weak accept but some additional points need to be addressed**", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- In table 1, although it is clear how one-step action space can be retrieved, it is not clear for multi-step. It also seems that multi-step is also not clearly defined, but I assume it is when d > 1?\n- I assume that every time that the generalized framework is used, only synchronous policy/low-level control is considered?\n\n- It is not clear how the full sweep of Figure 3 was done. Was there only one run per entry?\n Is it simulation only? Why choose grid search (it is usually an inefficient way to search)?\n How was the search space chosen?\n Please report uncertainty when possible.\n\n- For section 4.2, are the action space normalized? what is the original probability distribution used by PPO? (what happens if you replace PPO by SAC which uses a squashed Gaussian that should be less affected by action space limits)?\n\n- For figure 5, random rollout corresponds to rollouts with randomly initialized policy?\n\n- Figure 6/ section 4.4: only one run per datapoint? It would be good to normalize the performance (using random policy and max performance), because a difference of 0.002 in reward doesn't tell much if the performance changed a lot or not.\nAgain, please report uncertainty when possible.\n\n- minor: typo in Fig 6: \"no_udate\" -> \"no_update\"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths
- Highly relevant topic for all robot RL researchers
- Strong experimental design
- Nice discussion and limitations sections

Weaknesses
- Many details unclear
- Unclear if experiments report multiple runs
- Evaluating on multiple RL algorithms would have been nice to show generality
- Scope of action space a bit limited and also the related discussion needs to be improved
- Real robot experiments would strengthen the paper
- Unclear how to apply lessons learned in practice

# After Discussion Phase
All reviewers highly appreciated the additional information and improved experiments. Overall that improved the paper significantly. Reviewer 29AA still has 2 concerns that are straightforward to fix. Reviewer FMNG still maintains 2 suggestions on how to further strengthen the paper. While I agree with Reviewer 15bq that the results aren't as conclusive as one might have hoped, this is unfortunately typical for RL (whether a methods performs well or not is often highly task specific), and I believe the insights and new practical guidelines are a valuable contribution.
Decision: Accept