Action Space Design in Reinforcement Learning for Robot Motor Skills

Authors: julian  eßer
gabriel b margolis
oliver  urbann
sören  kerner
pulkit  agrawal
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:11:15

Abstract: Practitioners often rely on intuition to select action spaces for learning. The choice can substantially impact final performance even when choosing among configuration-space representations such as joint position, velocity, and torque commands. We examine action space selection considering a wheeled-legged robot, a quadruped robot, and a simulated suite of locomotion, manipulation, and control tasks. 
We analyze the mechanisms by which action space can improve performance and conclude that the action space can influence learning performance substantially in a task-dependent way. Moreover, we find that much of the practical impact of action space selection on learning dynamics can be explained by improved policy initialization and behavior between timesteps.

Main paper: 
 introduction: 
 Reinforcement learning (RL) is a powerful tool for synthesizing robot motor skills. However, practitioners must carefully select the action space where learning occurs, a decision often guided by intuition. For instance, a wheeled robot might be associated with a wheel velocity action space, a legged robot with joint positions, and a manipulator with Cartesian space targets. For some wellstudied tasks, the field has converged on common action spaces. For example, position control action spaces are widely adopted for learning legged locomotion [1, 2, 3] . However, to actuate the robot's motion, these position commands must be converted into torques by a feedback law, both to perform simulation and control the real robot. This raises several questions: What properties of position control make it particularly useful for legged locomotion tasks instead of directly learning to act with torques? Is position control helpful for all types of robot tasks, or are there other, potentially more effective, action spaces for systems with different dynamics? How might other tasks that have not been extensively studied benefit from different action space design choices?The historical motivations for considering joint position or velocity control instead of torque differ from their roles as action spaces in reinforcement learning. Classical systems aim for precise regulation to simplify and stabilize high-level control tasks. In contrast, RL aims to maximize cumulative rewards over time, which might involve deliberately altering position and velocity tracking errors to achieve the overall objective. The key point raised originally by Hwangbo et al. [1] is that the position target in RL is state-indexed and not time-indexed as typical in non-learning approaches. Still, a difference often arises in performance among action spaces due to their impact on the system dynamics and the learning dynamics. Just as PD controllers support high-bandwidth control beneath a slower model-based optimization [4, 5] , a learned policy might benefit from a high-frequency lowlevel control loop, achieving better performance at a lower frequency. On the other hand, the action space may also experience benefits in RL that are separate from those in model-based control. For example, the action space can alter the policy's exploratory behavior or architecture-related biases.Position Delta Position TorqueAllegroHand ✓ Ant ✓ Anymal ✓ AnymalTerrain ✓ BallBalance ✓ Cartpole ✓ FrankaCabinet ✓ Humanoid ✓ ShadowHand ✓ (a) evoBOT (b) Go1Figure 1 : Diverse action spaces in the OmniIsaacGymEnvs suite (left) are present across multiple robotic tasks. We aim to guide practitioners in selecting action space for new tasks by analyzing a subset of the suite (highlighted) and the evoBOT and Go1 platforms (right) as a case study.Contributions Through this lens, we examine the contribution of action space selection on the Unitree Go1 quadruped, the evoBOT hybrid robot [6] , and the OmniIsaacGymEnvs task suite [7] .We pose four novel questions that haven't been explored in prior work:1. How does the choice of action space impact learning in systems with distinct dynamics?We find that even for the same locomotion objective, two robots with distinct dynamics have opposite trends with regard to action space selection. Section 4.1 2. Is action space selection mostly about tuning exploration? We find that for some tasks, the performance gap between action spaces can be mostly recovered by intervening in the initial exploration behavior, while for others, this aspect is less influential. Section 4.23. Is action space selection mostly about expressive capacity? We find that despite a significant gap in reinforcement learning performance between action spaces, performant policies can be re-expressed in all those spaces with nearly equal performance. Section 4.34. How important is the policy behavior between timesteps? We evaluate the impact of highlevel and low-level control frequency across action space representations and find that the tuning of these parameters is coupled. Section 4.4 
related work: 
 Action Spaces Robots operate within a physics-defined action space, where for instance electric actuators use current to track torque via a high-frequency control loop [4] . While reinforcement learning policies can directly output torques [8, 9] , many studies suggest alternative action spaces such as joint position [10, 2, 3] , joint velocity [11, 6] , or task-space setpoints [12] , which are transformed to torques through feedback laws. Research indicates that action space choices significantly affect robot learning in various contexts, including character animation [13, 14] , manipulation [15, 16] , and flying robots [17] . While position control is often favored, some studies suggest joint velocity may be more effective for specific tasks. The OmniIsaacGymEnvs suite [7] underscores the complexity of action space implementation, featuring a balanced mix of action types. Our work focuses on configuration-space action spaces due to their relevance in sim-to-real RL.Inductive Bias through Action Space Design Action space selection serves as an inductive bias in reinforcement learning, influencing the policy's hypothesis space before learning starts [14] . This can also be viewed as a form of environment shaping [18] or a method of policy initialization [19] .Temporal Aspects in Action Space Design Parameters like control frequency can impact learning as well; lower frequencies may simplify learning by shortening trajectories [20] . Additionally, gain choices for setpoint tracking influence performance and sim-to-real transfer [21, 22, 23] . Techniques like action filtering [24] and chunking [25] also can enhance efficiency in long-horizon tasks. 3 Method 
a generalized parameterization of action space: 
 In a generalized form for RL, the transformation T from the policy output a t , state s t , and historical information h t := ∑ k=[0..t-1] a k into motor torque commands τ t can be expressed as:τ t = T(a t , s t , h t )This formulation includes task-space action spaces as well as configuration-space action spaces. Here we focus on configuration space control where actions are expressed in terms of the joint position q t , velocity qt , and torque τ t . Table 1 lists a set of common action spaces that fall in this category. If the command update and torque update operate at the same frequency (i.e. f Ctrl = f RL ), we can combine them to express any of these action spaces as a linear mapping from the state and action to a torque. This leads us to the following general form for configuration-space action spaces, which is a special case of the formulation above:τ t = T J (a t , s t , h t ), T J (a t , s t , h t ) = c 1 q t + c 2 qt + c 3 a t + c 4 h t .We enumerate action spaces represented by this parameterization in Table 1 . Each action space is simply a linear function of the state and action, independentlyly for each joint. Under this parameterization, we can see some simple relationships among the action spaces that may be unintuitive. Delta position control with a one-step integrator is equivalent to torque control with a damping term. Delta velocity control with a one-step integrator and torque control are also equivalent.Figure 2 illustrates how the parameters [c 1 , c 2 , c 3 , c 4 ] may be interpreted as a final linear layer φ A (•) of the policy network π(s t ). The choice among action spaces corresponds to a choice of manual initialization for these policy weights. This interpretation encourages us to consider whether the benefit of the action space is architectural, i.e. allowing the policy to learn parameters of this form results in better performance, or initialization-related, i.e. proper initialization of the network parameters seeds good performance.Torque Update[c 1 c 2 c 3 c 4 ] Position q des t ← a t τ t ← K p (q des t -q t ) -K d qt [-K p -K d K p 0] Delta Position (MS) q des t ← a t + q des t-1 τ t ← K p (q des t -q t ) -K d qt [-K p -K d 0 K p ] Delta Position (OS) q des t ← a t + q t τ t ← K p (q des t -q t ) -K d qt [0 -K d K p 0] Velocity qdes t ← a t τ t ← K d ( qdes t -qt ) [0 -K d K d 0] Delta Velocity (MS) qdes t ← a t + qdes t-1 τ t ← K d ( qdes t -qt ) [0 -K d 0 K d ] Delta Velocity (OS) qdes t ← a t + qt τ t ← K d ( qdes t -qt ) [0 0 K d 0] Torque τ t ← a t - [0 0 1 0]Table 1 : Seven special cases for the generalized parametrization of action spaces. The command update relates the policy output a t to the desired state (q des t , qdes t , or τ t ). The torque update relates the command to the desired state. MS = Multi-step, i.e. the action is applied as a delta to the previous desired state (e.g. q des t-1 ) which is equal to the sum of all previous actions h t ; OS = One-step, i.e. the action is applied as a delta to the current state.The generalized parameterization of action spaces indicates they are equivalent to a linear function of state and policy output (see Fig. 2 ). However, this does not account for policy behavior between timesteps when the linear function operates at a higher frequency than policy evaluation, which is common in real systems. Higher-frequency modules can help bridge the sim-to-real gap, particularly in tasks where torque control is less important. For instance, a well-tuned low-level position controller can mitigate the need for accurate torque modeling. In force-sensitive applications like locomotion, successful methods achieve a low torque sim-to-real gap through direct drives or learning-based system identification. For further analysis, we denote the following frequencies:Physics Frequency ( f Phys ): The frequency of the physics step in simulation, ideally as high as possible, but balanced against computation speed. This frequency is crucial for simulating dynamics accurately while managing resource constraints in real systems.Control Frequency ( f Ctrl ): The frequency of torque updates (see Table 1 ), determines how frequently actions can change. A lower control frequency can limit the variability of action sequences but may enhance learning stability by shortening trajectories, thus simplifying credit attribution during training. We adopt the methodology from [20] to evaluate different control frequencies while adjusting the training parameters to keep the effective discount factor and batch size constant.Learning Frequency ( f RL ): The frequency of command updates (see Table 1 ), which, when lower than the control frequency, allows the same action to persist over several timesteps. This can improve credit attribution and speed up learning by reducing the number of policy evaluations, even as torque updates occur multiple times based on local state information.Policy Training We utilize the Proximal Policy Optimization (PPO) [26] reinforcement learning algorithm, known for its effectiveness in sim-to-real motor control [1, 27] . Details on policy training and network architecture are provided in the appendix. We train on five continuous control tasks, including three from the OmniIsaacGymEnvs suite (BallBalance, Cartpole, FrankaCabinet) [7] and two custom locomotion tasks for evoBOT and Go1. The main focus is on evoBOT and Go1, where the reward combines velocity tracking and regularization terms. Training parameters and reward formulations are adapted from previous works [3, 6] , with all terms independent of action representations for fair comparison. Simulation is conducted using NVIDIA Isaac Sim. c3= 10 15 20 30 c1=-40 c2=-3.0 0.47±0.00 0.50±0.01 0.53±0.00 0.54±0.01 c1=-40 c2=-2.0 0.48±0.00 0.50±0.00 0.53±0.00 0.56±0.01 c1=-40 c2=-1.0 0.49±0.00 0.51±0.01 0.56±0.01 0.68±0.01 c1=-40 c2=-0.5 0.51±0.01 0.53±0.00 0.63±0.02 0.61±0.01 c1=-40 c2=0.0 0.47±0.01 0.50±0.02 0.49±0.02 0.51±0.02 c1=-30 c2=-3.0 0.49±0.01 0.51±0.01 0.53±0.01 0.54±0.00 c1=-30 c2=-2.0 0.50±0.00 0.53±0.00 0.55±0.01 0.57±0.01 c1=-30 c2=-1.0 0.51±0.00 0.55±0.00 0.57±0.01 0.62±0.02 c1=-30 c2=-0.5 0.51±0.01 0.57±0.01 0.59±0.01 0.60±0.01 c1=-30 c2=0.0 0.49±0.00 0.50±0.02 0.48±0.02 0.52±0.01 c1=-20 c2=-3.0 0.51±0.01 0.53±0.00 0.54±0.00 0.56±0.00 c1=-20 c2=-2.0 0.51±0.00 0.53±0.01 0.54±0.01 0.59±0.01 c1=-20 c2=-1.0 0.52±0.01 0.56±0.01 0.58±0.01 0.63±0.00 c1=-20 c2=-0.5 0.53±0.00 0.57±0.00 0.59±0.01 0.62±0.00 c1=-20 c2=0.0 0.53±0.00 0.59±0.02 0.59±0.02 0.59±0.01 c1=-10 c2=-3.0 0.54±0.01 0.55±0.00 0.55±0.00 0.57±0.01 c1=-10 c2=-2.0 0.53±0.01 0.54±0.00 0.54±0.00 0.59±0.01 c1=-10 c2=-1.0 0.52±0.00 0.54±0.01 0.56±0.02 0.64±0.01 c1=-10 c2=-0.5 0.53±0.01 0.57±0.00 0.63±0.00 0.65±0.01 c1=-10 c2=0.0 0.55±0.00 0.60±0.01 0.62±0.01 0.63±0.02 c1=0 c2=-3.0 0.71±0.01 0.78±0.00 0.84±0.01 0.94±0.00 c1=0 c2=-2.0 0.76±0.02 0.88±0.01 0.94±0.02 0.97±0.03 c1=0 c2=-1.0 0.94±0.01 0.99±0.00 1.00±0.00 0.99±0.01 c1=0 c2=-0.5 1.00±0.00 0.99±0.01 1.00±0.00 0.97±0.03 c1=0 c2=0.0 0.99±0.01 1.00±0.00 0.99±0.01 0.98±0.01(a) evoBOT c3= 10 15 20 30 c1=-40 c2=-3.0 0.22±0.01 0.26±0.01 0.39±0.00 0.36±0.01 c1=-40 c2=-2.0 0.54±0.03 0.62±0.01 0.64±0.01 0.58±0.02 c1=-40 c2=-1.0 0.91±0.00 0.96±0.00 0.96±0.02 0.85±0.01 c1=-40 c2=-0.5 0.94±0.01 0.99±0.01 0.84±0.02 0.71±0.02 c1=-40 c2=0.0 0.91±0.01 0.84±0.02 0.46±0.06 0.27±0.05 c1=-30 c2=-3.0 0.17±0.01 0.27±0.01 0.39±0.00 0.43±0.09 c1=-30 c2=-2.0 0.55±0.01 0.64±0.00 0.64±0.00 0.57±0.07 c1=-30 c2=-1.0 0.96±0.01 0.99±0.01 0.97±0.01 0.80±0.03 c1=-30 c2=-0.5 0.98±0.01 0.97±0.01 0.77±0.02 0.66±0.05 c1=-30 c2=0.0 0.94±0.00 0.76±0.05 0.51±0.04 0.30±0.15 c1=-20 c2=-3.0 0.16±0.03 0.34±0.02 0.44±0.06 0.65±0.04 c1=-20 c2=-2.0 0.47±0.01 0.65±0.01 0.64±0.00 0.78±0.00 c1=-20 c2=-1.0 0.95±0.01 1.00±0.01 0.96±0.00 0.68±0.02 c1=-20 c2=-0.5 1.00±0.01 0.94±0.02 0.78±0.03 0.60±0.00 c1=-20 c2=0.0 0.95±0.01 0.71±0.02 0.52±0.09 0.52±0.05 c1=-10 c2=-3.0 0.04±0.03 0.32±0.17 0.40±0.18 0.69±0.02 c1=-10 c2=-2.0 0.62±0.01 0.70±0.04 0.67±0.03 0.67±0.01 c1=-10 c2=-1.0 0.95±0.01 0.82±0.03 0.73±0.02 0.62±0.03 c1=-10 c2=-0.5 0.96±0.04 0.74±0.01 0.64±0.02 0.53±0.05 c1=-10 c2=0.0 0.72±0.02 0.62±0.01 0.57±0.01 0.43±0.06 c1=0 c2=-3.0 0.52±0.02 0.59±0.02 0.61±0.07 0.48±0.04 c1=0 c2=-2.0 0.61±0.01 0.59±0.02 0.54±0.03 0.51±0.02 c1=0 c2=-1.0 0.56±0.03 0.59±0.04 0.61±0.05 0.54±0.03 c1=0 c2=-0.5 0.61±0.00 0.53±0.03 0.50±0.01 0.48±0.02 c1=0 c2=0.0 0.54±0.02 0.40±0.09 0.39±0.07 0.43±0.05(b) Go1Figure 3 : Full sweep over the generalized action spaces with 100 different parameterizations (c 4 = 0). We report the mean and standard error of the normalized return across three seeds for each table entry, each representing one unique combination of the generalized action space.Robot Platforms To evaluate the choice of action spaces, we use two robots in our case study:The Unitree Go1 and the evoBOT. The Unitree Go1 is a quadrupedal robot with 12 electric actuators standing 40 cm tall and weighing 12 kg. The evoBOT is a dynamically unstable robot based on the principle of a compound inverted pendulum containing 8 electric actuators, standing 80 cm high (without the arms) and a total weight of 50 kg.To transfer the trained policies from simulation to the real world, we use standard domain randomization techniques. To model uncertainty in the robot's sensing and actuation, we add Gaussian noise to the sensor observations and implement a lag model to simulate processing delays. To make the robot robust to varied terrains and disturbances, we also randomize the ground friction randomization and randomly push the robot during training. The details of the domain randomization parameters and ranges for each task are given in Section 6. 
results: 
 In this section, we evaluate our research questions through experiments in simulation. To verify that our environment is reflective of a sim-to-real scenario, we also transferred the highest-performing learned policies from Section 4.1 to the Go1 and evoBOT robots in the real world.We train policies for the Go1 and evoBOT covering 100 sets of parameters c 1 , c 2 , c 3 in the generalized action space parameterization (Section 3). All policies are trained in simulation. The results are reported in Fig. 3 . Our aim in this experiment was not to find the most optimal action space for each task, but to broadly characterize the trends in the action space parameters for two robots with different dynamics. To accomplish this, it wasn't important to use the most efficient search method (e.g. bayesian optimization) but instead cover a wide variety of action spaces to expose trends. Therefore, we opted for a simple grid search. The ranges for the grid search were chosen to span the settings provided in OmniIsaacGymEnvs as the tuned default action spaces for similar robots.First, we observe that the optimal action space can strongly depend on the learning task. Even for locomotion environments, the Go1 and evoBOT have almost opposite trends in terms of how adjusting the action space parameters impacts their performance. Second, we see that the learning dynamics can be very sensitive to the choice of action space parameters. For evoBOT, introducing nonzero c 1 corresponding to a position control loop substantially weakens performance, as does increasing c 2 corresponding to the damping parameter. For Go1, the robot cannot achieve a high reward without a sufficiently high c 1 . The difference is due to a difference in each robot's natural dynamics. The Go1 benefits from a bias towards a nominal standing pose since learning gravity compensation is a key part of its task and the optimal strategy involves oscillating the joints around their nominal pose. The evoBOT on the other hand requires continuous rotations of the wheel to locomote so the bias towards a nominal position introduced by nonzero c1 can harm performance.Different action spaces yield different initial exploratory behavior, illustrated in Fig. 4 , which plots the joint position, velocity, and acceleration during random rollouts. The initial behavior is thought to be impactful for learning some tasks; for example, Hwangbo et al. [1] observed that for quadruped locomotion, the position control action space results in standing behavior under common network initializations while commanding torques directly will bias the robot to wriggle and fall. To experimentally evaluate the impact of the initialization, we propose an intervention to alter the initial exploration of the policy. We randomly initialize teacher policy πc 1 ,c 2 ,c 3 ,c 4 θ Tand student policy πc 1 ,c 2 ,c 3 ,c 4 θ S. The student policy is trained using supervised learning to imitate the randomly initialized teacher's torque outputs but with a different internal action space:min θ S E[(π c 1 ,c 2 ,c 3 ,c 4 θ T (s) -π c 1 ,c 2 ,c 3 ,c 4 θ S (s)) 2 ], for s ∼ π c 1 ,c 2 ,c 3 ,c 4 θ T(s) obtained from rolling out the teacher in the training environment. This will incentivize the rollouts of the student operating in one action space to match the initial exploratory behavior of another. The green highlighted blocks in Fig. 4 show that the random rollouts from the student policy adopt similar characteristics to each teacher.To evaluate the impact of initial exploration on the dynamics of learning, we evaluate four configurations in Fig. 5 : Torque from Scratch, Torque with PD-like Initialization, Position from Scratch, and Position with Torque-like Initialization, where X with Y -like initialization means we pretrain the X policy to imitate rollouts from a randomly initialized Y policy, then train the X policy with reinforce-Figure 5 : Initial exploratory behavior plays a major role in the performance of different action spaces. For dashed lines, the policy underwent a pretraining stage where it was trained through imitation learning to match the initial random rollouts sampled from another action space. Mean and standard error of normalized returns are reported across three random seeds. ment learning. For Go1, torque policy with position-like initialization boosts the final performance close to training with a position-controlled policy. Conversely, for evoBOT where the torque action space is more performant, pre-initializing a PD controller with torque-like initialization results in a gain, although not a full recovery of final performance. We also consider the FrankaCabinet, Ball-Balance, and Cartpole tasks from the OmniIsaacGymEnvs suite. We find that for some tasks (Go1, evoBOT), the performance gap between action spaces can be partly or fully recovered by intervening in the initial exploration behavior, while for others, this aspect is less influential. Surprisingly, FrankaCabinet obtains the highest performance from a torque policy with PD-like initialization, suggesting it may benefit from initially exploring like a PD policy and later exploring like a torque policy. This seems intuitively reasonable since the Franka can benefit from gravity compensation at the start derived from the bias of PD control and later explore a different space of movements to open the drawer.The complementary view of action spaces being tools for exploration would be that they have different expressive capacities due to their different architecture. We find this is not true for policies that run at synchronized frequency, and we can train a policy with any of position, delta position, or torque action space to match a high-performing policy trained in another action space using distillation (see Table 2 ). This is naively true because neural networks are universal function approximators; however, it illustrates that our comparison is devoid of factors that would make the policies impossible to imitate, such as varied behavior between timesteps, which we evaluate in Section 4.4.We examine the impact of temporal aspects on policy performance across different action spaces (see Fig. 6 ). Policies are trained for evoBOT and Go1 with control frequencies ranging from 25 to 200 Hz, and their performance is evaluated based on total reward. All experiments maintain a fixed physics frequency of 200 Hz and implement the corresponding RL frequency via frame skips of [1, 2, 4, 8] . Results indicate that control frequency significantly affects performance, with higher frequencies generally yielding better rewards in torque control mode for evoBOT, while lower frequencies also perform well for Go1.Additionally, the behavior between physics substeps impacts final policy performance. We analyze scenarios where torques are recomputed at each physics substep ( f Ctrl = f Phys ) versus when they remain fixed until the next RL update ( f Ctrl = f RL ). Our findings suggest that updating torques at the physics level leads to superior performance for most action spaces and tasks, especially in position control, likely due to the effects of stiffness and damping in PD control. Returnposition_f Ctrl = f Phys position_f Ctrl = f RL velocity_f Ctrl = f Phys velocity_f Ctrl = f RL torque_f Ctrl = f Phys torque_f Ctrl = f RL (b) Go1Figure 6 : Temporal behavior of RL frequencies and torque updates significantly influences policy performance across action spaces. Performance is evaluated across three seeds, with mean and standard error of normalized returns.Normalized Return evoBOT Go1Teacher 1.00 ± 0.00 1.00 ± 0.07 Position Student 1.00 ± 0.00 1.00 ± 0.13 Velocity Student 1.00 ± 0.00 1.03 ± 0.16 Torque Student 1.00 ± 0.00 0.99 ± 0.18  
practical guidelines for action space selection: 
 While our findings generally support that the choice of action space is specific to the robot and task, we distill some overall guidelines to assist practitioners when selecting action space for new tasks:1. When selecting an action space for learning, first consider the robot's dynamics and the types of movements you expect to be required. If keeping the joints close to a nominal posture requires some torque, a position gain may be helpful. If the joints require continuous rotation, consider a torque action space.2. The choice of action space has a considerable impact on the initial exploratory behavior of the policy. You can gain intuition for this by visualizing the position, velocity, and torque of random rollouts. Combining the initial behavior of one action space with the representation of another is possible through an initial imitation stage.3. In general, select the action space gains that provide the most advantageous learning dynamics for your task. If you need to deploy the final policy using a different set of gains, you can use teacher-student learning to perform the conversion.4. Policy behavior between timesteps can influence performance, but with mixed trends. It's generally preferable to run the policy at a higher frequency and without any frame skip. 
limitations and future work: 
 Although our work identified trends in action space design across multiple tasks, these trends do not fully explain the differences in learning dynamics. For example, initializing the policy to mimic the initial exploration of another action space improved performance in several tasks but did not yield identical learning curves, indicating that some factors remain unexplained. While we enhanced our mechanistic understanding of how action space selection affects learning, we could not leverage this understanding to stabilize training or introduce a new initialization scheme that consistently improves performance across tasks. This underscores the challenge of creating robust training that is less sensitive to environment design, as action spaces introduce strong, task-specific biases.Future work could focus on establishing metrics to predict whether an action space is suitable for new tasks before training, potentially accelerating automated environment design, such as using LLMs [28] . Additionally, it is important to note that our analysis was restricted to the behavior of action spaces in combination with the PPO algorithm. While we believe our conclusions are useful since PPO is widely adopted for sim-to-real tasks, further research is needed to explore how these results generalize to other RL algorithms. Table 7 : Reward for evoBOT locomotion (adapted from [6] for improved velocity tracking and to eliminate dependence on action representation, e.g. action change penalty is replaced by other smoothness terms). 
 
 Appendix:  


Decision: Strengths
- Highly relevant topic for all robot RL researchers
- Strong experimental design
- Nice discussion and limitations sections

Weaknesses
- Many details unclear
- Unclear if experiments report multiple runs
- Evaluating on multiple RL algorithms would have been nice to show generality
- Scope of action space a bit limited and also the related discussion needs to be improved
- Real robot experiments would strengthen the paper
- Unclear how to apply lessons learned in practice

# After Discussion Phase
All reviewers highly appreciated the additional information and improved experiments. Overall that improved the paper significantly. Reviewer 29AA still has 2 concerns that are straightforward to fix. Reviewer FMNG still maintains 2 suggestions on how to further strengthen the paper. While I agree with Reviewer 15bq that the results aren't as conclusive as one might have hoped, this is unfortunately typical for RL (whether a methods performs well or not is often highly task specific), and I believe the insights and new practical guidelines are a valuable contribution.

Reviews: review: title: title: Review of paper
paper_summary: paper_summary: This paper assesses the role of action space design on the performance of robotic control policies trained using reinforcement learning. The work aims to understand if and how the choice of action space affects learning dynamics. Through a variety of simulated experiments, the work assesses the extent to which the action space affects initial exploration, a model’s expressive capacity, and the role of multi-timestep action formulations. The work concludes that the optimal action space is strongly dependent on the particular task, and that this can only be partially explained by differences in exploration.
main_review: main_review: Strengths
* The paper attempts to shed light on a very important aspect in robot learning - action space design. It is well known to be critical but it is rare to see research papers dedicated to the topic. Insights on this topic are likely to be very useful for the robot learning community.
* The introduction does a good job of motivating this research, and clearly lays out both the main questions the authors seek to answer and their main findings.
* I appreciated the generalized parameterization of action space formulation. This is original and insightful. It also enabled the authors to isolate the effect that action spaces have on exploration which was very interesting.
* Strong experimental design. The experiments are creative and insightful. They are effective at providing evidence to answer the main questions posed in the paper. I especially appreciated the analysis and discussion in sections 4.2 and 4.3.
* A thorough limitations section
* I appreciated the discussion in section 3.2 on temporal aspects of action space design

Weaknesses
* Lack of clarity and detail in multiple places. As-is I have a number of questions about the method, tasks, experimental setup and training parameters, and results (see questions for rebuttal). This is the core weakness of this paper.
* Narrow scope of action spaces considered. Whilst the authors consider multiple action spaces, they ignore a key choice of whether to choose an action space in end effector / leg space or joint space, and instead focus solely on joint space, without justification. For example, in manipulation end effector space remains a common choice.
* No real robot experiments, although the authors mention do provide a convincing argument that their results will hold in the real world (lines 218 - 219). 
* Limited impact: Whilst this line of research has the potential to be very impactful, I think the paper as it stands is likely to have limited impact because there remain a number of open questions that limit the recommendations that the authors can give to those wishing to apply this work. Making progress on metrics that could be used to predict in advance of training whether an action space is likely to be useful (as discussed by the authors in lines 214-215) would be one way to increase the likely impact of this work.
* The related work is reasonably thorough, however the discussion of action spaces in lines 54-59 is missing a discussion of end-effector/leg space control vs joint space control (+ sub variants including delta-position control), as well as the choice of whether to remain in a continuous action space or to discretize. Additional choices such as heavily constraining action space ranges are also missing. Finally, I would have liked to see a discussion of how this work differs from prior work on action space design.
questions: questions: Related work 
* How does this work differ to prior work on action space design?

Method
* Fig 2. h_t is undefined
* How do c1, c2, c3, and c4 in fig 2 relate to [c1, c2, c3, c4] in table 1?

Experimental setup
* What are the tasks that are being evaluated? What is the maximum and minimum reward per task? 
* What are the policy training details - network architecture, number of training steps, number of training runs
* Sim-to-real - please give specific details. What parameters were domain randomized, what were the ranges, how much noise was added to the observations, how much latency?

Results
* 4.1
  * Line 139: What does a “full sweep” mean? I see that c1, c2 and c3 are varied but c4 appears constant. Is that the case and what was it’s value?
  * Was training in simulation or the real world?
  * What are your hypotheses about why there is such a difference between the Go1 and evoBOT embodiments?
* Fig 3: What does “maximum reward mean”? What is the maximum possible score?  Is the reported score a mean value? Over how many reported training runs? What was the variance of these results (RL is known to exhibit high variance)?
* Table 2: Is the reported score a mean value? Over how many training runs?
* Fig 5: What do the different color lines represent?
* Fig 6. Is there something wrong with the y axis of both charts? The value appears to be very low compared to Fig 3 and I assume this is the same task.
* How does the supplementary video relate to the paper? There are no real robot experiments in the paper and the video is of an evoBot wheeling around a space

Minor
* Line 31: This sentence doesn’t make sense
* Line 157: The teacher and student policy presumably have different action spaces, it would be helpful to make this explicit.
* Line 160: There is no bottom row to the referred figure
* Line 161: Should the be Fig 5 instead of Fig 4
* Line 196: effect → affect
limitations: limitations: 3
review_summary: review_summary: This paper is on an interesting and relevant topic, with nicely designed experiments, but poor clarity as the paper stands. Additionally, the action space scope is somewhat narrow, ignoring a key choice between end effector / leg space and joint space control.  The main reasons for my recommendation are (a) clarity which could be addressed during rebuttal and (b) narrow action space scope. Given there is no algorithmic, dataset, or system design novelty and the experimental results are in simulation, I wish to see a deeper exploration of action spaces and more insight, for example by including in this work the metric proposed in the limitations section.  **Post-rebuttal update: The revisions provided during rebuttal substantially improved the paper. As a result I am revising my recommendation to weak accept**
title: title: Important topic; concerns around generalizability, limited scope and insufficient runs
paper_summary: paper_summary: This paper studies the impact of action space in RL for robot control. It introduces a general reparameterization of the action space,  enabling the representation of common action space choices as a linear mapping from state and action to torque. Through experiments conducted in both simulated and real-world robots, the paper shows that action space design affects control performance in a task-dependent manner. Additionally, it demonstrates that performance difference is largely due to policy initialization and the interplay of control frequency and torque update frequency.
main_review: main_review: Strengths:

- The paper addresses an important and often overlooked topic, as the selection of action space in previous research has often been ad hoc.
- It introduces a general parameterization of action space, enabling the representation and comparison of common action spaces within a unified framework.
- Experiments are conducted on both simulated and real-world robots.

Weaknesses:

- It is standard practice to report statistics from multiple runs in RL. However, the paper does not provide these details, and it seems that the results presented in the plots are based on a single run.
- The experiments only apply one algorithm, PPO, making it unclear whether the conclusions can generalize to other RL algorithms or are specific to PPO.
- The scope of this work is about “understanding” the impact of action space. But it lacks practical insights on “how“ to select action space for different tasks.
- The results in Figure 4 indicate a high dependency on the specific task, making it challenging to draw broad conclusions.
- The writing could be improved for better clarity, as detailed in "questions for rebuttal".

Minor:

- Typo on page 7, “instead RL-level” → “instead of …”
- Should the y-label in Fig.4 and 6 be return instead of reward?
questions: questions: - Table 2 caption mentions an “optimal teacher policy”: how was this policy obtained?
- As shown in Fig. 1, none of the three simulation environments use position-based action space. What’s the rationale of picking these environments?
- Line 160 refers to “The bottom row of Fig. 4”,  is this plot missing or incorrectly referenced?
- Line 175, mentions “exploring like a PD policy and later exploring like a torque policy”.
    - can you elaborate on what it means to be exploring like a PD policy, or exploring a torque policy?
    - In PPO, exploration typically reduce gradually. Is there more evidence to support the claim that the observed benefits come from exploration?
- In Sec. 4.4, Line 200 mentions performance increase from more frequent torque update. I wonder if this increase can be attributed to a more accurate physics simulation from the faster update, rather than the action space itself.
- I’m having trouble understanding the x-axis label “Decimation” in Fig. 6. Why is it not simply labeled “frame skip”?
limitations: limitations: 2
review_summary: review_summary: **Updated Post-Rebuttal**: The paper addresses an important topic, but the results are inconclusive despite that the authors have made some improvements during the rebuttal.
title: title: Interesting paper, but needs clarification
paper_summary: paper_summary: The paper discusses the impact of action space selection on the performance of reinforcement learning for robot motor skills. Through a unified formulation, it explores the mechanisms by which action space influences performance, notably during the initial exploration phase and when using different control frequencies.
main_review: main_review: The paper tackles an interesting problem and the unified formulation to describe the different action spaces is a good contribution.
Overall, the paper goes in the right direction but lack clarity on some points and needs more polishing.

For example, it should be more explicitly mentioned that it considers only "configuration" based action space (no Cartesian/impedance control).
It also seems that Table 1 (and the generalized framework) is only valid when the policy network and low-level control are running at the same frequency (d=1).

It is not clear if all the results with evoBot and Go1 quadruped are in simulation or on real robots (or both?).
(I detail my different questions/remarks to improve clarity in the next section "questions for rebuttal")

In the results, no uncertainty is reported, which should be the case at least for simulation.

Unfortunately and as noted by the authors, the main takeaway message is that action space selection do have an impact on performance, but the paper provides no guideline to choose which action space to use in a given situation.
questions: questions: - In table 1, although it is clear how one-step action space can be retrieved, it is not clear for multi-step. It also seems that multi-step is also not clearly defined, but I assume it is when d > 1?
- I assume that every time that the generalized framework is used, only synchronous policy/low-level control is considered?

- It is not clear how the full sweep of Figure 3 was done. Was there only one run per entry?
 Is it simulation only? Why choose grid search (it is usually an inefficient way to search)?
 How was the search space chosen?
 Please report uncertainty when possible.

- For section 4.2, are the action space normalized? what is the original probability distribution used by PPO? (what happens if you replace PPO by SAC which uses a squashed Gaussian that should be less affected by action space limits)?

- For figure 5, random rollout corresponds to rollouts with randomly initialized policy?

- Figure 6/ section 4.4: only one run per datapoint? It would be good to normalize the performance (using random policy and max performance), because a difference of 0.002 in reward doesn't tell much if the performance changed a lot or not.
Again, please report uncertainty when possible.

- minor: typo in Fig 6: "no_udate" -> "no_update"
limitations: limitations: 3
review_summary: review_summary: I recommend a weak rejection of the paper due to several issues that need to be addressed, including a lack of clarity on certain points and in the presentation of results. Overall, while the paper makes a good contribution by exploring the impact of action spaces, it needs further refinement before it can be considered for publication. **post-rebuttal update: I updated my score to weak accept but some additional points need to be addressed**