A Primer on SO(3) Action Representations in Deep Reinforcement Learning
Authors: Martin Schuck
Sherif Samy
Angela P. Schoellig
Venue: ICLR 2026 Poster (2026)
Keywords: RL
SO(3)
3D rotations
Action Representations
Deep RL
robotics
Primary area: reinforcement learning
TL;DR: We analyze the effects of different SO(3) action representations for 3D rotations in deep RL.

Abstract: Many robotic control tasks require policies to act on orientations, yet the geometry of SO(3) makes this nontrivial. Because SO(3) admits no global, smooth, minimal parameterization, common representations such as Euler angles, quaternions, rotation matrices, and Lie algebra coordinates introduce distinct constraints and failure modes. While these trade-offs are well studied for supervised learning, their implications for actions in reinforcement learning remain unclear. We systematically evaluate SO(3) action representations across three standard continuous control algorithms, PPO, SAC, and TD3, under dense and sparse rewards. We compare how representations shape exploration, interact with entropy regularization, and affect training stability through empirical studies and analyze the implications of different projections for obtaining valid rotations from Euclidean network outputs. Across a suite of robotics benchmarks, we quantify the practical impact of these choices and distill simple, implementation-ready guidelines for selecting and using rotation actions. Our results highlight that representation-induced geometry strongly influences exploration and optimization and show that representing actions as tangent vectors in the local frame yields the most reliable results across algorithms. The project webpage and code are available at amacati.github.io/so3_primer.

Published as a conference paper at ICLR 2026
A PRIMER ON SO(3) ACTION REPRESENTATIONS IN DEEP REINFORCEMENT LEARNING
Martin Schuck
Sherif Samy
Angela P. Schoellig
Technical University of Munich
University Girona
Technical University of Munich
ABSTRACT
Many robotic control tasks require policies to act on orientations, yet the geom- etry of SO(3) makes this nontrivial. Because SO(3) admits no global, smooth, minimal parameterization, common representations such as Euler angles, quater- nions, rotation matrices, and Lie algehra coordinates introduce distinet constraints and failure modes. While these trade-off's are well studied for supervised learn- ing. their implications for actions in reinforcement learning remain unclear. We systematically evaluate SO(3) action representations across three standard con- tinuous control algorithms, PPO, SAC, and TD3, under dense and sparse rewards. We compare how representations shape exploration, interact with entropy regu- larization, and affect training stability through empirical studies and analyze the implications of different projections for obtaining valid rotations from Euclidean network outputs. Across suite of robotics benchmarks, we quantify the prac- tical impact of these choices and distill simple, implementation-reade guidelines for selecting and using rotation actions. Our results highlight that representation- induced geometry strongly influences exploration and optimization and show that representing actions as tangent vectors in the local frame yields the most reli- able results across algorithms. The project webpage and code are available at simacai.github.io/see
INTRODUCTION
Accurate reasoning over 3D rotations is core requirement for machine learning algorithms ap- plied in computer graphies, state estimation and control. In roboties and embodied intelligence, the problem extends to controlling physical orientations through learned actions, e.g., in manipulation policies that command full task-space poses or acrial vehicles that regulate attitude. These tasks rely on trained policies with action spaces including rotations in SO(3).
Dealing with rotations is especially challenging because the underlying manifold curved, and there exists minimal parameterization that maps R3 to SO(3) and is globally smooth, bijective and non-singular. This restriction has led to multiple parameterizations, each with its own trade- offs (Macdonald 2011 [Barfoo|] 2017). Euler angles minimal and intuitive but suffer from order dependence, angle wrapping, and gimbal-lock singularities. Quaternions are smooth and numer- ically robust with simple unit-norm constraint, but double-cover SO(3). Rotation matrices are a smooth and unique mapping, but are heavily paeettrramteri and require orthonormalization. Viewing SO(3) a Lie group, one can use tangent spaces, i.e., the Lie algebra m of skew-symmetric matrices, together with the exponential and logarithm maps to represent orientations. Tangent spaces are locally smooth, but globally exhibit singularities at large angles (Sola et al.) 2018). Irrespective of the choice of parameterization, any minimal 3-parameter chart must incur singularities, and global parameterizations that avoid singularities are necessarily redundant and constrained.
Applications in deep learning that require reasoning over rotations and orientations have renewed interest in this topic by adding another perspective: irrespective of any mathematical properties, what is the best representation to learn from data in SO(3)? Are low-dimensional representations necessarily more efficient, and does the double-cover of some representations harm training perfor- mance? Several works have explored these questions in the supervised setting (Zhou et al. 2019 Peretroukhin et al. 22200; Brégier 202II) Geist et al. (2024) offer an excellent overview that summa- rizes the most common representations, links mathematical properties of representations to observed

---

Published as a conference paper at ICLR 2026
performance gains, and gives concrete recommendations on supervised tasks like rotation estimation
or feature prediction.
What is still missing is general, systematic evaluation of SO(3) representations in deep reinforcement learning (RL). While the intuitions and results from prior studies on input/obser- vation representations for orientations also apply to the RL setting, the most suitable SO(3) action representation remains unclear. Action representation requires special attention, as it shapes the exploration dynamics induced by stochastic policies and exploration noise, and has implications for action clipping to comply with actuation constraints. Prior works have proposed specific action rep- resentations for a narrow set problems and algorithms (Alhousani et al. 2023a (). Schuck et al. (2025a) recently attempted to tackle the general question of best action and observation representa- tions for RI., but limited their investigation Deep Deterministic Policy Gradients (DDPG) under sparse rewards.
In this paper, we study three widely used continuous-contr algorithms: Proximal Policy Opti- mization (PPO) (Schulman al. 2017). Soft Actor-Critic (SAC) (Haarnoja et al. 2018), and Twin Delayed Deep Deterministic Policy Gradients (TD3) (Fujimoto et al.) 2018). We evaluate action representations under dense and sparse rewards and focus phenomena specific to RL rather than supervised learning. We show how representations shape exploration, interact with entropy regular- ization, and affect convergence stability. Finally, we quantify their practical impact across standard robotics benchmarks. In summary, our contributions are as follows:
1. We analyze the most popular RL algorithms for continuous control, PPO, SAC, and TD3, under action spaces that include orientations. These algorithms are extensively used in robotics research train policies deployed on physical hardware in the real world, and thus are particularly relevant.
2. We investigate why different action representations yield different training performance. Beyond intuitions properties such smoothness uniqueness, show how observed performance and sample-efficiency differences are attributed to the map between Euclidean network outputs and SO(3). Our analysis highlights the implication of representation- induced action projections on exploration, action scaling, and regularization techniques.
3. We offer concrete guidelines for choosing policy representations and handling representation-indueee effects. Building on insights gained in our empirical studies and in three benchmarks on three different robot platforms, we cover algorithm- and representation-deeendeed pitfalls and how to mitigate them.
This paper aims to make orientation control in RL easy to get right. Consequently, we prioritize clarity, common pitfalls, and ease of implementation manifold optimization. We hope this will help practitioners make a conscious decision on action representations and advance the training of policies with full pose control.
exhaustive mathematical treatment of
2 REPRESENTING SO(3) ACTIONS IN DEEP RL
The set of all 3D rotations forms the Lie group SO(3). appears in control, graphics, state esti- mation, and in RL tasks where policies must command orientations, such manipulation with full end effector pose control or drone control. Multiple representations exist to parameterize SO(3), each with its own properties. In the following, outline representations, geometric properties, and learning phenomena that matter when projecting neural network outputs to valid rotations and training policies that act SO(3).
2.1 THE SO(3) MANIFOLD
Unlike translation in R which is flat, commutative, and globally parameterized, SO(3) is a com- pact, curved, and non-commutative manifold M that admits no global, smooth, minimal chart. Its anisotropic geometry implies bounded, periodic angles and topological constraints that make action representation difficult: minimal coordinates introduce singularities, global coordinates are redun- dant and constrained, and tangent-space coordinates are only locally valid.
2

---

Published as a conference paper at ICLR 2026
Table I: Properties of common SO(3) representations used for actions.
Representation
A Action
Dim.
Cover
Smooth Singularities
Constraints R. R det R =
Matrix R
AR
9
single
Quaternion q
double
||||22
Euler angles (0. 0, v) Lie algebra m
A ( 0, v)
3
multi
multi
Formally defined
SO(3) - R E RRR=1 I, det R R- 11
Rax3
3D rotations have multiple representations that all map to (3)3). Table lists a selection of common representations and their respective properties. The Lie algebra denotes the tangent space T&M of SO(3) at the origin. Conversions between tangent increment vectors & E R and SO(3) are realized by the capitalized exponential and logarithmic maps Exp R M and its inverse Logg .M + RR Section A.5 contains example code demonstrating how to realize both Exp and Log maps. For an in-depth review Lie theory and SO(3) representations, we refer Solà et al. (2018) and Geist et al. 2024), respectively.
2.2 GLOBAL VS DELTA ACTIONS
There are two ways define orientation actions in deep RL. The most straightforward way of view- ing SO(3) actions is interpret them as desired orientation in the global frame E. The environment dynamics steer us towards that orientation, e-g., through a low-level controller. In the following, we will call these global actions.
However, the group structure (0(3) also permits us to view the action as an intrinsic delta rotation with respect to the current state of the agent, e.g. for rotation matrices R R, R.R. with delta action RAa- Changing the viewpoint makes actions independent from the global frame and thus potentially aids generalization. We will explore the benefits of changing the viewpoint in sections and
2.3 MULTI-COVERS, SINGULARITIES, DISCONTINCITIES
Action representations should be unique avoid multi-man modal solutions in policy outputs and targets. Non-injective parameterizations create equivalent actions for the same physical rotation, complicat- ing exploration, entropy regularization, and representation with uni-modal policies. In addition to uniqueness, representations should vary smoothly under small physical rotations. Intuitively, we want representations of actions that lead similar rotations lie close in Euclidean space.
Quaternions realize double-cover from the 3-sphere S(3) to SO(3), with and -q representing the same rotation. Enforcing a hemisphere convention (e.g., nonnegative scalar part) removes the ambiguity but introduces a branch discontinuity on the equator where the scalar part is zero and can cause abrupt sign flips along trajectories.
Lie algebra coordinates use the exponential map Exp R + SO(3) which wraps around infinitely often along each axis for t + 2 i N. Restricting m to a principal branch with angle |r| [0, T)) limits overlap but leaves a cut locus at 0 æ where log is discontinuous and the axis is not unique.
Euler angles triple after choosing standard ranges, whereas specific configurations admit infinitely many repre- sentations. The classical singularity ("gimbal lock") arises when the first and third rotation align and collapse into a combined angle. Independent of this, angle wrapping at +m introduces discontinuities.
many-to-one map that is not fixed k-to-1 cover. Most rotations have a unique
2.4 PROJECTIONS
Feedforward policies produce Euclidean outputs that do not satisfy manifold constraints by con- struction. Rotation matrices must obey R R I and det R 1, and quaternions must have
3

---

Published as a conference paper at ICLR 2026
unit norm. Therefore, actions must be projected from raw outputs to valid group elements. These projections can be inserted as differentiable layers in the actor, enabling backpropagation almost everywhere. For quaternions, given E IR, normalize q | which is smooth except 0.
For matrices, the singular value decomposition (SVD) projects M E R = UEVT to the closest rotation via
R = (1, (UV )) V)
which is differentiable except at degenerate SVDs (Schönemann) 1966). Tangent-space and Fuler- angle outputs need no feasibility projection, though magnitudes should be limited to permissible ranges by squashing network outputs through, e.g-, tanh activation functions, such that actions clamped to |r| < and Euler angles to (- and (- respectively.
For TD3's deterministic policies, this suffices to guarantee actions lie in SO(3). Stochastic policies, on the other hand, are often parameterized as multivariate Gaussians. Applying the projection to each sampled action guarantees on- on-manifold actions, but it also warps the action distribution and renders log probabilities intractable. РPO and SAC rely on accurate log probabilities, and closed- form corrections for normalization on S(3 or SVD-based projections are not readily available.
In this paper, we adopt a practical compromise: we project the mean inside the network wherever possible and sample in the ambient Euclidean space. The sampled, off-manifold actions are pro- jected again within the environment to a valid rotation. This approach keeps training compatible with standard log-probability computations while ensuring feasibility at execution time.
2.4.1 UNIT-ROTATION CENTERING
Policy neural networks for RL are designed to initially produce zero-centered output distributions actions. Counterintuitively, the projections for quaternion and matrix representations map outputs around zero to wide range of rotation actions (see section A.I.T). This is particularly relevant for delta actions, as the agent must first learn the unit rotation that prevents it from rotating.
One possible remedy is custom policy network that adds the unit rotation the action mean. We discuss the performance impact on delta matrix and quaternion representations in section 3.3 and provide ablations on corrections well as advanced benchmarks in section A.3 Local tangent vectors and delta Euler angles center around the unit operation and are thus unaffected.
2.5 ACTION SCALING
Orientation control is particularly interesting for physical systems such as robot arms drones. These systems have bounded angular rates, which motivates policies with limited rotation magni- tudes. Rate limits for global orientation targets have to be enforced by an underlying controller or the environment dynamics.
can scale the incre- ment before mapping to SO(3). For tangent vector "T E R,3, this is trivially achievable by limiting the output norm. Intuitively, we interpret tangent vectors as vectors attached to the local tangent space of the current orientation. One consequence of viewing SO(3) actions as delta rotations is that it mitigates discontinuities from wrapping or cut-locus singularities for sufficiently small delta rotations. Delta Euler angles are less straightforward to scale, since the change in orientation magni- tude depends on the current orientation. Scaling is either overly conservative or requires a complex chart of orientation-dependee
On the other hand, delta rotations in the local frame, as introduced in section 2.2
normalizations.
Quaternion and matrix representations can be scaled uniformly via geodesic operations. Using the exponential map presented in section 2.11 R = Exp(a Log R) scales rotations to a maximum angle of a, but introduces branch choices and non-smooth points at In practice, delta actions in the tangent space with norm control provide well-behaved and straightforward mechanism for action scaling.
3 CONTROLLING PURE ORIENTATIONS
We first study policies that control pure rotations to isolate the effects action representations. We compare PPO, SAC, and TD3 in an idealized environment with only rotational dynamics and
4

---

Published as a conference paper at ICLR 2026
orientation state. Our analysis tests hypotheses about how representations influence exploration, entropy regularization, and stability, and clarifies what matters for learning SO(3) actions.
3.1 ENVIRONMENT SETUP
Formally, model an episode goal-conditioned MDP M = (S, P,r,) with state space = SS(33 x SO(3) consisting of states 8, = (Re Rg where Rg is the current orientation and R2 is goal orientation fixed per episode. Following Geist et al. (2024),
use flattened rotation matrices as observation representations everywhere. Goal-conditioned environments allow us to also analyze sparse reward learning with Hindsight Experience Replay (HER) (Andrychowicze al. 2017 for off-policy algorithms (SAC and TD3).
The action space, our object of interest, is configurable. Actions can describe global desired orientations in any of the aforementioned repre- sentations or delta rotations. For global actions R.,, the deterministic environment transition dynamics are formulated as
optinel
Rii1
R
Ra:
if (R,,, Ra) < Ormax otherwise
Rc+1
Rd Exp
Goonr R.R
Log (Ry (eo"RR))
R
(3)
M
with the geodesic distance d(R1, R2) arccos tx [u(R) Ra) Ra)-1) Intu- itively, equation 3 takes the shortest path towards the desired orientation R. with maximum step length of amaz. Figure visualizes the en- vironment dynamics. Dense rewards rleuse
The agent ro- tates at max Qmaz dians from the current state R& the next state Ri+1 towards the de- sired state Ra The goal is to rotate into Rg.
Figure 1:
d(( R4, R2y are defined as the negative angle to the goal. Sparse rewards are when the angle between state and goal d( R;, Rg) S 0.1 and 1 everywhere else. Termi- nation occurs after a fixed step limit of 50.
3.2 PERFORMANCE COMPARISON
We benchmark PPO, SAC, and TD3 in the pure-rotation environment using four action parameteri- zations: rotation matrices, unit quaternions, tangent-space (rotation vectors), and Euler angles, each evaluated as global and delta actions. All other components remain fixed: network architectures, training budgets, observation spaces, and reward definitions are identical across conditions. The results are presented in table 2 with the best results highlighted in blue, and the second-bes shown in bold. Results are averaged over runs each. See section A.2 for the training curves, section A.4 for discussion on Zhou et al. (2019)* 's representation, and section A.8 for hyperparameters.
Across algorithms and reward formulations, the delta tangent vector representation almost always results in the best final policy with minor variances between runs. Global matrix representations achieve the second-best performance, except for SAC with sparse rewards, where they exhibit poor performance. Other representations often perform poorly, particularly in sparse reward environ- ments, despite using HER.
3.3 EXPLAINING THE EFFECTS OF SO(3) ACTION REPRESENTATIONS
The choice of action representations significantly impacts policy performance and training stability. We now dive into the reasons why this is the case, and draw conclusions for training policies on SO(3) actions. This section is organized hypotheses: we state conjectures based on the intuitions from section 2 analyze if these intuitions match our empirical results, and conduct ablations that explain any deviations.
Hypothesis / Smooth, unique representations converge faster and lead to superior policies. Our first hypothesis is a common assumption based on the intuition that neural networks better fit smooth functions (Barron 1993). In addition, the predominant policy network architectures either parametrize a uni-modal distribution (PPO, SAC) or a single, deterministic action (TD3). In this setting, multi-modis action representations should produce conflicting gradients, which harm per- formance. Consequently, rotation matrices should be the best action representation in SO(() among the selected ones they are both unique and smooth.
5

---

Published as a conference paper at ICLR 2026
Table 2: Results for the idcalized rotation environment.
PPO dense
SAC dense
TD3
sparse
dense
sparse
R
-5,4 0.2
-4.7 +: 0.3
-29.4 + 0.7
-.7 + 0.2
-6.4 + 0.5
AR
-12.3 1.1
-5.1 0.3
-31.0 1.5
-.9 + 0.3
-20.7 = 13.4
Aq
-11.5 + 1.8
0.5
-30.2
0.6
-9,2 = 1.5
-22.1 1.8
-.3
-2..3
0.4
-21.6 12.9
TT
0.5
1.5
-38.5
0.84
-32.3
"T
0.2
-.3
-79
33.5
-6,9
(b, 0, 1b)
0.6
+.7
-35.2
-7.3
-16.2
A(o, 0,
0.5
-5.8
-15.7
-7.4 0.9
-31.2 + 13.1
that this is only partially true. The global matrix representation does well in table except for SAC and sparse rewards. Based on the smoothness argument, delta ma- trices should be equally performant but consistently achieve lower performance. The performance difference originates from the fact that delta representations must learn the connection between the current orientation and the goal, instead of only relying on the goal.
In experiments,
While smooth, quaternions display weaker performance due to the double-cover. We can show that for both dense and sparse rewards, the critic learns the multi-modal reward distribution and thus produces conflicting policy gradients if actions are sampled from both hemispheres of S(3) (see section A.
The exceptions to our intuition are policies formulated in the tangent space of the local frame. While they are not free of singularities nor discontinuities, the maximum step angle Oman limits the policy to region where the tangent space is unique, has no discontinuities, and the Exp mapping is almost linear. The singularities and discontinuities at the cut locust are always out of the policy's reach since the space is attached to the local frame. Combined with not requiring projections and lower dimensionalityy local tangent increments outperform global matrix representations even though they must learn the relation between the current frame and the goal.
Tangent vectors in the Lie algebra follow our intuition and produce mixed results, as do global and delta Euler angles due to severe discontinuities and singularities.
Conclusion Uniqueness and smoothness benefit learning, but the properties do not have to hold globally for agents with limited angular step sizes. What matters is that discontinuities and singularities of reach for the action space, e.g., in the local tangent space.
Hypothesis 2 Representations influence the exploration dynamics the SO(3) manifold.
While the research community has proposed advanced exploration tech- niques improve speed of convergence (Houthooft et al. 2016 Plap- pert et al. 2018), the most common mechanisms are Gaussian stochastic actions (PPO and SAC) or Gaussian/uniform exploration noise (TD3). Samples from these distributions are generally off-manifold. Projecting the perturbed actions back onto their representation manifold (see sec- tion 2.4) produces action distributions that can concentrate around small regions of the action space and harm exploration.
Samples Li algebrs
Figure shows this concentration for random Fuler angle samples that concentrate around the singularities of the representation. The conse- quences are most noticeable in the sparse reward environments, where sparse rewards prevent the policy from converging early and agents ex- plore longer (see section A.2 All representations except local tangent vectors and global matrices for TD3 perform poorly on these tasks.
Figure 2: Distribution of Euler angles sam- pled from N (0,0)) and squashed with tanh. The resulting orientations are visualized in the 3D Lie algebra m. Note that samples also lie within the spherical space.
Additional ablation studies analyzing the replay buffers during training and the distribution of successful goals show that agents' success closely correlates with the exploration distribution. Representations with more

---

Published as a conference paper at ICLR 2026
spread, i.e., matrix and local tangents, thus advantageous. See section A.1.1 for compari- son of all distributions, and section A.1.4 for more details on Euler angles.
Conclusion The projection onto SO(3) warps common exploration distributions, which sig- nificantly impacts convergence. Euler angles and quaternions most affected by this, ma- trices to some degree, and local tangent spaces least.
Hypothesis 3 Standard entropy regularization leads to suboptimal policies on SO(3).
As seen above, action projections warp random distributions on the action vector. Apart from explo- ration, this has consequences for entropy regularization. Maximizing entropy without accounting for the representation manifold may incentivize actions with less randomness after projecting (see e.g. section A.. Figure 2 where high-variance Gaussian distribution of angles is mapped to narrow distribution in (3), demonstrates this well. This hypothesis only applies to PPO and SAC, since TD3 is missing an entropy term.
The effect not strong enough for dense rewards to have a strong impact on the results in table However, in sparse rewards, performance for representations with projections on SAC is significantly worse compared to TD3, whereas the dense variants produce near-identical results. We conclude that the entropy maximization leads to significantly worse exploration behavior than the noise in TD3.
To confirm this, we test all action representations with increased entropy coefficients. The full set of experiments for PPO and SAC can be found in section A..17 Our results show that entropy maxi- mization drives actions towards larger norms Euclidean space. These do not, however, correspond to more random actions because of the maximum step angle Omazx and instead lead to more stable rotation directions. Scaling delta tangent actions to the range of allowed values mitigates this ef- fect and contributes to the improved performance of SAC in section A.1.5 Quaternion and matrix representations do not have a similar mitigation and thus perform poorly.
For Euler angles, elevated entropy levels result in attraction towards singularities, but entropy coefficients have to be increased by two orders of magnitude to make the effect visible, and thus should not be an issue in standard parameter regimes.
Conclusion Increased entropy regularization drives actions larger magnitudes, but fails to increase the diversity of matrix and quaternion actions. influences exploration, particularly in sparse reward environments, but does not change the final policy. Attractions to Buler singularities only become relevant with extremely high entropy bonuses.
Hypothesis 4 Unit rotation-centered policies improve performance for delta actions. Section 2.4.1 remarks that due to the projection onto S3)3) small, zero-centered actions can re- sult in large spread of rotations for quaternion and matrix representations. Centering the output around the unit rotation should lead to less erratic initial exploration. In addition, agents do not have to learn the group-specific representation of the unit operation, i.e., unit quaternions and rotation matrices.
Dence Reword Rotacion Contered Actions
-10
Heward
-00
2g
Aguent
Samples
x10
We test the hypothesis by adding the constant unit rotation to the policy mean. While F shows a clear performance improvement over the non- centered delta actions, we find that this mostly holds for PPO, and the same modification produces mixed results for TD3 and SAC. For details on the implementation and the complete set of results, including benchmarks from section refer to section A.3
Figure 3: PPO learning curves for unit rotation- centered delta actions using delta quaternions.
Conclusion Centering delta actions around the unit rotation improves the performance of quaternion and matrix representations in PPO. For SAC and TD3, the results are not conclu- sive. Delta tangent and Fuler actions unaffected as they centered by construction.
7

---

Published as a conference paper at ICLR 2026
Hypothesis 5 Scaling actions to the range of permissible rotations boosts performance. As outlined in section tangent vector increments in the local frame can easily be scaled. Restricting the network output the range of possible rotations should be more efficient because the agent does not need to learn that actions with the same direction and magnitudes larger than Omma lead to the same outcome. Furthermore, it removes discontinuities at the cut locus from the action space and allows more fine-grained control in the relevant action ranges.
We can see the effects in all three algorithms. Ablation studies with unscaled tangent increments and dense rewards consistently exhibit a performance difference of around 11.5 1.5 compared to scaled tangent vectors across PPO, SAC, and TD3. The effect smallest for PPO. where the policy initial- ization around zero prevents frequent sampling of unscaled actions that reach the cut locus. Agents trained on sparse rewards exhibit similar decrease in performance. More importantly, however, sparse reward agents sometimes take longer to converge to near-optimal policy in TD3, and in rare cases show degraded performance in SAC (see section A.1.5). SAC's failures are caused by the entropy bomus driving actions to regions of the representation with discontinuities and singularities as outlined above.
Conclusion Scaling tangent vectors to the range of permissible angles improves performance and stability. For PPO, pay special attention to the log standard initializations.
3.4 PRACTICAL RECOMMENDATIONS
We summarize recommendations for practitioners as follows:
Prefer delta actions in the tangent space of SO(3). They avoid projection, and for per-step rotations below I radians, the cut locus and Exp/Log singularities do not affect training.
Dense rewards can mitigate representation-specifee failures while sparse rewards amplify them. Pay special attention to representations sparse rewards.
Exploration in the local tangent space is relatively well-behaved. Common strategies like starting with small, zero-centered actions have adverse effects for quaternion and matrix representations.
Global rotation matrix and quaternion representations can outperform deltas because they need not learn the relation between agent pose and goal. This advantage may vanish when relative object poses matter.
Delta Euler angles are better than absolute Buler angles, but generally poor choice.
For additional effects and their explanation, section A.1
ROBOTIC BENCHMARKS
In real-world problems, rotation actions in SO(3) rarely act on purely rotational dynamics in section Instead, they appear in a larger context, such as controlling a robot arm's pose and gripper grasp objects, or controlling a drone's orientation and total thrust for trajectory tracking. In this section, we validate the relevance and transfcrability of our findings by applying the different action parameterizations on three robot benchmarks. Here, we focus on the most promising action combinations, i.e., global matrix and quaternion representations, and delta tangent and Euler angles.
The first benchmark demonstrates the effect of action representations on PPO drone control. We evaluate performance on two tasks: trajectory tracking and drone racing. In the first task, the agent must track a predefined figure-8 trajectory, standard benchmark in drone control extensively used to compare the performance of different algorithms. For GPU-accelerated training, we use a vectorized version of the safe-control-gyr benchmark (Yuan et al. 2022; Brunke et al., 2022). In the second task, we use similarly adapted version of the IROS 2022 Safe Robot Learning Competition (Tetteert et al.) 2025) to train an agent in autonomous drone racing. The aim is to cross four gates fast possible in the correct order while avoiding obstacles. PPO is the current state- of-the-art for drone control, particularly drone racing, using reinforcement learning (Song et al.] 2021 Kaufmann al. 2023). Following Geist et al. (0024), convert all SO(3) observations to
8

---

Published as a conference paper at ICLR 2026
Drone Trajectory Tracking
Drone Racing
60
100
Reward
Reward
20-
200
-20
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.5
1.0
1.5
255
Samples
x10
Samples
x10
Figure 4: Achieved reward for the trajectory tracking (left) and drone racing competition (right)
tasks across action parameterizations. Shaded areas indicate the standard deviation across 25 seeds.
rotation matrices. As in section we only change the action representation used by the policy. For details on the choice of hyperparameters, refer to section A.8
In figure 4 action representations significantly impact convergence speed on both benchmarks, Ac- tions in the local tangent space consistently outperform other representations by converging faster and achieving higher rewards. Surprisingly, Euler angles are second before rotation matrices and quaternions. The reason is the limited range of angles required in the tasks. The drone cannot devi- ate too much from the upright orientation without crashing, and hence the policy remains in a region of (3) where Euler angles are still well-behaved. In contrast, absolute quaternion and matrix actions are highly random at initialization (see section section[A.I.I), which leads to fast crashes and limits progress. Ablations in section A.3 show that for these unstable environments, unit-centering (see section 2.4.1 and hypothesis in section 3.3 is a key factor for performance. is highly advisable use relative and unit-centered representations this class of systems.
Next, we benchmark action representations on RoboSuite (Zhu et al. 2020), a simulation framework implementing a suite of manipulation environments leveraging MuJoCo (Todorov al. 2012)) The reference baseline uses SAC with shaped rewards on nine tasks spanning from single-arm block lifting complex tasks such as peg-in-hole tasks with arms. As in section 3 we only change the action representation used by the policy. We follow the requirements outlined in the benchmark, training for 5M steps across five random seeds each using operational space control (Khatib) 2003) to convert from policy actions to joint torques. Figure presents the results. We report the mean performance and standard deviation as fraction of the maximum achievable reward in percent.
Robosuite Benchmark
Achieved Reward (%)
60
25
Average over
Block Lifting Black Stacking
Pick-and-
Not Assembly
Door Opening
Table Wiping
Two Arm Lifting Two Arm Pogi Ho-e
Two Arm Handover
Task
Figure 5: Achieved reward across the RoboSuite benchmark fraction of the maximum possible reward. Error bars denote the standard deviation across five seeds.
As expected, global actions do well on SAC with dense rewards. The dense reward compensates for exploration issues. Notably, quaternions outperform the matrix representation on several tasks. Contrary to section 3. local tangent actions, while competitive on most tasks, do not exceed the performance of quaternions. Global actions may benefit tasks requiring the arm move into a few select poses. Overall, the narrow performance gaps between these representations within the same task and larger performance gaps between tasks indicate that other factors, such reward design and overall task difficulty, dominate the benchmark.
In the last benchmark, robot arm control to include pose goals. We extend the agent's 's action space to include the gripper orientation and employ HER with TD3, the successor of the previously used DDPG (Lillierap al.
adapt the setup from Andrychowicz al. (017)) for goal-conditioned

---

Published as a conference paper at ICLR 2026
Reach Robot Arm Orientation
Pick and Place Cube Orientation
0-
Reward
20-
Reward
A,o.0,
*T
40
50
0
0.0
0.5
1.0
1.5
2.5
Samples
x10
Samples
x10
Figure 6: Achieved reward on teachOrient (left) and PickAndPlacses laceorrieneeeen (right). Both tangent and matrix action representations converge fast for the reach task, with quaternions second and Fuler angles last. On the harder pick and place task, the local tangent space representation significantly outperforms other representations both in performance and convergence speed.
2016). Analogous to the original reach and pick and place tasks, we define a reach task where the agent must lift its end effector into a target position and orientation, and pick and place task where the goal is to place the cube into a randomly sampled target position and orientation. The remaining fetch tasks, sliding and push, cannot easily be modified to include orientation goals and are thus omitted. For more details on the environment design, refer to section A.7
As in the previous benchmark, we analyze switching the SO(3) action parameterizations of the pol- icy. The results, averaged over five runs, are shown in figure 6 Policies with matrix and tangent representations quickly converge
near-perfect policy, quaternions follow slightly delayed, while Euler angles significantly lag. On the second task, the tangent representation again clearly outper- forms other representations at 69.8% success rate, with matrix second at 54.1%, quaternions third at 46.7%, and Euler angles last at 32.3%. Here, the combined randomness of the initial cube and target orientations requires the policy to cover a significant part of SO(3). Consequently, the differ- ences between representations become more pronounced, seen by the 2x increase in success rates between Euler and tangent representations.
5 CONCLUSION
Action representations on (3) shape exploration dynamics, entropy rewards, and smoothness of policies in deep RL. In this paper, we established that the choice of representations alone impacts performance and convergence speed. We analyzed the behavior of popular representations for PPO, SAC, and TD3, showed how the choice of representation affects learning, and gave clear recommen- dations for practitioners looking to train agents for orientation control. Delta actions in the tangent space offer the best performance, especially for tasks that cover all or a large subset of SO(3). While there are viable alternatives for specialized domains (e.g., where only few fixed orientations are required and global quaternion and matrix representations can be competitive, or with orientations close to identity, where Euler angles are an alternative), the tangent space overall serves as an effi- cient general representation across algorithms and tasks.
One limitation of this paper is its restriction state based observations and small networks. Ob- servations do not affect the action space; thus, our results likely still apply. However, this requires empirical evidence. addition, we did not consider discrete action space algorithms, which open up entirely new questions, e.g. discretization schemes and the required density of the SO(3) cover. We also noticed a lack of suitable benchmarks that require control over the full SO(3) manifold. Our extension to the HER environments could act starting point to build up standard set of tasks focusing on this capability. Finally, diffusion policies have found widespread adoption for imita- tion learning in robotics. A similar study suitable representation choices for diffusion may reach significantly different conclusions due to diffusion models' molti-moddllty and noise processes.
5.1 REPRODUCIBILITY STATEMENT
The code to reproduce our experiments is publicly available at crttub.coo/amaccenn primer includes the training seripts and instructions for running the experiments describe.
10

---

Published as a conference paper at ICLR 2026
5.1.1 ACKNOWLEDGMENTS
We thank SiQi Zhou and Marcel Rath for their helpful feedback on the manuscript, and Jan Brüdigam for the discussions on SO(3) representations and their impact on RL training. This work was supported by the Robotics Institute Germany under BMBF grant 16ME0997K, and by the Hum- boldt Professorship for Robotics and Artificial Intelligence.
REFERENCES
Naseem Alhousani, Hatice Kose, and Fares J. Abu-Dakka. Reinforcement Learning for Orientation the Lie Algebra. In 2023 3 Signal Processing and Communications Applications Conference (SIU), pp. 1-4, 2023a.
Naseem Alhousani, Matteo Saveriano, Ibrahim Sevine, Talha Abdulkuddus, Hatice Kose, and Fares J. Abu-Dakka. Geometric Reinforcement Learning for Robotic Manipulation. IEEE Access, 11:111492-111580 2023b.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, OpenAI, and Wojciech Zaremba. Hindsight Experience Replay. In Advances in Neural Information Processing Systems, volume 30, 2017.
Timothy D. Barfoot. State Estimation for Robotics. Cambridge University Press, 2017.
A.R. Barron. Universal Approximation Bounds for Superpositions of Sigmoidal Function. JEEE Transactions Information Theory, 39(3):33--945, 1993.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for Jlyper-Paramee Optimization. Proceedings of the 25th International Conference on Neural Information Pro- cessing Systems, pp. 2546-2554, 2011.
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela Schoellig. Safe Learning Robotics: From Learning-Basee Control to Safe Rein- forcement Learning. Annual Review of Control, Robotics, and Autonomous Systems, pp. 144, 2022.
Romain Brégier. Deep Regression on Manifolds: 3D Rotation Case Study, In 2021 International Conference on 3D Vision (3DV), pp. 166-174, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the 35th International Conference on Machine Learning, pp. 1587-1596, 10-15 15 Jul 2018.
A. René Geist, Jonas Frey, Mikel Zhobro, Anna Levina, and Georg Martius. Learning with 3D rotations: Hitchhiker's guide to SO(3). Proceedings of the 41st International Conference on Machine Learning. 2024.
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861-1870, 10-15 Jul 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: Variational Information Maximizing Exploration. In Proceedings of the 30th International Con- ference on Neural Information Processing Systems, NIPS' 16, pp. 1117-1125, 2016.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Mueller, Vladien Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620:982-987, 08 2023.
Oussama Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal Robotics and Automation, 3(1):33-55: 2003.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference Learning Representations, 2016.

---

Published as a conference paper at ICLR 2026
Alan Macdonald. Linear and Geometric Algebra. Alan Macdonald, 2011.
Valentin Peretroukhin, Matthew Giamou, David M. Rosen, W. Nicholas Greene, Nicholas Roy, and Jonathan Kelly, A Smooth Representation of SO(3) for Deep Rotation Learning with Uncertainty. In Proceedings of Robotics: Science and Systems (2S),20), Jul. 12-16 2020.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter Space Noise for Explo- ration. International Conference on Learning Representations, 2018.
Peter H. Schönemann. A Generalized Solution of the Orthogonal Procrustes Problem. Psychome- frika, 31:1-10, 1966.
Martin Schuck, Jan Brudigam, Sandra Hirche, and Angela Schoellig. Reinforcement Learning with Lie Group Orientations for Robotics. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 14369-14376, 2025a.
Martin Schuck, Alexander von Rohr, and Angela Schoellig. hty.sppttial.ransfomm::: Differen- tiable Framework-AAgostic 3D Transformations in Python. In EurIPS, Workshop on Differen- tiable Systems and Scientific Machine Learning, 2025b. URL httpss://rxxv..or////a
2511. 18157
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. ArXiv,
2017.
Joan Solå, Jérémie Deray, and Dinesh Atchuthan. A micro Lie theory for state estimation in robotics. ArXiv, abs/1812.01555 2018.
Yunlong Song, Mats Steinweg, Elia Kaufmann, and Davide Scaramuzza. Autonomous Drone Rac- ing with Deep Reinforcement Learning. In 2021 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pp. 1205-1212, 2021.
Spencer Teetaert, Wenda Zhao, Antonio Loquercio, Siqi Zhou, Lukas Brunke, Martin Schuck, Wolf- gang Hönig, Jacopo Panerati, and Angela P. Schoellig. Advancing Reproducibility. Benchmarks, and Education With Remote Sim2real: Remote Simulation to Real Robot Hardware. IEEE Robotics & Automation Magazine, 32(1):117-123, 2025.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt. and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.
Zhaocong Yuan, Adam W. Hall, Siqi Zhou, Lukas Brunke, Melissa Greeff, Jacopo Panerati, and Angela P. Schoellig. Safe-Control-Gymi A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in Robotics. IEEE Robotics and Automation Letters, 7(4): 11142-11149, 2022.
Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. On the Continuity of Rotation Representations in Neural Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Marthn Abhishek Joshi, Kevin Lin, Soroush Nasiriany, and Yifeng Zhu. robosuite: A Modular Simulation Framework and Bench- mark for Robot Learning, ArXiv, 0ttp/2009.12293. 2020.
12

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.I
ADDITIONAL EFFECTS
This section discusses additional effects and presents extended ablation studies related to section 3.3
A.1.1 PROJECTIONS OF NOISE SAMPLES
The properties of each representation affect the mapping of Euclidean noise samples when projected onto SO(3). We showcase how noise samples from identical distributions result in entirely different rotation distributions when projected. Then, we evaluate the effect of projecting action samples on PPO and SAC, two algorithms that utilize probability densities computed based on a Gaussian assumption.
Samples algebra n
Samples Lie aigetro m
Samples
Samples Lie di digebra '
Lie aigetra
(a) R ~ (T(N 0.3))
(b) n (ON( 2))
(c) ~ ITN(O, 0.3))
(d) ~ I(N 2))
Samples algebes
Samples in algebra
Samples ie aigshes
Sumple ie giehex
(e)* 4T ~~ (U,
(1) 8 ~ (0, 2))
(g) (6.0, v) ~ (O, (h) (b. 0, I((N( 2))
Figure 7: Samples from squashed Gaussian distribution projected onto the manifold using the projections IT outlined in section 2.4 Each action representation has its own characteristic distribution after sampling. Samples are visualized as 3D points in the sphere of the Lic algebra m.
Figure 7 depicts the projection of noise sampled from squashed Gaussian distributions onto SO(3), visualized in 3D through the Lie algebra m. The rotation matrix projection yields a distribution almost independent of the noise level, resembling uniform distribution. Tangent vectors remain normally distributed for small noise magnitudes, but concentrate at the boundaries of the 3D tangent space because of saturation. Quaternions display a uniform distribution similar to rotation matrices for small noise levels. Indeed, without clipping. projecting zero-centered Gaussian noise leads to a uniform cover of S3(3). At larger noise levels, the distribution of rotations concentrates along a nar- row sub-manifold. Finally, Euler angles act similarly to tangent vectors at smaller noise magnitudes, but concentrate around the two curves corresponding to the singularities for larger magnitudes.
For stochastic policies used in PPO and SAC, a natural question is whether projecting sampled actions to the valid representation manifold speeds up convergence, the critic never sees off- manifold actions. Ablation runs in figure g show this is false. Projections lead to a significant (s(ass) used in its clipped surrogate ob- performance loss in PPO, because the probability ratios T0
jective no longer matches the ratio of the unprojected action. Newly computed probabilities use projected actions, while the stored probabilities of the old policy are based non-projected actions. SAC remains unaffected as action probabilities are re-computed online during policy updates, with projections applied afterwards. However, we do not observe any performance improvements.
13

---

Published as a conference paper at ICLR 2026
PPO Reward. Gotal Actions
SAC: Dense Roward Glbbal Actions
-10-
Rewward
Reward
R2
R2
-30
-30
0.0
0.4
0.6
0.8
1.0
1.2
Samples
Samples
Figure 8: Learning curves for PPO (left) and SAC (right) evaluating the effect of projecting action samples onto the manifold for the global quaternion and rotation matrix action representations using dense rewards the idealized environment.
Based on our ablations, we recommend not projecting the actions of PPO and SAC after sampling, and instead relying the environment for the action projection. Introducing projections offers performance gains in the best case, and has the potential to severely harm convergence.
A.1.2 EULER ANGLE NONLINEARITIES AND DISCONTINUITIES
Fuler angles show very inconsistent performance across algorithms and benchmarks. This section analyzes why this the case and relates their performance to the nonlinearities and discontinuities outlined in section
In the idealized environments, Euler angles perform poorly almost everywhere. The only excep- tion PPO with dense rewards. The reason is that the relation between Euler angle rates, and by extension delta Euler angles, of a fixed magnitude and the induced angular change in orientation be- comes increasingly nonlinear, making harder to learn orientation control orientations far from the identity.
In PPO with dense rewards, the network initialization has adjusted to produce incremental actions narrowly focused around zero at the beginning of the training. Otherwise, agents do not converge successful policy. Adapted initialization partially alleviates the issue and leads suboptimal, but stable policy.
While delta Fuler angle chart highly nonlinear at large angles, is almost identical to the incremental tangent representation at small changes around the identity. We can see this in the drone control benchmarks in section 4 where A 0, v) performs nearly as well as "T. Stable drone flight only requires states in small region of (3(),, and the angular changes in the trajectory task are minimal. Hence, the performance of Euler angles is not surprising. In the drone racing task, drones flying more aggressively, and thus the performance gap to " increases.
In the RoboSuite benchmark, performance varies significantly, from being worse than other repre- sentations having minor differences. We attribute this to varying levels of pose control required to solve the tasks. On ReachOr ent and ReachP SeachPickAndp ckAndP lace, which cover a large subset of SO(3), (Ø) 0, v)) is again the worst among all policies.
Practitioners should avoid Euler angles for SO(3) action representations. Delta Euler angles can be successful on tasks that only require small angular changes, because this avoids the heavy non- linearities and singularities at larger angles. Examples are drone control at moderate speed and manipulation with minor orientation adjustments. However, they offer no advantages over local tangent increments "T, and yield worse performance as the required coverage of SO(3) increases.
A. 1.3 PROJECTING SAC'S 's ACTOR OUTPUT
SAC uses squashed Gaussian policy parameterization, which prevents the projection of the mean onto the manifold of its representation. Samples are generated using
To (s) tanh (u N (nr (s) oo (s))).
14

---

Published as a conference paper at ICLR 2026
Importantly, the squashing is applied after sampling. Consequently, projecting the mean before sampling will reduce the range of mean values the distribution can sample from. E.g., normalized Euler angles of [0, 0, as mean will become approximately [0, O, 0.76,, which makes reaching some orientations in S0(3) infeasible. Projecting actions after sampling does not improve performance shown in appendix A.I.1 Therefore, we keep SAC's actions completely off-manifold.
Combining off-manifold mean actions and SAC's ' maximum entropy formulation results in several entropy-related issues discussed in section A...7 with quaternions and rotation matrix represen- tations. However, we adhere to the commonly used policy parameterization for SAC due to its widespread use, squash actions once after sampling, and rely on the environment correctly project the policy's 's actions.
The mean actions can be projected in PPO and TD3 as samples are left unbounded (PPO) or use additive exploration noise with clipping (TD3).
A.1.4 EXPLORATION WITH EULER USING DELTA ACTIONS
Delta Euler angles outperform their global counterparts, but struggle in tasks that require full ori- entation control. Here, we show how wrapping Euler angles around the singularities at 0 = - dominates exploration behavior during training and leads to poor coverage of SO(3).
Starting from an orientation whose pitch angle 0g = and sampling normally distributed actions from.N(O. (0, o), the distribution of the next pitch angle 0 remains approximately normally distributed. However, if 0o is non-zero (e.g. 0o = )) samples exceeding = will wrap around. Thus, the 3z adds to that of I g] Therefore, the probability density of [ 3] outweighs density of 0, in 2. 4
that of [0, ]- Hence, the agent more likely to move closer to the singularity rather than away from it. This effect continues until the singularities, where one half of the Gaussian distribution is
mirrored onto the other half as shown in figure 9
Distribution of Pitch Angles from (H...0.3)
Distribution Pitch Angles from HER Buffer
Percentage Samples (30)
Percentage of Samples (1%)
Achieved gools 25%)
15
Achieved goals (25-50%)
Achieved goals (50-100%)
10
1.00
0.00
Pitch Angles (rad Xx/2
Pitch Angles (rad xx/2)
Figure 9: Distribution of pitch angles sampled from Gaussian distributions with different means and a standard deviation of 0.3 (left) and from the achieved goals stored in the HER replay buffer during training using SAC with delta Euler angles and sparse rewards in the idealized environment (right).
In the absence of dense reward that provides immediate feedback to policies about the quality of their actions, this effect severely slows down convergence and results in the high variance displayed by both SAC and TD3 for sparse rewards section A.2 Since policies are initialized randomly and rely on noise for exploration, large percentage of their initial trajectories end up near the singularities following this random walk. Accordingly, by re-labeling experiences through HER, they learn to reach these singularities consistently.
However, due to the highly nonlinear and discontinuous behavior of Euler angles near these points, policies fail to explore further regions of the goal space, resulting in the observed results for SAC and TD3. By inspecting the distribution of pitch angles for achieved goals stored in the replay buffer for SAC in figure 9 it is clear that points quarters of the training process. Only later on, during the second half of the training process, does the distribution of goals become more balanced.
the singularities dominate the first and second
15

---

Published as a conference paper at ICLR 2026
A.1.5 SCALING TANGENT VECTORS
Section 2.5 outlines how tangent vectors can be scaled to only encompass the range of possible rotations with a limited angle maaz. In section 3.3, we explain how this helps avoid the cut locus of the tangent space at large action norms. Here, we present the training curves of our ablations in figures 10 11 and 12 As previously stated, the final mean reward of scaled policies is slightly increased by 1.5 to
to more fine-grained control. Much of the unscaled policies' action space lies outside the Omaz limit and thus maps to the same actions.
S&C. Donce Reward. Tongent Compor
S&C. Sporse Reward, Tangent Comparison
-10
Reward
-20
-00
-30
*r unscaled Samples
P unscalled Samples
-00
0.0
0.2
0.4
10.6
0.8
1.0
1.2
13.0
03.2
0.4
0.6
0.8
1.0
1.2
Samples
Samples
x10'
Figure 10: Learning curves of the scaled and unscaled incremental tangent vector representation for SAC. We show the worst five among 50 unscaled runs to emphasize that the increased variance stems from small number of runs that fail to make significant progress.
Reward. Tangent Comparison
Sparse Tangent Comparison
-10-
Reward
20
-20
"r unscaled
-40
"p unscalld
Samples
Samples
-00
Samples
x10'
Samples
x10'
Figure 11: Learning curves of the scaled and unscaled incremental tangent vector representation for TD3. Again, show the worst five among 50 unscaled runs to demonstrate that the increased variance stems from a small number of runs with significantly slower convergence rates.
PPO: Desse Revard, Tangent Comparison
Howerd
-20
-30
" unscaled Samples
Samples
x10
Figure 12:
Learning curves of the scaled and unscaled incremental tangent vector representation for PPO. As before, we show the worst five among 50 unscaled runs. Dense rewards prevent conver- gence issues in PPO.
The effect is smallest for PPO, where the initialization of the policy leads to actions close to zero. For SAC and TD3, the effect is slightly more visible in dense environments. Agents in sparse environments infrequently converge with significant delay (TD3) fail converge (SAC), causing
16

---

Published as a conference paper at ICLR 2026
the increase in variance across The entropy bonus drives actions towards larger magnitudes around the cut locust before learning a reasonable policy, which causes the complete failures in SAC. Dense rewards prevent this by balancing the entropy bonuses with continuous signal towards successful policies from the beginning.
A.1.6 CONFLICTING POLICY GRADIENTS
Given the uni-modal policy parameterization used by all three studied algorithms, the full double- cover quaternions and partial overlap of the Lie algebra m for || > л, harm the learning process. Multiple optimal actions produce conflicting gradients that pull the policy in different directions. In the case of the quaternion double-cover, these actions point in opposite directions, yielding opposing gradient signals during policy updates.
Q-value
-1.2
Expl mTo)
-2.39
Rotation
Exp(ar,)
Exppp=xr
9 Exp((17))
(a) SAC Q-values along the double-cove of quaternions for sparse rewards.
Q-value
-3.208
Expl anl)
-1.337
Rotation
y Exp(x7,)
Expl (xx))
Esp((r,) -9
(b) TD3 Q-values along double-cover quaternions for sparse rewards.
Figure 13: Learned Q-values of SAC's and TD3's critics for sparse rewards using the global quater- nion action representation in the idcalized environment. We sample equally-spaced actions dis- cretely along the geodesic connecting q (s) and -q that passes through the goal orientation the S(3) manifold by applying the Exp of the unit-norm rotation vector Tg pointing in the direction of the goal. The Q-function clearly shows that the critics learn the multi-modal distribution.
Actor-critic algorithms such as SAC and TD3 rely on the critic for computing the policy gradient. Hence, conflicting gradients only appear if the critic does learn the bi-modal Q-function. To test this happens in practice, we analyze the learned Q-values between q (s) and q as shown in figure 13 We sample equally-spaced actions discretely along the geodesic connecting q and -q that passes through goal orientation on the S(3) manifold by applying the Exp of the unit-norm rotation vector Ty pointing in the direction of the goal. Both critics have learned nearly the same Q-values for -q and q, although the value for the actual action q is still slightly higher in both cases. Since the double-cover points in the opposite direction in Euclidean space, the critics will produce conflicting gradients for actions sampled on the hemisphere
(s)
Lastly, although not visualized here, the results directly apply to PPO's advantage estimates if both action representations are sampled during the same rollout.
A.1.7 EXTENDED RESULTS FOR ENTROPY REGULARIZATION IN PPO AND SAC
In this section, study the effects of varying entropy levels on PPO and SAC in the idealized en- vironment, For PPO, we scale entropy coefficients relative to the optimal value determined through hyperparameter tuning. SAC's target entropy is scaled relative to its default value of -di (AA).
Due to the non- Euclidean nature of rotation representations, increased entropy might not always lead to better exploration. For instance, vectors of larger magnitude in the tangent space attain
17

---

Published as a conference paper at ICLR 2026
higher directional stability. Thus, from the perspective of entropy maximization, actions of large magnitude are more attractive than their smaller counterparts. This correlation between entropy and action magnitudes in the tangent space be clearly in figure [4] Policies that fail to reach zero norm oscillate near the goal indefinitely due to a lack of sufficient exploration near the identity. This effect be mitigated by constraining the maximum rotation magnitude; hence, always recommend scaling local tangent increments. The effectiveness of this mitigation depends on the magnitude of the maximum rotation angle Omaz with smaller values being more beneficial.
PPO: Dense Reword. ta Actions
S&C: Dence Reward, Dela Actions
Entrogy coef. x0.25
Action form (xor
0.8
Entropy coef. x1
Actin Nor (xc)
0.8
Entropy coef. X41
0.6
0.6
0.4
0.4
Target entropy x11.25
0.2
0.2
Target entropy x1
Target entropy
20
Environment Stops
Environment Steps
Figure 14: Action norms for different entropy levels for PPO (left) and SAC (right) using the un- scaled delta tangent action representation in the idealized environment. Actions are evaluated across 3000 episodes with different goal orientations set at magnitude of æ. The agent's initial orientation always initialized to the identity rotation.
very similar effect manifests when using quaternions. Large imaginary components (x, y, =) make the rotation direction the effect of perturbations to the real component w on the rotation's magnitude. This follows from the relation that the rotation's 's magnitude 0 2 tan (V + y² /ww For SAC, this effect more prominent due to its mean actions remaining off off-manifold, allowing them to exceed unit norm.
robust noise perturbations. In addition, the norm of (ar, y, constrains
Fuler angles do not suffer from the same issues. Increased entropy coefficients lead to slight increase in action magnitudes, but this increase plateaus quickly, unlike in tangent and quaternion representations, which showcase an almost-monotonic effect. A likely explanation is that Euler angles become more non-linear as Euler angles become larger, which makes learning harder (see section A.1.2). At high entropy levels 100x above the baseline, singularities attract agents trained with Euler angle action representations. However, their performance deteriorates considerably in this parameter regime.
SAC: Dunn Roward Global Actions
SAC Spars Revand, Giubal Actions
Target entropy x0.25
Target entropy x0.25
Target entropy x1
2.5
Target
Target entropy xd
Action Norm
Target entropy x4
2.0
2.0
1.5
1.5
1.0 10
1.0
0
20
30
20
50
10
20
30
10
30
Figure 15: Action norms for different entropy levels for SAC across the dense (left) and sparse (right) reward settings using the global rotation matrix action representation in the idealized environment.
Lastly, rotation matrices with PPO are not affected by any entropy relatpd issues since the Frobenius norm of a 3D rotation matrix is constrained by the structure of the SO(3) manifold, such that |R]| = V3. However, due to SAC's off- manifold actions, entropy maximization provides an undesired incentive for policies towards increasing the Frobenius norm of their actions. This effect, which contributes to the poor performance matrix policies compared to TD3, can be seen clearly in figure 15 To explain why SAC's actions remain off-manifold, refer to section A.1.3
18

---

Published as a conference paper at ICLR 2026
A.2 FULL EXPERIMENTAL RESULTS ON THE IDEALIZED ENVIRONMENT
We test PPO with global and delta actions on dense rewards. Training conl-condi policies with sparse rewards makes little progress without HER; our analysis omits it. Since SAC and TD3 are off-policy algorithms compatible with HER, we train them with global and delta actions on sparse and dense rewards. All experiments use 50 runs per representation to ensure the significance of the results. Hyperparameters are tuned according to section A.8
This section completes the results presented in section 3.2
A.2.1
PPO RESULTS
Figure [6 shows the results. Rotation matrix representations yield the best performance for global actions. Despite its lower dimensionality, the quaternion representation is less successful. We tribute this to the double-cover as established in section A.1.6 Perhaps surprisingly, tangent incre- ments in the Lic algebra perform better than quaternions. Euler angles reach approximately the same performance as quaternions.
For delta actions, tangent vectors in the local frame display superior performance to other represen- tations, with Euler angles as a close second. Matrix and quaternion representations struggle to learn a good policy. Reducing the randomness the start of the training does not help these represen- tations, because vectors centered around O will still yield vastly different actions by projecting the mean onto the manifold.
PPO Danse Reward, Global Actions
P20: Donce Reward, Datta Actions
Reward
Reward
-20
-20
R
AR
-30-
-30
2q
06.0.31
00).0,0
4
Samples
Samples
x10'
Figure 16: PPO learning curves for dense rewards using global (left) and delta (right) action repre- sentations in the idcalized environment.
We note that the performance of the Euler representation is highly dependent on initializing the policy log-standards to a small value of 22. With default value of 0 used in many reference implementations, agents cannot learn any reasonable policy. This trick is specific to the Euler angle representation (see Aoction/A.1.2
We recommend the use of the delta tangent representation for PPO. It displays a reduced variance compared to global matrix actions, achieves a slightly better final performance, and avoids issues with projections small actions early on during training.
A.2.2 SAC RESULTS
For dense rewards and global actions, the rotation matrix representation converges fastest. We at- tribute this to the uniqueness and smoothness of the representation. Quaternions are second with decreased convergence stability because of the multi-modality introduced by the double-cover of S(3) (see section 1.6). For delta actions, the local tangent space representation significantly out- performs others and performs better than global actions. Attaching the tangent space to the local frame always puts the cut-locus on the farthest side from the current orientation and prevents from impacting uniqueness and continuity. All results are shown in figures 17 a [8
The results for sparse rewards differ significantly from the dense case. While the matrix represen- tation remains the best for global actions, its performance is well below the optimum. The other representations improve more slowly and remain worse in performance. An analysis of the trained policies reveals that the entropy maximization in combination with the inability to project actions
19

---

Published as a conference paper at ICLR 2026
(see section A.1.3) learns to maximize the entropy regularization before the critic provide meaningful policy gradient. Since subsequent exploration is based on the behavior of the policy and not random noise (e.g.. in TD3), agents cannot recover from this collapse. Tangent and Euler policies are unaffected but still suffer from discontinuities and singularities.
Delta actions show similar severe degradation in performance for matrix and quaternion repre- sentations for the same reason as in the global case. Tangent spaces are unaffected and achieve near-optimal performance. Delta Euler angles outperform the matrix and quaternion representa- tions, but remain significantly below the tangent representation and experience the largest variation between training runs (see section A.1.4..
3AC: Cense Reward. Global Actions
SAC Dense Reward, Deta Actions
-10-
-10-
Reward
20
-30
Ag
0l)
A().0,0)
1.
1.2
Samples
X10
Samples
x10*
Figure 17: SAC learning curves for dense rewards using global (left) and delta (right) action repre- sentations in the idealized environment.
Ressed, Actions
SAC: Spannar Delta Action
10
-20
Reward
-20
(..0.0)
-10
-10
Ag
(
-50
0.0
0.2
0.1
0.1
0.6
0.8
1.
1.2
Samples
x10
Samples
x10
Figure 18: SAC learning curves for sparse rewards using global (left) and delta (right) action repre- sentations in the idealized environment.
As with PPO, recommend using the delta tangent space representation with SAC. If global action spaces are required, practitioners should opt for matrix or quaternion representation, but have to ensure that dense rewards are available to prevent agents from collapsing into degenerate action outputs.
A.2.3 TD3 RESULTS
As in SAC, the rotation matrix representation TD3 converges fastest and displays the highest performance for dense rewards and global actions in figure 19 Quaternions are second again, for the same reasons as for SAC. Again, the local tangent space representation significantly outperforms all others for delta actions, and achieves superior performance compared to global actions.
The results in figure 20 representations. The matrix representation has more apparent advantage in the global frame, while Euler and the Lie algebra representations struggle to learn a successful policy. the local frame, the tangent representation converges almost immediately the optimal policy. On the other hand, matrix, quaternion, and Euler angle representations display large variances between training runs and are often unable to find successful policy.
for for sparse rewards show the same results, but amplify the differences between
As in PPO and SAC, our recommendation for TD3 is to use the delta tangent space representation. If global action spaces required, practitioners should opt for matrix quaternion representa-
20

---

Published as a conference paper at ICLR 2026
tion. Contrary SAC, exploration is less of an issue with these representations due to the policy- independent random exploration noise. Hence, matrix and quaternion representations also work with sparse rewards.
Rewerd, Actions
TD3: Darne
Delba Actions
-10-
-10
Reward
T
-00
T
R
Ag 0))
(l))
10
1.2
Samples
X10
Samples
x100
Figure 19: TD3 learning curves for dense rewards using global (left) and delta (right) action repre- sentations in the idealized environment.
Sparse Revard, Gobal Actions
TD3: Sprrte Reward, Delta Actions
-10
-20
Reward
-90
-30
-30
R
-10
-10
Ag 0)).0,
(6..0,
-30
-50
1.2
Samples
x10
Samples
x100
Figure 20: TD3 learning curves for sparse rewards using global (left) and delta (right) action repre- sentations in the idealized environment.
A.3 EXTENDED DISCUSSION OF UNIT ROTATION IETATION-CENTEE CENTERED DELTA ACTIONS
Section 1.4.1 discusses that policy networks using quaternion and matrix representations are not centered around the unit rotation for delta actions, potentially leading to worse performance. To address this, the network output can be modified such that a zero output corresponds to the identity rotation unit quaternion qr). We investigate two distinct implementations of this centering strategy for quaternion and matrix representations and evaluate their impact on learning stability and final performance.
A.3.1 COMPARISONS OF IMPLEMENTATIONS
We denote the output of the neural network qnes and Rner, respectively. We define two variants for mapping this output to a pre-projection action que and Reee, which are subsequently projected to obtain the valid rotation.
Additive Bias.
The simplest approach is to add the identity element to the network output.
aoee
Onet + qI
R R
Rnet I
This variant ensures that if quet or Ragt R O, the resulting action close to the identity. However, this method limits the set of reachable actions. Assuming tanh activations bound One within |-1, 1]. the scalar component qpre is restricted to the range 20. Consequently, the policy cannot represent the conjugate unit quaternion of directly. A similar property holds for the matrix representation.
21

---

Published as a conference paper at ICLR 2026
Scaled Bias. To mitigate the range limitation of the additive variant, also implement the mapping with a scaling factor dependent on the identity element.
Jpre
Juet 3 (qr 1) 91
Rpe Rest @ (R
Here, C denotes element-wise multiplication and By scaling the network output, the policy can reach all possible quaternion configurations, including the negative hemisphere. However, this introduces non-uniform sensitivities, since the scalar com- ponent w and the diagonal elements of the rotation matrix, respectively, induce larger changes in the resulting rotation magnitude compared the additive variant.
tensor of ones with matching dimensions.
A.3.2 RESULTS ON THE IDEALIZED ENVIRONMENT
We benchmark both implementations against the standard, uncentered parameterizations in ide- alized environment. Results are averaged over 50 runs as in the main text.
PPD: Rewer for Centered
10
10
Reward
20
R
Aq
AR
Ratiie
Alba
H
Samples
x10
Samples
x10"
Figure 21: PPO learning curves for unit rotation-centert delta actions using delta quaternion (left) and matrix (right) action representations in the idealized environment.
For PPO, the modified relative actions provide clear benefit. Both converge slightly faster than their absolute variants and significantly improve over the uncentered delta actions (see figure 2T). Comparing the two variants, the additive mapping achieves slight performance advantage over the scaled one. For quaternions, only the additive variant achieves similar performance to the absolute representation. For the matrix representation, both variants converge lower performance than the absolute representation.
Retation Actions
for Unit Resation-Cnntanel
-10-
-10
Reward
-20
-20
R
Aq
AR A Ragatt
Aquatti
Alpard
A
1.2 x10
Samples
x10
Samples
Figure 22: SAC learning curves for unit rotation-centered delta actions and dense rewards using delta quaternion (left) and matrix (right) action representations in the idealized environment.
The results for SAC present contrast to PPO. As can be seen in figure 22 in dense rewards, the scaled variant slightly improves over regular delta and absolute actions for quaternions, whereas the additive one leads significantly worse performance. The matrix representations are almost unchanged compared to the default implementation. In sparse rewards (see figure 23). the additive version performs slightly worse compared to the default delta orientation, whereas the scaled version slightly outperforms it. None of the three, however, yields a successful policy. The same is true for the rotation representation.
22

---

Published as a conference paper at ICLR 2026
SAC: Reward for Unit or Concered
SAC: Reward for Unit Contared
10
-10
AR
-20
Aquitiin
Reward
-0
A Razssie
A Raceol
-30
-30
-40
-00
0.0
0..2
014
00.5
1.0
1.2
0.0
01.2
0.4
0.6
0.8
1.0
1.2
Samples
Samples
Figure 23: SAC learning curves for unit rotation-centered delta actions and sparse rewards using delta quaternion (left) and matrix (right) action representations in the idealized environment.
TD3: Dense Reward for Unit Rotat Cencered Actions
TD3: Dense Revard for Unit Rocat Centered Actions
-10-
Rewarrd
Rewurd
H
Aq
-30
Aoeeen Aguard
-30
Ressov
Ru
0.0
0.22
014
0.6
0.8
1.0
1.2
0.0
01.2
0.4
0.6
0.8
1.0
1.2
Samples
Samples
Figure 24: TD3 learning curves for unit rotation-centered delta actions and dense rewards using delta quaternion (left) and matrix (right) action representations in the idealized environment.
TD3 shows a similarly mixed picture in figure 24 where both variants slightly underperform with dense rewards compared the default quaternion implementation, and have significantly increased variances between runs. The matrix representation shows no real difference between the default implementation and the improved versions. In sparse rewards, both modified quaternion representa- tions outperform the baseline, but still exhibit significant variance and fall short of the consistency and performance of the absolute representation. The modified matrix representations in figure [25 significantly outperform the baseline and approach the performance of the absolute representation, but have a higher variance between runs.
Sparse Roward
Sporse
20
Reward
-20
R
Aq
AR Rassun
-40
-40
-00
Aquart
-00
Samples
x10'
Samples
x10*
Figure 25: TD3 learning curves for unit rotation-centered delta actions and sparse rewards using delta quaternion (left) and matrix (right) action representations in the idealized environment.
Our experiments indicate that while unit-centering addresses some of the issues associated with delta rotations, its practical impact varies by algorithm. The performance difference is negligible for many configurations. The notable exceptions
PPO, where centering is crucial for competitive performance of delta actions, and SAC, where the scaled variant is favorable for quaternions. In the specific case of PPO, observe slight advantage of the additive variant over the scaled one.
23

---

Published as a conference paper at ICLR 2026
Trajectory Tracking: Unit-biased Relative Actions
Trajectory Tracking: R Unit-biased Relative Actions
9
2g
Reward
100
0-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Samples
Samples
Figure 26: PPO learning curves for the drone trajectory tracking environment using delta quaternion (left) and matrix (right) action representations with unit rotation bias.
Drone Racing: q Unit-biased Relative Actions
Drone Racing: Unit-biased Relative Actions
40
Reward
201
-20
Ag qi
1.5
2.0
Samples
x10'
Samples
x10'
Figure 27: PPO learning curves for the drone racing environment using delta quaternion (left) and matrix (right) action representations with unit rotation bias.
Revisiting Unstable Systems with Unit-biased Relative Actions. section 4 we hypothesized that zero-centered actions are particularly beneficial for unstable robotic systems, such as drones, where maintaining the initial attitude is safer than a random rotation. Centering actions around the unit rotation should theoretically allow quaternion and matrix representations to stabilize and converge faster on these tasks.
Since the additive variant performed best for PPO, selected it to validate this hypothesis on the trajectory tracking and drone racing benchmarks. We compare this centered variant against the global action baseline for both quaternion and matrix actions. For fair comparison, we retune the hyperparameters using the same sweep settings as for the baseline.
Our results confirm the hypothesis. The modified relative quaternion actions in figure 26 demon- strate dramatically improved convergence speed compared to uncentered variants. Similar holds true for matrices, although neither manage to outperform the local tangent representation. By initializing the policy output effectively at the identity rotation, the drone is less prone to erratic initial behavior that leads immediate crashes, thereby allowing the agent to gather meaningful experience earlier in the training process.
While less dramatic, results the drone racing task in figure 27 clearly show the trend. We conclude that for unstable systems, the unit-centering of representations is a key influence on performance.
A.4 COMPARISON OF ROTATION MATRIX AND 6D REPRESENTATIONS
Beyond the standard parameterizations discussed in the main text, several previous studies have explored suitable rotation representations for learning. One influential representation, particularly noted for its properties in optimization and supervised learning contexts, the continuous 6D rep- resentation proposed Zhou et al. (201).
The idea behind this representation is that a full 9-element rotation matrix over- wver-paaameterize. Defining two orthogonal column vectors is sufficient reconstruct the complete orientation without
24

---

Published as a conference paper at ICLR 2026
losing the uniqueness and smoothness of the representation. Since rotation matrix belonging to SO(3) describes a right-handed coordinate system, the third axis entirely defined by the cross product of the first two. Implementation-wise, the policy outputs 6D vector R that is treated as two raw 3D vectors. To ensure orthogonality and unit norm, we then apply the Gram-Schmidt process to these vectors.
While this representation enjoys a lower dimensionality than the standard flattened rotation matrix, Geist al. (2024) point out that it empirically falls short of the standard SVD-projected matrix representation on several tasks. Since the arguments regarding continuity and smoothness apply similarly to both the 6D representation and standard matrices, and given that rotation matrices are more prevalent in standard robotics pipelines, we selected matrices for our primary comparison. In this section, we provide a direct comparison demonstrate that the performance of the 6D repre- sentation is indeed comparable to, but rarely strictly better than, the standard matrix representation in reinforcement learning settings.
A.4.1 RESULTS ON THE IDEALIZED ENVIRONMENT
We benchmark the 6D representation against the standard matrix representation across all algorithms with runs per variant as the main text.
Danar Renard,
10
Howard
200
R
AR
A60
Samples
x10
Figure 28: PPO learning comparison between the 6D representation and rotation matrices in the idealized environment for both absolute (straight lines) and delta (dotted lines) actions.
For PPO, the 6D representation slightly outperforms the standard matrix representation in terms of both convergence speed and final performance when using absolute actions. Conversely, for delta representations, the 6D parameterization falls short of the matrix baseline (see figure 28
The results for SAC, visualized in figure 29 present a similar picture. In the dense reward setting, the performance of relative and absolute representations is practically indistinguishable between the two parameterizations. In the sparse reward setting, we observe a minor improvement for the 6D representation across both absolute and relative modes. However, is important note that neither
SAC Denee Rrwardd 6D vs. Matric Actions
BAC Spane Revord, 6D Matrix Actinns
10
10
Reword
Reward
20-
-30
R
A.R
AR
RD
6D
A6D
A6D
-50
0.0
0.2
0.6
0.8
1.0
1.2
Samples
X10
Samples
x10"
Figure 29: SAC learning comparison between the 6D representation and rotation matrices in the idealized environment for absolute (straight lines) and delta (dotted lines) actions on dense (left) and sparse (right) rewards.
25

---

Published as a conference paper at ICLR 2026
7D3: Donce Rewerd,
Actions
TDS: Sparse Reward. 30 Matri Actions
-10
-10-
Reward
-0
-30
-30
60
-40
26D
36D
0.0
0.22
0.4
0.55
1.0
1.2
0.0
0.4
0.6
0.8
1.0
1.2
Samples
Samples
Figure 30: TD3 learning comparison between the 6D representation and rotation matrices in the idealized environment for absolute (straight lines) and delta (dotted lines) actions on dense (left) and sparse (right) rewards.
the matrix nor the 6D representation successfully learns high-quality policy under these sparse reward conditions, sharing the same failure modes and training characteristics.
In TD3, the standard matrix variants slightly outperform their 6D counterparts generally. The only meaningful difference exists between the delta matrix and the delta 6D actions under sparse rewards, where the matrix representation converges consistent with our main results, both representations yield relatively poor policies in this setting.
better policy average (see Figure 33)). However,
Overall, the performance curves of matrix and 6D representations very similar across with absolute actions shows a slight preference for the 6D representation, whereas TD3 generally favors the standard matrix representation. possible explanation for this result is the increased bustness of the full matrix representation to large disturbances such as the uniform exploration noise used TD3, compared the Gram-Schmice reconstruction used in the 6D mapping. However, emphasize that these performance differences are relatively minor and should not be overinterpreted as definitive advantage for either representation.
PPO
Since the 6D representation slightly outperformed its matrix coun- terpart on PPO (see figure 28 we rerun the robot benchmarks that use PPO for trajectory following and drone racing.
Robot Benchmarks for PPO
Trajectory Tracking: 6D vs. Matrix Actions
Drone Racing: 6D vs. Matrix Actions
40
6DD
Reourd
20
.00
6D
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
1.5
2.0
2.5
Samples
Samples
Figure 31: PPO learning comparison between the 6D representation and rotation matrices on the trajectory following task (left) and on drone racing (right).
The results figure 31 display an inconclusive picture. While the 6D representation improves notably on the trajectory following task, the deviations between runs are large. Other effects such as unit-centering of delta actions (see section A.3) are significantly more consistent in improving convergence. The results racing again in line with intuition that the two representations perform very similar. Although matrix marginally outperforms the 6D representation, the difference not significant.
26

---

Published as a conference paper at ICLR 2026
A.5 IMPLEMENTATIONS
Even when the underlying concepts clear, putting (3())-specific operations into code can be difficult. We explicitly outline some of the important operations mentioned in the paper to help practitioners transfer the results.
We detail how we rotate around tangent vectors *T in the body frame, implement exponential and logarithm maps for rotation scalings, and perform differentiable matrix orthonormalization inside neural network layers. All components are implemented using standard scientific Python libraries only. Note that there is ongoing work within SciPy (Virtanen et al.) 2020) toward rotation routines compatible with most deep learning frameworks, further simplifying implementations (Schuck et al., 2025b).
Listing I: Rotate an orientation by a local tangent vector
import numpy as np from spt..spppaall.aassfatoo.... import Rotation R
4 current_orientation
R. R.deenitt())
5 Action in 1-1, " from the policy network
6 action
().ni(fommmm) animorr(-1, 1, size-3)
Apply the tangent vector to the current orientation as delta action
next_orientation current_orientation R. roov (action)
Listing 2: Limiting rotations to maximum angular change
import numpy np
from scipy wat.yyysspaaall..asftfoooo import Rotation
4 juzrent_orientatiin = R. R.identity()
5 Limit the rotation to a maximum of pi/10 radians caa__rotticnnto np.pi / 10
Calculate the next orientation
the rirettion of the global reference
8 action mp.array(IO, 1, 0, 01) # Example quaternion action
9 Compute the difference between current and target orientation delta = current prvrenttorrentationn inv() R. R. quat (action) Check if the rotation angle exceeds the maximum allowed. If so, scale
12 the magnitude to at most the maximum allowed rotation
B3 scale = mp.minimum (1, max_rotaticn / delta. magnitude
Apply the scaled delta to the current orientation. Power on rotation is equivalent to log -> scale exp next_orientation current_orientation delta
Listing 3: Differentiable matrix orthonormalization in PyTorch
import torch
def orthenormalization (x: terch. Tensor) ->
u, _ vh wwrch..inalg...mm
5 Ensure det without in-place changes to (breaks backprop)
6 unorm torch. zeros_ ikk (u)
7
unorm[..., :2] u vl..., :2]
unorm[..., 2] ul ..., 21 forch.det. (u 8 vh)
return unorm è vh
action torch,randn (3, 3) normaaction = orthonormalin zat ion (action)
Orthogonal, det 1
27

---

Published as a conference paper at ICLR 2026
A.6 ENTROPY ON THE MANIFOLD
The standard entropy bonus term employed in PPO and SAC is
H-
Σ log (2)
(5)
A policy that parametrizes rotation actions quaternions with zero norm and unit variance all action dimensions will lead to a uniform distribution on the S(3) sphere after normalization and thus uniform distribution of rotations with maximum entropy. If shift the mean of tu towards 1, the entropy in equation remains constant, while the actual entropy of the rotation distribution declines. The decline is evident because the entropy of the distribution the sphere
[
[ p(x) log (d(((r) 5(3)
H[X]
(6)
with dli as the surface area measure and random variable X E S(3) has its maximum at p(r) 27 One can show this the global maximum with variational argument where the entropy functional is concave in p.
The implication for training RL agents is that agents can increase their mean to higher values to concentrate the distribution of rotations in SO(3) while still receiving the same entropy bonus as uniformly distributed rotations, thus effectively breaking entropy regularization.
A.7 FETCH ORIENTATION ENVIRONMENTS
The ReachOri Lent and Pick PickAndP ceer r ent environments (see figure 32 used in section benchmark TD3 are directly inspired by the Fetch environments (Andrychowicz et al., 2017 However, to make them compatible with the extended scope of environments, made some modifications, which are listed below.
The most fundamental change is the extended action space. In addition to the arm's position and finger joints, agents also control the arm's 's orientation. The maximum angular change of the arm between two environment steps is z radians, as in our idcalized rotation control environments.
goal orientation has extended the environment's goal. the case ReachOrient, this target orientation is sampled by randomly rotating the start orientation by at most ī radians. We limit the goal orientation to prevent infeasible configurations that are physically infeasible for the arm. PickAndP 1 acceriennt samples its orientation goal similarly with respect to the cube orienta- tion. As in the original environment, half of the goal positions are sampled uniformly above the table, and half sampled on the table itself. Goal orientations on the table are sampled be phys- ically feasible. The cube is randomly rotated at the start of each episode by sampling which side is facing down and then rotating uniformly around its s-axis.
Example rollout of the p ckAndP aceOrient environment. The agent has pick up the cube and place it into the same position and orientation as the goal frame. The frame located to the right of the robot arm indicates the goal pose.
Figure 32:
Changes in the goal formulation also require change to the reward function. Agents re- ceive reward of 0 if the position and orientation of the arm (Reachorient) or cube
28

---

Published as a conference paper at ICLR 2026
Table Number runs per environment for hyperparameter optimization
Environment
Number of Runs
Idealized Rotation Environments
100
Drone Trajectory Tracking
100
Drone Racing
(eickkndPlaceOrleett) are both within their respective tolerances of 5cm and To radians, and everywhere else.
Andrychowicz et al. (017) do not consider orientation control, and thus it has no influence on the choice of robot. However, the Fetch robot shows a limited range of reachable position and orienta- tion targets. We thus exchange the robot with the Franka FR3, one of the most common robot arms in robotics research. The gripper remains from the original environments to limit implementation variations.
A.8 HYPERPARAMETER CHOICES
The hyperparameters used for each algorithm and action representation are optimized per environ- ment using Bayesian optimization (Bergstra al., 201T). Table lists the number of runs used for hyperparameter tuning across each environment.
We determine separate sets of hyperparameters for the idealized rotation environment on the sparse and dense reward settings. However, delta and global actions of the same representation share their hyperparameters. The action viewpoint does not significantly change the choice of hyperpa- rameters in trial runs. Across the Fetch environments,
reuse the same hyperparameters used by Andryyhhwwcc al. (2017) to maintain consistency with prior results. Similarly, for the Robo- Suite (Zhu et al.) 2020) benchmark, we adopt the same original hyperparameters used for SAC.
All our experiments use identical MLPs for both policy and value networks. For PPO, we use 2- layer networks with 64 units per layer and tanh activations. The only exception is drone racing, where we find that using 128 units per layer, as per (Kaufmann et al., 2023). is beneficial. Note that similar increases in the number of units per layer across other tasks with PPO negatively affect performance. Both SAC and TD3 use 3-layer networks with 256 units per layer and ReLU activa- tions. However, for SAC's RoboSuite benchmark, we use 2-layer networks instead, following the architecture originally used in the benchmark for full pose control.
A.9 USAGE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) for wording and proofreading individual sections. addi- tion, they were used as programming aid to create the TikZ figures. The ideation phase made no use of LLMs.
29


Reviews:
[{"review_id": "ZVHJLgPVIu", "reviewer": "Reviewer_1zXm", "summary": "This paper studies how the representation of rotation in actions affects RL training. The authors compare different types of SO(3) action representation in a range of RL environments, including an ideal rotation control environment and some more realistic robotic benchmarks. From the results, the authors recommend using tangent vectors for practitioners.", "questions": "Please refer to the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The SO(3) action representation is a commonly encountered problem by RL practitioners, especially in the field of robotic control. This work provides valuable guidance for the design of the action space and environments.\n\n2. The impact of different rotation representations is studied in a diverse set of benchmarks, and all the hypotheses are accompanied by detailed analysis. They all make the conclusions convincing.", "weaknesses": "1. The considered scope of this work is somewhat limited. Only the simple MLP network + Gaussian noise as policy representation is studied. Other popular design choices, such as discretizing actions into bins and learning a categorical distribution over the discrete action space could also be investigated.\n\n2. The insights/takeaways from experiments could be better presented. In Section 3.3, the authors state several hypotheses but the conclusions are not highlighted. Since the results are quite mixed for different representations, rewards and algorithms, I think the presentation of the analysis could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VwYihSFI6R", "reviewer": "Reviewer_a6NH", "summary": "The paper analysis how SO(3) rotation actions should be parameterized to facilitate reinforcement learning using PPO, SAC, and TD3. In particular, the paper analyzes how using the changes in rotation as action representation could benefit learning. First the paper discusses the different rotation representations and how to predict changes in rotation from deterministic / stochastic policy networks. In a simply toy example, the authors show that indeed the different rotation representation affect learning in particular for sparse reward settings. Afterwards, the authors discuss different hypothesis on why certain rotation representations show better performance. In particular, the authors consider the effect of smoothness on learning, interplay of rotation projections and exploration / entropy regularization, and action scaling. Finally, the paper provides extensive experimental benchmarks that underline the importance of rotation representations for reinforcement learning in SO(3).", "questions": "Please see Comment 1 above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "The paper looks at a critical issue for the reinforcement learning community that so far has been mostly overlooked. The paper is well structured, clearly written, and the experiments are extensive and thorough. Considering how rotation projection affects exploration in RL is quite clever. I generally enjoyed reading the paper and consider it an excellent fit for ICLR.", "weaknesses": "**Comment 1: Each delta rotation representation requires special treatment to perform well.**  \n\nIn Section 2.4, the authors outline how the Euclidean network outputs are mapped onto the rotation manifolds. For deterministic networks, the action are simply projected onto the respective manifold via a differentiable projection layer. For stochastic policies (as far as I can guess from the very brief description), the network mean is projected onto a rotation manifold, then points are sampled around this projected vector in Euclidean space, and these points are also projected onto the rotation manifold. **These approaches on obtaining rotations from deterministic/stochastic policies  are the basis of all of the subsequent discussion and therefore quite important.** \n\nThe problem is that neural networks are designed around zero-centered activations which with standard parameter initialization output zero-centered output distributions. This becomes a problem, if we want to fairly compare rotation action representation using rotation changes. The **unit rotation for lie algebra rotations and Euler angles is a zero-vector**. However, the unit rotation for quaternions and rotation matrices equals one or several unit vectors. In turn, if a network has to directly predict changes in quaternions or rotation matrices, then it has to first learn the unit rotation operation while for lie algebra rotations this is clearly not the case. In the paper (Table 2), the authors conclude that using changes in rotation is only sensible for the Lie Algebra rotation and Euler angles while predicting changes of quaternions and rotation matrices shows drastic drops in performance. I bet this is because of the aforementioned difference in the representation's respective unit rotation vectors. \n\nI also suspect that Lie Algebra changes in rotation could be particularly well suited for modelling rotation actions. However, if setup correctly, I am positive that modeling changes in rotation via quaternions and rotation matrices would not show such drastic decreases in performance as reported by the authors if setup \"correctly\". As the author's paper acts as a primer for the RL community, not carefully evaluating how changes in rotations need to be parameterized could lead to rushed conclusions.\n\nTo provide a fair comparison between different representations of changes in rotation, I suggest you **model changes in quaternions / rotation matrices by adding the network output to the unit  rotation and then project the resulting vector onto the rotation manifold**. For stochastic policies, you also could add the network mean to the unit rotation and then sample in Euclidean space around this mean.\n\nThe authors have now the following courses of action:\n\n1) Make a convincing argument why my above assessment is flawed (e.g. you did exactly the approach I have described above). If convincing, I increase my rating to an accept.\n2) Extend the discussion and experiments in Section 2 and 3 to consider the above proposed approach to predicting changes in Quaternions / Rotation matrices. If done thoroughly, I increase my rating to a strong accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hx45Wg4Oq3", "reviewer": "Reviewer_gmAW", "summary": "Authors investigated the importance of 3D rotation representations in the context of reinforcement learning. Authors further discussed 4 rotation representations, i.e., rotation matrix, quaternion, Euler angle, and Lie algebra, in absolute and relative coordinates. Extensive simulation experiments are performed on drone control as well as manipulation problems. Authors concluded that on average, the relative Lie algebra is the best performant representation for RL applications. To explain the advantage of relative Lie algebra, various hypothesis are presented.", "questions": "Table 1 is very helpful for explaining different rotation representations. Could authors add similar table to summarize the representations on Table 2? E.g., R stands for rotation matrix, \\delta R stands for relative rotation matrix.\n\nHow does Figure 2 visualize SO(3) rotation? Figure 2 seems visualized a S2 sphere, which only has 2 DoFs, but SO(3) has 3 DoFs.\n\nWhy Figure 3 only compared relative rotation representations? What do absolution representations perform?\n\nFor Figure 4, an additional bar that shows the average of each representation would be helpful for overall comparison.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Despite previous studies on rotation representations in the context of supervised learning, this paper explores rotation representations in the RL setup, which is an important topic but is ignored by previous works. This paper provided extensive comparisons on different RL benchmarks, ranging from drop control to manipulation. This paper also compared different RL algorithms, including stochastic and deterministic algorithms. The hypothesis point interesting explanation of the advantages of relative representations.", "weaknesses": "One important baseline seems messing: 6D rotation representation [1], where the neural network first outputs two 3D vectors, then uses Gram-Schmidt to normalize these vectors and thus reconstructs the rotation matrix. How would this baseline, and perhaps it’s relative version, performs in the RL tasks?\n\n[1] On the Continuity of Rotation Representations in Neural Networks. Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, Hao Li.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)