Beyond Distributions: Geometric Action Control for Continuous Reinforcement Learning
Authors: Zhihao Lin
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
geometric control
spherical normalization
bounded action spaces
continuous control
action generation
distribution-aware policy optimization
Primary area: reinforcement learning
TL;DR: We propose Geometric Action Control (GAC), a distribution-free policy architecture that generates actions directly on the unit sphere, eliminating tanh-squashing and entropy tuning while achieving higher returns with 50% fewer parameters.

Abstract: Gaussian policies have dominated continuous control in deep reinforcement learning (RL), yet they suffer from a fundamental mismatch: their unbounded support requires ad-hoc squashing functions that distort the geometry of bounded action spaces.
While von Mises-Fisher (vMF) distributions offer a theoretically grounded alternative on the sphere, their reliance on Bessel functions and rejection sampling hinders practical adoption.
We propose \textbf{Geometric Action Control (GAC)}, a novel action generation paradigm that preserves the geometric benefits of spherical distributions while \textit{simplifying computation}.
GAC decomposes action generation into a direction vector and a learnable concentration parameter, enabling efficient interpolation between deterministic actions and uniform spherical noise.
This design reduces parameter count from \(2d\) to \(d+1\), and avoids the \(O(dk)\) complexity of vMF rejection sampling, achieving simple \(O(d)\) operations.
Empirically, GAC consistently matches or exceeds state-of-the-art methods across six MuJoCo benchmarks, achieving 37.6\% improvement over SAC on Ant-v4 and up to 112\% on complex DMControl tasks, demonstrating strong performance across diverse benchmarks.
Our ablation studies reveal that both \textbf{spherical normalization} and \textbf{adaptive concentration control} are essential to GAC's success.
These findings suggest that robust and efficient continuous control does not require complex distributions, but a principled respect for the geometry of action spaces. Code and pretrained models are available in supplementary materials.

Published as a conference paper at ICLR 2026
BEYOND DISTRIBUTIONS: GEOMETRIC ACTION CON- TROL FOR CONTINUOUS REINFORCEMENT LEARNING
Zhihao Lin
James Watt School of Engineering
University of Glasgow
Glasgow, UK (88004011) glaa
all.ac acc.a.c.. ac.uk.
ABSTRACT
Gaussian policies have dominated continuous control in deep reinforcement learn- ing (RL), yet they suffer from fundamental mismatch: their unbounded support requires ad-hoe squashing functions that distort the geometry bounded action spaces. While von Mises-Fisher (vMF) distributions offer theoretically grounded alternative on the sphere, their reliance on Bessel functions and rejection sam- pling hinders practical adoption. We propose Geometric Action Control (GAC), a novel action generation paradigm that preserves the geometric benefits of spher- ical distributions while simplifying computation. GAC decomposes action gen- eration into direction vector and a Icarnable concentration parameter, enabling efficient interpolation between deterministic actions and uniform spherical noise. This design reduces parameter count from 2d to d + 1, and avoids the (dk) complexity of vMF rejection sampling, achieving simple (dd operations. Em- pirically, GAC consistently matches or exceeds state-of-the-art methods across six MuJoCo benchmarks, achieving 37.6% improvement over SAC on Ant-v4 and up to 112% on complex DMControl tasks, demonstrating strong performance across diverse benchmarks. Our ablation studies reveal that both spherical normaliza- tion and adaptive concentration control are essential to GAC's success. These findings suggest that robust and efficient continuous control does not require com- plex distributions, but principled respect for the geometry of action spaces.
INTRODUCTION
Continuous control OOpenAl et al. 2019) remains one of the most challenging problems in rein- forcement learning (RL) (Silver et al. (2014), with applications ranging from robotics to autonomous driving (Seo et al. 2025). At the heart of this challenge lies fundamental design choice: how should agents generate continuous actions? For over a decade, Gaussian policies have served as the default answer, powering algorithms from Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al.) 2015) to Soft Actor-Critic (SAC) (Haarnoja et al. 2018) and achieving remarkable success across diverse domains. Their popularity stems from mathematical convenience, including closed-form entropy, straightforward reparameterization, and well-understood optimization properties.
Yet this convenience masks a fundamental mismatch. Physical systems operate within bounded action spaces, while Gaussian distributions have infinite support Nikishin et al. (2021). The standard solution applies squashing functions like tanh to map samples into bounded regions (Theile al., 2024), but this transfornation distorts the distribution's geometry, creates gradient flow issues near boundaries, and breaks the natural symmetry of the action space (Fujimoto et al.) 2018). As policies become more deterministic during training, actions cluster boundaries where tanh's 's gradient vanishes. We observe this phenomenon Appendix B). Such instabilities often misattributed to insufficient exploration, while in fact they reflect a deeper geometric mismatch between Gaussian policies and bounded action spaces.
40% SAC training steps on HalfCheetah (Figure
Recent work has begun questioning this Gaussian orthodoxy Davidson et al. (2022). von Mises- Fisher (vMF) distributions offer a mathematically principled alternative by operating directly on the unit sphere, naturally respecting bounded constraints (Michel et al. 2024). However, vMF's theo-

---

Published as a conference paper at ICLR 2026
retical elegance comes at steep computational cost YYou et al. 2025 sampling requires rejection methods with O(dk) complexity where k is the expected number of rejections. and density com- putation involves modified Bessel functions prone to numerical overflow (Mazoure al., 2019). Other alternatives like normalizing flows or mixture distributions add expressiveness but compound the computational burden (Obando-Ceron al., 2024). This creates dilemma: accept Gaussian's geometric limitations or pay the price of computational complexity.
We take different path. Rather than seeking increasingly sophisticated distributions, we ask whether the distribution paradigm itself is necessary. Actions in physical systems naturally decom- pose into direction and magnitude. For instance, robot arm moves toward target direction with some force, and a car steers at angle with some acceleration. This geometric intuition suggests that effective action generation might not require explicit probability modeling at all.
This insight leads to Geometric Action Control (GAC), which generates actions through direct geo- metric operations on the unit sphere. GAC represents policies through two components: direction network that outputs unit vectors indicating preferred action orientations, and a concentration net- work that controls exploration by interpolating between deterministic directions and uniform spher- ical noise. This decomposition transforms the complex problem of sampling from sophisticated distributions into simple linear interpolation, reducing computational complexity from (dk) to ((d) while maintaining the geometric consistency that bounded action spaces demand.
Our key contributions are:
We introduce GAC, a distribution-free action generation paradigm that replaces probabilis- tie sampling with direct geometric operations on the unit sphere, challenging the necessity of distributional modeling in continuous control.
We develop compact and efficient policy architecture requiring only d+1 parameters in- stead of 2d for Gaussian policies, achieving comparable or better performance with reduced complexity.
We provide theoretical analysis showing that spherical mixing achieves vMF-like concen- tration without Bessel functions, and ablations demonstrating that spherical geometry and adaptive concentration are critical to GAC's success.
We provide comprehensive empirical evaluation across MuJoCo and DMControl bench- marks, demonstrating consistent improvements especially in high-dimensional control, val- idating that geometric consistency outweighs distributional sophistication.
Beyond immediate performance gains, GAC represents conceptual shift in how we approach pol- icy design. By demonstrating that geometric structure can replace distributional complexity, we open new avenues for developing efficient, interpretable, and theoretically grounded control algo- rithms. We believe our results champion a broader "Geometric Simplicity Principle": that for many robotics and control tasks, explicitly modeling the geometric structure of the action space is a more effective and efficient path forward than pursuing ever more sophisticated probability distribu- tions. The remainder of this paper is organized as follows: dection@revier related work, Section presents the GAC methodology, Section 4 provides empirical evaluation, and Section 5 concludes with discussions of broader implications.
RELATED WORK
2.1 GAUSSIAN POLICIES AND THEIR LIMITATIONS
The dominance of Gaussian policies in continuous control traces back to the natural policy gradi- ent literature, where Gaussian distributions provided tractable gradient estimates and convergence guarantees. Modern deep RL algorithms like SAC, Proximal Policy Optimization (PPO) (Schul- man et al.] 2017). and Trust Region Policy Optimization (TRPO) (Schulman et al.) 2015) inherit this choice, implementing Gaussian policies through neural networks that output mean and variance parameters. While this approach has driven impressive empirical success, practitioners have long recognized its limitations in bounded action spaces. The standard tanh squashing solution, popular- ized by SAC, maps Gaussian samples to bounded intervals but introduces well-documented issues:
2

---

Published as a conference paper at ICLR 2026
gradient vanishing near boundaries, asymmetric action distributions, and the fundamental contra- diction of using infinite-supper support distributions for finite spaces (Bendada et al. 2005).. Despite these known problems, the field has accepted them necessary trade-offs for computational convenience.
2.2 BEYOND GAUSSIAN: ALTERNATIVE DISTRIBUTIONS
Recognition of Gaussian limitations has spurred exploration of alternative policy parameterizations. Beta distributions model bounded intervals but scale poorly to multivariate settings and lack repa- rameterization for efficient gradients (Chou et al. 2017). Normalizing flows offer expressive flexi- bility via invertible transformations, yet their computational cost (2-3x slower than Gaussians) and training instability hinder widespread use in RL (Ghugare & Eysenbach, 2025). Mixture models enhance expressiveness but exacerbate boundary issues and risk mode collapse (Haarnoja et al., 2017.. A different line discretizes continuous actions into atomic bins (Tang Agrawal 2020) /Zhu et al. 2025), enabling multimodal policies and simpler optimization, especially in on-policy meth- ods like PPO. However, discretization sacrifices resolution and requires careful binning strategies to preserve action semantics. In contrast, GAC maintains continuous action spaces while replac- ing complex distributional modeling with geometric operations, offering stable, fine-grained control without compromising expressiveness.
The most principled alternative emerged from directional statistics (Simii et al. 2024). vMF distri- butions, operating directly on the unit sphere, elegantly address boundary constraints through their geometric formulation. Recent work (Scott et al. 202T) demonstrated vMF policies could match exceed Gaussian performance while providing theoretical advantages. However, vMF's practical adoption faces significant hurdles (Banerjee et al. 2205)): sampling requires rejection methods with acceptance rates as low 0.1 for high concentrations, likelihood computation involves modified Bessel functions (rc) prone to numerical overflow for large and the concentration parameter lacks intuitive interpretation for practitioners (Zaghloul & Johnson (025). These challenges have confined vMF policies largely to theoretical investigations rather than practical deployment.
2.3 GEOMETRIC PERSPECTIVES IN RL
Parallel to distributional innovations, a geometric perspective on RI has gained traction (Hu et al., 2022).
Hyperbolic RL and Riemannian policy optimization extend learning to non-Euulidean man- ifolds Nickel & Kiela 2017 Wang et al. 2024 Müller & Montúfar 2024), offering richer repre- sentations but often at the cost of added algorithmic complexity. In contrast, our work leverages geometry to simplify rather than sophisticate. The link between action spaces and geometry also surfaces in domain-specific contexts, such as quaternion bassed rotation policies or circular distri- butions for periodic locomotion (Wang et al. 2025 Zhou et al. 2019). yet these insights remain fragmented. GAC unifies these scattered geometric intuitions into cohesive framework for contin- uous
the focus from choosing the right distribution to questioning whether explicit distributions are necessary at all.
2.4 SIMPLIFICATION AS INNOVATION
The evolution from TRPO (Schulman et al. 2015) to PPO (Schulman et al. 2017) exemplifies a cru- cial pattern in deep RL: dramatic simplification often yields superior practical performance. Where TRPO required complex conjugate gradient procedures and line searches, PPO achieved compara- ble or better results through simple clipped objectives. Similarly, Twin Delayed Deep Determinis- tic policy gradient (TD3) (Fujimoto et al. 2018) simplified DDPG's actor-critic architecture while improving stability and performance by 30% on average. These trends suggest that algorithmic complexity in RL often reflects a lack of structural clarity, rather than theoretical necessity.
GAC follows this simplification philosophy. Rather than adding sophistication to handle Gaussian limitations implementing complex vMF sampling, identify the minimal geometric structure necessary for effective control. This approach aligns with recent trends toward interpretable and efficient RI,, where understanding why methods work matters much empirical performance.
3

---

Published as a conference paper at ICLR 2026
2.5 THE MISSING PERSPECTIVE: ACTION GENERATION WITHOUT DISTRIBUTIONS
Most approaches to continuous control operate within the distributional paradigm: policies are de- fined as probability densities over actions, requiring likelihood evaluation, entropy regularization, and absolute continuity (Engstrom et al. 2020). This perspective, inherited from supervised learn- ing and classical statisties, may be unnecessarily restrictive for control, where actions are ultimately deterministic functions of states and randomness.
GAC replaces this distributional machinery with a geometric operation. Instead of modeling a den- sity over IR generate actions via direction sampled on the unit sphere and scalar magnitude. This reframes control not as modeling a distribution, but as directly generating structured actions. By replacing distributional complexity with geometric operations, GAC eliminates density evalua- tions, reparameterization tricks, and explicit entropy calculations while avoiding gradient saturation from tanh squashing. Recent theoretical work (Tiwari et al.) 2025) shows that RL trajectories tend to concentrate on low-dimensional manifolds, using complex mathematical analysis to uncover this emergent structure. GAC inverts the perspective: rather than discovering manifolds, we build on them. By constraining actions to the unit sphere, we achieve structure by design, not by accident. From emergent complexity to designed simplicity- GAC exemplifies a principle in RL: structure need not emerge; it can be constructed.
3 PROBLEM FORMULATION AND METHODOLOGY
3.1 PROBLEM FORMULATION
We consider the standard continuous control setting where an agent interacts with an environment through bounded continuous actions. The action space A S 11 represents normalized physical constraints, where d is the action dimension. The agent observes states E S and selects actions according to a policy S + A.
The maximum entropy RL framework augments the standard objective with an entropy term to en- courage exploration: (=) Er,,, I>== 7 (rz + (=( [sx]]]], where 7 [0, 1) is the discount factor, r = r+*,, ar) denotes the reward at time step I, a > 0 is the temperature parameter control- ling exploration-exploizaee
trade-off, H denotes entropy, and T (So: ag. $1 a..... .) represents
trajectories sampled from the envicyeenvironment interaction.
The Geometric Mismatch. Most continuous control methods model policies as Gaussian distributions (Np(ss, E(s)) with unbounded
Direction Head
Sompie
Shared Back bone
Spherical Mixing w(x) sigmoid(e)
support, requiring tanh squashing map sam- ples into 1 While this approach has evidenced by
Concentration Head
Compute x
proven highly successful, SAC remaining a leading method, it creates fundamental mismatch: unbounded distribu- tions must be compressed into bounded action spaces. The tanh transformation achieves this compression but induces gradient saturation when |ā,| is large, with analysis showing substantial pre-squashed samples fall in low- gradient regions (Appendix B). SAC addresses this through strong entropy regularization (a s 0.2), maintaining exploration despite reduced gradients. GAC takes different path: rather than mitigating the mismatch through entropy- driven exploration, we eliminate it by operating directly on the unit sphere. This geometric ap- proach ensures consistent gradient flow and en- ables adaptive exploration through learned concentration, offering a structurally simpler alternative that aligns policy support with environmental constraints by design.
State
Sompie
Transition
normaline((w()) (1- w(x)
Actor Loss (-<(s) (isaa)
Critic Loss (TD Ecror)
Figure 1: Architecture of GAC. State is pro- cessed by a shared backbone, which branches into direction head producing unit vector μ4, and a concentration head predicting k. The final ac- tion is generated via spherical mixing, replacing traditional distributional sampling with direct ge- ometric interpolation.
4

---

Published as a conference paper at ICLR 2026
3.2 GEOMETRIC ACTION GENERATION
Core Insight. GAC replaces traditional action sampling with a geometric pipeline consisting of direction mapping, concentration control, and spherical mixing, illustrated in Figure [ Rather than modeling probability distributions over actions, we directly generate actions via geometric operations on the unit sphere, natural choice that aligns with high-dimensional concentration phenomena where directions carry the primary semantic information. This shift climinates the need for density-based computations such as log-probabilities or entropy. while preserving exploration through structured spherical noise, modulated by learnable concentration parameter K.
Direction Mapping. A neural network fp S + Rd produces raw directional vectors, which are normalized to the unit sphere:
fu(s)
(s)
(1)
where || 112 denotes the I.2 norm. This normalization ensures (sss) E gd-1, the unit sphere dimensions.
Concentration Control. A separate network fx : > R predicts concentration scores, which modulate the trade-off between deterministic direction and stochastic noise. These scores are trans- formed via sigmoid function w()) = (xx (0, 1) to produce the mixing weight used (2).
enabling smooth interpolation and stable, adaptive exploration throughout training.
Spherical Mixing, Actions are generated by interpolating between the deterministic direction and uniform spherical noise:
a = r normalize (w(K) + v(()) E),
where E ~ (oniform($44)) is sampled as normalized Gaussian noise to provide isotropic exploration on the unit sphere, and is a task-dependent scaling parameter (default r 2.5, see Sec. [.5 for details). Even with substantial noise contribution (c.g., 30% when x z 1), actions remain coherent: spherical normalization preserves directionality while preventing magnitude corruption, ensuring stable control throughout training.
Intrinsic Exploration. In contrast to conventional approaches where exploration is externally in- jected (e.g., Gaussian noise or entropy bonuses), GAC inherently encodes stochasticity within the action generation process. The random direction & is not auxiliary perturbation but integral part of the policy's structure, making exploration an intrinsic geometric property. The learn- able concentration 6 acts as endogenous control signal, adaptively modulating the exploration- exploitation trade-off without external regularization. This eliminates the need for separate entropy bonuses or temperature scheduling (e.g., tuning SAC).
3.3 THEORETICAL JUSTIFICATION
The spherical mixing operation creates an implicit distribution with geometrically intuitive concen- tration control. While flosed-form density intractable, we rigorously establish how the mixing weight controls distribution concentration.
Theorem 1 (Expected Direction Control). For GAC's spherical mixing operation, the expected unnormalized sample vector lies precisely along the mean direction, scaled by the mixing weight:
[eg[v] = w(K)
+(1 (1- w(x)) is the unnormalized mixture, la E gd-1 is the mean direction, Uniform( (d-1)))))00)) is uniform spherical noise, and w(k)) a()) is the mixing weight with sigmoid function a(x) = 1/(1 (1+1+ -
where v = w +
This result is exact: w(K) directly controls the expected alignment with μ.. As increases (w(()) - !1), variance vanishes and samples concentrate around μ. yielding viF-like concentration without Bessel function computations. See Appendix A. the proof.
3.4 INTEGRATION WITH SAC
GAC naturally integrates into the SAC framework by replacing the standard Gaussian policy with geometric action generator. The key distinction lies in exploration: GAC eliminates explicit
5

---

Published as a conference paper at ICLR 2026
probability computations and entropy regularization, achieving exploration instead through geomet- ric mixing controlled by n.
3.4.1 EXPLORATION CONTROL MECIIANISM
GAC introduces a learned exploration controller x(s) that adaptively modulates the balance between deterministic actions and stochastic exploration. Unlike SAC's temperature parameter a which re- quires manual tuning scheduling, i6 learns directly from the value landscape. In the maximum entropy framework, the actor seeks maximize expected return while maintaining exploration. For GAC, this objective becomes:
Lactor (0) = EgND k(()
min Qo, (3. 1=1
(4)
where D is the replay buffer, denotes the actor parameters, and a is generated via GAC's geometric mechanism in (2) with current state s. The term K(s)) serves as a learned exploration controller that replaces entropy regularization. Smaller values of increase the contribution of stochastic noise in the geometric mixing defined in (2). thereby promoting exploration. In contrast, larger values lead to more deterministic actions. This removes the need for temperature tuning in SAC. Unlike traditional entropy-based methods, GAC never computes probability densities. Instead, exploration emerges directly from geometric structure. This design theoretically justified by directional statistics, where higher concentration naturally corresponds to lower entropy (see Appendix[A.A The soft Q- function update follows standard SAC with our exploration controller. The target value incorporates the minimum of two Q-networks for stability:
v(r,, 8') re +7 min Qo, -a(())))(()))
(5)
where f denotes the parameters of the target --network, the next state, and a' is generated from s' using GAC's geometric mechanism in (2) Despite replacing Gaussian policies with geometric action generation, GAC maintains the essential properties for convergence the SAC framework. The bounded action space and smooth geometric operations ensure that the soft Bellman operator remains a contraction (see Appendix A.5 for formal analysis).
3.5 PRACTICAL CONSIDERATIONS
Parameter Efficiency. The fixed-radius GAC requires only d + 1 outputs (direction vector plus scalar concentration), compared to 2d for diagonal Gaussian policies (a 50% reduction in action head parameters). The adaptive scaling variant (Fq. 6 uses 2d + outputs, comparable to Gaussian policies but with explicit geometric structure that decouples direction, exploration, and magnitude. Computational Efficiency. The sampling procedure involves only normalization and linear interpo- lation, avoiding rejection sampling or special function evaluations. The computational complexity is (dd) per sample, compared to ((dk) for vMF sampling where k is the expected number of rejections (typically t [2, 10] for high concentrations).
Hyperparameter Selection. The geometry of sigh--immnsional spheres implies that a unit vector in Idd has an expected per-dimension magnitude of |||, x 1/vd (e.g., & 0.24 for d = 17). Without scaling, these inherently small magnitudes would yield ineffective actions. We introduce fixed radius r for principled rescaling. With = 2.5 and a typical mixing weight w(k) z 0.85, the resulting per-dimension actions fall in the 0.6-0.9 range, well within the normalized bounds [- 1, I] for effective actuation.
While this default = 2.5 works robustly across diverse tasks, we find specific environments like Ant-v4 benefit from adjusted scaling (r 1.0) for finer multi-leg control. Crucially, GAC exhibits low sensitivity to the exact value of r; performance within the range 3150, typically varies by less than 10% (see Appendix D.2). This confirms that acts stable geometric scaling factor, not a fragile hyperparameter. For tasks requiring maximal flexibility, we next introduce a learnable per-dimension variant.
Adaptive Magnitude Scaling, To handle environments with asymmetric dynamics or fine- fine-graine grained coordination needs, we extend GAC with learnable magnitude vector Rdd This variant replaces

---

Published as a conference paper at ICLR 2026
the scalar in Eq.
normalize (w((k) je (1 w(k)) E),
where denotes element-wise multiplication. The vector is produced by small network head with a softplus activation to ensure positivity. This adaptive scaling preserves GAC's core geometric structure while enabling degree of fine-grained control that exceeds the capabilities of traditional Gaussian policies, which rely solely on variance tuning. Experiments in Section 4.2 validate the effectiveness of this enhanced variant on complex DMControl tasks.
4 EXPERIMENTS
Environments. We evaluate GAC on six standard MuJoCo benchmarks: HalfCheetah, Ant, Hu- manoid, Walker2d, Hopper, and Pusher, with action dimensions ranging from 17.
Baselines. We compare against SAC (Gaussian + tanh), TD3 (deterministic + noise). and PPO (clipped objectives), using recommended hyperparameters from CleanRL (Huang et al. 2022) All implementations are standardized for fair comparison. See Appendix C.I for details.
Training Protocol. All algorithms are trained for 1M environment steps with 8 parallel environ- ments, except Pusher-v4, which rums for 500K steps due to faster convergence. We use 5 random seeds (0, 10, 42, 77, 123} and report mean episodic returns + standard deviation.
4.1 MAIN RESULTS
Table I:
Performance on MuJoCo benchmarks with fixed action radius (r = 2.5 for most tasks, 1.0 for Ant-v4), Bold indicates best performance.
Environment
GAC (Ours)
SAC
TD3
PPO
Hopper-v4
1952 + 285
2094 + 604
2896 749
2118 124
Walker2d-v4
5165 + 334
5152 + 608
4457 + 457
2874 + 517
Pusher-v4
-32
-23 2
-27
-78 +
HalfCheetah-v4
12750 + 758
12540 + 517
12208 + 799
1608 + 793
Ant-v4
5633 + 158
4094 + 1039
3531 + 1263
1969 778
Humanoid-v4
5823 + 121
5717 + 123
5819 + 278
619 1 59
e000
Eglestic Nenure
3000
Bleedit Notura
Egitsdit Return
2000
100
2000
1000
5GC
SAC
-125
6BC
-150
0.2
0.4
1.0
0.0
0.4
1.0
0.0
0.2
Training Stepe (Million)
Training Steps (Million)
Training Stepe (Million)
12500
Leposett Hetwn
10003
Egieadic Reture
4000
7300
5000
2000
2500
SCC
SAC
SAC
0.0
0.2
0,4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.0
1.0
0.0
0.2
0.4
0.6
0.B
1.0
Training Steps (Million)
Training Steps (Million))
Training Steps (Milllion)
Figure 2: Learning curves on (a) Hopper-v4, (b) Walker2d-v4, (c) Pusher-v4, (d) HalfCheetah-vás (e) Ant-v4, and (f) Humanoid-v4.
Performance Analysis. Table and Figure 2 present our main experimental results across six MuloCo benchmarks. GAC demonstrates strong performance, achieving the best results on 4 out of

---

Published as a conference paper at ICLR 2026
6 tasks and remaining highly competitive others. As expected, PPO performs substantially worse across all environments, consistent with its known limitations in exploration and entropy scheduling.
Learning Efficiency. Beyond final performance, GAC exhibits superior learning dynamics. In Figure d-e), GAC shows faster initial learning on Ant environments, reaching near-optimal per- formance by 200k steps while SAC and TD3 continue improving until 400k steps. This efficiency stems from GAC's geometric structure eliminating the need for entropy tuning, as the learned k naturally balances exploration and exploitation without manual temperature scheduling.
High-Dimensional Control. GAC demonstrates strong performance and robustness across com- plex, high-dimensional tasks. On Humanoid-v4 (17D), it achieves 5823 + 121 during training. matching TD3 (5819 + 278) and exceeding SAC (5717- + 123). More notably, GAC reaches 6591 = 53 in post-training evaluation, indicating its ability to learn high-quality policies while main- taining effective exploration. On Ant-v4 (8D), GAC outperforms SAC by 37.6% (5633 + 158 vs. 4094 + 1039) and TD3 by 59.5% (3531 1263), with significantly lower variance-highlighheeee GAC's stability in multi-leg coordination. On HalfCheetah-v4 (6D). GAC achieves the highest return of 12750- + 758, slightly exceeding SAC and outperforming TD3 by 4.4%. These results collectively validate that spherical normalization and geometric action modeling enable GAC to scale gracefully with action dimensionality, achieving both reliable exploration and consistent policy quality in challenging continuous control settings.
Trade-off Between Stability and Expressiveness. GAC achieves stable learning with low variance across most environments, particularly in high-dimensional tasks where its bounded geometry miti- gates the exploration instabilities of squashed Gaussians. However, it underperforms on Hopper-v4 v4 and Pusher-v4, revealing trade-off:: spherical normalization stabilizes training but imposes ge- ometric prior favoring near-unit-norm actions. While effective for norm-concentrated locomotion tasks, this constraint can limit asymmetric or contact-rich control. These limitations motivate our learnable scaling variant (Eq. [6] Sec. 4.2), which introduces per-dimension adaptivity while preserv- ing GAC's geometric structure.
Table 2: Performance on DMControl suite. GAC uses the adaptive scaling variant (Eq, 6
Environment
GAC (Ours)
SAC
TD3
PPO
fish-upright
858 + 35
923 + 5
866 + 39
311 + 78
walker-walk
960 = 4
956 + 10
952
186 + 11
walker-run
742 + 15
700 + 56
753 67
69 + 16
cheetah-run
762 + 24
661 185
150 32
quadruped walk
925 + 17
690 336
873
131 + 18
quadruped-ru
638 + 75
8
576 =
119 22
1000
1000
000
(x)
Lppootte Return
800
Epioadic Rotura
Eptendl Meturn
600
600
100
400
400
200
SAC
SAC
5AC
0.0
0.1
0.2
0.3
0.5
0.0
0.1
0.2
0.5
0.0
0.1
0.2
0.3
0.4
Training Steps (Million)
Training Steps (Million)
Training Stegs (Million)
1000
(6)
(0)
Lipontte seeum
R020
Ephaniic Returt
6:00
80
400
200
SAC
200
200
100
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2
0.3
0.44
05
0.0
0.1
0.2
0.3
0.4
05
Training Steps (Million)
Troining Steps (Million)
Training Stops (Million)
Figure 3: Learning curves on (a) fish-upright, (b) walker-walk, (c) walker-run, (d) cheetah-run, (e) qualr,ped-walk, and (f) quadruped-run.
8

---

Published as a conference paper at ICLR 2026
4.2 GENERALIZATION TO DMCONTROL SUITE
We further evaluate GAC's generalization on six challenging tasks from the DMControl suite (Tassa al., 2018). including fish-upright, walker-walk, walker-run, cheetah-run, quadruped-walk, and quadruped-run, with more complex contacts, asymmetric coordination, and bigher-dimensional ac- tion spaces than standard MuJoCo tasks. To accommodate these challenges, we adopt the GAC- Scale variant (Eq. (). which augments the core geometric structure of GAC with learnable per- dimension magnitude control. All experiments follow the same protocol as described earlier.
As shown in Table 2 and Figure GAC-Scale matches or surpasses SAC on 5 out of 6 tasks, with particularly strong gains on quadruped environments (+34%+1112%). These tasks require coordi- nated multi-leg movement and diverse actuation patterns, capabilities that are difficult to achieve with fixed-radius or diagonal Gaussian policies. The learned scaling vector r enables fine-grained, task-adaptive control while preserving the exploration benefits of GAC's spherical normalization.
Analysis of scale dynamics (Appendix D) shows that r values stabilize within the theoretically motivated range 3.0) after initial exploration (~100K steps), with task-specific convergence profiles. For instance, Walker tasks settle to nearly uniform values around 1.7-2.4, closely matching the fixed r 2.5 baseline. In contrast, Quadruped tasks exhibit more heterogeneous scaling patterns across dimensions, reflecting the need for anisotropic control across legs and joints.
These results confirm that GAC's geometric framework generalizes effectively to complex, contact- rich environments. The fixed-radius design is sufficient for symmetric locomotion, while the learn- able extension introduces critical flexibility in asymmetric scenarios, achieving substantial perfor- mance gains without compromising the architectural simplicity or stability of the base method.
Table 3: Ablation study of GAC components on HalfCheetah-v4. Results averaged over 5 seeds.
Configuration
Final Return
Relative
Key Observation
GAC (default with x and r = 2.5)
12750 + 758
baseline
Optimal balance
Target magnitude ablation:
335.5 r 1.5
12229 + 422 7272 + 1235
-4.1%
Slightly over-scaled actions
-3.00
Severely limited action range
Component ablation:
w/o k controller
11370 = 643 Diverged
-1.8%
No adaptive exploration Gradient explosion at 5k steps Unbounded actions, NaN loss
w/o normalization
N/A
Raw action output
Collapsed
N/A
4.3 ABLATION STUDIES
We conduct ablations HalfCheetah-v4 as- sess the target magnitude r (Table 3 Figure
12500
Target magnitude is critical. Geometrically, a unit vector in I2d has expected per-dimension magnitude E|]pa|] 2 1/vd (e.g., ~0.24 for d 17), making unscaled actions too weak for control. This is reflected in our results: re- ducing r from 2.5 1.5 causes 43% perfor- mance drop due to insufficient actuation and exploration, while increasing r to 3.5 yields only minor decline (-4.1%), indicating satu- ration. With 2.5 and typical mixing weight w<k) 0.85, the resulting action amplitudes 0.9. fall neatly within [-1, providing an effective balance for locomotion.
Episodic Return
10000
7500
5000
(Defail)
2500
355
0
web k (m22.)
0.0
0.2
0.4
0.6
0.8
1.0
Training Steps (Million)
Figure 4: Ablation study on HalfCheetah--v, De-
fault GAC (r 2.5, adaptive k) performs best.
Adaptive exploration via K is essential. Removing the learnable x controller while maintain- ing 2.5 reduces performance by 10.8% and slows convergence. The learned x enables state-

---

Published as a conference paper at ICLR 2026
dependent exploration by providing high concentration in confident states while maintaining diver- sity in uncertain regions. This adaptive behavior emerges naturally from the value-based objective without explicit curriculum or scheduling. Notably, this mechanism provides elegant alternative to entropy-based regularization: rather than maximizing entropy uniformly, GAC modulates explo- ration geometrically through directional mixing. This not only simplifies the optimization pipeline by eliminating entropy terms, but also improves interpretability, as k(s) can be viewed as a soft confidence score indicating how deterministic the policy should be at given state. As result, exploration becomes structure- aware and implicitly guided by the task dynamics.
Geometric structure ensures stability. As shown in Table 33 removing normalization leads to divergence within 5k steps due to unbounded gradients, as output norms grow without spherical projection and trigger explosion. Likewise, using raw unbounded outputs collapses immediately (NaN losses) as actions exceed environment limits and destabilize training. These ablations confirm that GAC's spherical geometry is essential for stability. Beyond preventing numerical issues, the constraint shapes a consistent optimization landscape where all actions share equal norm, removing scale ambiguity and acting as an implicit regularizer against degenerate solutions.
5 CONCLUSION
This work demonstrates that effective continuous control does not require complex probability dis- tributions. GAC achieves competitive superior performance across diverse benchmarks using geometric operations on the unit sphere: a = r. normalize (w(x) 44 + (1 - w(k))= E). By replac- ing distributional modeling with direct geometric mixing, we reduce parameter count by 50% while improving performance, achieving 37.6% gains over SAC on Ant-v4 and 1 112% on quadruped-run, with competitive or best results on out of 12 tasks across MuJoCo and DMControl benchmarks.
The success of GAC validates a broader principle: respecting the geometric structure of action spaces can be more effective than sophisticated probabilistie machinery. Our fixed-radius design (r = 2.5) works well across symmetric locomotion tasks by learning correct action directions, while the learnable per-dimension scaling variant (Eq. 6 provides additional flexibility for asymmetric scenarios, demonstrating GAC's udaptability across diverse control challenges.
Our method eliminates the computational burden of density calculations, Bessel functions, and re- jection sampling, while avoiding the gradient pathologies of able concentration parameter provides adaptive exploration without explicit entropy computation, demonstrating that exploration-exploitaiinn balance can emerge from geometric structure rather than information-theoreiic regularization.
Gaussians. The learn-
L Limitations and Future Work. While GAC demonstrates strong empirical performance, several avenues remain for investigation. The geometric prior (spherical manifold) proves particularly effee- tive for coordinated locomotion but shows limitations on asymmetric tasks (e.g.. Hopper's single- leg dynamics, Pusher's 's contact-rich manipulation). Future work could explore adaptive geomet- ric structures that interpolate between spherical, elliptical, and unconstrained manifolds based on task characteristics. The theoretical connection between our geometric exploration mechanism and informationn-theortie quantities, while empirically validated through strong performance, warrants deeper mathematical analysis. Additionally, extending the geometric approach to discrete or hybrid action spaces presents exciting challenges for general-purpose control.
Despite these open questions, GAC's success suggests that the Geometric Simplicity Principle, which replaces probabilistic complexity with geometric structure, could transform other areas of RL. Future work might explore geometric approaches to value function approximation, hierarchical control, or multi-agent coordination. By showing that simple geometric operations can replace complex probabilistic frameworks, GAC challenges the prevailing assumption that sophisticated distributions are necessary for continuous control and opens new directions in geometric RL.
Ultimately, control is not about predicting densities, but about choosing directions. GAC shows that when geometry is respected, simplicity not a compromise- but a strength. We hope this work inspires further efforts to rethink RL through the lens of geometric structure, not just statistical modeling.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering on the unit hy- persphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(46): 1345-1382, 2005.
Walid Bendada, Guillaume Salha-Galvan, Romain Hennequin, Théo Bontempelli, Thomas Bouabça, and Tristan Cazenave. Exploring large action sets with hyperspherical embeddings using von mises-fisher sampling. arXiv preprint arXiv:2507.005/8 2025.
Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 834-843, 2017.
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyperspher- ical variational auto-encoders. arXiv preprint arXiv:/80440000 2022.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference Learning Representations (ICLR), 2020.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1587-1596, 2018
Raj Ghugare and Benjamin Eysenbach. Normalizing flows are capable models for rl. arXiv preprint arXiv:2505..3357, 2025.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. arXiv preprint 170.008/70 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 1861-1870, 2018.
Jiang Hu, Ruicheng Ao. Anthony Man-Cho So, Minghan Yang. and Zaiwen Wen. Riemannian natural gradient methods. arXiv preprint arXiv:2207.0722 007287, 2022.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mchta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: http://09990770 2015.
Bogdan Mazoure, Thang Doan, Audrey Durand, R. Devon Hjelm, and Joelle Pineau. Leveraging exploration in off- off-policy algorithms via normalizing flows. arXiv preprint 1905:1995500893 2019.
Nicolas Michel, Giovanni Chierchia, Romain Negrel, and Jean-Françon Bercher. Learning repre- sentations on the unit sphere: Investigating angular gaussian and von miscs-fisher distributions for online continual learning. arXiv preprint w.Xi:22306003344 2024.
Johannes Müller and Guido Montúfar. Geometry and convergence of natural policy gradient meth- ods. Information Geometry, 7(1):485-523, 2024. ISSN 2511-249X.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
Evgenii Nikishin, Romina Abachi, Rishabh Agarwal, and Pierre-Luc Bacon. Control- oriented model-based reinforcement learning with implicit differentiation. arXiv preprint arXiv:2106.033 03273, 2021.

---

Published as a conference paper at ICLR 2026
Johan Obando-Ceron Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures experts unlock parameter scaling for deep RL. arXiv preprint wrXhv:2402.08600 2024.
OpenAl, Marcin Andrychowicz, Bowen Baker, Maciek Chociej. Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szy- mon Sidor, Josh Tobin, Peter Welinder, Lilian Weng. and Wojciech 7aremba. Learning dexterous in-hand manipulation. arXiv preprint arXiv: 1808.00111111 0008000777777.0 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Tyler R. Scott, Andrew C. Gallagher, and Michael C. Mozer. von mises fisher loss; An exploration of embedding geometries for supervised learning. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp- 10592-10602, 2021. doi: 10.1 1.11///CCC4992 2021.01044.
Younggyo Seo, Jafar Uruç, and Stephen James. Continuous control with coarse-to-fine reinforce- ment learning. In Proceedings of the Conference on Robot Learning (CoRL), pp. 2866-2894 2025.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proceedings of the 31st International Conference Machine Learning (ICML), pp. 387-395, 2014.
Viacheslav Sinii, Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, and Sergey Kolesnikov. In-context reinforcement learning for variable action spaces. arXiv preprint arXiv:2312.13377, 2024.
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5981-5988, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801 arX1v:1801.00690, 2018
Mirco Theile, Lukas Dirnberger, Raphael Trumpp. Marco Caccamo, and Alberto L. Sangiovanni- Vincentelli. Action mapping for reinforcement learning in continuous environments with con- straints. arXiv preprint arXiv:2412.04337,, 2024.
Saket Tiwari, Omer Gottesman, and George Konidaris. Geometry of neural reinforcement learning in continuous state and action spaces. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Shijie Wang, Haichao Gui, and Rui Zhong. Attitude estimation via matrix fisher distributions on so(3) using non-unit vector measurements. Automatica, 179: 12444, 2025.
Zhangyu Wang. Lantian Xu, Zhifeng Kong, Weilong Wang, Xuyu Peng, and Enyang Zheng. A geometry-aware algorithm to learn hierarchical embeddings in hyperbolic space. arXiv preprint arXiv:2407. 16641, 2024.
Kisung You, Dennis Shung, and Mauro Giuffrè. Learning over mises mises-fisher distributions via wasserstein-like geometry. arXiv preprint arXiv:2504. 14/64, 2025.
Mofreh R. Zaghloul and Steven G. Johnson. Efficient calculation of modified bessel functions of the first kind, (2). for real orders and complex arguments: Fortran implementation with double and quadruple precision. arXiv preprint arXiv:2505.007770 2025.
12

---

Published as a conference paper at ICLR 2026
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3738-5746, 2019.
Yuanyang Zhu, Zhi Wang, Yuanheng Zhu, Chunlin Chen, and Dongbin Zhao. Discretizing contin- uous action space with unimodal probability distributions for on-policy reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 36(6): 1285- 1297, 2025. doi: 10.1099/TNNLS222444467777
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS
A.1 PROOF OF THEOREM
Proof. Consider the unnormalized mixture vector:
v == uy + (1 w)
where μ1 Sd-1 is the deterministic mean direction, Uniform (S 1) is uniform spherical noise, and w (R) E [0, 1] the mixing weight.
Due to the symmetry of the uniform distribution on gd-1, any uniform random vector on the sphere has zero expectation:
Eg 0.
Therefore, the expectation of the mixture vector is:
Eg Egg[u [ur + (1 w){ wp + - w)gEg( wps.
This holds exactly for any 2, with no approximation error.
A.2 CONCENTRATION ANALYSIS
While Theorem 1 characterizes the expected direction of the unnormalized mixture, it does not capture how tightly the samples are concentrated around this direction. We therefore analyze the cosine similarity between normalized samples and the mean direction quantify concentration. Let v/lvv/22 denote the normalized mixture. where || |la is the L2 norm.
V
The cosine similarity between v and μ measures directional alignment:
v vTH μ
(vv
(10)
||
For the numerator:
v = (wi + (1 w)E)
pp # + (1 w)E
(11)
w 2 + (1 w)
w + -1- w μ.
where the last step uses || = 1 since μ E sd-1
Taking expectations over the uniform distribution:
Eglv T p4] w + (1 w\)E [4] =
(12)
since Eg [4] = 0, as a fixed unit vector and a random unit vector on the sphere are uncorrelated in expectation under uniform sampling.
To estimate the norm, we assume that the inner product fa T & remains small, which is typically the case when is uniformly sampled from the sphere. This leads the approximation:
| Im + (1 w) w² + (1 w)² + 2w(1 w R u² (1-
(13)
This simplification holds well in practice as confirmed by our empirical results (see Appendix A.3)
For action spaces (d >>>>> 1), concentration of measure implies that g f concentrates tightly around zero with high probability. Under this regime:
Eícos <(v, ()) ~
w
(14)
+ (1--w)
For typical operating ranges where [0.6, 0.99 (corresponding to (0.555). this quantity closely tracks w itself. For example, when w = 0.9, the ratio equals 0.994, validating our use of w(xs) effective concentration parameter.
14

---

Published as a conference paper at ICLR 2026
Table A.1: Concentration control validation: theoretical vs, measured. The close match between
measured concentration and theoretical w validates Theorem
K
Weight w
Concentration
Angle Std. (*)
2.0
0.119
0.091
39.30
-1.0
0.269
0.253
36.3
0.0
0,500
0,678
19.7
0.5
0,622
0.875
9.1
1.0
0.731
0,956
5.0
2.0
0.881
0.994
1.7
KT-
K- Cerommaationn 0,678
x-665 Coremnaation 0,75
N-T Contentiaion E556
Ciressratio 0001
Figure A.I: 3D visualization of GAC sample distributions for E (- 0, 0.5, 1}. Arrows indicate target direction μ. Colors represent cosine similarity with μ (blue=low, red=high). Higher values produce more concentrated distributions.
A.3 EMPIRICAL VALIDATION
To complement our theoretical analysis, we empirically verify that the concentration parame- ter к provides direct and monotonic control over the sample distribution's properties. For each R E (-2, 1, 0.0, 0.5, 1.0, 2.0}. we generate 500 samples using the GAC mechanism and measure their key concentration metrics, summarized in A.ble visualized in Figure A.I
Metrics Explanation:
Weight w()s:: The mixing weight w(x) = (K.) that controls interpolation between deter- ministic direction and uniform noise. This is deterministic given k.
Concentration: The empirical mean cosine similarity between normalized samples and the target direction fa, measuring how aligned the final actions are with the intended direction.
Angle Std.: Standard deviation of sample-j4 angles (in degrees), measuring the spread of the distributionnssalll values indicate more concentrated (less exploratory) behavior.
Empirical Validation. We empirically validate the theoretical result in Theorem by measuring the directional concentration of sampled actions under different values. Results show strong agree- ment between measured and theoretical concentration (correlation ss 0.95), with angular standard 22 (high exploration)
deviation decreasing monotonically from 39.3 (ncar-deterministie).. These results confirm that GAC achieves precise, vMF-like concentration con- trol through simple geometric operations without requiring modified Bessel functions or rejection sampling. The monotonic relationship between and directional concentration further validates our approach to structured exploration.
1.7 at i.
wwithuut
A.4 EXPLORATION CONTROL. AND ENTROPY CONNECTION
A.4.1 EXPLORATION CONTROL MECHANISM
GAC's concentration parameter x functions as an adaptive exploration controller that learns when to explore versus exploit based on the value landscape. Unlike traditional entropy regularization that requires computing probability densities, x directly modulates the geometric mixing between deter- ministic and stochastic components. While GAC implicitly defines mixture distribution through
15

---

Published as a conference paper at ICLR 2026
this geometric operation, it erucially avoids any density or entropy computation during optimization. The effectiveness of x as an exploration signal emerges from three complementary perspectives:
1) Geometric Perspective. As K increases, the mixing weight w(k) (k) increases monotoni- cally from I, causing samples to concentrate progressively around μ.. Empirically, the angular standard deviation decreases from 39.3º at -2 to 1.7 at 2 (Table A.T). directly demon- strating decreasing distributional uncertainty.
2) Information- Theoretie Perspective. For spherical distributions, concentration and entropy fundamentally inversely related. The vMF distribution provides a theoretical benchmark:
HyMF = log Ca(k) - k.A((K) ~ -K d log K const,
(15)
where Ca(k) is the normalization constant and Ax(x:) the mean resultant length; the approximation holds for large with the dominant linear term -K justifying our exploration term as higher-order terms contribute little in practice.
Note: The relationship H -K serves conceptual motivation rather than rigorous derivation. GAC uses -x directly as an exploration signal without ever computing actual entropy. This approx- imation provides theoretical intuition for why -n effectively balances the method's 's success does not depend on this mathematical correspondence.
but
3) Empirical Validation. Our experiments confirm that effectively captures exploration pres- sure:
The correlation between w((k) and measured concentration exceeds 0.95 (Figure A.1
As k increases: w(x)) = (K) + (more determinissicdexplootttiee))
As λ decreases: w(k) = (K) + (more stochastic/explorattry)
GAC with & achieves superior performance across all benchmarks (Table
k's smooth effect on exploration supports stable policy optimization
A.4.2 THEORETICAL CONNECTION TO ENTROPY
While GAC operates without computing distributions, the exploration controller x exhibits nat- ural connection to entropy in directional statistics. For vMF distributions on the unit sphere, the differential entropy is given by:
HyMP = log Ca(() - KAA((K) ~ - d-1 log k const,
(16)
is the mean resultant length, with Io denoting the modified Bessel function of the first kind. The asymptotic behavior of Aa(k) is:
where (a(K))
(2n) (()) is the normalization constant and Aa(s)) I
4/2-1
fa/2())
d-1 O(x
Aa(k) ~ 1
(17)
2x
Substituting into H.MF and simplifying, obtain:
d-
H.MF ~ - 2
log k. const.
(18)
The leading term -x dominates, with logarithmic corrections diminishing in relative magnitude for practical & values. This validates using -K as an exploration controller that faithfully re- flects the tradcoff between concentration and uncertainty. While GAC does not explicitly follow the vMF distribution, it inherits the same qualitative dependency between and sample concentra- tion through its spherical interpolation mechanism. This connection validates why effectively in the SAC framework. Empirical results Appendix A.3 further
balances apploration-eplloiiii confirm the effectiveness of this surrogate.
Key distinction: While this mathematical relationship exists, GAC fundamentally differs from entropy-regullarized methods:
16

---

Published as a conference paper at ICLR 2026
Traditional SAC: Computes H[x] -Ellog [aass)] requiring explicit densities
. GAC: Uses -K as exploration signal without any distributional computation
This allows GAC to achieve entropy-like regularization benefits through purely geometric opera- tions, eliminating computational overhead while maintaining theoretical grounding.
A.5 SAC CONVERGENCE WITH GAC
Theorem 2: GAC maintains key properties required for SAC-style convergence under standard
regularity conditions.
Remark: We provide sketch of the key arguments. A rigorous convergence proof would require extensive measure-theoretic analysis beyond the scope of this work. Our empirical results across diverse environments provide strong evidence for convergence in practice.
Proof Sketch. We establish that GAC maintains the key properties required for SAC convergence.
Soft Bellman Contraction. The soft Bellman operator with GAC takes the form:
(s, a) (s a) + y (s)),
(19)
where the soft value function incorporates our exploration controller:
V (s) = Eg (g((((s n(s), a generated via (2)
(20)
The contraction property requires: (i) Continuity: GAC's s action generation is continuous in parameters:
Direction mapping pa()) (l is continuous for (u(() +
Mixing weight w<k) o (Rt) is Ch00 smooth
Action normalization ensures ||a|| = r (bounded)
(ii) Regularization: The exploration controller provides consistent exploration pressure:
[ < O,
(21)
encouraging exploration when concentration becomes excessive.
Under these conditions and assuming bounded rewards (r(8, S Rmaxx the operator 7 is y-contraction:
IT" T r11Q1 Q2l
(22)
Policy Improvement. GAC approximates policy improvement through gradient- optimiza- tion:
Lactor = EL-DD.(((s) a)s,
(23)
where a is generated using GAC's mechanism with random noise &. This objective drives the policy toward high-value regions (via the Q term) while maintaining exploration (via the g term). achiev- ing similar goals to SAC's entropy- regulay-re policy improvement without explicit distributional computations.
Convergence Benefits, GAC improves convergence through three key design choices:
Stable gradients: Spherical normalization avoids tanh-induced saturation, preserving di- rection gradients.
Bounded actions: All actions satisfy ||a]| = preventing value divergence from out-of- bound actions.
Adaptive exploration: The learnable x balances exploration and exploitation without ex- ternal schedules.
17

---

Published as a conference paper at ICLR 2026
Exploration-Exploiie Balance. The loss function induces implicit tradeoff between explo- ration and exploitation. The direct gradient 0C/0k = 1 promotes exploration by penalizing high R, encouraging lower concentration and increased action noise. In contrast, the Q-value term courages exploitation: when higher leads to better actions (and thus higher Q), gradients through Q(s, (s;; k)) push upward. This dynamic balance emerges naturally from actor-critic interplay, without explicit entropy terms or temperature tuning.
Natural Stabilization of K. Despite being unconstrained, remains bounded (E [0, 5] empirically) through:
Sigmoid saturation: w((k) = (x) flattens for large x, capping its effect.
Gradient feedback: High x: leads to deterministic actions; in noisy environments, this reduces Q, discouraging overconfidence.
No explicit clipping: Yet stabilizes naturally via loss dynamics.
These mechanisms ensure convergence stability without manual regularization.
B
GRADIENT FLOW ANALYSIS IN TANH-SQUASHED POLICIES
1.5
Effective Gradient (53.6%)
Probability Density
0.16
Gradient Saturation (46.4%)
Tanh Gradient
1.2
Gradient
0.12
0.9
0.08
0.6
0.04
0.3
0.00
7.5-5.0-2.3 0.0 2.5 5.0 7.5 Pre-squashed Action Values
0.0
Figure A.2: Distribution of pre-squashed Gaussian samples from a trained SAC policy. Red areas indicate saturated gradients tanh' (=:] < 0.05), with 46.4% of samples falling into these regions. Dashed lines show the [-1, tanh boundaries. This mismatch between unbounded Gaussians and bounded action spaces motivates GAC's direct geometric approach.
To understand the geometric mismatch between Gaussian distributions and bounded action spaces, we analyze the gradient flow through tanh squashing functions. The tanh transformation a tanh(ā) has gradient:
Ôa 0a
1 - t
(24)
which approaches zero as |ā| - oc, creating regions vanishing gradients.
We sampled pre-squashed actions from the SAC policy throughout training, particularly during the stable performance phase near convergence (0.8M-1.0M steps).Figure [A.2) visualizes the distribu- tion of these raw actions before squashing, color-coded by gradient magnitude.
Key Observations:
A substantial fraction (46.4%) of pre-squashed samples have gradient magnitudes tanh' tanh'(@) < 0.05, indicating severe saturation.
The distribution exhibits heavy tails beyond |ā| > 2.5, where gradient flow is minimal.
SAC compensates through entropy regularization (a zs 0.2), which maintains exploration diversity despite reduced gradients.
18

---

Published as a conference paper at ICLR 2026
This analysis does not imply SAC is ineffective, it remains highly successful in practice. Rather, it highlights a structural inefficiency in that sigmificant computational effort spent managing the mis- match between unbounded distributions and bounded spaces. GAC sidesteps this issue entirely by operating directly on the unit sphere, ensuring consistent gradient flow without requiring squashing entropy- drivee exploration.
Note: This visualization shows pre-squashed samples. The actual gradient flow during training modulated by reparameterization and entropy regularization, which prevent complete saturation. However, this analysis does not diminish SAC's effectiveness but highlights an opportunity for geo- metric alternatives like GAC that avoid this structural inefficiency entirely.
C
REPRODUCIBILITY
To ensure complete reproducibility and facilitate future research, we provide comprehensive imple- mentation details and open-source resources for immediate verification of our results.
C.I IMPLEMENTATION DETAILS
Network Architecture: GAC uses a shared backbone with separate heads for direction and concen- tration:
Backbone: Linear(obs.dim, 256) - ReLU - Linear(256,, 256) - ReLU Direction head: Linear(256,, action dim)
Concentration head: Linear(256, 64) - ReLU + Linear(64,
Key Hyperparameters:
Learning rates: 3 x 10-4 (actor), x 10-3 (critic)
Batch size: 256, Buffer size: 106
Target network update: = 0.005
Discount factor: 7 0.99
Action radius 2.5 for most tasks, 1.0 for Ant-v4
We adopt standard hyperparameters from CleanRL (Huang al. 2022) without task-specific tuning. highlighting that GAC's gains stem from structural design rather than careful optimization.
C.2 COMPUTATIONAL EFFICIENCY
Computational Efficiency: GAC's sampling requires only: 1). Two forward passes (direction and
concentration networks); 2). One normalization operation; 3). One linear interpolation; 4). One final
scaling. This yields approximately speedup compared to vMF sampling with rejection methods.
C.3 CODE AVAILABILITY
The complete source code for GAC is publicly available at:
The repository includes the full training implementation, hyperparameter configurations, and in- structions for reproducing experiments reported in this paper.
D ADAPTIVE ACTION SCALING ANALYSIS
To validate the effectiveness of adaptive per-dimension scaling (Eq.fo). analyze the learned action scales r E dd from two complementary perspectives: (1) cross-task convergence patterns across the DMControl suite, and (2) ablation study canonical locomotion task.
19

---

Published as a conference paper at ICLR 2026
D.1
PROSS-TASK SCALE CONVERGENCE
3.0
r (Action Scaling)
2.5
2.0
1.5
walker-walk
fish-upright
walker-run
quadruped-rum
1.0
cheetali-run
quadrueed-walk
0.0
0.1
0.2
0.3
0.4
0.5
Training Steps (Million)
Figure A.3: Training dynamics of learned action scales across DMControl tasks. Each line rep- resents the mean scale F for one task (averaged over all action dimensions and 5 seeds). Scales converge to task-specific values in [1.0, 3.0] after initial exploration (< (<10KK steps), demonstrating adaptive learning without manual tuning.
Figure A.3 shows the evolution of mean scales F= D 2i=1 Pr across DMControl tasks over 500K steps.
Key Observations:
1) Stable Convergence Across Tasks, After an initial exploration phase (<100K steps), all tasks exhibit stable scale convergence, indicating the geo-head successfully identifies appropriate action magnitudes without manual intervention.
2) Task-Dependenl Adaptation. Different morphologies learn distinct scale profiles:
Walker tasks (walk, run): Converge to F Rs 2.0-2.3, close to our fixed = 2.5 baseline, confirming the validity of uniform scaling for symmetric bipedal locomotion.
Quadruped tasks: Stabilize at F = 2.0-255, with more variance reflecting multi-leg coor- dination requirements.
Cheetah-run: Tight convergence to F 2 2.3, consistent with forward-acceleration dominant dynamics.
Fish-upright: Gradual rise from R 1.0 to x 2.2, balancing buoyancy control and orienta- tion stabilization.
3) Geometric Validity. The learned range [1.0.3.0] aligns with geometric constraints: since nor-
malized directions satisfy |ög| < scales r; [1.0, 3.0) produce per-dimension actions la,| S 3.0, comfortably within normalized bounds [-1, 1] action limit for MuloCo/DMComtrol, avoiding both under-actuation (r 1.0) and over-saturation (r 3.0).
D.2 ABLATION: FIXED vs. ADAPTIVE SCALING WALKER2D-V4
To rigorously validate the necessity of learnable versus fixed we conduct controlled ablation on Walker2d-v4, comparing:
Fixed r E (1.0, 2.0, (scalar radius)
Adaptive e & (learned per-dimension)
All experiments use 5 random seeds with identical hyperparameters except for the scaling strategy.
Results & Implications:
1) Marginal Gain on Symmetric Tasks. Table A.2 shows adaptive r achieves only 3-9% improve- ment over well-tuned fixed 3.00,3.0,, confirming our hypothesis that symmetric locomotion
20

---

Published as a conference paper at ICLR 2026
5030
Ephent Heewee
40.00
Adapiive Vallee
2200
Adaptie hid
3000
3.0
2000
1030
80
0.2
0.4
0.2 Training Stepe (Million)
Training Steps (Millioo)
Training Steps (Million)
(a) Learning curves
(b) Adaptive evolution
(c) Adaptive std(r.) evolution
Figure A.4: Ablation on Walker2d-v4: Fixed vs. Adaptive Scaling, (a) Learning curves show adaptive achieves the best final performance (5064 + 156) but only marginally outperforms fixed r 2.0 (4607) and = 3.0 (4919), while r = 1.0 (3602) significantly underperforms due to insufficient action magnitude. (b) The learned mean scale converges
1.7, between the r 1.0 and r 2.0 baselines. (c) The standard deviation std(r;) rises from = 0.7 to = 0.9, indicating cimension-specifee differentiation rather than uniform scaling collapse.
does not require per-dimension scaling. The small gain comes from fine-tuning rather than fun- damentally different geometry.
Table A.2: Final performance (500K steps) on Walker2d-và
Method
Return
Params
r 1.0 (fixed)
3602 + 130
2.0 (fixed)
4607 + 179
r = 3.0 (fixed)
4920 + 250
Adaptive
5064 + 156
+384 (geo-head)
2) Low Sensitivity of Fixed The 2.0 and 3.0 baselines differ by less than 7%, demon- strating that GAC is not sensitive to radius hyperparameters within a reasonable range. This validates the robustness of our default choice = 2.5 in the main paper (Section 3.5
3) Learned Scales Confirm Theoretical Design. Figure showss adaptive F converges to =s 1.7, consistent with the [1.0, 3.0] range observed in DMControl (Figure A.3). The increasing std(r;) (F.gureAA.4c) indicates the network learns dimension-speciffcc modulation rather than collapsing to uniform scaling, though the impact modest for this symmetric task.
4) Task Complexity Determines Necessity. Combining this ablation with DMControl results (Sec- we conclude:
tion 4.2p.
Simple/symmetric tasks (Walker2d, HalfCheetah): Fixed r suffices, saving 50% parame- ters and avoiding unnecessary complexity.
Complex/asymmetrrcc tasks (Quadruped, Digh-DoF manipulation): Adaptive unlocks 6-112% gains by enabling fine-grained per-dimension control.
This demonstrates that r is not hyperparameter requiring tedious tuning, but rather an euvironment-depeeddent geometric parameter that can be either fixed (for efficiency) learned (for flexibility) based on task structure.
D.3
SUMMARY
This two-level analysis validates GAC's design philosophy:
Macro-level (DMControl): Adaptive converges robustly across diverse morphologies, with task-specific profiles emerging naturally without manual tuning.
Micro-level (Walker2d): On symmetric tasks, fixed achieves near-optimal performance with fewer parameters, while adaptive r provides modest gains at the cost of added com- plexity.
21

---

Published as a conference paper at ICLR 2026
Together, these findings demonstrate GAC's flexibility: structural simplicity when sufficient, adaptive complexity when necessary. The fixed r 2.5 baseline in the main paper represents an efficiency-optmmized choice validated by both theoretical geometry and empirical convergence, while the GAC-Scale variant (Eq. (6) extends this framework to handle asymmetric scenarios without sacrificing stability.
E COMPUTATIONAL COMPLEXITY ANALYSIS
We provide detailed comparison of per-sample computational costs across GAC, SAC, and TD3: Operation Breakdown:
Table A.3: Computational complexity per action sample. GAC achieves efficiency comparable to deterministic methods while maintaining structured exploration.
Method
Core Operations
Cost per Sample
TD3
Forward pass Gaussian noise Sampling + tanh + log-det Jacobian
(
SAC
( with log overhead
GAC
Normalization spherical interpolation
(dd)
Detailed Cost Analysis:
SAC: Requires (1) Gaussian sampling ~ (H, 2²) [O(d)], (2) tanh squashing [O(d), (3) Jacobian correction log det(@ tanh /dā)l [O(d], and (4) entropy evaluation -BElog x| [O(d). Total: 4 x O(d) with non-trivial constant factors from logarithmic operations.
GAC: Requires (1) Direction generation (forward pass + L2 norm) [O(d)] (2) Spherical noise sam- pling (normalized Gaussian) [d)], and (3) Linear interpolation [O(d)] Total: x ((d) with lightweight operations (no logarithms special functions).
TD3: Baseline deterministic policy [O))) plus Gaussian noise [O()). Total: x (d).
Empirical Timing (NVIDIA RTX 3090):
SAC: 1.0x (baseline)
GAC:: 0.78 0.78x (22% faster than SAC)
TD3: 0.65x (35% faster than SAC)
GAC achieves near-TD3 efficiency while maintaining the structured exploration benefits of stochas- tic policies, eliminating the computational overhead of entropy regularization that SAC requires.
Parameter Efficiency:
SAC: 2d outputs (mean μ., log-std log r))
GAC (fixed r): d + outputs (direction concentration κ) R= 50% % reduction
GAC-Scale: 2d + outputs (direction μ4, concentration K, scales r) comparable to SAC
but with geometric structure
This analysis demonstrates that GAC's geometric approach achieves computational efficiency com- parable to deterministic methods while preserving the exploration benefits of stochastic policies.
22


Reviews:
[{"review_id": "WwyisOFNdf", "reviewer": "Reviewer_HAcA", "summary": "This paper proposes a novel action generation paradigm that preserves the geometric benefits of spherical distributions while simplifying computation. GAC decomposes action generation into a direction vector and a learnable concentration parameter. It focuses on an important topic, policy distribution. The experimental results show its efficiency across MuJoCo benchmarks.", "questions": "In Figure 2, two tasks do not have an advantage. Can the authors provide empirical results on more environments?\nHow to guarantee that the unnecessary dimension does not influence the optimal policy or the optimal policy theoretical analysis?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The motivation of this work is novel and interesting, which could be an important work in the RL community.\nGAC represents policies through two components: a direction network that outputs unit vectors indicating preferred action orientations, and a concentration network that controls exploration by interpolating between deterministic directions and uniform spher-\nical noise.\nIt takes a novel perspective on whether the distribution paradigm itself is necessary, which also promotes efficient exploration.\nThe paper offers thorough theoretical analysis, with clearly stated assumptions, theorems, and proofs.", "weaknesses": "More benchmarks may be needed to test and provide solid experimental results.\n\nRegarding the policy distribution, I recommend that the authors add the discretization policy distribution topic works.\n\nDiscretizing continuous action space for on-policy optimization, AAAI, 2020 \nDiscretizing Continuous Action Space With Unimodal Probability Distributions for On-Policy Reinforcement Learning, IEEE TNNLS, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AUD1lWsvAN", "reviewer": "Reviewer_iSQb", "summary": "The paper focuses on the issue of distorted sampling space caused by Gaussian policies in a bounded action space. To solve this, a common technique is to use \"squashing\" functions, such as tanh, to map the Gaussian samples into a bounded range, causing actions to cluster near the boundaries. While alternatives like von Mises-Fisher (vMF) distributions are theoretically grounded on the unit sphere, they are computationally expensive. The paper proposes to retain the geometric benefits of spherical distributions by interpolating between a deterministic action direction and a uniformly sampled unit sphere vector. The action follows the direction and gets multiplied by a magnitude scalar. The proposed algorithm shows comparative empirical results to SAC, PPO, and TD3 on six MuJoCo tasks.", "questions": "1. As shown in Figure A.2, 46.4% of all pre-squashed action samples fall into regions of \"Gradient Saturation\". However, why are the performances of baselines not deteriorated by this issue?\n\n2. What distribution is the current action following? Figure A.1, GAC samples from a subsurface instead of a unit ball. Will it cause an exploration issue?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a strong motivation and a clear presentation of the algorithm.\n\nIt also delivers a comprehensive experimental analysis, featuring multiple baseline comparisons, thorough ablation studies, and an in-depth examination of convergence behavior and the sampling landscape.", "weaknesses": "The algorithm introduces an extra hyperparameter, action magnitude, but the robustness analysis across tasks is missing. Though the algorithm is well-motivated, the performance improvement is limited at the cost of an extra hyperparameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9BfRnYeR3", "reviewer": "Reviewer_rszA", "summary": "The paper proposes Geometric Action Control (GAC), a distribution-free policy paradigm that generates actions via a mixing operation with a unit sphere, replacing traditional Gaussian policies to avoid gradient saturation. GAC performances near SOTA models on continuous control Mujoco benchmarks.", "questions": "Could the authors compare the overall computational complexity of GAC with deterministic and Gaussian based models, instead of only emphases the sample complexity?\n\nThis model assumes the range of the action space is mapped to a unit. What if the range of the action space is unknown? What if the best actions are outside the unit sphere for exploration", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is overall logically clear, presenting an interesting perspective on the action exploration in continuous control tasks. With this perspective, this paper reformulates the exploration distribution to a unit sphere. Compared to a policy with von Mises-Fisher distributial output, it reduces the sample complexity.", "weaknesses": "This paper seems to overstate its contribution by comparing to a distribution that is not commonly used in RL, especially on the point about sample complexity. It is better to have a clearer comparison between the proposed method with deterministic models and Gaussian based models for the \"Geometric\" feature and sample complexity, respectively. The \"Geometric\" feature in this work is similar to deterministic models, which is not clearly stated in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CEvXpd4YPH", "reviewer": "Reviewer_4Nrh", "summary": "The paper proposes a novel method for action sampling in continuous control environment. Authors point out that the common practice of sampling from a diagonal Gaussian have a fundamental mismatch: the unbounded support of Gaussian and bounded support of action space.\n\nTo this end, they propose to instead constrain the action space onto a unit sphere, where the network outputs a deterministic action $\\mu$ and state-dependent exploration paramter $w(\\kappa)$. As exploration is achieved simply by interpolating between predicted action ($\\mu$) and random noise ($\\epsilon$), the aforementioned mismatch is addressed, while still applicable to algorithms such as SAC.\n\nEfficacy of the method is shown by experimenting on 6 Mujoco environments, where they outperform naive SAC, TD3 and PPO on 4 of the 6 environments.", "questions": "- Line 234: Can $\\epsilon \\sim \\text{Uniform}(S^{d-1})$ be considered a normalized Gaussian noise? If not, could there be a better way to add noise that's centered on $\\mu$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Clear motivation**. Authors clearly state the issue of action space handling with Gaussian and the limitation of previous method (vMP), and propose a novel method for addressing the problem.\n- **Simplified SAC, theoretically grounded**. Authors show that their method can be integrated with SAC, which removes the need for temperature tuning. While this immediately raises the question of theoretical soundness, authors also provide the proof that the algorithm maintains contraction even after the simplification.", "weaknesses": "- **Over-limitation of action space**. The proposed method limits all actions to have the same magnitude. This raises the concern: fixing the action's magnitude removes the to flexibly choose between 'active motions' (high action magnitude) and 'passive motions' (low action magnitude). Indeed, authors show that the choice of magnitude $r$ is crucial to the performance.\n- **Limited results and evaluation benchmark**. Combined with above, the results of Mujoco do not seem very compelling. Compared to SAC, GAC outperforms in one environment (Ant-v4) but is also outperformed by SAC in Pusher-v4. To gain more persuasiveness, I suggest expanding the evaluation benchmark to e.g., DMC, Metaworld.\n- **No comparison with vMF**. Since vMF has been mentioned as prior work throughout the paper, it seems natural for them to appear as one of the baselines (despite its practical difficulty).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)