Distributions as Actions: A Unified Framework for Diverse Action Spaces
Authors: Jiamin He
A. Rupam Mahmood
Martha White
Venue: ICLR 2026 Poster (2026)
Keywords: deterministic policy gradient
actor-critic
continuous control
discrete control
hybrid control
action space
reinforcement learning
Primary area: reinforcement learning
TL;DR: We introduce a reinforcement learning framework that treats distributions as actions, enabling a unified policy gradient method (DA-PG) and algorithm (DA-AC) that performs well across different action spaces.

Abstract: We introduce a novel reinforcement learning (RL) framework that treats parameterized action distributions as actions, redefining the boundary between agent and environment. This reparameterization makes the new action space continuous, regardless of the original action type (discrete, continuous, hybrid, etc.). Under this new parameterization, we develop a generalized deterministic policy gradient estimator, Distributions-as-Actions Policy Gradient (DA-PG), which has lower variance than the gradient in the original action space. Although learning the critic over distribution parameters poses new challenges, we introduce Interpolated Critic Learning (ICL), a simple yet effective strategy to enhance learning, supported by insights from bandit settings. Building on TD3, a strong baseline for continuous control, we propose a practical actor-critic algorithm, Distributions-as-Actions Actor-Critic (DA-AC). Empirically, DA-AC achieves competitive performance in various settings across discrete, continuous, and hybrid control.

Published as a conference paper at ICLR 2026
DISTRIBUTIONS AS ACTIONS: A UNIFIED FRAMEWORK FOR DIVERSE ACTION SPACES
Jiamin He
A. Rupam Mahmood & Martha White Department of Computing Science University of Alberta
Department of Computing Science
University of Alberta
Alberta Machine Intelligence Institute (Amii) ji amin122 2 walberta.cc..
Canada CIFAR AI Chair, Amii
{a (armahmood, whireemeealleereeeeeeee
ABSTRACT
We introduce novel reinforcement learning (RL) framework that treats param- eterized action distributions as actions, redefining the boundary between agent and environment. This reparameterization makes the new action space contin- uous, regardless of the original action type (discrete, continuous, hybrid, etc.). Under this new parameterization, we develop generalized deterministic policy gradient estimator, Diitributtions-s-AAciii lower variance than the gradient in the original action space. Although learning the critic over distribution parameters poses new challenges, we introduce Inter- polated Critic Learning (ICL), a simple yet effective strategy to enhance learning. supported by insights from bandit settings. Building on TD3, strong baseline for continuous control, we propose a practical actor-critic algorithm, Distributions- as-Actions Actor-Critic (DA-AC). Empirically, DA-AC achieves competitive per- formance in various settings across discrete, continuous, and hybrid control.'
Policy Gradient (DA-PG), which has
INTRODUCTION
Reinforcement learning (RL) algorithms are commonly categorized into value-based and policy- based methods. Value-based methods, such as Q-learning (Watkins & Dayan, 1992) and its variants like DQN (Mnih et al., 2015), are particularly effective in discrete action spaces due to the feasibility of enumerating and comparing action values. In contrast, policy- methods are typically used for continuous actions, though they be used for both discrete and continuous action spaces (Williams, 1992; Sutton et al., 1999).
Polecy-based methods are typically built around the policy gradient theorem (Sutton et al., 1999), with different approaches to estimate this gradient. The likelihood-ratio (LR) estimator can be ap- plied to arbitrary action distributions, including discrete ones. continuous action spaces, one can alternatively compute gradients via the action-value function (the critic), leveraging its differentia- bility with respect actions. This idea underlies the deterministic policy gradient (DPG) algorithms (Silver et al., 2014) and the use of the reparameterizzttio (RP) trick for stochastic policies (Heess al., 2015; Haarnoja et al., 2018). These approaches can produce lower-variance gradient estimates by backpropagating through the critic and the policy (Xu et al., 2019).
Despite the flexibility of policy gradient methods, current algorithms remain tightly coupled to the structure of the action space. In particular, different estimators and architectures are often required for discrete versus continuous actions, making it difficult to design unified algorithms that generalize across domains. Although the LR estimator is always applicable, it often requires different critic architectures for different action spaces (Haarnoja et al., 2018; Christodoulou, 2019) and carefully designed baselines manage high variance (Greensmith et al.. 2004).
In this paper, we introduce the distributions-ca aattssss-s--aansssss-saa-s framework, an alternative to the classical RL formulation that treats the parameters of parameterized distributions actions. For Gaus- sian policy, for example, the distribution parameters are the mean and variance, while for softmax policy they are the probability values. The RL agent outputs these distribution parameters to the
Cooe available https: //ap::://///// teiithub.ccm com/ hmt.

---

Published as a conference paper at ICLR 2026
environment, and the action sampling becomes part of the stochastic transition. Importantly, distri- bution parameters are typically continuous, even when the underlying actions are discrete, hybrid, or structured. By shifting the agent-environment boundary in this way, we develop single continuous-action algorithm for a diverse class of action spaces. 2
To develop algorithms under the new framework, first propose the Distribatioons-a-AAciine Pol- icy Gradient (DA-PG) estimator, and prove it has lower variance than the corresponding updates in the original action space. This reduction in variance can increase the bias, because the critic can be harder to learn. We develop an augmentation approach, called Interpolated Critic Learning (ICL), to improve this critic learning. We then introduce a deep RL algorithm based on TD3 (Fujimoto et al.. 2018), called Distributions-as-Aciii estimator and ICI. We empirically evaluate DA-AC to assess the viability of this new framework and its ability to use single algorithm across diverse action spaces. DA-AC achieves competitive and sometimes better performance compared baselines a variety of settings across continuous, discrete, and hybrid control. We also provide targeted experiments to understand the bias- trade-off of different gradient estimators, and show the utility ICL for improving critic learning.
Actor- Critic (DA-AC), that incorporates the DA-PG
PROBLEM FORMULATION
We consider Markov decision process (MDP) (S, A, d. where S is the state space, A is the action space, : S x A + (S)) is the transition function, do E A(S) is the initial state distribution, r : S x A ((R) the reward function, and y is the discount factor. Here, A(X) denotes the set of distributions over a set X. We use T(as)) to represent the probability of taking action a E A under state s for policy æ.. The goal of the agent to find policy under which the below objective is maximized:
(TT = Ei=o As (S)), (=-.s-((-)..AA.)) [n R(+1] = E En [ Re+1].
(I)
where the second formula uses simplified notation that we follow in the rest of the paper. The (state-Jvalue function and action-value function of the policy defined as follows:
Un (s) En [ R
=], qr, a) = [R, tox A, = a].
(2)
In this paper, we consider actor-critic methods that learns a parameterized policy, denoted by ma. and parameterized action-value function, denoted by Qw- Given a transition (S., A,, Ret1, Se41), Qw is usually learned using temporal-difffrencee (TD) learning:
w - w + a [R+1 + 7Q ( Ae+1) As+1)
VQ) Ar),
(3)
where a is the step size, and A,+1 is sampled from the current policy: Ac+1 T0(-1S+41).
The policy is typically optimized using a surrogate of Equation (1):
j = Es, As (- @ (S,, Az)].
(4)
where d E A(S) is some distribution states. Below outline three typical estimators for the gradient of this objective.
The likelihood-ratio (LR) policy gradient estimator (Sutton et al., 1999) uses (oJ St, A) = Vo log Ro To Qw A), where A (-).)). Since the LR estimator suffers from high vari- ance, it is often used with the value function as a baseline:
&yR [mo: St, A) = V log no (A[S)) (S., A) - V(S))).
(5)
where V(S;) could either be parameterized and learned or be calculated analytically from Qw when the action space is discrete and low dimensional.
The deterministic policy gradient (DPG) estimator (Silver et al., 2014) is used when the action space continuous and the policy deterministic (TO : S - A), and uses the gradient of Q with respect to the action:
DPG i S.) VoTo(S,) V (S)
'Concurrently, Todorov (2025b) studied related formulation that considers a similar shift in the agent- environment boundary. Their work focuses on establishing a theoretical equivalence between stochastic and deterministic policy gradients in continuous action settings. In contrast, we emphasize algorithm design and empirical evaluation across diverse action types. A more detailed discussion is provided in Appendix A.
2

---

Published as a conference paper at ICLR 2026
Classical RL Framework
Distrbatiins-as-Action Framework
Agent
Env
Agent
Env
Ryl
A -ft-l
Syyl
S
Uppiate
Updatte
AuT
funcciio,
R.
R
Sampling occurs inside the agent.
Sampling is treated as part of the environment.
Figure 1: Comparison between the classical reinforcement learning (RL) framework and the framework. In the classical RL setting (col 1), the agent's s
proposed distributions-as-atiie policy Fe consists of Te. which produces the distribution parameters, and a sampling function that returns an action given these parameters. In the iisrributtins-aa-acctioo framework (col 2), the sampling function f is considered part of the environment, and the agent outputs the distribution parameters (SS) as its action. The sampled action A, may optionally be observable to the agent, though it is not required for the core formulation. This shift redefines the interface between agent and environment, potentially simplifying learning and enabling new algorithmic perspectives.
The reparamcterization (RP) policy gradient estimator (Heess et al., 2015) can be used if the policy be reparameterized (i.e., A go S,), p(-), where p(-) is prior distribution):
PPP JTT: St, Vego(c; St) T (S, A=
3 DISTRIBUTIONSSAS AASTTIIINNS---AAIIN FRAMEWORK
The action space is typically defined by the environment designer based on Nomain-sspecifi knowl- edge. Depending the problem, it may be more natural to model the action space discrete, con- tinuous, or a hybrid of both. In all cases, the agent's 's policy at a given state can often be interpreted first producing distribution parameters (8), followed by sampling from the resulting distribution. With slight abuse of notation, we denote Tag S + U as the part of the policy me that maps states to distribution parameters, and f(-lu) as the distribution over actions parameterized by E U. Note that li is usually continuous.
action A (())m((s)
In the classical RL framework, both Ho and f are considered part of the agent, as in the left of Figure 1. In this work, we introduce the distributions casennss-----eee as-actions framework: the agent outputs distribution parameters Re(s) its action, while the sampling process A ~ (s) is treated as part of the environment, depicted on the right in Figure 1. This reformulation leads to new MDP which the action space is the distribution parameter space 24. The transition and reward functions in this MDP become
p(s' s u) (ad A)), F(8, u) - EAws( [r(s, A)].
This gives rise to the distributions-as-action MDP (DA-MDP) (S, U, p, da, F, 7, We can define the corresponding value functions, and show they are connected to their classical counterparts.
(s = Ei R = (s, u) = Eg R Tūx (S:)
Uo ul.
(9)
The set 24 is continuous and compact, and the function f(alu) and its derivative continuous with respect u. When S A is continuous, the corresponding set is assumed to be compact; moreover, the functions p(s'le [s, a), do(s), r(s, a), #(s). f(alu), and their derivatives are also continuous with respect to s, or a, respectively.
Assumption 3.1.
Proposition 3.2. Under Assumption 3.1, J (s) uz (s) and qo (s, u) EA- [g= A)].
The proofs of Proposition 3.2 and all other theoretical results presented in Appendix C.
The main advantage of this framework is that it transforms the original action space into a continu- parameter space U, regardless of whether the underlying action space A is discrete, continuous,
3

---

Published as a conference paper at ICLR 2026
structured. This unification allows us develop generic RL algorithms that operate
tinuous transformed action space, enabling a single framework to accommodate a wide variety of settings, including discrete-continuous hybrid action spaces (Masson et al., 2016). For example, can apply DPG methods even in discrete action domains, where they were not previously applicable. We explore this direction in detail in Sections 4 and 5.
DISTRIBUTIONS AS ACTIONS POLICY GRADIENT ALGORITHMS
In this section, we introduce the Distributionn-as-AAN -Aciiios Policy Gradient (DA-PG), generalization of DPG for the distributions-as-aciiie framework. We show this estimator has lower variance, and then present practical DA-PG algorithm for deep RL.
4.1 DISTRIBUTIONS-AS-
POLICY GRADIENT ESTIMATOR
DA-PG is obtained by applying DPG to the distributions-as-actiinn MDP, as stated formally below. Assumption 4.1. The function (s)) and its derivative continuous with respect to 0.
Theorem 4.2 (Distributions-as-aeee the gradient of the objective ( Ei= Ex [ Re+1] with respect to 0 can be expressed
policy gradient theorem). Under Assumptions 3.1 and 4.1,
(oo( Eg-da (o) Vugno
VV (S, (l)==)()))
where daa = -r Ex ( = =}] is the (discounted) occupancy measure under mg.
The resulting gradient estimator of the surrogate objective J(Ho) = E.,, nd Qw (St. to( is
DPA-PGJ(TTss) S₁)
(S,) Vw U) (oo))
(10)
where @w is learned parameterized critic. Note that the DA-PG estimator shares the same math- ematical form as the DPG estimator (Equation (6)). However, the roles of the components differ: in DA-PG, the policy 7a outputs distribution parameters rather than a single action, and the critic estimates the expected return the entire action distribution, rather than for specific action.
In fact, DA-PG a strict generalization of DPG. When the policy is restricted to be deterministic, the distribution parameters effectively become the action, and the distributions as actions critic reduces to the classical action-value critic.
Proposition 4.3. IfUl A and f(-lu) the Dirac delta distribution centered at u, then and Q are equivalent to me and Qw, respectively. Consequently, the DA-PG gradient estimator becomes equivalent to DPG:
&D..G j(Fo: DPG i[TT: St).
Moreover, DPG's theoretical analysis can also be extended to the distributiinn-as-actioo frame- work. Appendix C.3, we generalize the convergence analysis of DPG to DA-PG, establishing theoretical guarantee that holds for MDPs with arbitrary action space types.
4.2 COMPARISON TO OTHER ESTIMATORS FOR STOCHASTIC POLICIES
We now compare the proposed DA-PG estimator with classical stochastic policy gradient methods, highlighting its variance and bias characteristics across action spaces.
DA-PG can be seen as the conditional expectation of both the LR (Equation (5)) and RP (Equa- tion (7)) estimators. This leads to strictly lower variance for stochastic policies.
Proposition 4.4. Assume Qw Quo in i (mo; A) and Qw am in DD ( St). Then, PG ( St) E A~ ( ( Sr, A)]. Further, if the expectation of the action- conditioned variance is greater than zero, then V V [(To: St)) < V J i St. A)).
Proposition 4.5. Assume A continuous, Qw Ina in [ Sr, and Qw = aoa in P-PA-G ((to Then, f (.) = Eenp [ j ; St, <)]. Further, if the expectation of the noise- induced variance is greater than zero, then V DA.GG ( VVRR ((To; (Too;
RP
4

---

Published as a conference paper at ICLR 2026
In discrete action spaces, the LR estimator typically requires carefully designed baselines to manage high variance, especially as dimensionality increases. While biased alternatives like the straight- through (ST) estimator (Bengio et al., 2013) continuous relaxations (Jang et al.. 2017; Maddison et al., 2017) exist, they sacrifice unbiasedness even when using a perfect critic. DA-PG avoids this trade-off, providing the first unbiased RP-style estimator with low variance in the discrete setting.
In continuous action spaces, DPG offers zero variance but assumes fixed stochasticity (i.e., no learn- able exploration). RP estimators allow for learning the stochastic parameters but exhibit higher variance. DA-PG offers the best of both worlds: it permits learning all policy parameters including those for stochasticity while retaining the zero-variance property per state.
Another direction to reduce variance is expected policy gradient (EPG; Ciosek & Whiteson, 2018; Allen et al., 2017). The idea to integrate (or sum) over actions, yielding zero-variance gradients conditioned on state: VE (TT:: St) T 8.) [Qw (S,, Ar)]- However, this estimator is only practical in low-dimensional discrete action spaces (Allen et al., 2017) or in special cases within continuous settings- such as Gaussian policies with quadratic critics (Ciosek & Whiteson, 2020). In contrast, our estimator VD ( Si) generalizes to a wider range of settings, including high-dimensional discrete, general continuous, and even hybrid action spaces.
Despite its lower variance, DA-PG may suffer from increased bias due to the increased complexity of the critic's s input space. For discrete actions, the critic Qw inputs vector of probabilities corre- sponding to discrete outcomes. For continuous actions, with Gaussian policies, the critic Qw inputs both the mean and standard deviation. This increased input dimensionality may make it harder to approximate the true action-value function, and if the critic is inaccurate, the overall benefit of lower gradient variance may be diminished an effect we examine empirically in Section 5.5.
4.3 INTERPOLATED CRITIC LEARNING
In this section, we propose method to improve learning the distributionn--s-action critic Qw- Similar to Equation (3), the standard TD update for Qw is
w - w + (Re+1 + + (S(+1))) Qw (S,, Uz)) V (w.
(1I)
This update, however, does not make use of the sampled action A, or its relationship to the resulting state and reward. If A, is made observable the agent, the transition itself can be leveraged to update the value alternative distribution parameters Ü. This is possible because the observed action A, could have been sampled from distributions parameterized by many such U.. As a result, we can update the critic off-distribution, i.e., at distribution parameters different from those used to generate the transition. This parallels off-policy learning, but operates over action distributions rather than distinet policies.
What, then, should we choose for 0,? To answer this, ask: what properties should the critic have to support effective policy optimization in parameter space? Our answer is that the critic should provide informative gradient directions that guide the policy toward optimality. For MDPs, there always exists a deterministic optimal policy (Puterman, 2014). Therefore, we assume the existence of some Ua; U, deterministic distribution corresponding to the optimal action A; for state S.. Ideally, the critic should exhibit curvature that points toward such optimal parameters U,+
One candidate for Ú, is UA the deterministic distribution parameters associated with the sampled action A, However, merely learning accurate values at UA, does not ensure that the critic has smooth curvature from U, toward high-value points. To encourage the critic to generalize better and provide smoother gradients, we propose using a linearly interpolated point between U, and A,=
Ú. wili+ + (1 A
We ~ Uniform 1].
(12)
The critic is trained to predict the value at Ú, using the following update:
w - w + (R +
- Qw (S,,
0t).
(13)
We refer to this approach Interpolated Critic Learning (ICL). Note that the ICL update omits importance sampling weights that correct for the mismatch between f(-(V,) and f(-1ū;), and is therefore generally biased. For simplicity, we do not include such corrections and leave a principled treatment of importance weighting and more advanced off-distribution updates to future work.
5

---

Published as a conference paper at ICLR 2026
Reword for Each Accion
Expected Reward Contorr Simplex
Learned Or be (Slandard Updatel
Learedd Cruic BCL Update)
100
Spawihdly
Kialyy
835
0.55
3.0
100
P.
Reward Runction
Dpecced Reward Contau
Learned Gttic Blandard updats)
Learned Citce Updala)
0.8-
05-
ast marrolo
3,64
3466
454
TEheloy
1.3
-A3
-25
0.12
2424
038
3.44
-6.5
-35
1000
-30
-0.3
Figure 2: Visualization of the reward function (col 1), expected rewards of distribution pa- rameters (col 2), and learned critics using the standard update in Equation (11) (col 3) and the Interpolated Critic Learning (ICL) update in Equation (13) (col 4) in policy evaluation (PE). Top: K-Armed Bandit. Bottom: Bimodal Continuous Bandit. With access only samples from the PE policy (the fixed policy being evaluated), the standard update estimates values accurately at that pol- icy but fails generalize beyond it. In contrast, the ICL update learns critic that captures curvature information useful for policy optimization.
To further provide intuition on ICL, we conduct a policy evaluation experiment bandit problems, shown in Figure 2 (column 1). Figure 2 (column 3) and (column 4) show the learned crities using the standard update in Equation (11) and the ICL update in Equation (13), respectively. The critic learned by ICL has more informative curvature. As a result, the policy can be updated toward high- value regions more easily. In the continuous action case, the learned critic is sufficient to update the policy toward near-optimal distribution parameters. More details can be found in Appendix D.2.
4.4 DISTRIBUTIONS-AS-ACC ACCTINN ACTOR-CRITED
Since DA-PG is derived from DPG, we base our practical algorithm on TD3 (Fujimoto et al., 2018) (), a strong DPG-based ofl-policy actor-critic algorithm for continuous control. We replace the clas- sical actor and critic with their Distributions-as-actii (Equation (10)) and the ICL critic loss (Equation (13)) to update them, respectively. We omit the actor target network, it does not improve performance (see Appendix D.4). The pseudocode for the resulting algorithm, Distributions Citiiattioos----ttiiin Acctins Cittr-Critic (DA-AC), provided in Appendix G.
counterparts and use the DA-PG estimator
EXPERIMENTS
In this section, we conduct experiments to investigate DA-AC's empirical performance in continuous (Section 5.1), discrete (Sections 5.2 and 5.3), and hybrid (Section 5.4) control settings. In addition, we examine the effectiveness of the proposed interpolated critic learning in Section 5.5. Unless otherwise noted, each environment is run with 10 seeds, and error bars shaded regions indicate 95% bootstrap confidence intervals.
5.1 CONTINUOUS CONTROL
We use OpenAl Gym MuJoCo (Brockman et al., 2016) and the DeepMind Control (DMC) Suite (Tunyasuvunakool.. et al., 2020) for continuous control. From MuJoCo, environments: Hopper-v3, Walker2d-v3, HalfCheetah-v3, Ant-v3, and Humanoid-v3; from DMC, the same 15 environments D'Oro et al. (2023). Details about these environments are in Appendix D.4. We run for I million (1M) steps in each environment.
the commonly used
Algorithms We use TD3 (Fujimoto et al., 2018) as primary baseline, DA-AC is based it. We also include an off policc actor-critic baseline that uses the reparameterization (RP) estimator. This RP-AC algorithm closely resembles DA-AC but learns in the original action space and updates

---

Published as a conference paper at ICLR 2026
Relative Finst Perfomanco: DA-AG TD3
DNC kahk)
Narrakuee Natan D
DA-A2 12-40
RPMA
102
SAG
PP0
Figure 3: Relative final performance of DA-AC versus TD3 across 20 individual continuous con- trol tasks (col 1), and average normalized returns of DA-AC and baselines on MuloCo (col 2) and DeepMind Control (col 3) tasks. In individual task comparisons (col 1), results are averaged over 10 seeds per task. For average performance plots (cols 2-3), values are averaged over 10 seeds and tasks. Error bars show 95% bootstrap confidence intervals (CIs).
kal-unu
quaddrupod walk
quadiuppedraa cun
1000
Avurage Ralurn
800
Aouraqa Ralurn
600
800
Avwrage
400
400
200
200
0.4 Timsolap
Timesdep
185
Timwstop
165
100
humanad-stter
humanois-uwik
hamanaid--un
400
100
Avsrage Rueum
Avuraga Rutum
400
Average
800
75
2000
50
1200
100
0.0
0.2
0.4
08
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.0
0.8
1.0
Timestip
105
Timestap
166
Timestop
106
DA-AC
RP-AC
TDS
SAC
PPO
Figure 4: Learning curves in six DeepMind Control tasks with high-dimensional action spaces. Results are averaged over 10 seeds. Shaded regions show 95% bootstrap Cls.
the policy using the RP estimator. For consistency, DA-AC and RP-AC use the default hyperparam- eters of TD3 and a Gaussian policy parameterization. Implementations details and pseudocode can be found in Appendices D.4 and G, respectively. For reference, of SAC (Haarnoja et al., 2018) and PPO (Schulman et al., 2017). We focus on the single-stream learning setting,
also evaluate the performance
the original single-stream version of PPO as in Schulman et al. (2017).
Results Figure 3 shows per-environment performance for DA-AC and TD3 and the aggregated results across environments for all algorithms. From Figure (column 1), we can that DA- AC achieves better performance in more environments compared to TD3. From Figure 3 (columns 2-3), see that DA-AC achieves better overall performance, outperforming most baselines significantly in the DMC Suite, particularly in high-dimensional environments (see Figure 4).
5.2 DISCRETE CONTROL
Following Ceron & Castro (2021), we use Gym classic control (Brockman et al., 2016) and MinAtar environments (Young & Tian, 2019) for discrete control. We run each environment for 500k (classic control) or 5.M (MinAtar) steps.
Algorithms We include off-policy actor-critic baselines that resemble DA-AC. These baselines leam in the original action space and update the policy with different gradient estimators, including the likelihood-ratte (LR-AC) and expected (EAC) policy gradient estimators. Here, LR-AC uses state-value baseline analytically computed from action values. Although not common in prior work, also include variant that uses the through (ST) estimator (Bengio al., 2013), denoted

---

Published as a conference paper at ICLR 2026
Gyn 14 luas)
Minttrr lasks)
Disontied MujjOo is taiks)
Diwneloo DMC kolk)
66
D642 0T-40
LMAC
D6C
DQH
PP0
04-40 0T4G LAWC
DAG
OGAD
mmo
D4-46
07-40
D4-40
0146
LMA0
Figure 5: Average normalized returns of DA-AC and baselines on discrete control benchmarks, including classic control (col 1), MinAtar (col 2), discretized MuJoCo (col 3), and discretized Deep- Mind Control (col 4) tasks.
as ST-AC. This baseline is the discrete counterpart of RP-AC, serving performance reference for alternative RP-based methods. For comparison, we also evaluate the performance of Discrete SAC (DSAC; Christodoulou, 2019). DQN (Mnih al., 2015), and PPO (Schulman et al., 2017). The hyperparameters for DA-AC and X-AC baselines (LR-AC, ST-AC, and EAC) are adopted from the TD3 defaults and adjusted the corresponding benchmark based those of DQN. More details and pseudocode can be found in Appendices D.5 and G, respectively.
Results From Figure (columns 1-2), we can see that DA-AC is among the best-performing algorithms in both classic control and MinAtar, achieving comparable performance to DQN.
5.3 HIGH- DIMEESSIMNNIO DISCRETE CONTROL
We also conduct experiments on discrete control with high-dimensional action spaces. For this setting,
the same 20 environment from Section 5.1 but with discretized action space. Specifically, we discretize each action dimension into bins with uniform spacing. For exam- ple, the original action space in Humanoid-v4 is [-0.4, 0.4]*7, which is discretized to 0.4 x We run each environment for 1M steps.
(-1,-3 3: 3 0, 3 1)
Algorithms We use ST-AC, LR-AC, and PPO from the previous section baselines. EAC, DSAC, and DQN are excluded, as they are computationally infeasible in environments with high- dimensional discrete actions. In particular, DSAC-lik- like EAC-relies dates over the full action set; without the expected updates, it fails to learn. DQN would likewise require representing and maximizing over this action space, which is intractable. LR-AC learns an additional state-value function as a baseline, since analytically deriving it from the action-value function is prohibitive in this high-dimensional setting. We use the same hyperparameters as those in Section 5. More details can be found in Appendix D.6.
computing expected up-
Results As shown in Figure (columns 3-4), DA-AC's average performance is higher than all baselines in both benchmarks. Note that the performance of DA-AC and PPO is comparable to the original continuous action setting (see columns 2-3 in Figure 3). Figure 12 Appendix D.6 also contrasts the performance of DA-AC with discrete versus continuous actions.
5.4 HYBRID CONTROL
In addition to continuous and discrete control settings, we also evaluate DA-AC's performance in parameterized-action MDPs (PAMDPs), a hybrid control setting with parameterized actions (see Masson et al. (2016) for detailed discussion). We use 7 PAMDP environments from Li et al. (2022) and follow their experiment protocol. See Appendix D.7 for more details.
RAMDFs (7 tasks)
Auwrey Normated Deformeno
6.9
Algorithms We use PATD3 our primary baseline, DPG- based baseline specifically designed for parameterized action (PA) spaces. PATD3 builds PADDPG (Hausknecht & Stone, 2016) and incorporates clipped double Q-learning from TD3, making it suitable and directly comparable baseline for DA- AC, as both methods build on TD3. DA-AC, the distribution parameters include both the probability vector for the discrete actions and mean/log-std vectors for
DA-AC:
PATI3
POON
HIHON
Figure 6: Average normalized performance of DA-AC and baselines on hybrid control tasks.
8

---

Published as a conference paper at ICLR 2026
Shored Step
DA-AG weo 1a, Step 1000
DAAC Step 1000
BA-AC wie id, Step 2000
DAVAC Step 2000
0.5
0.004
Fy 100
Shared Stap
UA-ACC wile KL Step 1000
DA-AC Step 1900
DA-AC wro ICL Stup 2000
DA-AC Step 2000
05
0.
-2
Figure 8: Initial critic (col 1) and learned crities at different training stages for DA-AC w/o ICL (cols and 4) and DA-AC (cols 3 and 5). Top: K-Armed Bandit. Bottom: Bimodal Continuous Bandit. Red dots indicate the current policy parameters. DA-AC produces more accurate value estimates at deterministic distribution parameters-ccorrspooodd and the x-axis the continuous case and offers stronger gradient signals for policy optimization.
to the vertices in the discrete case
the continuous actions. We keep most hyperparameters the same TD3's default unless otherwise adjusted to align with PATD3. In addition, we also include PDQN (Xiong et al., 2018) and HHQN (Fu et al., 2019) additional baselines for reference. See Appendix D.7 for more details.
Results Figure shows the average normalized performance DA-AC and baselines. The learn- ing curves in each individual environment can be found Figure 16. We can see that DA-AC also achieves competitive or better performance than the baselines.
5.5 EFFECTIVENESS OF INTERPOLATED CRITIC LEARNING
We compare DA-AC and DA-AC w/o ICI., ablated version that uses the standard critic up- date (Equation (11)). As shown in Figure 7, DA-AC w/o ICL generally underperforms DA- AC across settings; the paired t-test in Ap- pendix D.8 similarly indicates a statistically
Performance Comparison Across Settings
Average Nomalized Retur
0.8
DA-AC
DA-AC wa ICL
0.4
significant positive mean improvement.
0.2
To provide further insights into the differences, to bandit setting where visualization and analysis are intuitive. We use the same ban- dit environments from Figure and run each algorithm for 2000 steps and 50 seeds. See Ap- pendix D.3 for hyperparameters and details.
Gym
Figure 7: Comparison between DA-AC and DA-AC w/o ICL in all settings.
Figure in the appendix shows the superiority of DA-AC over DA-AC w/o ICL, as well as the bias- variance trade-off incurred by different gradient estimators. To assess the impact of ICL on critic quality, we visualize the learned critics from a representative training run of DA-AC and DA-AC w/o ICL. Figure 8. In both discrete and continuous action settings, DA-AC yields a significantly improved critic landscape early in training.
CONCLUSIONS
We introduced the distributions-as-actii framework, redefining the agent-environment boundary to treat distribution parameters actions. We showed that the policy gradient update has theoreti- cally lower variance, and developed a practical deep RL algorithm called Distributions-as-e Asction Actor-Critic (DA-AC) based on this estimator. We also introduced date, ICL, tailored to this new setting. We demonstrated that DA-AC achieves competitive perfor- mance in diverse settings across continuous, discrete, and hybrid control.
improved critic learning up-

---

Published as a conference paper at ICLR 2026
This reframing allowed us develop continuous action algorithm that applies to diverse underly- ing action types. A key next step is to further exploit this reframing for new algorithmic avenues, in- cluding model-based methods, hierarchical control, novel hybrid approaches. See Appendix B for a discussion of how existing RI. algorithms can extended within this framework. There are also key open questions around critic learning. More advanced strategies for training the distributions- as-actions critic could be investigated, including off-distribution updates at diverse regions of the parameter space or using a learned action-value function Qw(s, a) guide updates of Qw (5, u). This will also open up new questions about convergence properties for these new variants.
ACKNOWLEDGMENTS
This research is supported by the Canada CIFAR Al Chairs program, the Reinforcement Learn- ing and Artificial Intelligence (RLAI) laboratory, the Alberta Machine Intelligence Institute (Amii), and the Natural Sciences and Engineering Research Council (NSERC) of Canada. Jiamin He also gratefully acknowledges Richard S. Sutton, Csaba Szepesvári, Bryan Chan, and Shivam Garg for valuable discussions and thanks the Digital Research Alliance of Canada for providing computa- tional resources.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide code release covering DA-AC implementations across all control settings at https: h //github. com/he jm37/ com/hejm37/dan jm37/da-acc Comprehensive hyperparameter choices and environment configurations documented in Appendix D. All reported metrics based on multiple seeds, with uncertainty quantified using 95% bootstrap confidence intervals.
REFERENCES
Cameron Allen, Kavosh Asadi, Melrose Roderick, Abdel-rahman Mohamed, George Konidaris, and 2017.
Michael Littman. Mean actor-critic. arXiv preprint
Amrit Singh Bedi, Anjaly Parayil, Junyu Zhang, Mengdi Wang, and Alec Koppel. On the sample complexity and metastability of heavy-tailed policy search in continuous control. Journal of Machine Learning Research, 25(39):1-58, 2024.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279 2013.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 308.3432, 2013.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2024.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv: 1606.00660000 0440, 2016.
Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insight- ful and inclusive deep reinforcement learning research. International Conference on Machine Learning. pp. 1373 1383. PMLR, 2021.
Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the Beta distribution. In International Conference Machine Learning, pp. 834 843. PMLR, 2017.
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint 209::90077 2019.
Kamil Ciosek and Shimon Whiteson. Expected policy gradients. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 32, 2018.
10

---

Published as a conference paper at ICLR 2026
Kamil Ciosek and Shimon Whiteson. Expected policy gradients for reinforcement learning. Journal of Machine Learning Research, 21(52):1-51, 2020.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference Learning Representations, 2023.
Zhou Fan, Rui Su, Weinan Zhang, and Yong Yu. Hybrid actor-critic reinforcement learning in pa- rameterized action space. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pp. 2279-2285, 2019.
Haotian Fu, Hongyao Tang, Jianye Hao, Zihan Lei, Yingfeng Chen, and Changjie Fan. Deep multi- agent reinforcement learning with discrete-continuous hybrid action spaces. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 2329-2335, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pp. 1587-1596. PMLR, 2018.
Evan Greensmith, Peter I. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates reinforcement learning. Journal of Machine Learning Research, 5:1471-1530, 2004.
Tuomas Haarnoj,, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
Nicklas A Hansen, Hao Su, and Xiaolong Wang. Temporal difference learning for model predictive control. In International Conference on Machine Learning, pp. 8387 8387-8406 PMLR, 2022.
Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. In International Conference on Learning Representations, 2016.
Jiamin He, Samuel Neumann, Jincheng Mei, Adam White, and Martha White. Revisiting mixture policies in entropy- regularzeed actor-critic. In NeurlPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists, 2025.
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learn- ing continuous control policies by stochastic value gradients. Advances in Neural Information Processing Systems, 28, 2015.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. CleanRL: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-Softmax. In International Conference Learning Representations, 2017.
Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. HyAR: Addressing discrete- contriuu- action reinforcement learning via hybrid action representation. In International Conference on Learning Representations, 2022.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforeement learning. In Inter- national Conference Learning Representations, 2016.

---

Published as a conference paper at ICLR 2026
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete distribution: A continuous laxation of discrete random variables. In International Conference on Learning Representations, 2017.
Warwick Masson, Pravesh Ranchod, and George Konidaris. Reinforcement learning with param- eterized actions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang, Bo An, Haipeng Chen, and Sanjay Chawla. S AC: Energy- based reinforcement learning with Stein soft actor critic. In The Twelfth International Conference Learning Representations, 2024.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Kirkeby Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charlie Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference Machine Learning, pp. 1928-1937. PMLR, 2016.
Ofir Nabati, Guy Tennenholtz, and Shie Mannor. Representation-drrieen reinforcement learning. In International Conference on Machine Learning, pp. 25588-25603 PMLR, 2023.
Paszke. PyTocch: An imperative style, high-performance deep learning library. arXiv preprint 11110:191,.01703, 2019.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in rein- forcement learning. Journal of Machine Learning Research, 25(318):1-63, 2024.
Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Matthew Kyle Schlegel, Volodymyr Tkachuk, Adam M White, and Martha White. Investigating action encodings in recurrent neural networks in reinforcement learning. Transactions on Machine Learning Research, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Tim Seyde, Igor Gilitschenski, Wilko Schwarting, Bartolomeo Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? Solving continuous control with Bernoulli policies. Advances in Neural Information Processing Systems, 34:27209-2777 27221, 2021.
Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via Q-learning. In The Eleventh International Conference on Learning Representations, 2023.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pp. 387-395. PMLR, 2014.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Sifre, George van den Driess- che, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 52947587)4444489, 2016.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances Neural Information Processing Systems, 12, 1999.
12

---

Published as a conference paper at ICLR 2026
Yunhao Tang and Shipra Agrawal. Discretizing continuous action space for on-policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5981-5988, 2020.
Emo Todorov. Dynamical system optimization. arXiv preprint
2025a.
Emo Todorov. Equivalence of stochastic and deterministic policy gradients. arXiv preprint arXiv:2505.23244 2025b.
Saran Tunyasuvunakool. Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm.control: Software and tasks for continuous control. Software Impacts. 6:100022, 2020.
Hado van Hasselt and Marco Wiering. Reinforcement learning continuous action spaces. In 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning. pp. 272-279. IEEE, 2007.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Gautham Vasan, Mohamed Elsayed, Seyed Alireza Azimi, Jiamin He, Fahim Shahriar, Colin Bellinger, Martha White, and Rupam Mahmood. Deep policy gradient methods without batch updates, target networks, or replay buffers. Advances in Neural Information Processing Systems, 37:845-891, 2024.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17(3):261-272. 2020.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992.
Ronald J Williams. Simple statistical srudient-following algorithms for connectionist reinforcement learning. Machine learning, 222992256 1992.
Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministic policy gra- dient: Convergence analysis. In Uncertainty in Artificial Intelligence, pp. 2159-2169. PMLR, 2022.
Jiechao Xiong, Qing Wang. Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu. Parametrized deep Q-networks learning: Reinforcement learning with discrete-continuou hybrid action space. arXiv preprint arXiv: 181.0:1810.06394, 2018.
Ming Xu, Matias Quiroz, Robert Kohn, and Scott A Sisson. Variance reduction properties of the reparameterization trick. The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2711-2720 PMLR, 2019.
Timothy Yee, Viliam Lisy. and Michael Bowling. Monte Carlo tree search in continuous action spaces with execution uncertainty. Proceedings of the 25th International Joint Conference on Artificial Intelligence, pp. 690-697, 2016.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint 1XXi:19033003166 2019.
Shenao Zhang, Boyi Liu, Zhaoran Wang, and Tuo Zhao. Model-based reparameterizztion policy gradient methods: Theory and practical algorithms. Advances in Neural Information Processing Systems, 36:68391-68419 2023.
Lingwei Zhu, Haseeb Shah, Han Wang, Yukie Nagai, and Martha White. g-exponential family for policy optimization. In The Thirteenth International Conference on Learning Representations, 2025.
Yuanyang Zhu, Zhi Wang, Yuanheng Zhu, Chunlin Chen, and Dongbin Zhao. Discretizing contin- uous action space with unimodal probability distributions for on on-policy reinforcement learning. IEEE Transactions Neural Networks and Learning Systems, 2024.
13

---

Published as a conference paper at ICLR 2026
A
RELATED WORKS
In this section, we provide an extended discussion of related work.
Value-based control When the action space is discrete, value-based algorithms are one of the most commonly used approaches (Watkins & Dayan, 1992; Mnih al., 2015; van Hasselt et al,, 2016; Hessel et al., 2018). By learning an action-value function, these algorithms extract policies using various greedy operators. While these methods have been effective in wide range of dis- crete control domains, their applications to continuous-action problems are limited, with only a few exceptions (Seyde et al., 2023).
Policy-hased discrete control Policy-based methods, including actor-critic algorithms, form another important class of approaches for discrete control (Williams, 1992; Mnih et al., 2016). These methods explicitly maintain a policy that outputs distribution parameters used construct a policy distribution from which actions are sampled. In the discrete case, these parameters correspond to the logits of a categorical distribution. Beyond the likelihood-ratio (LR) policy gradient estimator (Williams, 1992), straight-through (ST; Bengio et al., 2013), and Concrete (Maddison et al., 2017) estimators provide reparameterization based but biased gra- dient estimation. In contrast to these biased estimators, our distributions-as-aciii (DA) gradient estimator Equation (10) can be viewed as the first unbiased reparameterization (RP) estimator for discrete distributions.
(Jang et al., 2017) 7),
Policy-based continuous control For continuous control, pasie methods dominate the literature (van Hasselt & Wiering, 2007; Silver al., 2014; Lillicrap et al., 2016; Schulman al., 2017; Haarnoja et al., 201 8). The policy typically outputs the parameters of a parametric distribu- tion. Gaussian policies are the most common choice (Lillicrap et al., 2016; Schulman et al., 2017: Haarnoja et al., 2018), although many alternatives have been explored in different contexts (Chou et al., 2017: Bedi et al., 2024; Zhu et al., 2025; He et al., 2025). Optimizing these policies using classical policy gradient estimators (LR or RP) requires access either to analytical log-density function or a reparameterization function. In contrast, the DA gradient estimator in Equation (10) requires neither, enabling application to broader class of policies. Beyond parametric distributions, implicit policies built using more expressive generative models have also been studied (Haarnoja al., 2017; Messaoud et al., 2024). Our DA framework and estimator can be applied to these advanced policy classes as well, which suggests an interesting direction for future work.
Policy-based continuous control with discretization Another line of work discretizes the con- tinuous action space and then applies discrete-action control algorithms- policy-based meth- ods due to the high dimensionality of action spaces (Tang & Agrawal, 2020; Seyde et al., 2021; Zhu et al., 2024). While such approaches have shown strong benchmark performance, they may be undesirable in practice because the resulting control can be less smooth and more unstable (Seyde al., 2021), and the method often requires additional tuning of the discretization granularity (Tang & Agrawal, 2020). In this work, we treat discretized continuous control problems primarily as testbed for nigh-dimensional discrete control. Thus, we do not extensively compare continuous- vs. discrete-basee based methods for continuous control, as this is not our main focus. Nevertheless, we include such a comparison in Figure 12 for reference.
Policy-based hybrid control Beyond purely discrete or continuous settings, many real-world applications involve hybrid action spaces requiring the agent to control discrete and continuous variables simultancously (Masson et al., 2016: Xiong et al., 2018). These problems can often be modeled as parameterized action MDPs (PAMDPs; Masson et al., 2016), in which the agent selects a discrete action and its associated continuous parameters. Most standard discrete and continuous control algorithms are not directly applicable to PAMDPs and require additional adaptation or hy- bridization to handle such action structures. For example, DDPG (Lillicrap et al., 2016) and PPO (Schulman et al., 2017) have been modified to support hybrid actions (Hausknecht & Stone, 2016; Fan et al., 2019), and combinations of DDPG and DQN (Mnih et al., 2015) have also been explored (Xiong al., 2018; Fu et al., 2019). Unlike these methods, which patch together retrofit existing algorithms, our DA reframing directly converts hybrid control into a continuous control problem, enabling a simple, unified algorithm applicable to PAMDPs and even more general hybrid settings.
14

---

Published as a conference paper at ICLR 2026
Different from the traditional policy optimization perspective used in the methods above, representation-drives RL (RepRL) offers an alternative viewpoint (Nabati et al., 2023). Instead of optimizing policy parameters 0 by estimating gradients based on the values of sampled state-action pairs, RepRI. recasts the search for optimal 0 as a linear bandit problem by projecting 0 into a lower-dimensional representation [() and optimizing 0 based its expected value over states. While our proposed DA framework also redraws the decision boundary, it does so in fundamentally different way. RepRL retreats the decision boundary all the way bandit problem and, in some sense, treats even the policy network и as part of the environment. In contrast, the DA framework only reframes the distribution parameters themselves as the decision variables, viewing only the sampling function as part of the environment.
Representation-drivel RL
D-MDP and dynamical system optimization which closely related the distributionn--s-action (DA) MDP formulation considered in our work. Their primary focus is on using D-MDP to establish connections between stochastic and deterministic policy gradients in continuous action settings. In contrast, we emphasize the applica- bility of the DA-MDP formulation to diverse action spaces and develop practical algorithms under this perspective. While their work concentrates on theoretical characterization, we complement this viewpoint with empirical investigation, demonstrating the viability of DA-MDP-based algorithms across a variety of action space types. subsequent work. Todorov (2025a) further proposed a dynamical system optimization perspective that connects broader class of problem settings. Ex- ploring the relationship between our proposed algorithm and this dynamical systems view could be an interesting direction for future work.
Concurrently, Todorov (2025b) studied D-MDP,
B EXTENDING EXISTING RL ALGORITHMS TO THE DA FRAMEWORK
While we have explored only one model-free actor-critic algorithm under the proposed distributions- as-actions (DA) framework, many other algorithms in the classical RI. literature can also be extended to this setting. To facilitate future research, we outline several such directions below.
widely used mechanism for encouraging exploration in RL. and incorporating it into the DA framework represents promising avenue. We discuss two potential approaches for adding entropy regularization to DA-AC. The first approach is to augment the policy optimization objective (Equation (4)) with an entropy term. This requires adding an entropy component to the policy gradient estimator (Equation (10)):
Entropy regularization Entropy regularization
SDA PG j(To: St) = (S,) T Qu (St, (),, U)) + a [VVoH((((
(14)
where is the entropy coefficient and H(f) denotes the entropy of distribution f. Optionally, the critic may also incorporate entropy, yielding the Maximum Entropy RL formulation (MaxEnt; Haarnoja et al., 2018) 8). The standard MaxEnt critic update within the DA framework becomes
w w a (Re:
+ (St, V 2 U.).
(15)
The second approach specific to the MaxEnt setting and incorporates entropy directly into the reward:
R2- Re+1 (HH((-)
(16)
Under this reward shaping, the optimization problem coincides with the MaxEnt objective (Haarnoja et al., 2018). This method does not require modifying the actor or critic updates; only the reward is transformed. The entropy can be computed analytically when available or estimated via samples (e.g.. using log / (A, U.)). Understanding the trade-offs between these alternatives itself an interesting open question.
Model-based planning algorithms rithms can also be incorporated into the DA framework. A straightforward approach is to combine traditional model-based algorithms operating on primitive actions with DA-based value estimation.
Beyond model-free methods, model-based planning algo-
15

---

Published as a conference paper at ICLR 2026
For example, in discrete-action environments, one can apply Monte Carlo tree search (MCTS; Silver et al., 2016) over the primitive discrete actions while using DA for the critic.
A potentially selves. This would make possible to apply continuous-ace actiionn model-based planners- as continuous-action variants of MCTS (Yee et al., 2016), TD-MPC (Hansen al., 2022), model- based reparameterization gradient methods (Zhang et al., 2023) 202rccdirectly within the DA framework.
compelling direction is to learn model over the distribution parameters them-
Incompatibility with discrefe-structuur--aase algorithms By treating distributions tions, we might lose the ability to exploit certain convenient structures of the original action space- particularly discrete settings. While this choice allows the DA framework remain agnostic to the specifics of the primitive action space, it may still be desirable to leverage action structure when beneficial. Hybrid approaches that combine the DA framework with structure-awar algorithms, such as integrating MCTS with DA-based value estimation, provide one promising path forward.
C THEORETICAL ANALYSIS
We provide proofs of the theoretical results for the distributionn-as-aaee casooons-----cttiie framework and seaibuttioon---Acctin Policy Gradient (DA-PG) in the main text in Appendices C.I and C.2. In addition, we also extend convergence proof of DPG from Xiong et al. (2022) to DA-PG in Appendix C.3.
C.I PROOFS THEORETICAL RESULTS SECTION 3
Assumption 3.1. The set UI is continuous and compact, and the function f(alu) and its derivative are continuous with respect to u. When S AA is continuous, the corresponding set is assumed to be compact; moreover, the functions p(s' [s, a), do(s). r(s, a), F(s), f(alu), and their derivatives are also continuous with respect to 8, or a, respectively.
Proposition 3.2. Under Assumption 3.1, Un (8) ux(3) (3) and On (8, u) = Ba~s(-lu) (8, As]
Proof. Let be the policy in the original MDP that first maps to u #(s) and then samples ~ f(|lu). The state-value function Un (8) in the ceaabbutioon-s-acctiono MDP is defined as
T (5)
Σ (= = s],
K=
where Up #Sx)). From Equation (8), F(s,u) EA~s(-la) [r(s, (r(s,))) A)]. Also, the transition
p(8'|s, u) f(-l())))) [p(8'ls, (1, A)]. Consider trajectory So. Up, S1, U1.... in the distributions- as-actions MDP (DA-MDP). This corresponds to a trajectory So, Ao. AI. in the original MDP where A& f(-lu)) The expected reward time & the DA-MDP, given S₃ and UK = *Sx), is F(S,, (Sx)) E (
(S(S,, A+)]- The dynamics are also equivalent in expectation: For example, when S continuous, E [[S++1/ Se, U²] = 15,,, = EA,=f(- [s [s]]. Thus, the sequence of states and expected rewards generated under the DA-MDP identical in distribution to the sequence of states and rewards under æ in the original MDP. Therefore, (s) = (s).
For the action-value function (s( u),
Tx (s, u) Ex [F Un) + x (S) So = = U.U0 = u]
F(s,u) + + (S:)]
EAws(())) A)) TE A- [s (S))] (using Ux)
EA~s(-1)) [r(s, A) + 7 (
Ex-s(()) Ex R + Tu, =
A, A]]
Ex-s(-))) (s, A)]
The compactness and continuity assumptions in Assumption 3.1 ensure these expectations and value functions are well-defined.
16

---

Published as a conference paper at ICLR 2026
C.2 PROOFS OF THEORETICAL RESULTS SECTION 4
Assumption 4.1. The function ( and its derivative are continuous with respect to 0.
Theorem 4.2 (Distributions- cettributions-s--actiiio as-actions policy gradient theorem). Under Assumptions 3.1 and 4.1, the gradient of the objective ( Lt= Ex [ Re+i] with respect O can be expressed
( Es-daa (oto((S T Vu (S,U))u=
where da (8) = Er= Eep h[[(S: = }]] is the (discounted) occupancy measure under mg-
Proof. This theorem results from applying the deterministic policy gradient (DPG) theorem to the DA-MDP (S, U, d, 7), where no S + U acts as deterministic policy. The objective
function is ( - Ux (So)]-
Following the DPG theorem derivation (Silver et al. (2014), Theorem 1), for a general deterministic policy Ne + A. the policy gradient is
(oJ( -Bsddu (o) T Vaguo (S,
AS)A)poo))(
In our context:
The policy in the DA-MDP is #o(s).
The action space U, and actions are denoted by u.
The critic Ta (s, u) is the action-value function in this DA-MDP.
The state distribution drg (s) is the discounted state occupancy measure under policy ñe-
Assumptions 3.1 and 4.1 ensure that (s) and Tre (s, u) are appropriately differentiable and that the interchange of expectation and differentiation is valid. Substituting To for ne and qs. for quo yields the theorem's result:
VoJ(me) = Eswdd (V) Vu (S,
The notation Vo) Vudro in the theorem statement implies the appropriate vector or matrix product. e R$ and u E Rmm, then (oñ is an m x Jacobian, VuğRo is an m X 1 vector, and the product (s)) Vuğno results in the 1 gradient vector for ((Re).
Proposition 4.3. IfU = A and [(-bu) is the Dirac delta distribution centered at u, then To and Qw are equivalent to πe and Qw, respectively. Consequently, the DA-PG gradient estimator becomes equivalent to DPG:
&Da.PG j & [ Si).
Proof. The DA-PG gradient estimator given by Equation (10):
AD PG jiTo: St) = (o))(S)) (S.
Given the conditions:
2 A: The distribution- distribution-paramtteer space the action space.
2. f(l1u) = 5(- - u): Sampling ~ f(-lu) yields A = u.
Under these conditions, no (S:) outputs parameters U 14 which are directly actions in A. Thus, we can write (S.)) = #o (St), where (S{S,) E A.
Next, consider the DA value function [5. (St, U). From Proposition 3.2, q5. (St, U) (- Iza (S₁, A)]. Since f(A(U) 8(A U), the expectation becomes gzo (S₁ So,
(ag (St, U) Oro (Se, U) where U Eud A.
17

---

Published as a conference paper at ICLR 2026
This means the DA critic Qw US),U) is estimating the vetiun-value function qno U). Thus, can write (w( = (S, U), where U E A..
Substituting these equivalences into the DA-PG gradient estimator:
pD St) VoTo TV V (S,, A))
This is precisely the DPG gradient estimator (Equation (6)). Thus, pDA-PG DDPG St):
Proposition 4.4. Assume Qw = Ano in [ St. A) and Qw = qao in DD [ St). Then, J Si) (Aamto-\Ss.) J [=o; St, A)). Further, if the expectation of the action- conditioned variance is greater than zero, then V V( [ St)) < V (( J (mØ;; St, A)).
A)]. Given Qw Toe and Qw qug. this becomes Qw(S, (S,, = ((-10))) [ (S,, A)]. Note that Qw (St, A) and Qw,,, U) distinct critic functions. 'he use of w for both signifies that they are learned approximators. In the context of this proof, corresponding subset of w.
Proof. Proposition 3.2 states T5 (St, U) = A~ U) I (St,A)]
think of Qw and Qw as separate approximators, each utilizing
Starting with the DA-PG estimator (assuming continuous A: other cases similar),
DAG jaa; St) VoTo(S.) VS,
Voño(S.) VUEAwSHHU) [Q (St. (=M((S)
= Voña(S.)) Vu / f(AlU) dA
(==Fo(8)
VoTo(S.) VOJAAVU))Q A) A) Um=+((e)
= Voto(S.) T /
Qw (w dA
/ (cTo(S))VV[((() Q
(S A)
The differentiability under the integral sign is justified by Assumption 4.1. The last line follows from the chain rule, where (AlTa(a Vota(S.) ( (A
Using Te (AASs) = ( and the log-derivative trick, we can express the DA-PG estimator
PG i(mo: St)
V (S.
Ve log no A
E A (s))
Vo (oo0 A)].
The LR estimator is VLR ( Sr, A) Vo Volog no (A)s A) V(S.)). Its expectation is
E Ex-Ta(-15)) İLR (TT:: S., A)]. The term involving the baseline (S:) vanishes in expectation:
B VØ Volog no (A) = V(S,) (8 V log (A[(s))
=V(S.) / (oTO(A dA
VV (A() dA= V(8.V
18

---

Published as a conference paper at ICLR 2026
1 J To: St, A)] BANTo (:18) Vo lolo Toll This shows V s St) = E. ANaTai (8a)) VRR J(TT: St, A)]
Thus, (A-To(118e))
For variance reduction, let X J St, A) and Y V DDAPG Si). We have Y s ( (expectation over A). By the law of total variance: V(X)
E [ =o())) + (( (E[ To (o.))) This translates to ((To Sr A)) = Es. | (
+ v v( PG (TTe:
If s, V a (VIR J (TO: Sr, A)] (i.e., the action-conditioned variance is positive on average), then V V( PG [ Si)) ((ViRR(TT) St, A)).
Proposition 4.5. Assume A contimuous, Qw = q in ( St, e). and Qw = qa in D...PG ((), Then, D [ S.) = Ee-pl [ J(Te: St, c)) Further, if the expectation of the
variance is greater than zero, then V (VD (#o: v( (To; e)).
Proof. For the RP estimator, the action is generated as go (c; St), where e p(-). For con-
sistency with DA-PG notation, we can write A = g(e; U), where U = To (S,) E l represents all relevant learnable distribution parameters. Thus, the distribution f()(U) of the random variable induced by g(e; U) with p(-).
Similar to the proof Proposition 4.4, given the critics are the corresponding true action-value functions, we have
(St, U) = EA~ [ (S,, A)] = Bewp (Qw
where we use change of variables to express the expectation in terms the noise e. Now, we can express the DA-PG gradient as
DAA-G i St) Voño(S.) V (S,
Voño(S,) Vu Ve.
Voño(S,) Eeep (L (==o(()))
Ea-p (oto(S,) (((()
Ex-p VeQ (St, g(e: 7o (5:)))].
The differentiability under the integral sign is justified by Assumption 4.1. The last line follows from the chain rule, where VoQ VaQw(Se..f To (S:))) Vota(S,) V gf
On the other hand, the RP gradient is
RP [TTo St, = Vogo(e; (..ccssSS.TTVa (S)
VAQ Vog(e: VVQ = (S (o()))
where we use the chain rule again in the last equation: (oQw(Se.((c70(S.)) Tog(c;oo(S.)) TVA V A) (=)( Thus, we have
PD..PG j(Fo: S.) = Ee-p v (To
The variance reduction argument is similar to that in Proposition 4.4. Let X PRP (mTe Se, and Y PDD..G S.). We have Y [[] (expectation over e). By the law of total variance: ((x) = E V (XXS VB x S. E) el). This translates to
A J ()) Es. V. W.(V (v j ()).) + ((ooo)))))
If Es, (. ( S:)] > 0 (i.e., the noise-induced variance is positive on average), then v V(DAAPGG j S.)) V((+A'](TO;SS S(,
19

---

Published as a conference paper at ICLR 2026
C.3 CONVERGENCE ANALYSIS FOR DA-PG
We present a convergence result for the distributioon--s-aciiin policy gradient (DA-PG), which direct application of the convergence of the deterministic policy gradient (DPG; Xiong al., 2022). We assume an on-policy linear function approximation setting and use TD learning to learn the critic. See Algorithm 1 for the analyzed DA-PG-TD algorithm. We follow the notation of Xiong et al. as much as possible for comparison with their results. As an example, instead of using da, to represent the discounted occupancy measure under Ho as in other parts of this paper, we use the notation of Xiong et al. and refer to it as in this section.
Algorithm DA-PG-TD
1: Input: Oggy 0gg, Wo, Op. batch size M.
2: for 1= 1.....,.,1
do
3: for j == ..... M- do
4: Sample St.j ~ de.-
5: Generate _ = no (st.j)-
6: Sample st+1,j ~ p( p(118115j (t, and rt.j.
7: Generate U111, = no, (st+1.))
8: Denote Lt.j (Stji Wt
9: ōr.j = rt. + + 1.j) Wt Mars) we-
end for
11: W++1 Wi
=== ((()))
M-
12: for 0, 1, ..., M do
13: Sample 5/6,j vo..
14: end for
15: 0141 = 0: +
M E
02
VoTo, Vono. (bl,)) Twe
16:
end for
Following their notation, the parameterized policy denoted 7o and the objective function J(aa) (Equation (1)) is denoted as (0). The distributiios--a--ctiins policy gradient is
VeJ (0) Ex-ro [ Vuğno (s, =))=)
(17)
where vo(s) = E Ex, [T' = s)] is the discounted occupancy measure under Tg. We also define the stationary distribution of The to be ds(s) limr Σn Eg [U(S, s)] Under linear function approximation for the critic function, the parameterized critic can be expressed as Qw u) ((s, u) w, where á S ul + Rdd the feature function.
We will first list the full set of assumptions needed for the convergence result, followed by the convergence theorem. In addition, we incorporate the corrections to the result of Xiong et al. from Vasan et al. (2024), which extends the result to the reparamcterizattoon policy gradient. Following Vasan et al., the corrections are highlighted in red.
Assumption C.1. For any 01, f22 O E Id, there exist positive constants Lx. Lg and Ap. such that (1) || (s)--Top (+)] < L 1 1000-021 02 Vs E S: (2) /VoTo, (s) Voño, (s)ll < I 10, 02 Vs E S; (3)
the matrix V == Ev Veño(s) (s) is non singula with the minimal eigenvalue uniformly lower-bounded as Jmin (V)) 2 Au.
Assumption C.2. For any U1, U2 E U, there exist positive constants Lş. L,, such that (1) the distributions-as-actions transition kernel satisfies pps' [s, u,) ( pps''s uz)1 - L uzll, Vs, s' S; (2) the distributionn-as-aciiin wall Vs, s' S.
reward function satisfies |F u1) f(s, f(8,u2) < Lullur
Assumption C.3. For any u1, u2 E U, there exists positive constant L, such that || V (s,u1) u1} V. V. (8, w2)ll < LuI|ur u2/1, vo E Rd E S.
Assumption C.4. The feature function Φ x 14 + Rd uniformly bounded, i.e., ||[(, )l S Co for some positive constant C.g. In addition, we define A = Eds [0(x)(10<=)) - <(x)) and = Es, (o(x))())T] and assume that A and D are non-singular. We further assume that the absolute value of the eigenvalues of A are uniformly lower bounded, i.c., (a(A) 2 X for some positive constant AA
20

---

Published as a conference paper at ICLR 2026
Proposition C.5 (Compatible function approximation). fuuction estimator (s, u) is compatible with policy No, i.e,, (0) BBe, [Vo u) ul)==Ro())) if it satisfies the following two conditions:
1. (w u) (== (e)
2. " minimizes the mean square error Evy, [E(s; w) V. u) - Vulna (x, u) u))==))()))
T) E(s; w)), where E(s; w)
Given the above assumption, one can show that the distributions-as-atiio policy gradient is smooth (Lemma C.6), and that Algorithm converges (Theorem C.7).
Lemma C.6. Suppose Assumptions C. C-I.C.3 hold. Then the distributtions-a--cctiin policy gradient (()) defined in Equation (17) Lipschitz continuous with the parameter LJ, i.e., v0,, 02 E R
[V,( J(((2)) - Lsll0, 02l1.
where I, = (o (T++ Buuaa 1.g + Ti + IT I. 1-7
+ P.egiahh
1-7
Theorem C.7. Suppose that Assumptions C.1-C.4 hold. Let aw
48am
5
20
Aa Then the output of DA-PG-TD in Algorithm I satisfies
min
4hy 24761
min E
V [(.))l
+ M T
tcgh
EE(T]
where 0j = ao(1->)
88max
+ 1111]
11000 2 11, C2 = 480 2 (C2 +?, 961 1 ag
1116, +
Area
dow
da
72L 0 10g:
C3 = 1817 +
241 24.2 L7
24
1441.
with CA = -02 Cb = Rmax Car Cio
daw
Aaeu
Co Owe = xx)(1-7)*
C,
Li
L1 +
Cg
(L +
20g Leg
I-y L = L3 Cg = LF + Lip
Ne
x²11->)
RRuxx 1-9
Lu Lo and Ls defined in Lemma C.6, and we define
max lug w..ll
Remark C.8. Apart from the corrections highlighted in red, the convergence result retains the same mathematical form as the DPG convergence result (see Theorem I of Xiong et al. (2022)). How- ever, the associated constants differ, as they are defined with respect to the distributions-as-actie formulations of the MDP, policy, and critic. Notably. the satributiions-s--action policy class strictly generalizes the deterministic policy class. Consequently, this convergence result constitutes strict generalization of the DPG convergence result.
The proofs of Lemma C.6 and Theorem C.7 follow the same lines as that of Lemma and Theorem 1 of Xiong et al. (2022). We refer the reader to Xiong et al. for proofs and discussion and Vasan al. (2024) for details about the corrections.
D EXPERIMENTAL DETAILS
Our implementation builds upon a PyTorch (Paszke, 2019) implementation of TD3 from CleanRL (Huang et al., 2022), distributed under the MIT license.
Since the performance distribution in reinforcement learning (RI.) is often not Gaussian, we use 95% bootstrap confidence intervals (CIs) for reporting the statistical significance whenever applicable, as recommended by Patterson et al. (2024). We use hrtpy.stats.boot bootstrap with 10, 000 resamples from SciPy (Virtanen et al., 2020) to calculate the bootstrap Cls.
For all bar plots, we plot the final performance, which is computed using the average of the return collected during the final 10% training steps. We treat each seed in each environment inde- pendent trial and compute bootstrap Cls over the variability across both seeds and environments, thereby capturing both stochastic training variation and environment-specffic effects.
21

---

Published as a conference paper at ICLR 2026
D.1 POLICY PARAMETERIZATION AND ACTION SAMPLING
When the action space multidimensional, we treat each dimension independently. For simplicity, exposition will focus the unidimensional in the remaining of the paper.
We use the categorical policy parameterization: f(llpp.... )) where f(al[pı,.. "Ppp,""))))))))) I Pi ((=m)) is the probability mass function for the categorical distribution. For DA-AC, choose the probability vector [p,,-.. PPN) as the distribution parameters. We define the distribution parameters corresponding to an action A to be the one-hot vector U^= one hot.A..
Discrete action spaces
Continuous action spaces parameterization that is used in TD3: = clip()' + e, Omin: 0max) restrict the mean to be within [Amin: 0max] using a squashing function:
Assume the action space is [Amin; Omax - We use the Gaussian policy (O) Same as TD33,
" + 2
(amax Omin) + amins Up tanh
(18)
where logit, ja
E I the actor network's output for μ. While TD3 uses a fixed OTD3 0.1
Omin
we allow the learnable standard deviation to be within range o E [Tmin Tmax]:
Uo + * (log maa log mmin) + log minn
log a
tanh(logit,),,
(19)
where logit, E R. is the actor network's output for For RP-AC, the reparameterizztion function (c; 5,) clip(/4 (S.) + (g(S,) amax). C NN 1). For DA-AC, we choose the distri- bution parameters to be u [U,, Ua] 1-1, 1| that the parameter space is consistent across mean and standard deviation dimensions. Since we lower bound the standard deviation space to encourage exploration, we define the distribution parameters corresponding to action A to be UA = 2A 1]] to approximate the Dirac delta distribution, which corresponds to μ = A
and σ min.
For environments with hybrid action spaces, DA-AC simply uses the policy parameterizations described above for the corresponding discrete and continuous parts.
Hybrid action spaces
D.2 POLICY EVALUATION IN BANDITS
K-Armed Bandit
We K-armed bandit with K and a deterministic reward function: r(ai)= 0, raa)) 0.5, r ra3) 1.
We use continuous bandit with bimodal reward function that is deterministie. The reward function the normalized summation of two Gaussians' density functions whose standard deviations are both 0.5 and whose means are -1 and 1, respectively:
Bimodal Continuous Bandit
(
r(a)
+
We restrict the action space to be [amin: 4maxl [-2,2]. The standard standard deviation is con-
strained to [amin: mmax] [e-3, e). corresponding to
[log amiu, log maax] [..
(20)
Critic network architecture To be consistent with the RL settings, we use the same critic network architecture those in Appendices D.4 and D.6. Specifically, use two-layer MLP network with the concatenated state and action vector as input. We reduce hidden size from 256 to 16 and use a dummy state vector with a value 1.
Experimental details We keep the policy evaluation (PE) policy fixed and update the distributionn--a-actionn critic function for 2000 steps using either Equation (11) or Equation (13). In K-Armed Bandit, the PE policy is APs = upp = 11/3,1/33 1/3]; in Bimodal Continuous Bandit, the PE policy is TEP UPE [0. 0.5] (corresponding to μ 0 and log 0.0: Equations (19) and (20) for the calculation
log σ). The hyperparameters are the same as those of DA-AC in Table 4, except that the actor is kept fixed to the corresponding PE policy.
22

---

Published as a conference paper at ICLR 2026
D.3 POLICY OPTIMIZATION IN BANDITS
Environments
We use the same bandit environments as Appendix D.2.
Algorithms In addition to DA-AC and DA-AC w/ ICL, we also include LR-AC and RP-AC as a reference, as they should be quite effective in these settings because of a much simpler critic function. Note that our goal not to show that DA-AC can outperform other baselines in these toy settings, but rather to illustrate how ICL substantially improves critic learning in DA-AC. Here, LR-AC uses the average of the action values as the baseline. We also include LR-PG, RP-PG, and DA-PG, variants of LR-AC, RP-AC, and DA-AC that have access to their corresponding true value functions to remove the confounding factor of learning the critic.
Experimental Details We use the same critic network architecture in Appendix D.2. We use the same actor network architecture as those in Appendices D.4 and D.6. Specifically, we use two-layer MLP network with the state vector as input. We reduce the hidden size from 256 to 16 and use a dummy state tensor with a value of 1. The hyperparameters are in Table 4. For LR-PG, RP-PG, and DA-PG, the critic function is calculated analytically: otherwise, their hyperparameters are the same as their counterparts with learned critic function. See Figure 9 for learning curves.
Results with alternative learning rates for a more controlled comparison in Section 5.5, we note that interpolated critic learning (ICL) also improves the performance of DA-AC under other learning rates (see Figure 10).
While we choose fixed learning rate for all algorithms
Kohreed Burnk (K-3)
Breecll Our tenosa Band
D4-40
aaac DNAC
04
04-40
500
1930
1900
2000
500
1930 1500
200
Figure 9: Learning curves of DA-AC, DA-AC w/o ICL. and baselines the K-Armed Bandit (col 1) and Bimodal Continuous Bandit (col 2) tasks. Results are averaged over 50 seeds. Shaded regions show 95% bootstrap Cls. DA-PG exhibits slightly faster convergence than RP-PG-albeit the advantage of using lower wwwer-variance estimator when no bias is present. When the critic learned, DA-AC w/o ICL performs significantly worse than LR-AC and RP- AC, achieving highly suboptimal returns even by the end of training. This reflects the difficulty of learning an effective critic using the standard update, as discussed in Section 4.2. In contrast, although DA-AC initially learns more slowly than LR-AC and RP-AC, it substantially outperforms DA-AC w/o ICL and eventually matches LR-AC and RP-AC in the later stages of training.
Blinddal Contranse Band!
K-feraal
Dimvadi Cont Bondt
1.0
1.0
1.0
6.
JR-40
Jld2
3040
22-10
UAAG D64G
Li-0
04
D440
580
1008
2006
1000
120
2006
1008
1590
2006
1006
Triadr
Triadss
Tieades
Timaidas
Figure 10: Learning curves of DA-AC, DA-AC w/o ICL, and baselines using learning rates 0.001
(cols 1-2) and 0.1 (cols 3-4). Results are averaged over 50 seeds. Shaded regions show 95% bootstrap Cls. An aggressive learning rate of 0.1 often leads to premature convergence to suboptimal points for most algorithms. Consistent with Figure 9, ICL demonstrates improved performance for DA-AC when a more conservative learning rate is employed.
D.4 CONTINUOUS CONTROI.
Environments From OpenAl Gym MuJoCo, we use the most commonly used environments (see Table 2). From DeepMind Control Suite, we use the same 15 environments as D'Oro et al.
23

---

Published as a conference paper at ICLR 2026
MARIOU DS SARVE
4eac
100 10T
Figure 11: Average normalized returns with and without actor target network (ATN) on Mu- JoCo (col 1) and DMC (col 2) tasks. Values are averaged over 10 seeds and 5 (MuJoCo) or 10 (DMC) tasks. Error bars show 95% bootstrap Cls.
Raildive Firl Performence DA-AC (Continaaaa) va DA-AC (Diazeee))
(oacraa beter
Narrabed
Anve
Figure 12: Relative final performance of DA-AC with continuous actions versus with diserete actions across 20 individual MuJoCo and DMC tasks. Results averaged over 10 seeds per task.
(2023), which are mentioned to be neither immediately solvable nor unsolvable by common deep RL algorithms. The full list of environments and their corresponding observation and action space dimensions are in Table 3. Returns for bar plots are normalized by dividing the episodic return by the maximum possible return for given task. In DMC environments, the maximum return is 1000 (Tunyasuvunakool et al., 2020). For MuJoCo environments, we establish maximum returns based on the highest values observed from proficient RL algorithms (Bhatt et al., 2024): 4000 for Hopper-v4, 7000 for Walker2d-v4 -4,, 8000 for Ant-v4, 16000 for HalfCheetah-v4, and 12000 for Humanoid-v4.
Experimental details Similar to TD3, DA-AC and RP-AC also adopt uniform exploration phase. During the uniform exploration phase, the distribution parameters formly sampled from [-1, 1|2. These three algorithms use the default hyperparameters of TD3 (see Table 5). For SAC (Haarnoja al., 2018) and PPO (Schulman et al., 2017), we use the implementa- tions and tuned hyperparameters in CleanRL. (Huang et al., 2022) (see Tables 5 and 8, respectively). Learning curves in each individual environment can be found in Figure 13.
[M4e uo] are uni-
Impact of the actor target network We also investigate the impact of using an actor target network (ATN) in DA-AC and the baselines. While TD3 already employs an ATN, both DA-AC and RP-AC do not. We additionally test DA-AC w/ ATN and RP-AC w/ ATN and TD3 w/o ATN. From Figure 11, we can see that the actor target network does not have a significant impact in general.
D.5 DISCRETE CONTROL
Environments We use the same 4 Gym classic control (Brockman et al., 2016) and 5 MinAtar (Young & Tian, 2019) environments as in Ceron & Castro (2021).
Experimental details for Gym environments
We use the existing implementations and tuned hyperparameters of DQN (Mnih et al., 2015) and PPO in CleanRL (see Tables and 11, respec- tively). For DA-AC, ST-AC, LR-AC, and EAC, we adjust relevant off-policy training hyperparame- ters based on those of DQN, including batch size, gradient steps per step. network size, replay buffer size. We also disable double Q-networks to better align with DQN. See Table 6 for the updated pa- rameters from Table 5. We use a similar setup for Discrete SAC (DSAC; Christodoulou, 2019), as shown in the same table. Learning curves can be found in Figure 14.
24

---

Published as a conference paper at ICLR 2026
Experimental details for MinAtar environments The MinAtar setups for DQN and PPO adopted from their implementations and tuned hyperparameters for Atari (Bellemare et al., 2013) in CleanRL (sce Tables 10 and 12, respectively). Similar to the above, adjust relevant off off-po training hyperparameters based on DQN for DA-AC, ST-AC, LR-AC, and EAC (see Table 7). We similar setup for DSAC but decrease its uniform exploration steps according its hyperpa- rameters for Atari in CleanRL (see Table 7). For consistency, we use the same critic network for DA-AC, ST-AC, LR-AC, EAC, and DSAC, which takes actions as input. See the next paragraph for the joint encoding of CNN observation features and actions for these methods in this experiment. Learning curves can be found in Figure 14.
Joint encoding of CNN observation features and actions While concatenation is used for joint encoding of observations and actions for state-based observations in MuJoCo/DMC/Gyn environ- ments, it might not be efficient for encoding latent features and actions (Schlegel et al., 2023). Inspired by Schlegel al., we use the flattened outer product of the CNN observation features (with a dimension of 128) and the vectorized action representations (with a dimension of [A]] as the joint encoding. The action representations are the action probabilities for DA-AC, while they are one-hot embedding of actions for other algorithms. We then use an additional hidden layer with small number of hidden units in MinAtar) with negligible overhead to extract higher-level features.
D.6 HIGH- BIGH-DIMENSIONAL DISCRETE CONTROL
Details We use the same 20 environments as Appendix D.4. Similar to the continuous control case, we also include a uninform exploration phase for all discrete control algorithms. For LR-AC and ST-AC, the action is randomly sampled from uniform categorical distribution. For DA-AC, the logits of the distribution parameters (in this case, the probability vector) are sampled from ((0, 1) where N is the number of possible discrete outcomes. All algorithms use the default hyperparame- ters of TD3 (see Table 5). See Figure 15 for learning curves in each individual environment.
Comparison to continuous control We plot the relative final performance of DA-AC with continuous actions versus with discrete actions in Figure 12. We can see that the performance of DA-AC with discrete actions often compete with DA-AC with continuous actions.
D.7 HYBRID CONTROL
Environments We use 7 parameterized-aacion MDP (PAMDP; Masson et al., 2016) environ- ments from Li et al. (2022). Please see their Appendix B.1 for the descriptions of the environments.
Contrary to other settings, which report training episodes' return, we report performance in evaluation phases following Li al. (2022). During evaluation phases, DA- AC uses discrete actions with the highest probability for the discrete components and mean actions for the continuous components. We the implementations provided by Li et al. for baselines, including PADDPG (Hausknecht & Stone, 2016), PDQN (Xiong et al., 2018), and HHQN (Fu et al., 2019). All the baselines incorporate clipped double Q-learning from TD3, with PADDPG renamed to PATD3. The hyperparameters of DA-AC are adjusted according to those of the baselines (see Table 13). Since PDQN
Experimental details
per-environment tuned 7 in Li al., our results are slightly different than theirs as we use a fixed ? 0.99 for PDQN to be consistent with other algorithms. Learning curves can be found in Figure 16.
D.8 PAIRED t-TESTS FOR THE EFFECTIVENESS OF ICL
To assess the effectiveness of ICI, we conduct two-sided paired t-tests comparing DA-AC with and without ICL across benchmark settings. For each setting, tests are performed over paired seeds pooled across environments, where pairs share the same random seed. We additionally conduct paired t-test on the mean performance differences aggregated across the seven settings. Table reports the mean differences, t-statistics, p-values, and sample sizes. Overall, the results show that incorporating ICL yields statistically significant improvements in several settings and a positive mean improvement across settings.
25

---

Published as a conference paper at ICLR 2026
Table Paired t-tests comparing DA-AC and DA-AC w/o ICL across benchmark settings. For each setting, the test is conducted over paired seeds pooled across environments. The final row reports paired t-test on the mean performance differences aggregated across the seven settings.
Setting
Mean diff
p
MuJoCo
0.0760
3.306
0.0018
50
DMC
0.0157
1.115
0.2666
150
Discretized MuJoCo
0.1157
5.526
1.25 x 10
50
Discretized DMC
0.0383
2.149
0.0333
150
Gym
0,0510
1.570
0.1274
30
MinAtar
0.0158
0.409
0.6844
50
PAMDPs
0.0501
2.844
0.0059
70
Across Settings
0.0508
3.820
0.0088
7
D.9 COMPUTATIONAL RESOURCE REQUIREMENT
All training for bandits conducted local machine with AMD Ryzen 9 5900X 12-Core Processor. Each training run was executed using a single CPU core and consumed less than 256MB of RAM. Most runs completed 2000 training steps within 10 seconds.
All training for other tasks was conducted on CPU servers. These servers were equipped with diverse range of Intel Xeon processors. including Intel E5-2683 Broadwell 2.1GHz, Intel Platinum 8160F Skylake @ 2. 2.1GHz, and Intel Platinum 8260 Cascade Lake @ 2. 2.4GHz. Each train- ing run was executed using single CPU core and consumed less than 2GB of RAM. The training duration varied considerably across environments, primarily influenced by the dimensionality of the observation space, the complexity of the physics simulation, and, in the case of discrete ac- tion spaces, the dimensionality of the action space. For MuJoCo simulation tasks, most algorithms typically completed million training steps in approximately 7 hours per run. However, LR-AC required longer training period of roughly 9 hours due to the additional computational overhead of learning an extra neural network for computing the baseline.
Table 2: Observation and action dimensions of OpenAl Gym MuJoCo environments.
Environment
Observation dimension
Action dimension
Hopper-v3
11
Walker2d-v3
17
HalfCheetah-v3
17
Ant-v3
Humanoid-v3
376
17
Table 3: Observation and action dimensions of DeepMind Control Suite environments.
Domain Task(s)
Observation dimension
Action dimension
pendulum acrobot
swingup swingup
3
1
reacher
hard
2
finger turn hard
12
2
hopper stand, hop
15
4
fish swim swimmer6
24
5
swimmer
25
5
cheetah
run walker run walk, run stand, walk, run
17
6
24
quadruped
58
12
humanoid
67
24
26

---

Published as a conference paper at ICLR 2026
E HYPERPARAMETERS
Table 4: Hyperparameters for both continuous (col 3) and discrete (col 2) bandits that are different from Table 5. DA-AC is applied to both settings, denoted as DA-AC (C) and DA-AC (D), respec- tively. RP-AC uses the same hyperparameters as DA-AC (C); LR-AC uses the same hyperparameters DA-AC (D).
Hyperparameter
DA-AC (D)
DA-AC
Batch size
Learning rate (actor / critic)
0.01
Neurons per hidden layer
(16, 16)
Discount factor (7)
N/A
Replay buffer size
2000
Policy update delay (Na)
Uniform exploration steps
N/A
Learnable o range ([omin+ max))
N/A
3 ee
Table 5: Hyperparameters of actor-critic algorithms for both MuJoCo/DMC continuous (cols 3-5) and discrete (col 2) control environments. DA-AC is applied to both settings, denoted as DA-AC (C) and DA-AC (D), respectively, For simplicity, assume [Amin, [max] = [- for continuous control algorithms. RP-AC uses the same hyperparameters as DA-AC (C); LR-AC and ST-AC use the same hyperparameters DA-AC (D).
Hyperparameter
DA-AC (D)
DA-AC (C)
TD3
SAC
Batch size
256
Optimizer
Adam
Learning rate (actor / critic)
0.0003
0.0003 / 0.001
Target network update rate (r)
0.005
Gradient steps per step
Number of hidden layers
2
Neurons per hidden layer
(256, 256)
Activation function
ReLU
Discount factor (7)
0.99
Replay buffer size
1 x 106
Policy update delay (Na)
Uniform exploration steps
25, 000
5, 5,000
Learnable a range (fmmin+ max))
N/A
[0.03,
N/A
N/A
Target entropy
N/A
[AA]
Target policy noise clip (c)
N/A
0.5
N/A
Target policy noise (ōTDz)
N/A
0.2
N/A
Exploration policy noise (Fips)
N/A
0.1
N/A
27

---

Published as a conference paper at ICLR 2026
Table 6: Hyperparameters of actor-critic algorithms for Gym environments that are different from Table 5. EAC, LR-AC, and ST-AC use the same hyperparameters DA-AC.
Hyperparameter
DA-AC
DSAC
Batch size
128
Learning rate (actor / critic)
0.0003
Target network update rate (7)
0.01
Gradient steps per env step
(every env steps)
Neurons per hidden layer
(120, 84)
Replay buffer size
1 x 104
Policy update delay (Na)
Uniform exploration steps
12, 500
2,500
Target entropy
N/A
0.89].4]
Table 7: Hyperparameters of actor-critic algorithms for MinAtar environments that different from Table 5. EAC, LR-AC, and ST-AC use the same hyperparameters as DA-AC.
Hyperparameter
DA-AC DSAC
Batch size
32
Learning rate (actor critic)
0.0003
Gradient steps per step
(every 4 env steps)
Number of Conv. layers
Conv. channels
16
Conv. filter size
3
Conv. stride
1
Number of MLP layers
2
Neurons per MLP layer
(128, 8)
Replay buffer size
10a
Policy update delay (Na)
1
Uniform exploration steps
50, 000
4, 000
Target entropy
N/A
0.89|A|
28

---

Published as a conference paper at ICLR 2026
Table 8: Hyperparameters of PPO in MuJoCo/DMC continuous- and discrete-control environments.
Hyperparameter
PPO
Optimizer
Adam
Learning rate (actor critic)
x 10
Discount factor (?)
0.99
GAE parameter (A)
0.95
Rollout length (timesteps per update)
2048
Minibatch size
32
Number epochs per update
10
Number of hidden layers
2
Neurons per hidden layer
(64, 64)
Activation function
Tanh
Clipping parameter (c)
0.2
Entropy coefficient
0.0
Value loss coefficient
0.5
Max grad norm
0.5
Reward normalization
Enabled
Observation normalization
Enabled
Learning rate schedule
Linear decay
Table 9: Hyperparameters of PPO in Gym environments that are different from Table
Hyperparameter
PPO
Rollout length (timesteps per update)
128
Number of epochs per update
4
Table 10: Hyperparameters of PPO in MinAtar environments that are different from Table 8. Since
MinAtar already normalizes the observations and rewards, we disable the normalization wrappers.
Hyperparameter
PPO
Learning rate (actor critic)
2.5 10-4
Rollout length (timesteps per update)
128
Number of epochs per update
4
Number Conv. layers
1
Conv. channels
16
Conv. filter size
3
Conv. stride
1
Number MLP layers
Neurons per MLP layer
(128.)
Activation function
ReLU
Entropy coefficient
0.01
Reward normalization
Disable
Observation normalization
Disable
29

---

Published as a conference paper at ICLR 2026
Table 11: Hyperparameters of DQN in Gym environments.
Hyperparameter
DQN
Batch size
128
Optimizer
Adam
Learning rate
2.5 X 10
Discount factor (?)
0.99
Hard target network update
Every 500 env steps
Gradient steps per env step
(every 10 steps)
Number of hidden layers
2
Neurons per hidden layer
(120, 84)
Activation function
ReLU
Replay buffer size
1 104
Min replay size before learning
10,000
Linear e-greedy range
1.0 + 0.05
Linear e-greedy steps
2.5 10
Table 12: Hyperparameters of DQN in MinAtar environments.
Hyperparameter
DQN
Batch size
32
Learning rate
1 x 10
Hard target network update
Every 1000 env steps
Gradient steps per env step
(every 4 env steps)
Number of Conv. layers
1
Conv. channels
16
Conv. filter size
3
Conv. stride
Number of MI.P layers
1
Neurons per MIP layer
(128.)
Replay buffer size
10
Min replay size before learning
40,000
Linear e-greedy range
1.0 + 0.01
Linear e-greedy steps
5 x 10
30

---

Published as a conference paper at ICLR 2026
Table 13: Hyperparameters of actor-critic algorithms for PAMIP environments.
Hyperparameter
DA-AC
PATD3
HHQN
PDQN
Batch size
128
Optimizer
Adam
Learning rate (actor / critic)
0.0003
0.0001 10001/0.000
Target network update rate (r)
0.005
0.001 0.01
Gradient steps per step
Number of hidden layers
2
Neurons per hidden layer
(256, 256)
Activation function
ReLU
Discount factor (7)
0.99
Replay buffer size
1 x 10a
Policy update delay (N₂)
Uniform exploration steps
5, 000
N/A
Learnable range (famin: mmax))
[0.05, 0.2)
N/A
Target policy noise clip (c)
N/A
0.5
N/A
Target policy noise (ōyps)
N/A
0.2
N/A
Exploration policy noise (OrDs)
N/A
0.1
N/A
Ornstein Ubleen--hheebeckk noise
N/A
Enable
Linear e-greedy range
N/A
1.0 + 0.01
Linear e-greedy steps
N/A
1 10a
Max grad norm
N/A
0.5
31

---

Published as a conference paper at ICLR 2026
F
ADDITIONAL PLOTS
100%
300
700
2530
1003
003
Theasho
Tuare
100
Trrsstap
Trrsstip
400
800
120
100
0.2
Trisrap
Trussip
1900
walaavan
100
100
400
03
02
0.4
30
0.2
0.4
0.0
0.2
0.6
30
0.0
3.2
0.4
08
1.5
1.0
DA-AC
DA-AC wio ICL
RP-AC
TD3
SAC
PPO
Figure 13: Learning curves of DA-AC, DA-AC w/o ICL, and baselines in 5 MuJoCo and 15 DMC continuous control tasks. Results are averaged over 10 seeds. Shaded regions show 95% bootstrap Cls.
-153
110
12000 20000 40000 MICES
200030 2000
SCIECR
IRWERD
Irmoky
UnitaWWeeeet
Mini Grspade!
DA-AC
DA-AC wil ICL
ST-AC
LRAC
EAC
DSAC
DGN
PPO
Figure 14: Learning curves of DA-AC, DA-AC w/o ICL, and baselines in Gym and 5 MinAtar discrete control tasks. Results are averaged over 10 seeds. Shaded regions show 95% bootstrap Cls.
32

---

Published as a conference paper at ICLR 2026
200
2.5
Humanaddio
Oogar furt
1990
1000
600
Tarssdip
1e0
Timading
Trexdin
140
Trrsskp
160
200
Hoars
Tirssap
165
Trissip
160
600
800
400
Temosopp
Trmosky
1922
DA-AD
ST-AC
LRVAC
PPO
Figure 15: Learning curves of DA-AC, DA-AC w/o ICL, and baselines in MuJoCo and 15 DMC discrete control tasks. Results are averaged over 10 seeds. Shaded regions show 95% bootstrap Cls.
55900 2006
250000
1C0KS
29000
20006
Hort Move (mi))
Hand Move (eni)
Hord Mivo (ynk))
2000 18800 150000 20300
5000
150800 10000
200000
50300
130000
15000
2000
DA-AC
DA w/o ICL
PATD3
PDON
HHON
Figure 16: Learning curves of DA-AC, DA-AC w/o ICL, and baselines in 7 hybrid control tasks. Results are averaged over 10 seeds. Shaded regions show 95% bootstrap Cls.
33

---

Published as a conference paper at ICLR 2026
G PSEUDOCODE
G.1 DA-AC: DISTRIBUTIONSSAS-ACCIONS ACTOR-CRITIC
Algorithm 2 DA-AC for diverse action spaces
Input action sampling function f ((A) (see Appendix D.1 for f in different settings) Initialize parameters W1, W2, 0, W < W1, W2 t W2, replay buffer B Obtain initial state So for to 0 to T do
Take action A; ~ f(|U,) with U, = (S,), observe Re+1, S Add (S,, U,, A., S Re+1) to the buffer B
Sample a mini-batch B from buffer
Sample ú = wU + (1 w)U w ~ Uniform(0, 1), for each transition (S,U,A, R)
Update critics on B:
W; w; di R PR+yying minjee(1.)) Qw, (8', (o - (w.t)) Vw, Qw.
if = 0 (mod Na) then Update policy B:
- - o. VV (S,
Update target network weights:
W TW; + (1 7)
end if end for
G.2
TD3: TWIN DELAYED DEEP DETERMINISTIC POLICY GRADIENT
Algorithm 3 TD3 for continuous action spaces
Input exploration noise 0DD3, target policy noise TDD3, target noise clipping Initialize parameters W1, W2, O, W1 < W1, W2 + W2, 4- 0, replay buffer Obtain initial state So f 0 to T do
Take action A, (S,) 6, NO, TDS)), and observe R441, S441
Add (St, Ar, S Re+1) to the buffer B
Sample a mini-batch B from buffer B
Sample A' = (S") + e, clip(N (0, -c, c), for each transition AS R) in B
Update crities on B:
w; w; + ar (R+ y min Qw, (S' A') Qw. (S, A)) Vw Qw, (S, A)
if = 0 (mod N2) then Update policy on B:
- 0 0- (OTO V (S,
Update target network weights:
W; + TW; + (1- 7,
0 r0 + (1 7)0
end if end for
34

---

Published as a conference paper at ICLR 2026
G.3 RP-AC: ACTOR-CRITIC WITH THE REPARAMETERIZATION (RP) ESTIMATOR
Algorithm 4 RP-AC for continuous action spaces
Input reparameterization function go S x R + A (for Gaussian policies, see Appendix D.1)
Initialize parameters W1, W2, 0, W1 < W1, W2 t W2, replay buffer B Obtain initial state So for = 0 to T - do
Take action A, (t) (0, 1), and observe R4+1, Se+1
Add (S,, A, 1, 1 to the buffer
Sample mini-batch B from buffer B
Sample A' = ga(c; 5"), c ~ (0, 1), for each transition AS,A,S, B
Update critics on B:
w. + (R+ min Qw, ( Qw. (S, Vw. Qw, (S,A)
i 0 (mod Na) then
Sample ~ N0, 1) for each transition (S,A,S', in B Update policy on B:
レ 04 Vego(c
T ViQw, (S, A) i (e:8)
Update target network weights:
WF - TW; + (1 T)
end if end for
G.4 ST-AC: ACTOR-CRITIC WITH THE STRAIGHT-THROUGH (ST) ESTIMATOR
Algorithm ST-AC for discrete action spaces
Initialize parameters W1, W2, Wi < Wi. We f- W2, replay buffer B
Obtain initial state So for to T do
Take action A, (-1:)),, and observe R4+1, Se+1
Add (St, Ar, S Rx+1) to the buffer
Sample mini-batch B from buffer B
Sample A' ~ (-SS) for each transition (S, S" R) B
Update critics on B:
<- w, + (R+yy mingee1.2) Qw, (5', A
V Qw. (S. A)
i = 0 (mod Na) then
Sample Á (-(S), for each transition AS, R3 in B Use the Etraight-througe trick to compute Åg oot.hot(f + TT(.IS) TT(-1)) Update policy on B:
0- 0 VoTo(1S))
V (S, AS,
Update target network weights:
Wi TW; + (1 r)W;
end if end for
35

---

Published as a conference paper at ICLR 2026
G.5 LR-AC: ACTOR-CRITIC WITH THE LIKELIHOOD- RATIO (LR) ESTIMATOR
Algorithm 6 LR-AC for diverse action spaces
Initialize parameters W1, W2, 0, v, W1 WI, W2 Wa. replay buffer B Obtain initial state So
fort= 0 to T - 1 do
Take action A, ~ Tol (-)5)), and observe R4+1, Se+1
Add (St, A,, 1 Rx+1) to the buffer B
Sample a mini-batch B from buffer
Sample ~ (-(S) and A' ~ (-(S'') for each transition A,,, Rt in B
Update critics on B:
wi + (R min Qw, (5',A) Qw. AS) Vw. Qw, (S,A)
v an (Qmm (S, A) Vu(()) (.)
if (mod Na) then
Sample À ~ To(-) for each transition AS, RR) in B Update policy B:
Vo log (Qm, (5, - (v(())
0 θ 0
Update target network weights:
Wi TW; + T)W;
end if end for
G.6 EAC: ACTOR-CRITIC WITH THE EXPECTED POLICY GRADIENT ESTIMATOR
Algorithm 7 EAC for discrete action spaces
Initialize parameters W1, W2, W1 + W1, Wy +- W2. replay buffer B Obtain initial state So fort = 0 to T 1 do
Take action A, ~ (-)-)..), and observe Re+ Se+1
Add (S., A,, Sr+1. R2+1) to the buffer B
Sample a mini-batch from buffer
Sample ~ T-(-S)) and A' ~ me(-(S") for each transition AS, R) B
Update critics on B:
w. + w. + (R+ y minje(112) Qw, (S", A') Qw. (S, Vw. (S,A)
if 0 (mod N₂) then Update policy on B:
0-
ai Vo Σ
To(alS)Qw, (S,
aEA
Update target network weights:
W TW; + (1 r)W;
end if end for
36

---

Published as a conference paper at ICLR 2026
H USE OF LARGE LANGUAGE MODELS
Large language models (LLMs) were employed in a strictly auxiliary capacity during the prepa- ration this paper. Their use was limited to areas: (1) assisting with writing refinement by improving readability, grammar, and conciseness, without contributing to the technical content or conceptual development; and (2) supporting workflow tasks such as drafting or adjusting scripts for data processing and figure generation, with all outputs carefully reviewed and corrected by the authors. LLMs were not used for generating research ideas, conducting literature searches, or pro- ducing original technical material. Their involvement was confined to polishing communication and light implementation support.
37


Reviews:
[{"review_id": "bQqFhr2mAV", "reviewer": "Reviewer_5jaa", "summary": "This work focuses on the policy distributions and proposes a distributions-as-actions framework, an alternative to the classical RL formulation that treats the parameters of parameterized distributions as actions. By shifting this agent-environment boundary, they develop a continuous-action algorithm for a diverse class of action spaces and achieve lower variance. Across experimental results, this paper achieves competitive performance.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The motivation is novel and interesting, in which parameterized action distributions are an important topic in the RL community.\nThe reparameterization policy gradient estimator can reduce variance using interpolated critic learning.\nAnd under selected baselines, this work achieves competitive performance.", "weaknesses": "1. The baselines are weak, only SAC, TD3. More baselines should be considered in various settings across discrete, continuous, and hybrid control. The newest baseline was published in 2019.\n\n2. Compared with continuous actions, will discrete action spaces affect performance?\n\n3. The related works on parameterized action distributions are insufficient.\n\nDiscretizing continuous action space for on-policy optimization, AAAI, 2020 \n\nDiscretizing Continuous Action Space With Unimodal Probability Distributions for On-Policy Reinforcement Learning, IEEE TNNLS, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlJpXNZHHv", "reviewer": "Reviewer_8RUo", "summary": "Distributions as actions is a new way for representing the policy.\nInstead of predicting an action, the policy predicts the action distribution.\n\nThis work then presents how to integrate this into the env (the env has a sampler function) and into the critic (the critic estimates V of the distribution) and how to train the critic (since next-states arrive from sampling from the distribution and then simulating).", "questions": "When comparing to TD3, how does the policy performance change if sigma is fixed and it only learns the mean? (mimicing how TD3 learns).\nWhat happens if TD3 is trained like in FB-CPR (Tirinzoni et al). There the agent predicts the mean and the action is sampled through a gaussian distribution. The Q policy gradients can still flow through this sampling (reparam trick). In this comparison, TD3 could predict mean and std.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "DA-AC seems like a very promising framework.\nThe work itself is interesting and the concept is novel.\nResults show it outperforms pre-existing methods on simple tasks.", "weaknesses": "While the authors show the interpolated critic is an important feature and very nicely present how this helps the critic learn a better optimization landscape, it isn't clear what part of the agent is benefitting from this method.\n\nWhen comparing to TD3, how does the policy performance change if sigma is fixed and it only learns the mean? (mimicing how TD3 learns).\nWhat happens if TD3 is trained like in FB-CPR (Tirinzoni et al). There the agent predicts the mean and the action is sampled through a gaussian distribution. The Q policy gradients can still flow through this sampling (reparam trick). In this comparison, TD3 could predict mean and std.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aVriDlJqKM", "reviewer": "Reviewer_WRJw", "summary": "The paper proposes a unified framework for handling both discrete and continuous action spaces, which traditionally require distinct design choices in reinforcement learning. Instead of directly learning over the original action space defined by the MDP, the method learns policies and value functions over the distribution parameters from which actions are sampled. Because distribution parameters are typically continuous, both discrete and continuous action spaces can be handled in a unified manner. Central to the paper is the introduction of a gradient estimator defined on these parameters that exhibits lower variance but higher bias. This bias issue is effectively mitigated through an interpolated critic learning mechanism. Taken together, these advantages appear to be the primary factor driving the strong empirical performance of the proposed approach across a range of tasks.\n\nThe work evaluates the method on tasks with diverse action space types and demonstrates consistent performance advantages.", "questions": "As noted in the weaknesses section, I would encourage the authors to provide a more explicit discussion of how this work relates to prior literature.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and comprehensive. While the idea may appear straightforward at first glance, the authors provide thorough motivation, analysis, implementation details, and experimental results, which I find convincing.", "weaknesses": "Based on my reading, the paper does not include a dedicated related work section where the connection between this approach and prior research is explicitly discussed. While the authors do reference commonly used policy gradients in Section 2 and baseline algorithms in the experiments, I would be interested in a deeper discussion of previous efforts that also attempt to learn policy distribution parameters—perhaps separately in continuous and discrete action spaces. Such a contextualization would help better highlight and clarify the contributions of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5vkzUXlHYZ", "reviewer": "Reviewer_Xv1R", "summary": "This paper introduces a framework for reinforcement learning called \"distributions-as-actions\". The authors try to redefine the boundary between the agent and the environment. Instead of the agent outputting an action from a potentially discrete or hybrid space, the agent outputs the continuous parameters of a probability distribution over actions (e.g., the probability vector for a categorical policy, or the mean and variance for a Gaussian policy). The environment is then responsible for sampling the final action from this distribution. This reparameterization transforms any RL problem into one with a continuous action space, allowing for the development of unified algorithms.\n\nThe authors derive a corresponding policy gradient estimator. To address the challenge of learning a critic over this new, more complex action space, they propose a simple and effective technique called Interpolated Critic Learning (ICL). Finally, they combine these ideas into a practical actor-critic algorithm, DA-AC, based on td3.", "questions": "- Could the authors elaborate on how entropy regularization could be effectively adapted to the distributions-as-actions framework? \n- How does one encourage diverse actions by acting in the continuous parameter space of that distribution?\n- What are the authors' thoughts on the compatibility of this framework with planning algorithms like MCTS, which rely on a discrete branching factor? \n- Does this reframing inherently limit the applicability to model-free actor-critic methods?\n- Regarding the PPO comparisons, can the authors provide more details on the implementation and tuning? For example, were the reported results for PPO based on a single-environment setup, or was it scaled with parallel environments as is common for achieving top performance?\n- The ICL technique uses linear interpolation between the policy's output parameters $U_t$ and the deterministic parameters $U_{A_t}$. Could you provide more intuition on this choice? For categorical policies where the parameter space is a simplex, does this linear interpolation respect the geometry of the space, or does it operate on the unconstrained logits", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea of treating distribution parameters as actions is a simple and interesting concept. It unifies diverse action spaces (discrete, continuous, hybrid) into a single, continuous control problem, potentially simplifying algorithm design\n2. The introduction of Interpolated Critic Learning (ICL) is a practical solution to the challenge of learning the critic in this new, higher-dimensional parameter space\n3. The authors test their method on over 30 tasks across continuous, discrete, high-dimensional discrete, and hybrid domains, consistently showing competitive performance", "weaknesses": "1. The paper presents the transformation from discrete to continuous action spaces as a clear benefit, but does not sufficiently discuss the potential downsides. Discrete action spaces have a specific structure that is often leveraged by advanced algorithms. For instance:\n- Entropy regularization is commonly used in RL for promoting exploration in discrete spaces. It is unclear how an equivalent and effective exploration strategy would be implemented in the continuous parameter space. Simply encouraging entropy over the parameters (a vector of probabilities) is not the same as encouraging entropy in the resulting categorical distribution.\n- Methods like mirror descent are well-suited for optimization over the probability simplex. By moving to an unconstrained Euclidean space of parameters, these geometric advantages are lost.\n- Planning algorithms like MCTS are fundamentally built on discrete action branching. The proposed framework makes it difficult to see how such methods could be integrated. A deeper discussion of what is \"lost\" in this transformation and how the framework might address these challenges would significantly strengthen the paper.\n2. The paper should better position itself with respect to other works that also re-frame RL problems through representation learning. For example, \"Representation-Driven RL\" (Nabati et al.)  also learns a latent space for policies and uses bandit algorithms to guide exploration in that space. While the mechanism is different (RepRL learns a representation of the entire policy, whereas this paper redefines the action at each step), the high-level goal of shifting the problem into a more structured latent space is shared. Discussing these connections would provide a richer context for the paper's contribution.\n3. The comparison to PPO, a very strong on-policy baseline should be more convincing. PPO's performance is known to be highly dependent on implementation details and computational budget (e.g., total environment steps, number of parallel environments). Without a more detailed analysis or comparison against a highly-tuned PPO baseline known to achieve SOTA on these tasks, it is difficult to be certain that DA-AC's superior performance isn't an artifact of a weakly-scaled baseline. A stronger claim could be made by showing that DA-AC is more sample-efficient or reaches a higher asymptotic performance in a \"computationally unconstrained\" setting where both algorithms are given unconstrained resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)