Learning in complex action spaces without policy gradients
Authors: Arash Tavakoli
Sina Ghiassian
Nemanja Rakicevic
Venue: Accepted by TMLR ()

Abstract: While conventional wisdom holds that policy gradient methods are better suited to complex action spaces than action-value methods, foundational work has shown that the two paradigms are equivalent in small, finite action spaces (O'Donoghue et al., 2017; Schulman et al., 2017a). This raises the question of why their computational applicability and performance diverge as the complexity of the action space increases. We hypothesize that the apparent superiority of policy gradients in such settings stems not from intrinsic qualities of the paradigm but from universal principles that can also be applied to action-value methods, enabling similar functions. We identify three such principles and provide a framework for incorporating them into action-value methods. To support our hypothesis, we instantiate this framework in what we term QMLE, for Q-learning with maximum likelihood estimation. Our results show that QMLE can be applied to complex action spaces at a computational cost comparable to that of policy gradient methods, all without using policy gradients. Furthermore, QMLE exhibits strong performance on the DeepMind Control Suite, even when compared to state-of-the-art methods such as DMPO and D4PG.

Published in Transactions on Machine Learning Research (06/2025)
Learning in complex action spaces without policy gradients
Arash Tavakoli Riot Games
atavakoli@@roolggmee.com
Sina Ghiassian Spotify
sinag
Nemanja Rakiéević
mukicenic@google..c com
Google Mind
Reviewed on OpenReview: https: openreview. net/ forun? id= 40
Abstract
While conventional wisdom holds that policy gradient methods are better suited to complex action spaces than action-value methods, foundational work has shown that the two paradigms equivalent in small, finite action spaces (O (O'Donghuue al., 2017; Schulman et al., 2017a). This raises the question of why their computational applicability and performance diverge as the complexity of the action space increases. We hypothesize that the apparent superiority of policy gradients in such settings stemss not from intrinsic qualities of the paradigm but from universal principles that can also be applied to action-value methods, enabling similar functions. We identify three such principles and provide a framework for incorporating them into action-value mothods. To support our hypothesis, we instantiate this framework what we term QMLE, for Q-learning with maximum likelihood estimation. Our results show that QMLE can be applied to complex action spaces at computational cost comparable to that of policy gradient methods, all without using policy gradients. Furthermore, QMLE exhibits strong performance on the DeepMind Control Suite, even when compared to state-of-the-art hods such as DMPO and D4PG. We make our code publicly available.
1 Introduction
In reinforcement learning, policy gradients (and their actor-critic variants) (Sutton et al., 1999) have become the backbone of solutions for environments with complex action spaces (Dulac-Arnokd al., 2015; OpenAI et al., 2019; Vinyals et al., 2019; Ouyang et al., 2022). We use the term "complex action spaces" broadly to refer to those for which enunerating actions or brute-force operations are computationally intractable, such as domains with multi-dimensional continuous high-dimensional discrete actions (Hubert al., 2021). In contrast, action-value methods- methods--primaril.. primarily, variants of Sarsa and Q-learning have traditionally been confined to tabular-action models for small and finite action spaces. However, where applicable, such as on the Atari Suite (Bellemare et al., 2013; Machado et al., 2018), action-value methods are frequently the preferred approach over policy gradient methods (Kapturowski et al., 2023; Schwarzer et al., 2023).
In recent years, foundational research has shown that the distinction between action-value and policy gradient methods is narrower than previously understood, particularly in the basic case of tabular-action models in small and finite action spaces (see, e.g., Schulman et al., 2017a). Notably, O'Donoghue et al. (2017) established direct equivalence between these paradigms, connecting the fixed-points of action-pretd pettin--referencess of policies optimized by regularized policy gradients to action-values learned by action-value methods. These insights invite further exploration of discrepancies emerging as action-space complexity increases.
What core principles underpin the superior computational applicability and performance of policy gradient methods in such settings? In this paper, we identify three such principles. First, policy gradient methods leverage Monte Carlo (MC) approximations for summation integration over action spaces, enabling

---

Published in Transactions on Machine Learning Research (06/2025)
computational feasibility even in environments with complex action spaces. Second, they employ amortized maximization through a special form of maximum likelihood estimation (namely, the policy gradient itself), iteratively refining the policy to favor high-value actions without requiring brute-force arg max over the action space. Third, scalable policy gradient methods employ action-in architectures for action-value approximation, implicitly facilitating representation learning and generalization across the joint state-action space.
Are these principles exclusive policy gradient methods? We argue that these principles can be adapted to action-value methods, thereby bridging the scalability and performance gap between the two paradigms. Instead of using MC methods for summation or integration as in policy gradient methods, we propose using MC methods to approximate the arg max operation, making action-value methods computationally scalable for complex action spaces. Moreover, explicit maximum likelihood estimation can enable caching and iterative refinement of parametric predictors for amortized arg max approximation. Lastly, action-in architectures can be employed not only to scalably evaluate limited set of actions in any given state, but also to enable representation learning and generalization across both states and actions.
To test our arguments, we introduce Q-learning with mazimum likelihood estimation (QMLE). Given that QMLE builds upon Q-learning, DDPG (Lillicrap et al, 2016) serves as its closest counterpart among commonly used policy gradient methods for contimouus-action tasks, and thus is our primary bascline for comparison.
Through systematic ablation studies ($5.3), we show that each identified principle individually contributes to improved scalability or performance of action-value learning. Benchmarking QMLE against DDPG on diverse milti-dimensional continuous control tasks, ranging from 1 to 21 action dimensions, we demonstrate that incorporating all three principles collectively enables QML.E achieve competitive performance ($5.2). These results provide evidence that the identified principles are indeed fundamental to the broad applicability of policy gradient methods in complex action spaces, and that they are not intrinsically lied to the policy gradient paradigm and can be effectively adapted into action-value learning.
Beyond multi-diimensional contimuous-actinn scenarios, we demonstrate the applicability of QMLE to large, high-dimenssonal discrete-action ones by employing a 3-bin discretization scheme on our continuous control benchmarks. Our results illustrate QMLE's robustness in managing increasing complexity due to the curse of dimensionality, effectively scaling from 2 to 38 action dimensions (corresponding to discrete action sets ranging from only 32 9 up to 338, approximately 1.35 quintillion, actions).
The idea of using sampling- based approximation of the arg max in value-based methods has been explored in earlier works. For example, Tian et al. (2022) studied the combination of value iteration and random search in discrete domains, with a tabular mechanism for tracking the best historical value-maximizing action in each state. Kalashnikov et al. (2018) introduced the QT-Öpt algorithm, which employs a fixed stochastic search via the cross-entropy method to approximate arg max in Q-learning. Closely related to QMLE is the AQL algorithm by de Wiele et al. (2020), which integrates Q-lcarning with entropy-regularizzed MLE to approximate value-maximizing action distribution. While QMLE shows superior performance relative to QT-Opt and AQL in complex action spaces ($C), our emphasis in this work is less on algorithmic novelty and more on dissecting the core principles that bridge the gap between the two paradigms.
2 Background
2.1 The reinforcement learning problem
The reinforcement learning (RL) problem (Sutton & Barto, 2018) is generally described Markov decision process (MDP) (Puterman, 1994), defined by the tuple A, P. R), where S is state space, A is an action space, P: x A + A(S) is a stato-transition function, and R S x A x S + ARR) is a reward function. The behavior of an agent in an RL problem can be formalized by a policy л : S + A(A), which maps a state to a distribution over actions. The value of state s under policy π may be defined as the expected
1A denotes distribution.
2

---

Published in Transactions on Machine Learning Research (06/2025)
discounted sum of rewards: (T) = Px 2 s), where y E (0, 1) is discount factor used to exponentially decay the present value of future rewards."
The goal of an RL agent is defined as finding an optimal policy x¹ that maximizes this quantity across the state space: V= 2 V for all т. While there may be more than one optimal policy, they all share the same state-value function: V* = === Similarly, we can define the value of state s and action a under policy w: (" a) = Ex, P.R. 1 2'r141|s0 |so s, ag = a]. Notice that the goal can be equivalently phrased as finding optimal policy 't' that maximizes this alternative quantity across the joint state-action space: Q" 2 for all Same before, optimal policies share the same action-value function: Q" Q²=00 =²=
The state and action value functions are related to each other via: (*(s)= = Eu (T a a) (als), where we use Σ to signify both oummattion and inlegration over discrete or contimuous actions. For all MDPs there is always at least one deterministic optimal policy, which can be deduced by maximizing the optimal action-valus function: arg max, (s a) in any given state s. It is worth noting that there may be cases where multiple actions yield the same maximum value, resulting in ties. By breaking such ties at random, considering all conceivable distributions, we can construct the set of all optimal policies, including both deterministic and stochastic policies. Regardless of the optimal policy, the optimal state-value and action-value functions are related to each other in the following way: V*(s)= = max, Q*(s. Similarly, the optimal state-value function can be used to extract optimal policies by invoking the Bellman recurrence: arg max, R/rt + >*V*(*++1 = s]- However, this requires access to the MDP model, rendering the sole optimization of state-values unsuitable for model-free RL.
2.2 Action-value learning
Optimizing the action-value function and deducing an optimal policy from it seems to be the most direct approach solving the RL problem in mreel-free manner. To this end, we first consider the Bellman recurrence for action-values (Bellman, 1957):
(=(,,,)= = E Ii+1 + 7Q (+)))) = 8,0 = x,P,R
(1)
where æ is in general a stochastic policy and ac+1 ~ T(114411. By substituting policy with an optimal policy x" and invoking Q" (s,s, arg max, a)) = max. (* a), we can rewrite Eq. 1:
Q*
JE [re+1 + max (5 a |s s, s, - a].
(2)
P,R
a'
The method of temporal differences (TD) (Sutton, 1988) leverages equations (1) and (2) to contrive two foundational algorithms for modcl-free RL: Sarsa (Rummery & Niranjan, 1994) and Q-lcarning (Watkins, 1989). Sarsa updates its action-value estimates, Q(),,at) by minimizing the TD residual:
(r41 + (Q(#+11 - Q(s(,((s))
(3)
whercas Q-learning does so by minimizing the TD residual:
(ree1 7 max Q(s411, Q(s), 0+).
(4)
Both algorithms have been shown to converge to the unique fixed-point Q" of Eq. 2 under similar conditions, with one additional and crucial condition for Sarsa (Watkins & Dayan, 1992; Jaakkola et al., 1994; Singh et al., 2000). Namely, because Sarsa uses the action-value of the action chosen by its policy in the successor state, the action-values can converge to optimality in the limit only if it chooses actions greedily in the limit: limg-roo (als) aamarg max,, Q(sa")) This is in contrast with Q-lcarning which uses its maximum aclion-value in the successor state regardless of its policy, thus liberating its learning updates from how it chooses to act. This key distinction makes Sarsa an on-policy and Q-learning an off-policy algorithm. As a final point, the action-value function can be approximated by a parameterized function Q, such as a neural network, with parameters w and trained by minimizing the squared form of the TD residual (3) or (4).
Discounts are occasionally employed to specify the true optimization objective, whereby they should be regarded as part of the MDP. However, more often discounts serve as a hyper-parameter (van Seijen al., 2019).
3

---

Published in Transactions on Machine Learning Research (06/2025)
2.3 Policy gradient methods
Unlike action-value methods ($2.2), policy gradient methods do not require action-value function for action selection. Instead they work by explicitly representing the policy using a parameterized function 9, such as a neural network, with parameters 0 and only utilizing action-value estimates to Iearn the policy paramcters. To demonstrate the main idea underpinning policy gradient methods, we start from the following formulation of the RL problem (ef. $2.1):
- arg max E T.P.
(s.))
(5)
The objective function in this formulation is the expected state-value function, where the expectation is taken over the state distribution induced by policy и and state-transition function P. This problem can be solved approximately via gradient- optimization. In fact, this forms the basis of policy gradient methods. Accordingly, the policy gradient theorem (Sutton et al., 1999) proves that the gradient of the expected state-value function with respect to policy parameters governed by:
V E VT
V
Σ (e)((s a,P
E x.P
(6)
By using an estimator of the above expression, denoted VJ(O), policy parameters can be updated via stochastic gradient ascent: - 0 + aV)) where a is a positive step-size. It is important to note that, like Sarsa ($2.2), policy gradients are on-policy learners: applying one step of policy gradient updates the policy paramcters 0 + o' and thereby the policy - x', thus inducing a different action-value function Q" + + and a different state distribution.
There have been attempts to extend policy gradients to off-policy data (Degris et al., 2012). The most common approach in this direction is to use deterministic policy gradients (DPG; Silver et al., 2014):
(5t)
/ Q" (st, - ((:))
E x,P
v
E R.P
(7a)
V E Q (st, (s.))
(7b)
M.P
E
(7c)
This is similar to Eq. 6 with the difference that here we replace the general-form policy <alss) with a deterministic and continuous policy 5(a r(s),, where å denotes the delta function whose parameters are given by (s)) Moreover, this derivation only holds in continuous action spaces and, as such, we substitute our general-form notation Σ for both summation and integration with f to specify integration over continuous actions. The expression (7b) is then derived from (7a) by invoking the sifting property of the delta function and (7c) is deduced from (7b) by applying the chain rule, yielding gradient with respect to actions (denoted Va) and another with respect to policy parameters 0 (denoted as before by the shorthand V).
To implement an off-co method using DPG, we must make two key changes to the true deterministic policy gradient (7). First, the deterministic policy which is the target of optimization by DPG generally differs from the behavior policy (als)) that the agent uses to interact with and explore the environment. Therefore, we must modify our notation to reflect this distinction:
E a= V((s)
where y denotes the parameters of the delta function 8 and the expectation is computed with respect to the state distribution induced under behavior policy π and state-transition function P. Second, our estimator Q > '" must be differentiable with respect to actions. This is typically achieved by training parameterized function Q by minimizing the squared form of the TD residual:
(re.1 + 7Q ( (0
Q(+),
(9)
This expression can be viewed as substituting Q( (($441)(((44)) ((5(41)) for max, Q($ a) the TD expression (4), which is used by Q-learning.
4

---

Published in Transactions on Machine Learning Research (06/2025)
2.4 Maximum likelihood estimation
Suppose have data set {(2,,)) 8:)} drawn from unknown joint distribution p(z,y) y), where random variables 2; and respectively represent inputs and targets. Frequently, problem scenarios involve determining the parameters of an assumed probability distribution that best describe the data. The method of maximum likelihood estimation (MLE) addresses this by posing the question: "For which parameter values is the observed data most likely?". In this context, we typically start by representing our assumed distribution using a paramcterizod function f. such as a neural network, with paramcters O. Hence, Φ = f(x) serves our estimator for the distributional parameters in x. For example, & contains K values in the case of categorical distribution with K categories, and contains means μ and variances in the case of a multivariate heteroscedastic Gaussian distribution. We will denote the probability distribution that is specified by parameters Φ = f() as zy)zz). The problem of finding the optimal parameters can then be formulated as:
arg max E p(2-))
log flu ((vlla:))
(10)
This problem can be solved approximately via gradient-based optimization by leveraging the log-likelihood gradient with respect to parameters 0:
E
V log [Wv [w):
(11)
p(a.))
By using estimates of the above expression, denoted VJ(0), we can iteratively refine our distributional parameters & via stochastic gradient ascent on O: 0 - 0 + ((), where is positive step-size.
3 The principles underpinning scalability in policy gradients
As we discussed in Section 2.2, both Sarsa and Q-learning require maximization of the action-value function: Sarsa relics on greody action-sclection in the limit for optimal convergence and Q-lcarning nceds maximizing the action-value function in the successor state to compute its TD target. Additionally, both Sarsa and Q-learning need action-value maximization in the current state for exploitation or, more generally, for constructing their policies (c.g. an e-greedy policy relies on choosing greody actions with probability 1 - and uniformly at random otherwise). However, performing exact maxinization in complex action spaces is computationally prohibitive. This has in turn limited the applicability of Sarsa and Q-learning to small and finite action spaces. On the other hand, policy gradient methods are widely believed to suitable for dealing with complex action spaces. In this section, we identify the core principles underlying the scalability of policy gradient methods and describe each such principle in isolation.
3.1 Approximate summation or integration using Monte Carlo methods
The scalability of policy gradients in their general stochastic form relics heavily on the identity:
E
Σ (s.
= E x.P
QTTT at)
Vi((als)) T(ag(sc)
(12)
= E
Q (st,
at V log ((a))s)))
where the middle expression is derived from our original policy gradient expression (6) by substituting importance sampling estimator in place of the exact summation or integration over the action space. 4 The rightmost expression is then derived simply by invoking the logarithm differentiation rule, where log denotes the natural logarithm. Consequently, using an experience batch the usual form {(se, ae, Tt+1, St+1)] with
3 Equivalent minimizing the KL-divergence between p(a, p(y|z)p(z) and p(x, = f(yl=)p(æ). Impprrtacee sampling is a Monte Carlo (MC) method used for sampling-bass based approximation of sums and integrals (Hammersley Handscomb, 1964).
5

---

Published in Transactions on Machine Learning Research (06/2025)
size n, we can construct an estimator of the policy gradient as follows:
1 L (st, an a log I(alls))
(13)
where Q" is the true action-value function under policy which itself needs to be estimated from experience, e.g. via (*((s,,))) az) x Te+1 + +sV(s)-1) with V serving as a learned approximator of V*.
Considering the fact that the policy gradient estimator (13) is founded upon replacing the exact summation or integration over the action space with an on-trajectory (single-action) MC estimator, we can construct a more general class of policy gradient estimators by enabling off- fifttrajector action samples to also contribute to this nunerical computation (Petit et al., 2019):
1 - +1
(Q ( V (agss)) +1 (m Σ
1
Q" (82, a;) V VlogT(a;)s))),
(14)
where m is the number of off-trajectory action samples a, (-1se) per state St, and the indicator term 1 [m>0) ensures that when m = 0, the second term vanishes, thus reducing to the original on-trajectory policy gradient estimator (13). It is important to note that using the on-trajectory estimator (13) applies broadly to both critic-free (e.g. REINFORCE) and actor-critic (e.g. PPO) methods, but using the off-trajectory estimalor (14) with m > O requires direct approximation of the action-values Q" by a function Q, e.g. a neural network trained by minimizing the squared form of the TD residual (3). As such, the latter can only be employed by Q-based actor-critic methods (c.g. DDPG).
A large portion of policy gradient algorithms rely on the on-trajectory estimator (13), including REINFORCE (Williams, 1992), A3C (Mnih et al., 2016), and PPO (Schulman et al., 2017b). To our knowledge, surprssingyy few algorithms make use of the generalized MC estimator (14), with AAPG (Petit et al., 2019) and MPO (Abdolmaleki et al., 2018) being our only references. On the flip side, methods that perform exact summation or integration over the action space are either limited to small and finite action spaces (Sutton et al., 2001;
Allen et al., 2017) or restricted to specific distribution classes that enable closed-form integration (Silver et al., 2014; Ciosck & Whiteson, 2018; 2020).
3.2 Amortized maximization using maximum likelihood estimation
In RL and dynamic programming, generalized policy iteration (GPI) (Bertsekas, 2017) represents class of solution methods for optimizing policy by alternating between estimating the value function under the current policy (policy evaluation) and enhancing the current policy (policy improvement). Sarsa is an instance of GPI, wherein the policy evaluation step involves learning of an estimator Q ~ Q" by minimizing the temporal difference (3) and the policy improvement step occurs implicitly by acting semi-greedily with respect to Q. Policy gradient methods share a close connection to GP1 as well (Schulman et al., 2015). They also alternate between policy evalustion (i.e. estimating Q R Q") and policy improvement (i.e. updating an explicit policy using an estimate of the policy gradient). Notably, one can instantiate a policy gradient algorithm by performing the policy evaluation step in the same fashion Sarsa. From this standpoint, the mechanism employed for policy improvement is the main differentiator between policy gradient methods and action-value methods like Sarsa. In the previous section, we illustrated how policy gradient estimation can be carried out in a computationally scalable manner. In this section, we delve into the question of how updating the policy using policy gradients achieves policy improvement, and how it does so in an efficient manner.
We start with recasting the log-likelihood gradient (11) using RL terminology, replacing the variables (,,y,i,f) with (s, (,,,,)) Moreover, we reinterpret the expectation computation to be under the joint visitation distribution of stato-action pairs within an RL context." 5 Subscquently, we contrast the reframed log- lieelioel gradient against the policy gradient (12):
V log (logs(a)(ss)
vs.
(0 az) (e) log (og
(15)
x.P
Log- lileeeelihodd gradient
policy gradient
Whiile the reframed lggliiel gradient is useful for comparison against the policy gradient, it does not per se specify a meaningful optimization problem. This is because the target distribution and its estimator are equivalent (Le. m(als)p(s)).
6

---

Published in Transactions on Machine Learning Research (06/2025)
(b)
Reward
Probability
5
4
3 Action
3
Action
Action
Action
(c)
(d)
π
π
T
n
Action
Action
Figure 1: Policy progression according to the true policy gradient in two distinct bandit problems: (a) reward function and (b) softmax-policy progression over time from random initialization to deterministic policy in a multi-armed bandit; (c) delta-policy progression in a continuous bandit problem with bimodal rewards; (d) varianee Gaussian Caussian-policy progression in the same continuous bandit problem. In (c) and (d), policy progressions overlay the reward function.
This comparison implies that policy gradients perform a modified form of MLE, wherein the log-likelihood gradient term is weighted by Q" for each state-action pair. This weighting assigns importance to actions according to the product of (* a) and log (a]s). Therefore, a single step of the true policy gradient updates the policy distribution such that actions with higher action-values become more likely. From this perspective, policy gradients can be construed as form of amortized inference (Gershman & Goodman, 2014). Each step of the true policy gradient improves the current approximate maximizer of interdependent action-value function, with the policy functioning as mechanism for retaining and facilitating retrieval of the best approxination thus far. To elucidate this, consider a basic one-state MDP (aka. mulli-armed bandit) with deterministic rewards (Fig. la). In such a setting, true action-valuee are independent of the policy and are equivalent to rewards: (a) = (a) = r(a) for all æ and a. For Icarning, we use a tabular policy function with a softmax distribution and update it using the true policy gradient in each step. These choices minimize confounding effects, allowing us study the way policy gradients achieve policy improvement in isolation. Figure 1b shows the progression of the policy distribution during training, starting from a random initialization until convergence. Early in training the policy captures the ranking of actions according to their respective action-values. In other words, sampling from the policy corresponds to performing a probabilistic arg sort on the action-value function. In the absence of any counteractive losses, such as entropy regularization, this process continues until convergence to a deterministic policy corresponding to the arg max over the action-value function.
We have discussed that policy gradients can be viewed an iterative approach to action-value maximization. However, they do not always yield the global arg max. This limitation is rooted in local tendencies of gradient- based optimization, affecting scenarios with non-tabular policy distributions (Tessler et al., 2019). Figures 1c,d respectively show progression of a delta policy and fixed-variance Gaussian policy in a continuous bandit with bimodal, deterministic rewards. In both cases, policy improvement driven by policy gradients results in local movement in the action space and thus convergence to suboptimal policies.
3.3 Representation learning via action-in architectures
There are two functional forms for constructing approximate action-value predictor Q: action-in and action-out architectures. An action-in architecture predicts Q-values for given statc-action pair at input. An action-out archilecture outputs Q predictions for all possible actions in an input state. Action-out, architectures have the computational advantage that a single forward pass through the predictor collects all actions' values in given state, versus requiring as many forward passes as there are actions a state by an action-in architecture. Of course, such an advantage only pertinent when evaluating all possible actions, considerable subset of them, in given state necessity that varies depending the algorithm. On
7

---

Published in Transactions on Machine Learning Research (06/2025)
the other hand, one notable limitation of action-out architectures is their incapacity to predict Q-values in continuous action domains without imposing strict modeling constraints on the functional form of the estimated Q-function (Gu et al., 2016).
Action-value methods typically use action-out architectures suchh as DQN (Mnih et al., 2015) and Rainbow (Hessel et al., (018)-while policy gradient algorithms with Q approximations rely on action-in architectures to handle complex action spaces such as DDPG (Lillicrap et al., 2016) and MPO (Abdolmaleki al., 2018). These choices reflect the distinct requirements of each family. In particular, standard action-value methods must evaluate all possible actions per state to perform maximization, making action-out architectures the more efficient choice from computational perspective. In contrast, policy gradient methods that rely on Q upproximation evaluate only one or a small set of actions in any given state ($3.1). Hence, using action-in architectures in the context of policy gradient methods is more computationally efficient in finite action spaces and one that functionally supports Q evaluation in complex action spaces.
So lar, we have compared action-in and action-out architectures from computational and functional standpoints. Now, we turn to fundamental but often overlooked advantage of action-in architectures: their capacity for representation learning and generalization with respect to actions. Specifically, by treating both states and actions as imputs, action-in architectures unify the process of learning representations for both. For example, when training an action-in Q approximator with deep learning, backpropagation enables learning representations over the joint state-action space. In contrast, action-out architectures are limited in their capacity for generalizing across actions (Zhou et al., 2022). This limitation arises because, although many layers may serve to learn deep representations of input states, action conditioning is introduced only at the output layer in tabular-like form. While some action-out architectures introduce structural inductive biases that support combinatorial generalization across multi-dinensional actions (see, e.g., Tavakoli et al., 2018; 2021), they do not capacitate action representation learning and generalization in the general form. Moreover, such architectures remain limited to discrete action spaces and are, generally, subject to statistical biases.
4 Incorporating the principles into action-value learning
In Section 3, we identified three core principles that we argued underpin the effectiveness of popular policy gradient algorithms in complex action spaces. In this section, challenge the conventional wisdom that policy gradient methods are inherently more suitable in tackling complex action spaces by showing that the same principles can be integrated into action-value methods, thus enabling them to exhibit similar scaling properties to policy gradient methods without the need for policy gradients.
In the same spirit as using an MC estimator in place of exact summation or integration over the action space in policy gradient methods ($3.1), the first principle that we consolidate into action-value learning substitutes exact maximization over the action space with a sampling- based approximation. Formally, we compute an approximation of max, Q(s, a) via the steps below:
Principle
Am {ailm ~ Awarchh
(16)
arg max Q(s, a) R arg max Q(+, a;) = =a max a.eA..
(17)
max Q(s a) x Q(s max)
(18)
where m 2 1 is the number of action samples in state s and Aeeach is probability distribution over the generally state-conditional action space A.. Without any prior information, opting for a uniform Asearch is ideal as it ensures equal sampling across all possible actions in a given state. This approach, with a constant m, allows for action-value learning at a fixed computational cost in arbitrarily complex action spaces, Sampled actions are only used internally to probe the Q-function for arg max approximation a given state, and not executed in the environment.
The next principle is equip aetion-value learning with a mechanism for retention and retricval of the best arg max approximation far, analogous to the policy function in policy gradient methods ($3.2).
Principle 2
8

---

Published in Transactions on Machine Learning Research (06/2025)
To do so, let assume we maintain memory buffer B = {(tt, {( where at denotes our best current arg max approxination in visited state In small and finite state spaces, the memory buffer itself can serve as basic mechanism for retention and retrieval via table-lookup (as used by Tian et al., 2022):
dt
MAx ((s)) if 8t in B otherwise
(19)
In this case, we can enable the reuse of past computations for amortized arg max approximations by modifying Eq. 16 in the following way:
Am = {amaa U {aiJm-1 Aeearch (As).
(20)
Then, we refine the arg max approximation via Eq. 17 and update the buffer B(s;) + af This approach does not achieve generalization across states, thus compromising its general efficacy in major ways. To enable a capacity for generalization, we resort to training a stotiiicnditional parameterized distribution function with MLE ($2.4). In other words, we train a parametric arg max predictor fo(((s)) by employing the ligelikel gradient (11) on the stored tuples {(s, a] Notably, this paradigm naturally supports training ensemble of such predictors, for example based different distribution families. Therefore, we rewrite Eq. 20 to explicitly incorporate ensemble of k parametric arg max predictors below:
Amo (A..))
Am ~ (-)(-s))
Am = U
(21)
Ame (-(se)
{amux)] (if a prior approximation exists)
The third, and final, principle is to combine action-value learning with action-in instead of action-out architectures in order to enable action-value inference in complex action spaces as well as representation learning and generalization with respect to actions ($3.3). While the other ingredients apply more broadly to both tabular and approximate cases, this last one is only relevant in conjunction with lunctional approximation. Appendix B.1 provides a neural network architecture from our experiments that exemplifies the action-in approach.
Principle 3
5 Experiments
To evaluate framework, we instantiate Q-learning with marimum likelihood estimation (QMLF) as example of integrating the adapted core principles (54) into approximate Q-lcarning with deep neural networks (Mnih el al., 2015). Appendix A presents the QMLE algorithm in general form. Our illustrative st study ($5.1) employs simplified implementation of this algorithm. Appendix B provides the details of the QMLE agent used in our benchmarking experiments ($5.2).
5.1 Illustrative example
In this study, we present a first illustrative example of how the combination of the three core principles enables QMLE to extend action-value learning to contimuous-action problems. Specifically, Principle allows approximate arg max computation over non-convex Q-surfaces defined on continuous (i.e. non-finite) action spaces, as demonstrated in a 2D action setting. Principle 2 facilitates iterative caching and refinement of the arg max approximation, allowing efficient reuse during and after training. Principle 3 enables function approximation of the Q-function over continuous actions, which is not possible with action-out architectures such as that used in DQN. Additionally, this example highlights that, with respect to Principles and 2, action-value learning can offer more flexible arg max computation than deterministic policy gradients (DPG). As a result, QMLE is able to both mimic and surpass the behavior of DPG, depending on the choice of the action sampling scheme.
We compare QMLE the deterministic policy gradient (DPG) algorithm in a continuous 2D bandit problem with deterministic and bimodal rewards (similar to that presented by Metz et al., 2019). This problem
9

---

model architecture, and Published in Transactions on Machine Learning Research (06/2025)
(a)
(b)
Maximizer Training
Ground Truth
DPG local gradient
QMLE [deita]
QMLE [delta]
approximata local
apprommate local & global
Uniform
Exploration
-10
-1.0
-06
03
06
10
Gaussian
1.0
-0.5
5.0
2.5
24
10
-1.0
Figure 2: QMLE with local sampling approximately subsurnes DPG and with added global sampling transcends DPG by cireumventing suboptimality, as examined in a continuous 2D bandit with two modes and under three canonical exploration strategies. The trajectory of delta distributions during training (yellow) with endpoints (green) overlay the respective learned Q D-functions at convergence.
sctting minimizes confounding factors by reducing action-value learning to supervised learning of rewards and eliminating contributions from differing bootstrapping mechanisms in the two methods. For an apples- to-apples comparison, we constrain QMLE to only a single parametric arg max predictor based on a delta distribution, mirroring the strict limitation of DPG to delta policics. We further simplify QMLE by aligning its computation of greedy actions with that of DPG. This ensures the only remaining difference between QMLE and DPG in how their delta parameters are updated, not in how their greedy actions are computed for constructing behavior policies. Both methods use the same hyper-paramede initialization across all experiments.
We examine two simplified variants of QMLE. The first one samples around the delta parameters for arg max approximations that are used as targets for MLE training, Precisely, we only allow samples Am drawn from (0(s) + +,,, where óg denotes the delta-based arg max predictor and is a zero-mean Gaussian noise with standard deviation of 0.001 (ef. Eq. 21). This akin to computing an MC approximation of asx))=)(()) in DPG (7c). The second variant additionally incorporates sampling uniformly over the full action space, corresponding Amg samples in Eq. 21. For brevity, refer the first variant as using local sampling, and the second using global sampling.
Figure 2a depicts the reward function of the bandit, or equally the ground-truth Q-function. Figure 2b shows the trajectory of delta distributions during training (yellow) until convergence (green), overlaid on the final learned Q-function. DPG (Fig. 2b, left) consistently converges to a local optimum, regardless of the exploration stratogy and despite the sufficient accuracy of learned Q-function. QMLE with local sampling (Fig. 2b, middle) behaves similarly DPG. On the other hand, QMLE with global sampling (Fig. 2b, right) converges to the global optimum across all exploration strategies.
10

---

Published in Transactions on Machine Learning Research (06/2025)
Cartpole SwUp
Cartpole SwUp Sparse
Pendulum SwUp
Point Mass Easy
Ball In Cup Catch
Reacher Hard
1000
Return
750
500
250
1M
2M
1M
0
1M
0
1M
0
2M
Finger Spin
Finger Turn Hard
Hopper Stand
Hopper Hop
Walker Stand
Walker Walk
1000
Return
750
500
250
2M
2M
1M
5M
1M
0
1M
Walker Run
Cheetah Run
Humanoid Stand
Humanoid Walk
Quadruped Run
Dog Walk
1000
Return
750
500
250
5M
5M
10M
10M
0
5M 0
5M
Env step
Env step
Env step
Env step
Env step
Env step
QMLE
DMPO
A3C [1e8]
DDPG [1e8]
D4PG [1e8]
Figure 3: Comparison of QMLE against its closest policy gradient counterpart, off-policy DDPG, with the canonical on-policy A3C included for reference. Performance of toatt--ofthhearrt methods DMPO and D4PG is also shown, representing a potential next frontier for QMLE. DMPO and D4PG incorporate advances such as distributional and N-step learning, which are not used in our current QMLE implementation. Performance curves (QMLE, DMPO) show mean undiscounted return + 1 standard error over seeds. Performance levels (A3C, DDPG, D4PG) report the mean undiscounted return, averaged over 100 evaluation episodes per seed and across seeds. The number of seeds per agent is detailed in Table 2.
This study illustrates key properties of QMLE with respect to DPG: subsumption, where QMLE with local sampling approximates DPG updates, and transcendence, where global sampling allows QMLE to overcome the local lendencies of policy gradients and surpass DPG.
5.2 Benchmarking results
In this section, we evaluate QMLF on 18 continuous control tasks from the DeepMind Control Suite (Tassa et al., 2018). Figure shows learning curves of QMLE alongside the learning curves or final performances of several baselines, including state-of-the-art methods DMPO (Hoffman et al., 2022) and D4PG (Barth- et al., 2018), as well as the canonical (on-policy) A3C (Mnih et al., 2016) and (off-policy) DDPG (Lillicrap et al., 2016). Results for DMPO (12 tasks) are from Seyde et al. (2023), while those for A3C, DDPG, and D4PG (16 tasks) are from Tassa et al. (2018).
With the exception of the Pinger Turn Hard task, QMLE consistently performs between DDPG and D4PG. Notably, it matches or outperforms DDPG on 14 out of 16 tasks, with DDPG being the closest counterpart from the policy gradient paradigm to QMLE. Moreover, QMLE substantially exceeds the performance of A3C across all tasks. This is despite QMLE being trained 10 to 100x fewer steps compared to A3C, DDPG, and D4PG. While QMLE competes well with DMPO in low-dimensional action spaces, it trails in higher-dimenssonal ones. Nonetheless, the strong performance of QMLE in continuous control tasks with up to 38 action dimensions, all without policy gradients, in and of itself testifies to the core nature of our identified principles and their adaptability to action-valle methods.
Appendix C provides supplementary benchmarking results against alternative policy gradient and action-value methods.
11

---

Published in Transactions on Machine Learning Research (06/2025)
5.3 Ablation studies
We conducted ablation experiments to assess the contribution of each principle from Section to QMLF's performance and scalability, Here, we provide a concise summary of our findings and refer the reader to Appendix D for the learning curves and additional details.
In Appendix D.1, we study the impact of the number of action samples used for approximating the value- maximizing action. We find that reducing the number of samples from 1000 to as few as 2 has negligible effect on final performance, highlighting a low sensitivity to sampling budgets. Although using significantly more samples improves accuracy, the benefits often taper off due to amortized maximization, which enables reuse of previous approximations. Consequently, one can significantly reduce inference costs by lowering the sample count without sacrificing performance, a practically valuable property for large-scale lasks or when using large models where each forward pass is computationally expensive.
In Appendix D.2, we ablate amortized maximization by removing parametric arg max predictors and relying solely on uniform sampling. Performance degrades significantly under this condition, particularly in higher- dimensional action spaces. This study confirms the critical role of amortization when sampling budgets are limited relative to the action-space complexity, an inevitable problem scenario across real-world domains where action-space complexity quickly outpaces feasible sampling budgets.
Finally, in Appendix D.3, we replace our action-in architecture with action-out variant. This ablation proves computationally infeasible in large discrete action spaces and underperforms even in moderate action spaces, underscoring the importance of action-in architectures for scalability and generalization.
Taken together, these ablations support that each principle individually contributes to improved scalability or performance of action-value learning.
6 Conclusion
In this paper, we distilled the success of policy gradient methods in complex action spaces into three core principles: MC approximation of sums integrals, amortized maximization using special form of MLE, and action-in architectures for reprosentation learning and generalization over actions. We then argued that these principles are not exclusive to the policy gradient paradigm and can be adapted to action-value methods. In turn, we presented framework for incorporating adaptations of these principles into action-value methods. To examine our arguments, we instantiated QMLE by implementing our adapted principles into approximate Q-learning with deep neural networks. Our results showed that QMLE performs strongly in continuous control problems with up to 38 action dimensions, largely outperforming its closest policy gradient counterpart DDPG. These results provided empirical support for the core nature of our identified principles and demonstrated that -alion methods could adopt them to achieve similar qualities, all without policy gradients. In comparative study using DPG and two simplified QMLE, variants, we highlighted a key limitation of policy gradients; namely, their reliance on local updates in contimuous parameter spaces, which can lead to convergence to suboptimal solutions in multimodal or non-convex value landscapes. By contrast, QMLE leverages flexible action sampling and explicit maximization to overcome this limitation. This study serves as a motivator for a shift from policy gradients toward action-value methods with our adapted principles. It also offers potential explanation for the improvements observed over DDPG in our benchmarking experiments.
References
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Rémi Munos, Nicolas Heess, and Martin Ried- miller. Maximum posteriori policy optimisation. In International Conference on Learning Representations, 2018. URL https://openrevier wtwp:rrwwwie.eeeooouuunun.. wwtp:/oruum..dssSAlll..
Cameron Allen, Kavosh Asadi, Melrose Roderick, AbdeIrahman Mohamed, George Konidaris, and Michael Littman. Mean actor critic. Xie preprint 1rXio:1709.00500 1709000503 2017.
12

---

Published in Transactions on Machine Learning Research (06/2025)
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffre E. Hinton. 1xXiu:66770066 2016.
Layer normalization.
arXiv preprint
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations, 2018. URL https: httes://pppeeeee hptp//eeppeeeeeeeeeeeeeeeeeeeeee wwwpeeiwwwwwweeeeeeeeeeeeeeeeeee ferm/forum?tti ?id= SyZipzbCb.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 -299 2013.
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedliggs of Machine Learning Research, pp. 449-458 PMT.R, 06-11 Aug 2017. html.
URL
mlr. www.ppees//e......
Richard E. Bellman. Dynamic Programming. Princcton University Press, 1957.
Dimitri Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 4th edition, 2017.
Kamil Ciosek and Shimon Whiteson. Expected policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609 wtt.a/aaaivvaa.aaa/aa//a URL sctps://ojst /tap::///js.aaai.orrr
index. wttp:ppp....taa...a.e/e/eeee////
Kamil Ciosek and Shimon Whiteson. Expected policy gradients for reinforcement learning, Journal of Machine Learning Research, 21(52):1 51, 2020. URL http:://aal..rrrpppprss///// wttpppappers///111110022 html.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. Proceedings of the AAAI Conference on Artificial Intelligence, 15:746-752, 1998.
Djork-Arné Clevert Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In International Conference on Learning Representations, 2016.
Tom Van de Wiele, David Warde-Farley, Andriy Mnih, and Volodymyr Mnih. Q-learning in enormous action spaces via amortized approximate maximization. ar preprinl arXiw:2001.08116 2020.
Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. In John Langford and Joelle Pincau (eds.), Proceedings of the 29th International Conference on Machine Learning, pp. 457-464, Edinburgh, Scotland, July 2012. Omnipress.
Gabriel Dulac-Arnoll Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Théophane Weber, Thomas Degris, and Ben Coppin. Deep reinforcement lcarning in large discrete action spaces. arXie preprinl urXiv:1512.00 07677779, 2015.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Alcksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In International Conference on Machine Learning, 2024. URL https:// http:p////ppppooooooeeee////oad/
Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnacve. Growing action spaces. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. PMLR, 13 18 Jul 2020. URL
.ml. httpa//p........er.eeeeeeeeeeeee ppeessvvi9// 1atass//1111/ffuuaa200000 html.
Mchdi Fatemi and Arash Tavakoli. Orchestrated value mapping for reinforcement learning. In International Conference Learning Representations, 2022. URL
13

---

Published in Transactions on Machine Learning Research (06/2025)
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, 1. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Cur- ran Associates, Inc., 2016. URL https://proceeddngs neurips wwwrips.cc/paprr.rccro.... httppppeeeeeeeee//eeeeeeeeeeeeee saapperffieesppppppee fites//paper//211//11.
pdf.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-crilic methods. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1587-1596. PMLR, 10-15 Jul 2018. URL ttttps:/pproceedingg mlr. pww.s//esss//// ht///fjjjjmo fmjoom818..... html.
Samuel J. Gershman and Noah D. Goodman. Amortized inference in probabilistic reasoning, In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 36, 2014.
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep Q-learning with model-based acceleration. In Maria Florina Balean and Kilian Q Weinberger (eds.), Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2838, New York, New York, USA, 20-22 Jun 2016. PMLR. URL attps://proceeddngg mlr. press/v48/ heee//ul6....0. html.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URL https://proceeddnng amltiiinnnmmm.......mmm..r wttpsemww........eeeeeeeeeeeeeee press/v80/ haarnoj a18b. html.
John M. Hammersley and David C. Handscomb. Monte Carlo Methods. John Wiley & Sons, 1964.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. doi: 10.1109/CVPR. 2016.90.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. Proceedings of the AAAT Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609 htt.aaaaivv22...1..... v32i1.11796. URL https://ojs aaai org/index. wwtpndx...pp/aaeee http///At/aariiileeeee/////
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines.
2018.
Matthew W. Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Momchev, Danila Sinopalnikov, Piotr Stańczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, Léonard Hussenot, Robert Dadashi, Gabriel Dulac-Aroodd Manu Orsini, Alexis Jacq, Johan Ferret, Nino Vieillard, Seyed Kamyar Seyed Ghasemipour, Sertan Girgin, Olivier Pietquin, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Abe Friesen, Ruba Haroun, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Frcitas. Acme: A research framework for distributed reinforcement learning. arXiv preprinl ar (exxx:2006.00977 2022.
Shengyi Huang, Rousslan Fernand, Julien Dossa, Chang Yc, Jeff Braga, Dipam Chakraborty, Kinal Mchta, and João G.M. Araújo. Cleanrl: High-quality single-file file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274) 2022. URL http://jmlr org. prgg/apprs/ v23/21-1342.n htmm.
Thomas Hubert, Julian Schrittwicser, Joannis Antonoglou, Mohammadamin Barckatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings
14

---

Published in Transactions on Machine Learning Research (06/2025)
of Machine Learning Research, pp. 4476-4486. PMLR, 18-24 Jul 2021. URL https://proceedngs... mml. press//113/hhuert222 .hmml.
Tommi Jaakkola, Michael I. Jordan, and Satinder P. Singh. On the convergence of stochastic iterative dynamic programming algorithms. Newral Computation, 6(6):1185-1211, 1994.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for vision- -asond robotic manipulation. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (cds.), Proceedings of the 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. press/v87/kaal httpe//@8//allaaniiooll/. html.
PMLR, 29 31 Oct 2018. URL haop:://proceedinng mml.
Steven Kapturowski, Victor Campos, Ray Jiang, Nemanja Rakicevic, Hado van Hasselt, Charles Blundell, and Adria Puigdomenech Badia. Human-level Atari 200x faster. In International Conference on Learning Representations, 2023. URL https://openreview http://rwweeeeeeeeeourriieeeeeee met//forrmm//d-e-cccyHHHB....
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
Zechu Li, Tao Chen, Zhang- Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel Q-learning: Scaling off-policy reinforcement learning under massively parallel sinulation. In Audreas Krause, Emmn Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19440 19459. PMLR, 23 29 Jul 2023. URL https: httpp::/ppoceeee /twp:////cccccco....moo .ml.iinnnmm.......m.m..m..m..... wttpsssmw...peeeeeeeeeee ptwppe/e/e/ei22////22 html.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Hoess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiiee R. Fergus, S. Vishwanathan, and R. Garnett. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https / /ttp///eepogc....e//ee/p/e cetpp....ers.....aapper files/ /ttpee/eeeeeee003334446666666e-0
environments. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach,
Paper.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Fnvironment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523 562, 2018.
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep RL. arXio preprint arXiv: 1755. 05035, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement. learning. Nature, 518(7540):522 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning, In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp- 1928 1937, New York, html.
New York, USA, 20 22 Jun 2016. PMLR. URL https://prroceednggs mlr, parees/v48/mniiiie
Vinod Nair and Geoffrey E. Hinton. Rectified lincar units improve restricted Boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, pp. 807-814, Haifa, Isracl, 2010. Omnipress.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Brendan O'Donoghue, Rémi Munos, Koray Kavukenoglu, and Volodymyr Mnih. Combining policy gradient and Q-learning. In International Conference on Learning Representations, 2017. URL
OpenAT, Tlge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving Rubik's cube with a robot hand. arxiv preprint arxiv:19100 1910007111.,, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. URL https:/pproceeding. nwwrips wwtppperrpfiiiieeeeeee files/ pdf.
Fabio Pardo. Tonic: A deep reinforcement learning library for fast prototyping and benchmarking. arxiv preprinl
2020.
Fabio Pardo, Arash Tavakoli, Vitaly Levdik, and Petar Kormushev. Time limits in reinforcement learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4045 4054. PMLR, 10-15 15 Jul 2018. URL https: /tttps://prort htttps:/proceeddngg .mlr. www.mmlr.........mm..mmmmmmmmm patsss/v80/pardoll html.
Benjamin Petit, Loren Cuulahl-Culleton, Yao Liu, Jimmy Smith, and Pierre-Luc Bacon. All-action policy gradient methods: A numerical integration approach. arXiv preprint 191::1191000909, 2019.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley &
Sons, 1994.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder de Witt, Gregory Farquhar, Jakob Foerster, and
Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-11, 2020. URL http://jmlr. http:///innnoooogppeeeeeee arg/paperss/21/200088. html.
Gavin A. Rummery and Mahesan Niranjan. On-line Q-learning using connectionist systems. Technical report CUED/T-INPENG/TI //TT 166, Department of Engineering, University of Cambridge, 1994.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference Learning Representations, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. URL https: https://proceeedngg.. .mr. http://eedd.n.........eeees////s wtt.pprss//////ccnnneen htmml.
John Schulman, Xi Chen, and Picter Abbeel. Equivalence between policy gradients and soft Q-learning. arXiv preprint XiX:::704.06: 2017a
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint ar 1xix:1107.0033 2017b.
Max Schwarzer, Johan Samir Obando Ccron, Aaron Courville, Marc G. Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level efficiency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 30365-30380 PMLR, 23-29 Jul 2023. URL https:
// //proceedings. mlr. httpa/inww.....eeee////////////e pttpppr///22//ccaaeeeeeeeeeeeeee html.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. uncertainty estimation with probabilistic neural networks. In International Conference on Learning Representations, 2022. URL http:://pparrrreeeeeeee//rrrtaap
On the pitfalls of heteroscedastic
Tim Seyde, Igor Gilitschenski, Wilko Schwarting. Bartolomco Stellato, Martin Riedmiller, Markus Wulfmeier, and Daniela Rus. Is bang-bang control all you need? Solving continuous control with Bernoulli policies. In M. Ranzato, A. Beygekimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. ran Associates, Inc., 2021. URL htap:::/pproocedine neurips. cc/paper wtt.ss/pppaee/2////110 e4bb66..0066630caaa6646262226669
Cur-
Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via Q-learning, In International Conference on Learning Representations, 2023. URL https://openrevvev. http:p/awrrrrreeeeeeoeoroonaeaaa net./toruu/1dds5000AAeee.
Tim Seyde, Peter Werner, Wilko Schwarting, Markus Wulfmeier, and Daniela Rus. Growing Q-networks: Solving continuous control tasks with adaptive control resolution. In Alessandro Abate, Mark Cannon, Kostas Margellos, and Antonis Papachristodoulon (eds.), Proceedings of the 6th Annual Learning for Dynamics and Control Conference, volume 242 of Proceedings of Machine Learning Research, pp. 1646-1661. PMLR, 15 17 Jul 2024. URL https://prooceedngss mlr. setras//242//eee htmml.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31sl. International Conference on Machine Learning, volume 32 of Proceedinggs of Machine Learning Research, pp. 387-395, Beijing, China, 22-24 Jun 2014. PMLR. URL https://procceddngs.
wwwpediinnnnmmm.m...mmmmmem .ww. wtl..ppees/vv///s1ee saweeee//////eeee///////eaee/eee htmml...11.................
Satinder P. Singh, Tommi Jaakkola, Michael L. Littman, and Csaba Szepesvári. Convergence results for single-step on-policy reinforcementllearnning algorithms. Machine Learning, 38(3):287 308, 2000.
Peter Sunehag, Guy Lever, Audrumas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning. arXie preprint arXiv: 706.05296, 2017.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9 44 1988.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018.
Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Newral Information Processing Systems, volume 12. MIT Press, 1999. URL https://ppett http://///ccc......//// -aaeer .df.
Richard S. Sutton, Satinder Singh, and David McAllester. Comparing policy-gradient algorithms. http: // //incompleteidess....ooooo pttp://pppppeee//S/taapuaaaaaaaa 2001.
Yumhao Tang and Shipra Agrawal. Discretizing continnous action space for on-policy optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 3444))598--5988, Apr. 2020. doi: 10.1609/ 111169/aaai.vva baai.v34i044.605 URL https://oja aaai. oarai.or///a..aao..aeaaa..oooara index. pttp:///AA.///aattcll//eee//////
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin Riedmiller. DeepMind Control Suite. arXiv preprint arXiv:1801.00099, 2018.
Arash Tavakoli, Fabio Pardo, and Petar Kormushev. Action branching architectures for deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/ naai.v321..11999 URL https://ojes aaai. http:///.....p////./aataaaaaaaaa
17

---

Published in Transactions on Machine Learning Research (06/2025)
Arash Tavakoli, Mehdi Fatemi, and Petar Kormushev. Learning to represent action values hypergraph on the action vertices. In International Conference on Learning Represenlations, 2021. URL https: //ppeneeview.nee/eeeeeeeeeeeeeee
Chen Tessler, Guy Tennenholtz, and Shie Mannor. Distributional policy optimization: An alternative approach for continuous control. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d d'Alchó-Buc, E. Fox, and R. Garnell (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 1352 1362. Cur- ran Associates, Inc., 2019. URL btttps:/pproeeedee wwwrgsp...rip... cc/paper_ files/paper/2209/f11e 000777/6633220015511444440300130 Papee . pdf.
Tian Tian, Kenny Young, and Richard S. Sutton. Doubly-asynchronous value iteration: Making value iteration asyuchronous in actions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and
A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 5575-5585. Cur- ran Associates, Inc., 2022. URL
newteeirdpnnnnnnnnps..s... htw.sgss....iss.....pprrrrrrrr cc/paper ftles//paper2222/tttee/ 0ee66228474888877000004887770112 Confapprr-oofeeeneee.p.p
neurips.
Hado van Hasselt. Double Q-learning, In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010. URL https:/ hptpss////occceees..... wwwrgsp...rpp... wttpppacccppppppppppeeer.re fttlss//aaaere200///iee pdf.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1)22044-2000, Mar. 2016. doi: 10.1609/
vaai.v30i1.10295, CRL https://ojs httts://ojs.aaa..... aaai. httpa//ww.ppp/...aaaaaaaa///e///
Harm van Seijen, Mchdi Fatemi, and Arash Tavakoli. Using logarithmic mapping to enable lower discount factors in reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. AAlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32, pp. 14134-14144. Cur- ran Associates, Inc., 2019. URL https:/ meueeiiiinnnnnnuipss..ee. ccc/paper_fille/paper////////// (aa23feccc23300a4444003333366 waw.eaa2000..55aaaaeper.rr.r
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias PfasT, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent. reinforcement learning. Nature, 575(7782):350 350-354, 2019.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning. Tn Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1995-2003, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https:// httpp:://proceedings mlr. prwss//48/ /at////agff666 htmll..
Christopher J. C. H. Watkins. Learning from delayed rewards. PhD thesis, University of Cambridge, 1989.
Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3):279-292) 1992.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229 256, 1992.
Zhiyuan Zhou, Cameron Allen, Kavosh Asadi, and George Konidaris. Characterizing the action-generalizaaton gap deep Qllearnin. arXiv preprint ar arxi:2200.. 2055880888888 2022.
18

---

Published in Transactions on Machine Learning Research (06/2025)
A Q-learning with maximum likelihood estimation
In this section, we present the Q-learning with mazimam likeldhood estimalion (QMLE) algorithm. Specifically, our presentation is based on integrating our framework ($4) into the deep Q-laarning algorithm by Mnih et al. (2015). In line with this, we make use of experience replay and target net that is only periodically updated with the parameters of the online network. Importantly, we extend the scope of the target network to encompass the arg max predictors in QMLE. Although the algorithm does not mandate the use of action-in Q approximators per se, such architectures become necessary for addressing problems with arbitrarily complex action spaces ($3.3).
Algorithm details the training procedures for QMLE. Notably, the algorithm is flexible regarding the composition of the ensemble of arg max predictors. For instance, the ensemble can consist of a combination of continuous and discrete distributions for problems with contimuous action spaces. QMLE introduces several related to its action-sampling processes. These include the sampling budgets for target maximization, Mtarget: and greedy action selection in the environment, Mgrredy. Additionally, QMLE uses sample allocation ratios (po, . pk), where Po corresponds to the proportion of the budget allocated to uniform sampling from the action space, and A1 through Pt correspond to the proportions assigned to the eusembble of k parametric arg predictors.
To effectively manage training inference costs in QMLE, we recommend allocating larger budget to Migredyy than to Mtarget- Since Mgreedy is used at most once per interaction step, inereasing it incurs relatively little computational burden. In addition, more accurate arg max approximations during training interactions can lead to higher quality data for learning, making this increase particularly beneficial. In contrast, each training update requires Maaggt X N₂ inferences the target Q-network, where N₂ is the batch size. This makes increasing Mearget much more costly in terms of training inference costs. On that account, choosing
Algorithm 1: QMLE algorithm.
Input:
sampling budgets mearges, Migreddy and ratios (po, pk. } (I is the # of arg max predictors) Input: initial model parameters w, (0,, ......, 0u); step sizes Ckgs Gargmax
Input: : target update frequency N batch size No; replay period K; interaction budget No T
Initialize target parameters , (0, } + w. (0.}i, accumulators A, = {A.}t 0 Initialize memory buffer B = Ø for episode E (..,.... Ne) do
Observe initial state 50 for E {0, l,..., 7'- 1} do
with probability s do
Sample action ~ Uniform(A..)
otherwise do
Generate actions Agreedy using (0.)1, {m, p, x Mgreedy 1/2 in Eq. 21,
Approximate greedy action de using Qu, st, Agt
Agresdy in Eq. 17
Observe re+1, Sc+1, 7t41 from environment given ae, set max +- az
Store transition (St, az, To41, Te,71 8t+1, 8141, opp)) in B
if 0 0 mod K then for E (1, No) do Sample random transition (sj, dy. T541, 1541,75411 max, from B
Generate actions Atarget using (0, }4, (m = p: x maarget }o a (prior) in Eq. 21
Approximate target-maximizing action 4y+1 using Qu 83+1, A target Eq. 17 Set mmax +- aj+1 and update B
Compute squared TD residual = + T4 (8)+1, a - (s),),a))))
Compute MLE losses H using parameters (0,) and target a+f1
Accumulate parameter-changee 4g <- Ag + VwLy, (A, (- A, + Vo, L
Update parameters + Nr 0g Aqo {0. +
Oargmax A 1
Reset accumulators A, - (A.)
Update target parameters w (0, } w, (0:)t every N time steps
Terminate episode on reaching a terminal state, where 7441 = 0
19

---

Published in Transactions on Machine Learning Research (06/2025)
moderate Maarget allows for computational tractability with larger batch sizes. Remarkably, moderate Mtarget could also help reduce the overestimation of action-values (van Hasselt, 2010; van Hasselt et al., 2016). Also, assigning a smaller Maarget relative to Mgreedy is further justified because target maximization benefits from additional amortization. Specifically, each time a transition is sampled from the memory buffer for experience replay, we use the previously stored arg max approximation as a prior. This approximation is then recalibrated and updated in the memory buffer for the next time that the transition sampled for replay.
B Experimental details
This section details the specific QMLE instance that we evaluated in our benchmarking experiments. We adopted prioritized experience replay (Schaul al., 2016), in place of the uniform variant that was described in Algorithm 1. Furthermore,
deployed QMLE with two arg predictors: based delta distribution over the continuous action space, and another based on factored categorical distribution defined over finite subset of the original action space (Tang & Agrawal, 2020).
To build the discrete action support, we applied the bang-off-bang (3 bins) discrctization scheme to the action space (Seyde et al., 2021). For sampling from the delta-based arg max predictor, we always included the parameter of the delta distribution as the initial sample. Any additional samples were generated through Gaussian perturbations around this parameter using small standard deviation.
Sections B.1, B.2, and B.3 provide details around the model architecture, aaper-parametery. and implementa- tion of QMLE in our benchmarking experiments, respectively. Section B.4 details the number of seeds per agent and the computation of our learning curves.
B.1 Model architecture
Argmax Predictor
Argmax Predictor 2
Q-value
MLP
MLP
Les
-
LayerNorm
Embedding
Embedding
Observation
Action
Figure 4: Schematic of the model architecture used with QMLE for benchmarking experiments. Dashed lines indicate paths without gradient flow during backpropagation.
Figure depicts the model architecture of QMLE in our benchmarking experiments. The model begins with two separate streams, one for the observation inputs and the other for the action inputs. The outputs of these streams are then concatenated and jointly processed by the Q-value predictor. Furthermore, the output of the observation stream is separately processed by each arg max predictor.
20

---

Published in Transactions on Machine Learning Research (06/2025)
In the observation stream, apply linear embedding layer with 128 units followed by residual block (He et al., 2016) that maintains this width and uses rectified linear unit (ReLU) activation (Nair & Hinton, 2010). The residual block is succeeded by layer normalization (LayerNorm) operation (Ba et al., 2016) and exponential linear unit (ELU) activation (Clovert et al., 2016).
In the action stream, we apply a linear embedding layer with 128 units. The output of the embedding layer is then directly followed by LayerNorm and ELU activation.
The outputs from both streams are concatenated and passed through joint observation-action residual block with 256 units and ReLU activation. Subsequently, we apply LayerNorm and ELU activation. The outputs are then linearly mapped to a single scalar, representing the predicted Q-value.
The output the observation stream also used as input to the two arg max predictors. To avoid interference, we prevent backpropagation from the arg max predictor streams through the shared observation stream. Each arg max prodictor stream leverages hidden multilayer perceptron (MLP) layer with 128 units and ReLU aclivation.
In the arg max predictor stream based the delta distribution, produce output per action dimension. Each output is passed through hyperbolic tangent (Tanh) activation to yield a contimuous value constrained within the support of each action dimension in our benchmark. the arg max predictor stream based on the factored categorical distribution, we produce three outputs per action dimension. We apply the softmax function to the outputs for each action dimension, producing multiple softmax distributions over a bang-off off- bang discrete action support.
B.2 syper-ppraaeters
Table provides the hyper-parameters QMLE in our benchmarking experiments.
Table 1: QMLE hyper-parameterss in our benchmarking experiments.
Parameter
Value
Mtrget
100
Mgreedy
1000
Po (uniform)
0.9
P1 (delta)
0,01
P2 (factored categorical)
0.09
step sizes ag Clarg max update frequency
0.0005
10
batch size
256
training start size memory buffer size target network update frequency
1000
1000000
2000
loss function
mean-squared error
optimizer
Adam (Kingma & Ba, 2015)
exploration
0.1
discount factor
0.99
time limit
1000
(Tassa et al., 2018)
truncation approach
partial-episode bootstrapping (Pardo et al., 2018)
importance sampling exponent
0.2
priority exponent
0.6
21

---

Published in Transactions on Machine Learning Research (06/2025)
B.3 Implementation
Our QMLE implementation is based on DQN script from CleanRL (Huang et al., 2022) and incorporates prioritized experience replay adapted from Stable Baselines (Hill et al., 2018), both available under the permissive MIT license.
support reproducibility, we release the implementation used in our benchmarking experiments al: https://github. cww/coom//////e /qml
B.4 Seeds and performance
All curves show the mean undiscounted return across seeds, with one standard error. Performance levels of DDPG, D4PG, and A3C are averaged over 100 episodes per seed, after 100M environment steps of training. Table 2 lists the number of seeds used per agent, grouped by result source.
Table 2: Number of seeds used in benchmarking experiments.
Agent.
Trials
QMLE
Dog and Humanoid tasks
10 all other tasks
Results from Seyde et al. (2023)
DMPO
DQN
10
Results from Tassa et al. (2018)
A3C 15
DDPG
15
D4PG
5
Results from Pardo (2020)
MPO
10
SAC
10
TD3
10
PPO
10
TRPO
A2C
10
Results from de Wiele et al. (2020)
AQL
QT-Opt 3
C
Supplementary benchmarking results
Figures 5 and provide comparisons of QMLE with a range mainstream policy gradient methods. The baseline results are due to Pardo (2020).
Figure compares QMLE with policy gradient methods that use action-value approximation: MPO (Abdolmaleki et al., 2018), SAC (Haarnoja et al., 2018), and TD3 (Fujimoto al., 2018).
Figure compares QMLE with policy gradient methods that use state-value approximation: PPO (Schulman al., 2017b), TRPO (Schulman et al., 2015), and A2C (Mnih al., 2016).
Figure 7 compares QMLE with QT-Opt (Kalashnikov et al., 2018) and both the discrete and continuous action variants of AQL (de Wiele et al., 2020). The baseline results are taken from de Wiele et al. (2020).
22

---

Published in Transactions on Machine Learning Research (06/2025)
Cartpole SwUp
Cartpole SwUp Sparse
Pendulum SwUp
Point Mass Easy
1000
750
Return
500
250
0
1M
0
2M
1M
1M
Env step
Env step
Env step
Env step
Ball In Cup Catch
Reacher Hard
Finger Spin
Finger Turn Hard
1000
Return
750
500
250
0
1M
0
2M
0
2M
2M
Env step
Env step
Env step
Env step
Hopper Stand
Hopper Hop
Walker Stand
Walker Walk
1000
750
Return
500
250
0
1M
5M
1M
1M
Env step
Env step
Env step
Env step
Walker Run
Cheetah Run
Humanoid Stand
Humanoid Walk
1000
Return
750
500
250
0
5M
0
5M
0
5M
MM
Env step
Env step
Env step
Env step
Quadruped Run
1000
750
Return
500
QMLE
MPO
SAC
TD3
250
5M
Env step
Figure Learning curves of QMLE against MPO, SAC, and TD3.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Cartpole SwUp
Cartpole SwUp Sparse
Pendulum SwUp
Point Mass Easy
1000
750
Return
500
250
0
1M
2M
1M
1M
Env step
Env step
Env step
Env step
Ball In Cup Catch
Reacher Hard
Finger Spin
Finger Turn Hard
1000
750
Return
500
250
1M
2M
2M
0
2M
Env step
Env step
Env step
Env step
Hopper Stand
Hopper Hop
Walker Stand
Walker Walk
1000
Return
750
500
250
0
1M
5M
1M
0
1M
Env step
Env step
Env step
Env step
Walker Run
Cheetah Run
Humanoid Stand
Humanoid Walk
1000
750
Return
500
250
5M
5M
5M
0
5M
Env step
Env step
Env step
Env step
Quadruped Run
1000
750
Return
500
QMLE
PPO
TRPO
A2C
250
0
5M
Env step
Figure 6: Learning curves of QMLE against PPO, TRPO, and A2C.
24

---

Published in Transactions on Machine Learning Research (06/2025)
Humanoid Walk
1000
250
AQL-Gaussian
QT-Opt
0
10M
200M
Env step
Humanoid Stand
1000
Return
750
500
250
MQL-Gaussian
QT-Opt
10M
200M
Env step
Cheetah Run
1000
750
500
5M
200M
Env step
Walker Run
1000
750
500
250
200M
Env step
Hopper Hop
1000
750
500
0 5M
200M
Env step
QMLE
AQL-Autoregressive
AQL-Gaussian
QT-Opt
Figure 7: Comparison of QMLE with QT-Opt and AQL.
25

---

Published in Transactions on Machine Learning Research (06/2025)
D Ablation studies
In this section, we present ablation studies to evaluate the impact of the principles in our framework on the performance of QMLE.
D.1 Approximate maximization
Figure 8 shows the learning curves for QMLE with sampling budgets of 2 and 1000. Expectedll, increasing the number of samples for Q-maximization improves performance by yielding more accurate estimates of the TD target and greedy actions. Nevertheless, amortization dampens the negative impact of undersampling by enabling reuse of past computations over time.
Cartpole Sw
Point Mass Easy
Ball In Cup Catch
Finger Spin
Walker Stand
1000
Return
750
500
250
1M
1M
1M
Env step
Env step
Env step
Env step
Env step
m=2
m=1000
Figure 8:
Comparison of QMLE with sampling budgets of 2 and 1000.
D.2 Amortized maximization
Figure 9 compares the performance of QMLE against its ablation without amortized maximization. In this experiment, QMLE employs a delta-based arg max predictor, while its ablated variant relies solely uniform sampling for arg max approximation. We use the same sampling budgets of mtarget = migreedy = 2 for both variants, with QMLE allocating its budgets equally between uniform sampling and the delta-based arg max predictor (Puniform = Pdelta = 0.5), and the ablated variant allocating them entirely to uniform sampling (Puniform = 1).
The action spaces range from 1-dimensional (lelmmost) to 6-dimensional (rightmost) for the considered problems. The results demonstrate that amortized maximization significantly improves performance, particularly as the complexity of the action space increases.
Cartpole SwUp
Point Mass Easy
Ball Cup Catch
Finger Spin
Walker Stand
1000
Return
750-
500
250
0
1M
0
1M
0
1M
1M 0
1M
Env step
Env step
Env step
Env step
Env step
QMLE [continuous]
no amortization
Figure 9: Comparison of continuous variant of QMLE with and without amortized maximization.
D.3 Action-in architecture
We compare the performance of QMLE with action-in and action-out architectures. Since action-out Q- approximators are not readily compatible with continuous action spaces, we examine both agents on the bang-off-hang (3 bins) discretized versions of the considered environments.
26

---

Published in Transactions on Machine Learning Research (06/2025)
The QMLE variant with action-in architecture employs arg max predictor based factored categorical distribution, with the same sampling budgets and uniform sampling ratio in Table but with Pdelta 0 and Plactored categorical = L. On the other hand, exact maximization is performed for the ablated variant as a forward pass through an action-out Q-approximator collects all actions' values in a given statc. Thorefore, using an action-out architecture in the ablated variant obviates the need for learned arg max predictors or any approximate maximization altogether. That is to say, when inference with an action-out architecture is compulationally feasible, performing exact maximization should also be feasible given that its cost is generally negligible compared to that of inference. This, in effect, reduces the ablated variant to DQN.
Figure 10 shows the learning curves for QMLE and DQN.
In cimes-dime action spaces, such as Finger Spin and Walker Walk with 2 and 6 action dimensions respectively, where DQN is computationally tractable, both QMLE and DQN achieve similar final performance levels. However, QMLE performs more sample-efficiently due to the use of an action-in architecture, which enables generalization across actions.
In higher-dimensional action spaces, DQN becomes computationally intractable, resulting in out-of- memory errors or exceeding computational time raintsaints. Tn contrast, QMLE performs stnglyy in these environments, including Dog Walk with 3 x 1.35 x 10 discrete actions, underscoring the benefits of action-in architectures both in terms of computational scalability and generalization across enormous action spaces.
Finger Spin
Walker Walk
Quadruped Run
Humanoid Walk
Dog Walk
1000
Return
750
500
250
0
2M
1M
0
SM
0
10M
0
5M
Env step
Env step
Env step
Env step
Env step
QMLE [discrete]
DQN
Comparison of QMLE with DQN where DQN represents the ablation of action-in architectures, and in turn all three principles, in QMLE. Dashed lines indicate out-of-memory errors or excessive computational demands for DQN.
Figure 10:
E
Future work
E.1
Combining with other improvements
In this paper, we integrated our framework into the deep Q-learning algorithm of Muih et al. (2015), in a proof-of-concept agent that we termed QMLE (Algorithm 1). In our benchmarking experiments, we further combined QMLE with prioritized experience replay (Schaul et al., 2016; see details in Section B). While this setup is relatively basic compared to the advancements in deep Q-learning, served our purpose of demonstrating the general competency of action-value methods in complex action spaces without involving policy gradients. We anticipate that a purposeful integration with advancements in deep Q-learning could significantly improve the performance of our QMLE agent. For instance, fundamental methods that be trivially combined with QMLE include N-step returns (Sutton & Barto, 2018; Hessel et al., 2018) and distributional learning (Bellemare et al., 2017), similarly to the critics in DMPO and D4PG. Certain methods, including double Q-learning (van Hasselt, 2010; van Hasselt et al., 2016) and dueling networks (Wang et al., 2016) may not be directly applicable or relevant to QMLE, underscoring the importance careful integration. We are particularly excited about using a cross-cntropy classification loss place of regression for training Q approximators (Farebrother et al., 2024), as well as combining with ideas introduced by Li al. (2023); Schwarzer et al. (2023). Moreover, formal explorations into the space of value mappings (van Seijen et al.,
27

---

Published in Transactions on Machine Learning Research (06/2025)
2019; Fatemi & Tavakoli, 2022), particularly those that benefit Q-function approximation with action-in architectures, offer an intriguing direction for future work.
Since our approach employs maximum likelihood estimation (MLE) in disentangled manner (see discussions in Section 3.2), makes it trivial to incorporate advances from supervised learning for training the parametric arg max predictors. To provide an example, advancements heteroscedastic uncertainty estimation, such that introduced by Seitzer et al. (2022), can be readily applied to model state-conditional variances for Gaussian arg max predictors.
E.2 Multiagent reinforcement learning via CTDE
A problem scenario that could benefit from QMLE, and more broadly our framework, is multiagent reinforce- ment learning (MARL) under centralized training with decentralized execution (CTDE; Foerster et al., 2016; Lowe et al., 2017). Currently, the dominant class of solutions in this paradigm is based on combinations of deep Q-learning and value decomposition methods (Sunehag et al., 2017; Rashid al., 2020). These approaches decompose the Q-function into local utilities for each agent, aiming for the local arg max correspond to the global arg max the joint Q-function. However, maintaining this alignment requires imposing structural constraints that limit the representational capacity of the joint Q-approximator, which can lead to suboptimal decentralized arg max policies.
QMLE avoids these constraints by disentangling the process of approximating the joint Q-function from learning the decentralized arg max policies, allowing for a universal representational capacity of the joint Q-function while maintaining docentralized exccution. Instead of relying on a factored Q approximation, QMLE models the joint Q-function in unconstrained manner. Simultaneously, an arg max predictor (or ensemble of them) is separately trained for each agent, conditioned their respective observations. This approach allows for improved coordination between agents by preserving the full representational capacity of the joint Q-function. As demonstrated in Figure 11, in continuous variant of the "climbing" game (Claus & Boutilier, 1998), linear value decomposition (Sunehag et al., 2017) leads to a suboptimal reward of 5 due to its constrained capacity to represent the joint Q- Q-function as Q = U1+ U2. In contrast, QMLE, by accurately modeling the joint Q-function, enables decentralized arg max predictors that guide agents to the globally optimal reward of 11.
Ground Truth
QMLE
Q =
U2
U1
1.0
160
90
65
10
-30
U2
4.0
-10
-.0
180
a1
Figure 11: Comparison of QMLE with linear value decomposition in a continuous variant of the "climbing" game with two agents (Claus & Boutilier, 1998). Linear value decomposition leads to suboptimal reward of due to its limited representational capacity (Q - U1 + U2), whereas QMLE, by modeling the joint Q-function without such constraints, enables decentralized arg max predictors that guide the agents to the globally optimal reward of 11.
E.3 Curriculum shaping through growing action spaces
Growing of the action space as a form of curriculum shaping is an effective approach for improving learning performance in complex problems. Nonetheless, existing approaches, such as that presented by Farquhar et al. (2020), are restricted to discrete actions. Seyde et al. (2024) report improvements in sample efficiency and solution snoothness on physical control tasks by adaptively increasing the granularity of discretization
28

---

Published in Transactions on Machine Learning Research (06/2025)
during training. This is because coarse action discretizations provide exploration benefits and yield lower variance updates early in training, while finer control resolutions later on help reduce bias at convergence. However, due to the strict dependence of this approach on a class of action-out architectures (Tavakooi et al., 2021; Seyde et al., 2023), cannot ultimately transition from coarse discrotization to the original continuous action space.
On the other hand, QMLE can support learning with dynamically growing action spaces, including transitions from finite to continuous supports in continuous action problems. We show this capability in preliminary experiment, where start with coarse bang-off- bang discretization and later shift to the original continuous action space (Figure 12). This capacity positions QMLE, and more broadly our framework, as promising candidate for future research in the context of growing action spaces.
Cartpole SwUp
Hopper Hop
Walker Run
1000
750
Return
500
250
0
1M
0
5M
0
5M
Env step
Env step
Env step
curriculum
discrete
continuous
Figure 12: Learning curves for discrete, continuous, and discrete-to-contimuus ("curriculum") variants of
QMLE. Dashed lines mark the transition from discrete to continuous actions for the curriculum-based agents.
29


Reviews:
[{"review_id": "5NtQ3WySjQ", "reviewer": "Reviewer_rMvR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Organization: the main text is short, and a lot of important content and experiments are in the appendix. \n- Experimental results are not convincing: in Fig 3, proposed QMLE seems to underperform DMPO in many cases. \n- Missing computational scalability results: one of the main claims is the three principles identified are reasons for PG's scalability wrt action space complexity, and those principles can be applied to action-value methods too. However, there are no experiments showing as action space complexity grows, what happens to the performance of different approaches.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In RL problems with large, complex action spaces, this manuscript identifies three principles that enable the practical computational advantages of PG methods and applies them to action-value methods, making them more scalable.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- What does \"complex\" action spaces mean? How are you defining the complexity of action space? \n- Page 5 Eqn 14: the indices related to $m$ appear to be inconsistent, specifically $\\frac{1}{m+1}$ and $\\sum_{i=0}^{m-1}$\n- Page 6 Sec 3.2: the PG as presented in the paper seems to be actor-critic methods as it also includes the policy evaluation step. Pure PG does not have a value-based component for policy evaluation. In fact, it is concerning that \"actor-critic methods\" are not mentioned at all in the paper main text, given this paper studies the relationship between value-based and policy-based RL methods. \n- Page 6 Eqn 5: why do we want to maximize the policy log likelihood?", "broader_impact_concerns": "NA.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8E1UChcz68", "reviewer": "Reviewer_9Qh3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n1. Overall, the paper is well-written. I enjoyed the exposition in sections 1-3.\n2. The topic of the paper is important, and a portion of the RL community would benefit from the ideas in this paper. I personally would share the paper with colleagues. \n\n# Weaknesses\n\nThe paper aims to understand the core principles that bridge the gap between policy gradient algorithms and action-value algorithms, though this goal seems vague. The paper does a good job contrasting these two algorithms, though it remains unclear what it means to successfully \"bridge the gap.\" My interpretation is that (1) each of the three principles of policy gradient algorithms should *individually* improve the data efficiency of action-value learning, and (2) using *all three* principles in action-values algorithms should improve data efficiency more than using a subset of these principles. The experiments in the main paper explore (1), and the ablations in Appendix D explore (2). Since QMLE builds off of Q-learning, DDPG is the closest continuous-action counterpart (the paper mentions this later), QMLE vs DDPG is the comparison we care about most. The paper should more clearly articulate *what* the goal is and *how* the experiments demonstrate that QMLE achieves these goals.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies three core principles underlying the success of policy gradient methods and demonstrates that these principles can be adapted to action-value methods. The principles are: (1) Monte Carlo approximation for integration, (2) amortized maximization through maximum likelihood estimation, and (3) action-in architectures for state-action representation learning. The paper introduces QMLE, an action-value method that uses \n1. A sampling-based MC approximation of argmax Q(s,a)\n2. When sampling actions to approximate argmax Q(s,a), sample actions from a family of distributions that approximate current argmax Q(s,a). These functions are analogous to the agent's policy in policy gradient learning.\n3. An action-in architecture to compute Q(s,a) to enable state-action representation learning.\n\nQMLE matches or outperforms policy gradient methods like DDPG on many DeepMind Control Suite tasks, supporting the claim that action-value methods can be effective in complex action spaces when incorporating these universal principles.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The ablations in Appendix D are critical to the core claim of the paper -- namely, that we can improve action-value algorithms using each of the three principles outlined in the paper -- and should be moved to the main paper. The main experiments in section 5 show that all three principles together can improve performance. If you keep these experiments in the appendix, the main paper should call out their existence and summarize their findings.\n\n1. Clarification question: In Section 4, actions sampled from $A_m$ are only used to probe the Q function for the argmax, correct? These actions are never executed in the environment, correct?\n\n2. The illustrative experiment in Figure 2 was a bit unclear to me at first. This is the first time the term \"global sampling\" appears in the paper, though it's unclear was this refers to. My initial impression was that QMLE was finding the optimal solution because it was sampling off-policy, which allowed it to explore actions that DPG wasn't explore. After some thought, I think the core takeaway of this experiment is that QMLE is doing a better job approximating argmax Q(s,a) than DPG and that it is important to sample off-policy data *to probe Q(s,a) for the maximizing action*. I think that's the point that needs to be better emphasized here: DPG has a good enough approximation of Q*, but it's not finding a*. If this is the correct interpretation, it's worth emphasizing that this example focuses on principles 1 and 2.\n\n3. Also does the true policy gradient point towards the optimal policy in the illustrative example? \n\nThe changes below are more minor:\n\n4. Section 5: \"with DDPG being the closest counterpart from the policy gradient paradigm to QMLE.\" This should be emphasized earlier in the experiments.  \n\n5. Conclusion: \"we highlighted a key limitation of policy gradients and showed how QMLE could overcome it.\" Specify the key limitation.\n\n6. The experiment section or figure captions should specify what is plotted (mean with +/- 1 standard deviation). This information is currently in the appendix.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TX7gct9fcR", "reviewer": "Reviewer_2ZYF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n- The paper is well-written and easy to follow, with no significant technical inaccuracies aside from minor details.\n\n**Weakness**\n\n- The paper is somewhat lengthy and could benefit from conciseness.\n- The central claim is ambiguous. Specifically, it is unclear what is meant by a \"complex\" action space and AV methods.\n- The paper's scope may not appeal to a broad audience.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the following question: why do the computational applicability and performance of policy gradient (PG) methods and action-value (AV) methods diverge as the complexity of the action space increases?\n\nThe authors identify three principles that underpin the scalability of PG methods and hypothesize that AV methods can achieve comparable efficiency by incorporating these principles. To validate this hypothesis, they introduce QMLE, an AV method that integrates the three principles, and evaluate it through experiments on a 2D continuous bandit problem and the DeepMind Control Suite. The results demonstrate that QMLE performs on par with popular PG methods, concluding that AV methods combined with the three principles work.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Clarify \"complex\" action space**:  \n   - If \"complex\" refers to hybrid action spaces, please include experiments in environments with hybrid action spaces to demonstrate that QMLE performs on par with PG methods.  Also, it is necessary to cite papers discussing RL for hybrid action spaces, such as P-DQN and HyAR, and position this paper into an appropriate context.\n   - If \"complex\" refers to higher-dimensional continuous action spaces, please explicitly define what constitutes the \"complexity\" of an action space.\n\n2. **Clarify AV methods**:  \n   - Does \"AV methods\" refer to algorithms that do not use policy gradients? If so, how do other policy search algorithms fit into this definition?  \n   - Alternatively, does \"AV methods\" refer to algorithms that do not use a policy network? If so, how do classification-based policy iteration algorithms fit into this framework?\n\n3. **Improve conciseness of Section 3**:  \n   - Section 3 could be significantly condensed, as many of the stated principles appear to reiterate well-known facts about PG and reinforcement learning. For example, the equivalence between reward-weighted maximum likelihood estimation and PG has been established in prior work such as [Kober and Peters 2009](https://papers.nips.cc/paper_files/paper/2008/hash/7647966b7343c29048673252e490f736-Abstract.html).\n\nWithout clarifying 1 and 2, it is impossible to evaluate the soundness of the paper, which is the most important acceptance criteria in TMLR. After aforementioned revisions and confirming all claims are supported by evidences, I am inclined to acceptance.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the revised manuscript is clear, sound, and interesting and does not require any revision to be published in TMLR.
Decision: Accept (poster)