Actor-Free Continuous Control via Structurally Maximizable Q-Functions
Authors: Yigit Korkmaz
Urvi Bhuwania
Ayush Jain
Erdem Biyik
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Q-learning
Primary area: reinforcement_learning
TL;DR: Actor-free Q-learning in Continuous Action Spaces by Learning a "Wire-fitted Q-function"

Abstract: Value-based algorithms are a cornerstone of off-policy reinforcement learning due to their simplicity and training stability. However, their use has traditionally been restricted to discrete action spaces, as they rely on estimating Q-values for individual state-action pairs. In continuous action spaces, evaluating the Q-value over the entire action space becomes computationally infeasible. To address this, actor-critic methods are typically employed, where a critic is trained on off-policy data to estimate Q-values, and an actor is trained to maximize the critic's output. Despite their popularity, these methods often suffer from instability during training. In this work, we propose a purely value-based framework for continuous control that revisits structural maximization of Q-functions, introducing a set of key architectural and algorithmic choices to enable efficient and stable learning. We evaluate the proposed actor-free Q-learning approach on a range of standard simulation tasks, demonstrating performance and sample-efficiency on par with state-of-the-art baselines, without the cost of learning a separate actor. Particularly, in environments with constrained action spaces, where the value functions are typically non-smooth, our method with structural maximization outperforms traditional actor-critic methods with gradient-based maximization.

Actor-Free Continuous Control via Structurally Maximizable Q-Functions
Yigit Korkmaz'
Urvi Bhuwania!"
Ayush Jain Erdem Bryik
'Thomas Lord Department of Computer Science, University of Southern California, Meta AI
Abstract
Value-based algorithms comerstone of off- reinforcement learning due to their simplicity and training stability. However, their use has traditionally been restricted to discrete action spaces, as they rely estimating Q-values for individual state-action pairs. In continuous action spaces, evaluating the Q-value over the entire action space becomes computationally infeasible. To address this, actor-critic methods are typically employed, where a critic is trained on off-policy data estimate Q-values, and actor is trained to maximize the critic's output. Despite their popularity, these methods often suffer from instability during training. In this work, we propose purely aalue-based framework for continuous control that revisits structural maximization of Q-functions, introducing set of key architectural and algorithmic choices to enable efficient and stable learning. We evaluate the proposed actor-free Q-learning approach on a range standard simulation tasks, demonstrating performance and sample-efficiency on par with state-of-the- baselines, without the cost of learning a separate actor. Particularly, environments with constrained action spaces, where the value functions are typically non-smooth, our method with structural maximization outperforms traditional actor-critic methods with gradient-bar based maximization. We have released our code at http:://gihhub.oo/SSSS//CC
Introduction
Reinforcement learning (RL) has been highly effective in many domains. ranging from robotics to gaming and recommender systems [25, 29, 45, 1]. Value based RL methods such as Deep Q- Networks (DQNs) [34, 17] are widely used in discrete action spaces due their simplicity, training stability, and sample-efficiency. However, these methods require maximization over the action space, which is feasible only in discrete action spaces, requiring alternative approaches for continuous control.
In problems with continuous action spaces, policy-based methods are typically used, such DDPG [30]. SAC [16], and TD3 [1]]. These approaches employ an actor-critic architecture, where the actor (policy) is updated based on feedback from the critic (Q (Q-function) to select the optimal actions, and both components are usually parameterized by neural networks. While powerful, actor-critic methods often face instability due the coupled training of actor and critic, hyperparameter sensitivity, and predicting in high-ding sigh--immnsional action spaces. Moreover, gradient- when the action space is constrained, such when actions require to be within specific safety bounds [8, 24], because the gradient-ascending actor can only find locally optimal actions.
actor-critic methods struggle
Can develop an actor-free value-based algorithm that can efficiently select optimal actions in continuous domains? In this work, we propose to extend the DQN framework to continuous action spaces via Q-function representation that is structurally maximizable. This has been explored for
Equal Contribution, Equual Advising. Work performed in personal capacity outside of work Meta.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

quadratic models of the Q-function [15], but that has limited representation capacity. In contrast, our approach builds upon a prior technique that uses a set of control-points" anchor Q-function approximation [5, 12], such that the maximum is at one of the control-points. This direction was largely abandoned [51] due to poor benchmark performance and scalability complex environments. We revisit this direction with series of critical design innovations that enable practical and effective continuous Q-learning for the first time with function approximation of control-points.
We propose the Q3C algorithm, Q-eaarnin for continuous control with control-points, where our contributions include: (1) combining structurally maximizable Q-function with deep learning to find maxima easily in complex continuous action spaces; (2) an improved model architecture that simplifies optimization by separating the complexity of control-point generation and value estimation; and (3) algorithmic improvements that help make the algorithm robust across different environments with various action spaces and reward scales. We evaluate our approach on a range of continuous control Gymnasium tasks [50] where Q3C is on par with the performance of common deterministic actor-critic methods. Particularly, in complex environments with constrained action spaces, where gradient-basee basedd actor-critic methods struggle, Q3C consistently outperforms existing approaches. We also conduct ablation studies to measure and visualize the impact of each design component in Q3C.
2 Related Work
Off--policy actor-critic methods are widely employed for tasks with continuous action spaces. Among them, Deep Deterministic Policy Gradient (DDPG) [30] is particularly influential. As a deep variant of the deterministic policy gradient algorithm [44], DDPG combines a Q-function estimator with deterministic actor that seeks to maximize the estimated Q-function. However, the interaction between the actor network and the Q-function estimator often introduces instability, making DDPG highly sensitive to hyperparameters and difficult to stabilize.
Various improvements to address these shortcomings have been proposed, including multi-step returns [19], prioritized replay buffers [40], and distributional critics [6]. One notable extension, Twin Delayed Deep Deterministic Policy Gradient (TD3) [LI], introduces several key modifications, such as delayed updates for the policy, target networks, and the use of clipped double Q-learning. to significantly improve the stability and robustness of training, making TD3 one of the most widely adopted algorithms for continuous control. However, TD3 often performs suboptimally in environments with complex or non-convex Q-function landscapes. To address this, Jain et al. [24] propose augmenting TD3 with a successive actor framework that trains multiple actors to explore different modes of the Q-function. However, this approach introduces additional computational overhead and inference-time latency. Furthermore, the expressiveness of the actor's parameterization may still be insufficient to capture complex optimal action distributions. Finally, Soft Actor-Critic (SAC) [16] learns a stochastic policy that maximizes an entropy regulaiieee Q-function.
Evolutionary algorithms like simulated annealing [27], genetic algorithms [47]. tabu search [14]. and cross-entropy methods (CEM) [9] are deployed for global optimization in RL but often suffer from premature convergence at local optima in environments with complex Q-functions [20]. CEM approaches such as QT-Opt [25, 28, 26], GRAC [43], CEM-RL [35], and Cross-Entropy Guided Poli-
cies [46] are relatively effective but additionally introduce high computational workload, struggling with high-dimensional action spaces as sampling becomes progressively more inefficient [53].
-asue methods, as opposed to actor-critic methods, suffer from the challenge of finding the maximal action in continuous domains. Prior work approach this problem via various optimization techniques to minimize the Bellman residual [4], including mixed-integer programming [39], the
cross-entropy method [25]. and gradient ascent [3]]. While these approaches can be effective, they often result in either suboptimal local maximization or are computationally impractical [24]. Another line of work discretizes the action space and performs Q-weighted averaging over discrete actions [32]. Though effective in low-dimensional settings, this approach does not scale well to nigh-dimensional tasks. Gu al. [15], Wang al. [52] propose a different approach by constructing the state-action advantage function in quadratic form, allowing for analytical solutions to directly obtain the Q-value maximum. However, this formulation restricts the Q-function's 's expressivity to be quadratic in action space, restricting its practical applicability. Amos et al. [2] make similarly limiting assumption that the fufucctio universally convex in actions and use a convex solver for
action selection. Most similar to our approach, Asadi et al. [3] proposed RBF-DON, which learns

---

Q-functions without an actor by using a deep network with radial basis function (RBF) output layer, enabling easy identification of the maximum action-value. However, standard RBF interpolation does not guarantee that the maximum lies at a basis centroid. It has implicit smoothing trade-off between how expressive the Q-function can in modeling various local optima and how close basis centroid is to the true maxima. In contrast, the control-poine approximation framework of Q3C alleviates this trade-off, being notably more adaptable to arbitrarily shaped Q-functions, including those with multiple modes non-convexities. Thus, Q3C exhibits the ability to find the accurate maximizing action, which was the primary bottleneck of prior value-based methods.
Our work builds on less known but promising framework that approximates Q-values using control- points, also referred as wire-fitting interpolators [5, 12, 13]. While earlier studies included this approach as a baseline, they reported poor or unstable performance [51, 31]. In this work, we revisit this framework and propose several novel modifications that are crucial to stabilize learning, enabled by advances in deep learning and reinforcement learning. We demonstrate that, with these critical additions, control-point-basse based Q-function approximators become competitive with sate-of-the-an reinforcement learning algorithms in standard benchmarking environments and outperform them on constrained action space environments.
Preliminaries
3.1 Q-Learning
We consider a Markov Decision Process (MDP), where at each time step t, an agent observes the state 8 S and takes an action a₂ AA in the environment &. The agent receives a reward r(st, az), and transitions to the next state 8141 E S. The objective the agent is to learn a deterministic policy S A that maximizes the cumulative return discounted by factor of [0, 1) per time- step, R2 Eis y (r-), a:). Q-learning [48, 33] solves this by defining the optimal action-value function Q" (s(s a) as the maximum expected return achievable after taking action a at state 8,
(' a) max E [Re St = =s at a, a.
The greedy policy optimal, Λ² arg max Q" a), where Q" follows the Bellman equation,
(s. = EgIn r r y max Q"((s' a') ss
a'EA
Value iteration algorithms like Deep Q-Networks (DQN) [33] apply the Bellman equation as an iterative update to train a function approximation Q with weights 6, called the Q-network.
( mix d':0) Q( ((s.a: ) -
Letlmmnn (8)
d'EA Q(s
where p(s,a) a) is the probability distribution over states and actions in the collected training data.
3.2 Maximization in Q-Learning
The two max operations involved in deriving the greedy policy from Eq. (1) and evaluating the next state's 's best Q-value in Eq. (2) are casily computable in discrete action spaces by evaluating the Q- values of every action [33, 18, 34, 17]. However, they become infeasible in continuous action spaces. To address this, the deterministic policy gradient algorithm [44, 30, 11] learns an "actor" policy in addition to the "critic" Q-function. The actor is trained with gradient ascent of the Q-function landscape, resulting in a greedy policy that finds locally optimal actions. Furthermore, the max in Eq. (2) is avoided by replacing the max-Q formulation with the expected-Q of the actor [48, 44, 30].
However, the introduction of an additional actor brings several downsides: (i) additional hyperpa- rameters for the actor network, making reproducibility a challenge due to interaction between the learning of the actor and critic [21], (ii) computational overhead increased training memory, and (iii) actors trained with the policy-gradics can only find locally optimal actions [24].
3

---

4 Q3C: Q-learning for Continuous Control with Control-points
We aim to simplify Q-learning in continuous action spaces by proposing a lightweight actor-free approach that enables the maximization required in Equations (1) and (2). Our key insight is to learn structurally maximizable representation of the Q-network that alleviates the need for an actor. Previously, Baird and Klopf [5] introduced wire-fitting, general function approximation system that uses finite number control-points for fitting the Q-function. This design reduces the problem of finding the maximum over the entire action space to finding the maximum among the control-points. However, the direct application of wire-fitting to deep neural networks has led to poor results [51, 31]. We identify the crucial challenges that have limited deep wire-fitting and propose approach, Q-lcarning for continuous control with control-points (Q3C), to address them.
4.1 Function Approximation Using Control-points
The wire-fitting framework [5] facilitates finding the maximum of function U + R by maintaining a set of N control-points U. = {ui,-.. uv} and their corresponding f values V {yI... ). The value fu) is evaluated as inverse weighted interpolation smoothed with c 0,
E, U:W where w, u E, w.
1
f(u)
u,l + C (Ymax - Mi)
As shown in Figure 1, this formulation guarantees that f(u) the maximum of the function f is attained one of the control-points, E Ue such that i = arg max yk-
{us, stt)
{ur sta]
(ste yn
The smoothing parameter in our formulation affects the function's ' value only at non-maximal control-points. With small smoothing, the approximated function passes through all control-points, whereas with a large smooth- ing parameter, the function's value may differ slightly at some control-points. However, in all cases, it is still guaranteed that the maximum of the function lies at control-poine an advantage in highly non-convex Q land- scapes.
{MA, sc)
[ug 31
Control Points
U
Figure 1: Wire-fitting function. A Q- function represented by an interpolation of learnable control-points is structurally maximized at one of the control-poing
This function approximation is extended to model Q- functions Q(s, a) with the goal of finding the maximiz- ing action a for any given state s. Herein, the control- points and their values are learned as functions of As(s) (s) (s)s and Qc(s) {Q)(s).... (s)}. These sets of functions Ag and Q. are modeled with function approximations like neural networks, for example, a shared neural network backbone with 2N scalar output heads for (=))) and (s)s).
((s. a)= E. Qi(s) as) Σ, (,,)),
where (s) |a (s)
max (Qmax- (s))
This representation of the Q-function enables Q-learning in continuous action spaces with the ability find the maximal action via direct maximization over scalars (s).
arg max Q(+, a) = â, where j = arg max Q;-
(5)
In Appendix B, we prove the following proposition that replacing a neural network Q-function with wire- fitt-fitting interpolation in the Q-function preserves its universal approximation ability.
Proposition. Let A be compact action set and s be given state. For any continuous Q-function (a) == Q(s, a) and any > O, there exists a finite set of control-points (a,,..., aN} C A with corresponding values y; (a( such that the wire-fitting interpolator
f(a) = ΣN 11 Ui wa(a)
1
(x(aa
la asl max (vv v))' satisfies
1=1 wi(a)

---

Control Point
Wire-Fitting Interpolator
Generator
Q(+,a)
x(1) argnns, Qts, argunna, C
S
gΦ
(6, Oo)
Q(s,a)
h
(Gr Qu)
(a., Q1)
Q Estimator
t a
Figure 2: Q3C Architecture consists of components: (i) a control-point generator estimates the representative N control-point actions, (ii) a Q-estimator estimates the values of these N points, and (iii) a wire-fitting interpolator estimates the inverse-distance weighted Q-value of the given action.
- -s11s sup ((a) - Qs(a)) < aEA
Hence, the wire-fitting formulation can approximate any continuous (s::)) arbitrarily well.
Extension Deep Reinforcement Learning. Prior attempts on adapting the above wire-fitting framework to deep neural networks struggled on digh--dimensinal continuous control [31]. This is in line with prior works deep RI. [33, (0) that show the need for specialized stabilization techniques in the RI. algorithm and architecture. Therefore, we propose to (i) build on the exploration and function approximation tools of the TD3 algorithm [11] which analogously learns deterministic policies for continuous control, (ii) reduce optimization complexity of the control-point architecture (Section 4.2), and (iii) improve the robustness training across different environments (Section 4.3).
Building on TD3. We explore with Gaussian-noise added over the deterministic action selected by greedy maximization. We augment the training with twin Q-networks avoid overestimation and target networks to make the learning targets stationary in Eq. (2). To encourage generalization, we apply target policy smoothing to the maximizing action a' E A, obtained via Eq. (5).
4.2 Reducing Optimization Complexity of Control-points
While building TD3 provides the basic framework for deep wire-fitting, the control-point archi- tecture still suffers from complexities of optimization: (1) control-point's Q-value Qi(s) is learned without conditioning on its corresponding action a,(s) and must implicitly learn it in neural weights, and (2) Q-function's 's global expressivity with many control- poirotssoints comes the cost of local learning inefficiency because the backpropagated gradient is spread over all the control-poine
Action-conditioned Q-yalue Generation. Independently predicting the Q-values (s)s) from their control-points àz(s) lets the network assign very different values to identical or near-identical actions, destabilizing learning. We instead structure the control-point architecture into 2 stages; a control- point generator (s(s) outputs N control-point actions (s)s), for which a separate Q-estimator obtains the corresponding Q-values, Q((s) = ( à.). The idea of evaluating various control-point with the same Q-estimator is motivated from prior work action representation generalization [22, 23], and ensures consistency Q- Q-values and simplifies learning (see Figure 2).
Relevance-based Control-point Filtering. To enforce stricter locality than the soft weighting in Eq. (3), we evaluate Q(s, a) only using the top & control-point weights w; w; a) for the action discarding the remaining N k. This hard filter removes spurious influence from distant points, sharpens the local value landscape, and yields a simpler learning task. Empirically, choosing < N consistently improves both training stability and final performance across benchmarks.
5

---

Algorithm Q3C
Initialize control-point generators Gor Ioz and Q estimators hig 1a22 Initialize target networks 1/ $ Vi, ti
Initialize replay buffer
Define calc_q_value a, Φ, (b) < following Eq. (4)
Define get_action (s) t- Eq (5)
for t 1 do
Select Action
Select action with exploration noise and observe reward r and new state s'. Store transition tuple (s, a, s') in replay buffer
Update
Sample mini-batch of N transitions (s, u, from replay buffer
f get get_actton (s') + e, where 6 clip(N 0, 7), -6, +6)
(s + salc_q_valuee (8', 04 vi for i = 1,2
y f + 1.2
Qi(s, a) < calc_q_value a, d. th:) for = 1,2
Bellman (1., *h,) <- N -N- - (s. L Experation (0i) t Eq. (6) Calculate losses L( 12)) Rellmann Ois v/) Experation (0:) for = 1,2
Update parameters di, be for i = 1.2 if l mod
then Update target networks:
+ To: + (1 - 7) +)
1. - TW, , + (1 T 7
end
end for
4.3 Robustness of Training across Different Tasks
The performance of the wire-fitting framework may still degrade when reward scales, action ranges, or environment dynamics differ markedly across tasks. be a robust algorithm, Q3C must (i) maintain a diverse, well-spread set of control-points so the critic models representative slice of the action space in every state, and (ii) normalize the scales of the action distance and Q-value difference despite varying reward magnitudes across states and tasks.
Control-point Diversity. While action-conditioned Q-value generation disentangles Q-values from control-point generation, it does not ensure that control-points cover the action space. In practice, observe that control-poine often cluster the boundaries, limiting the expressiveness of the learned O-function (see Figure 6). While policies acting at extremes can perform well in certain scenarios [42], more uniformly distributed control-points offer richer representations and improved robustness when applying Q3C. To spread them out, we add pairwise separation loss with æ < 1:
1
Luegaration (0) N(N - 1)
where N = # of control-points
(6)
which is minimized when the points are uniformly dispersed. We add this loss to the Bellman loss in Eq. (2) weighted by hyperparameter, À (0, 1].
Scale-Aware Control-points and Q-values We normalize action spaces to [-1, 1] and use a tanh nonlinearity in the control-poin generator. In Eq. (4), while the action distances |a (s) are bounded in [0, 1]. the Q-value difference term e; (Umax Bt) can vary widely between environments, even with shared c, c. We (i) rescale each state's control-point values to Q; E [0, 1] within the QuiQuin and (ii) anneal the smoothing factor c; exponentially. This prevents Quia
weight term only, Q, = Qun Qumn
large rewards from overwhelming spatial information and keeps learning robust across diverse tasks. Combining all the aforementioned modifications, we present Q3C, pure value-based reinforcement learning algorithm for continuous control in Algorithm 1.

---

Table Standard Environments. Final performance on Classic Mujoco Environments shows that Q3C is comparable to TD3 while outperforming other baselines (mean + std).
Environment
NAF
Wire-Fitting
RBE-DQN
Q3C
Pendulum-vl
144.61 25.28
252.36 56.66666 63
3351.52 390.18
1443
23.86
159.53
16,
Swimmer-v4
300.70 + 125.64
20.63 + 12.62
313.63 + 106.23
92.38 + 44.97
316.40
14.75
Hopper-y4
3113.41 + 888.17
500.80
240.93
1957.50
1127.06
2189.37
1093. 13
3206.14 1: 407
Bipeda//Waller-v3 Walker-v3
309..2 + 10.94
-108.19 34.76
70.01 100.28
265.35
74.38
290.11
26.43
Walker2d-v4
4770.82 8770.82- + 560.16
2179.56 + 1034.59
2462.30 = 1095 41
781.58 +: 252 66
3977
+: 879.70
HaltCheetahh
0984.74 1076.58
3531.50
802.84
7546.23
6175.57
9044.93
9468.66
Ant-v4
5167.66 66 + 673.44
18. + 0.30
1154.59
+ 420.92
1674.03
964.60
3698.41 + 1314.88
Pendulum-vv
Swimmer-v4
Hopper-v4
BipedalWalker-v3
Averge Betur
-500
300
3000
200
-1000
100-
1530
10000
70000
10000 40000 N000
0.0
0.2
0.4
0.6
0.8
1.0
0.
0.7
0.4
0.6
0.0
0.0
0.4
066
0.6
Clobal Step
Global Step
Clobal Step
Clobal Step
Walker2d-ve
HaliCheetah-va
Ant-v4
10000
7500
5000
2500
2000
0.4
Global Step
Global Step
Global Step
TD3
NAF
Wire-Fitting
RBF-DQN
QBC
Figure 3: Standard Environments. Q3C comparable in performance TD3 and outperforms
other actor-free value-based baselines in most environments.
5 Experiments
We conduct experiments across range of continuous control tasks and baselines. Our goal is to eval- uate both performance in standard benchmark environments and robustness in settings with complex, multimodal Q-functions. Further details about the environments can be found in Appendix E.
Environments. We evaluate our method several tasks from the Gymnasium suite [50]. Specifically, we use Pendulum, Swimmer, Hopper, Bipedal Walker, Walker2d, Half Cheetah, and Ant tasks to cover a range task difficulty. all experiments, we use state-based observations and do not modify the reward function of the tasks. In addition, similar to prior work [24, 39], we create restricted versions of a subset of the environments, namely Inverted Pendulum, Hopper, and HalfCheetah. We follow the same procedure Jain et al. [24], where the action space is restricted by defining set of hyperspheres as the valid section the space. The actions outside this space are defined as invalid and have no effect on the environment. These restricted settings are designed to induce non-convex Q-functions, where local maximization fails to find the optimal action.
Baselines. We primarily compare Q3C against deterministic RL algorithms, including both actor- critic and value-based methods, as our approach results deterministic policy and follows TD3's exploration. Whenever possible, we use the official implementations from the stable- balle-baselines3 [37] and tuned hyperparameters from rlzoo3 [36] to ensure optimized baselines. We implement Q3C using the same
backbone to minimize implementation-leww differences between methods.
Q3C (Ours): Q-leaming for continuous control that learns structurally maximizable Q-function via a set of learned control-points with improved optimization and robustness for deep networks.
Wire-Fitting [12, 13]: Vanilla wire fitting Q-function approximation without our contributions.
7

---

Table Restricted Environments. Q3C outperforms TD3 and actor-free baselines (mean std).
Environment
TD3
NAF
Wite-Fitting
RBF-DON
Q3C
LnveredPendulummoxx
782.76 + 348.92
909.72
120.14
386,38
307.57
862.02 398.
10000 4337
HaltCheetahBox
2276.70 + 2036.39
4867.06 06-- 1487.69
- -2139.78 4702.22
2238.35 38 + 3227 .1
1503.33 1170.00
HopperBox
1406.83 1162.72
461.54 389.04
796.78
InvertedPendulumboxxe
HaliCheetahBox-v4
HopperBox-và
Average Return
250
4000
2000
500
2000
250
0.00
0.2
0.4
0.6
0.8
1.0
0
0.00
0.5
1.0
1.5
2.0
Global Step
Global Step
Global Step
leb
TD3
NAF
Wire-Fitting
RBL-DQN
Q3C
Restricted Environments. Q3C consistently outperforms TD3, RBF-DQN, and Wire- Fitting in restricted environments and largely outperforms NAF, Q3C converges at a higher reward with high stability while other algorithms stuck at local optima.
Figure 4:
TD3 [11]: A state-of-the-arr actor-critic algorithm that improves stability in continuous control by using double Q-learning, delayed policy updates, and target policy smoothing.
NAF [15]: A value-based algorithm for continuous action spaces that assumes the Q-function to quadratic form, allowing the optimal action be computed analytically without an explicit actor.
RBF-DQN [3] A deep Q-Jearning algorithm variant that uses radial basis functions to approximate Q-functions in continuous action spaces.
Evaluation Scheme. We train 10 different random seeds for each algorithm. Throughout training, we evaluate each method every 10000 steps by running 10 rollout episodes and report the average return. The curves correspond the mean, and the shaded region to one standard error across 10 trials.
5.1 Standard Environments
We present our quantitative results in Table and learning curves in Figure 3. Q3C achieves performance comparable to TD3 on most tasks, with the exception Ant-v4, where it performs suboptimally. Compared to the vanilla wire-fitting baseline, which lacks our proposed additions, Q3C achieves substantial improvements across all benchmarks, highlighting the impact of its components.
Other value-based algorithms than Q3C struggle in most environments. NAF consistently under- performs, likely due its restrictive inductive bias that the Q-function quadratic in the action space, which does not hold for complex control problems. RBF-DQN similarly achieves suboptimal performance, possibly because its function approximation cannot reliably recover the true maximizing action without excessive smoothing. Furthermore, it requires large number of centroids (~ 100) to achieve sufficient Q-function expressivity, limiting scalability to high-dimensional environments.
These results show that Q3C is viable alternative to deterministic actor-critic methods in common RL benchmarks and the fate-offtheearr actor-free method for continuous action spaces.
5.2 Restricted Environments
Environments with restricted action spaces help evaluate robustness of Q3C to complex Q-value landscapes [24]. These constraints induce sharp discontinuities the Q-values of nearby actions, because they result in significantly different outcomes and returns. As consequence, the Q- function becomes highly non-convex and difficult to optimize. Our experiments on restricted environments presented in Figure 4 and Table 2 show that TD3 performs significantly worse in

---

Table Ablations of Q3C. Final performance comparison shows every component is necessary to improve the converged performance of Q3C (mean + std).
Algorithm
Hopper-v4
BipedalWalker-v3
Walker2d-v4
HalfCheetah-ve
Q3C
3206.14 407.23
290.11 1: 26.43
3977.39 + 879.70
9468.66 1 949.01
Q3C CondQ
2329.9 + 610.1
286.0 1 46.7
3614.3 + 488.7
8385.9 + 564.9
Q3C Ranking
3036.5 371.4
179.8 187.3
3167.5 1167.2
8960.9 = 519.6
Q3C Div
1921.1 + 1117.9
-67.8 = 119.6
3174.3 = 1194.7
5282.7 = 1116.2
Q3C Norm
2915.3 + 366.6
261.5 = 83.4
2880.1 = 1327.5
8745.5 = 529.4
Wire- Wire-Fitting
1987.50 + 1127.06
70.01 100.28
2462.30 = 1095.4
7546.23 1234.31
Hopper-v4
BipedaWWalller-33
Walker2d-v4
HaliChectah-vd
Average Retur
2000
4000
3/00
5000
1000
1000
2340
Step
lxs
Global Step
Stop
Step
Q3C-CondO
Q3C-Div
QGC- Ranking
Q3C-Nerm
Wire-Fitting
QBC
Figure 5: Ablations Q3C. Every component of Q3C is validated for importance. Individual Q3C components complement each other and combined Q3C model visibly outperforms ablations.
restricted environments than their unrestricted counterparts. This is expected since gradient-based optimization of the policy can get stuck at local optima highly non-convex Q-functions. In contrast, Q3C consistently outperforms TD3 by avoiding restrictive policy parameterizations and instead leveraging direct maximization over learned Q-function.
Q3C similarly outperforms other value-based algorithms. The vanilla wire-fitting algorithm achieves minimal to performance in the restricted environments, highlighting the necessity of our additional design components. RBF-DQN and NAF achieve moderate performance but are still suboptimal. This is probably due to similar drawbacks to those outlined in Section 5.1 for standard environments, but are exacerbated with the increased complexity restricted environments. NAF performs marginally better than Q3C in HalfCheetahBox-v4 suggesting that approximation with quadratic function might be enough to sufficiently represent the Q-function in this environment.
Overall, these results highlight Q3C's ability to handle discontinuities in the action space. We also present extended comparisons against other popular RL algorithms Appendix D.2.
6 Ablations
To understand the contribution of each component of our method, we conduct ablation experiments by selectively removing one component time from Q3C. The ablated components include:
Q3C without conditional Q-value generation (Q3C-CondQ)
Q3C without control-point diversification (Q3C-Div)
Q3C without ranking of relevant control-points (Q3C-Ranking)
Q3C without normalization of Q-value differences in the wire-fitting kernel (Q3C-Norm)
We evaluate these variants on four environments Hopper, BipedalWalker. Walker, Walker2d, and HalfChee- tah-and and report learning curves in Figure 5, and the corresponding final reward values in Table 3. For reference we also include the vanilla wire-fitting approach in the plots. Further ablations on the design choices can be found in Appendix C.
Across all tasks, full Q3C outperforms all ablated variants, highlighting the importance of each component. While the impact of each feature varies by environment, removing any single component

---

Q3C without action conditioned Q-value and Control Point diversity
Q3C without Control Point divensity
Q3C
2000
00.0
30.5
0
200
2755
2000
2312
20
20
-10
-15
60
C. delendevattn
80 (loestin
Figure 6: Learned Q-Function Visualizations of Key Q3C Component Ablations. Conditional Q- VaVuuee Architecture and control-point Diversity Loss are Essential For Reasonable Q-Functions
generally results in performance and stability degradation. Notably, even the weakest ablated variants outperform vanilla wire-fitting, demonstrating the effectiveness of our contributions.
One exception is the Q3C-Div variant: in certain environments, removing the control-point diver- sification loss leads to a substantial performance drop. We attribbte this to the underlying Q-value landscapes of those environments, and the role of diversification in ensuring sufficient action space coverage, which is essential for the other components-such as relevance filtering and Q-value normalization--to operate effectively.
6.1 Visualizing the Effect of Q3C Components
As seen in Figure 6, without explicit constraints on control-points (left), the learned control-points suffer from stacking, regression to corner action spaces, and noticeably minimal distribution (Sec- tion 4.1). With the architectural modification to condition the Q-values on the control-poing (center), the control-points still stack towards the corners, but all control-points with the same action vector now retain the same Q-value, which is a key improvement in consistency (Section 4.2). When extend- ing this revised architecture with control-point diversity (Section 4.3), Q3C achieves both explicit consistency across all control-points and a solid control-point distribution that prevents premature collapsing endpoints and encourages expressivity.
Conclusion
In this work, we introduced Q3C, an actor-free Q-Learning approach for off-policy reinforcement learning in continuous action spaces. Our approach builds on the control-point function approximator, augmented with several key architectural innovations, such as action-conditioned Q-value generation, gelevance-based control-poin filtering, encouraging control-point diversity, and normalized scaling for robustness. We demonstrated that these additions enable the function approximator to match or exceed the performance of state-of-the-art online RL algorithms on standard benchmarks. Moreover, in constrained environments where only a subset of the action space is admissible, Q3C achieves robust performance while TD3's gradient maximization often fails.
Limitations and Future Work. Q3C simply adopts TD3's exploration scheme, and its sample efficiency can lag behind other baselines in certain environments. More work on exploration strategies could be promising, such as Boltzmann over the control- control-point point values. As a hybrid between DQN-style methods and actor-critic algorithms, Q3C offers flexibility to incorporate enhancements from both paradigms. Future work could explore integrating sample-efficiency improvements such as n-step returns [18], prioritized experience replay [40], or using batchnorm layers in the critic network instead of target networks [7]. Additionally, adapting the control-pont approximator to the offline RL setting could be an exciting direction: due to its inherent constraints on Q-value interpolation, may offer natural mitigation against overestination, common challenge in offline learning. Finally, we only employ Q3C on deterministic Q-learning, however, future work can investigate extending stochastic policies where the control-poin architecture models soft-Q function like SAC.
10

---

Acknowledgments
We thank the USC Center for Advanced Research Computing (CARC) for providing us with compute resources. Urvi Bhuwania was supported by CURVE fellowship from the USC Viterbi School of Engineering. This project was partially funded by Airbus Institute for Engineering Research (AIER). We appreciate the valuable discussions with anonymous reviewers and members of the USC Lira Lab. thank Norio Kosaka for starter code of restricted environments.
References
[1] M. M. Afsar, T. Crump, and B. Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys, 55(7), 2022.
[2] B. Amos, L. Xu, and J. Kolter. Input convex neural networks. International Conference on Machine Learning (ICML), 2017.
[3] K. Asadi, N. Parikh, R. E. Parr, G. D. Konidaris, and M. L. Littman. Deep radial-basis value functions for continuous control. In AAAI Conference Artificial Intelligence, 2021.
[4] L. Baird, Residual algorithms: Reinforcement learning with function approximation. In International Conference Machine Learning (ICML), 1995.
[5] L. C. Baird and A. H. Klopf. Reinforcement learning with nigh-dimensional continuous actions. Wright Laboratory, Wright-Pattersen Air Force Base, Tech. Rep. WL-TR-93-1147, 15, 1993.
[6] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning (ICML), 2017.
[7] A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranahvvili, T. Brox, and J. Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In International Conference on Learning Representations (ICLR), 2024.
[8] Y. Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh. A lyapunov- based approach to safe reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[9] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. A tutorial the cross-entropy method. Annals of operations research, 134(1), 2005.
[10] R. Franke and G. Nielson. Smooth interpolation of large sets of scattered data. International journal for numerical methods in engineering, 15(1)), 1980.
[11] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning (ICML), 2018
[12] C. Gaskett, D. Wettergreen, and A. Zelinsky. Q-learning in continuous state and action spaces. In Australasian Joint Conference on Artificial Intelligence, 1999.
[13] C. Gaskett, L. Fletcher, and A. Zelinsky. Reinforcement learning for a vision based mobile robot. International Conference on Intelligent Robots and Systems (IROS), 2000.
[14] F. Glover. Tabu search: tutorial. Interfaces, 20(4), 1990.
[15] S. S. Gu, T. P. Lillicrap, I. Sutskever, and S. Levine. Continuous deep q-learning with model- based acceleration. In International Conference on Machine Learning (ICML), 2016.
[16] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (ICML), 2018.
[17] H. v. Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In AAAI Conference on Artificial Intelligence, 2016.

---

[18] M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018.
[19] D. Horgan, J. Quan, D. Budden, G. Barth-Maron, M. Hessel, H. van Hasselt, and D. Silver. Dis- tributed prioritized experience replay. In International Conference on Learning Representations (ICLR), 2018.
[20] Hu, M. C. Fu, and S. Marcus. A model reference adaptive search method for global optimization. Operations research, 55(3), 2007.
[21] R. Islam, P. Henderson, M. Gomrokchi, and D. Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv: 1708:04183 2017.
[22] A. Jain, A. Szot, and J. Lim. Generalization to new actions in reinforcement learning. In International Conference Machine Learning, pages 4661-4672 PMLR, 2020.
[23] A. Jain, N. Kosaka, K.-M. Kim, and J. Lim. Know your action set: Learning action relations for reinforcement learning. In International Conference on Learning Represeniations, 2021.
[24] A. Jain, N. Kosaka, X. Li, K.-M. Kim, E. BIysk, and J. J. Lim. Mitigating suboptimality of deterministic policy gradients in complex g-functions. Reinforcement Learning Conference (RLC), 2025.
[25] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakr- ishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. In Conference on Robot Learning (CoRL), 2018.
[26] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2 002.100..2211 12, 2021.
[27] S. Kirkpatrick, C. D. Gelatt Jr, and M. P. Vecchi. Optimization by simulated annealing. science, 220(4598), 1983.
[28] K.-H. Lee, T. Xiao, A. Li, Wohlhart, I. Fischer, and Lu. Pi-qt-opt: Predictive information improves multi-task robotic reinforcement learning at scale. In Conference on Robot Learning (CoRL), 2023.
[29] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1), 2016.
[30] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016
[31] S. Lim, A. G. Joseph, L. Le, Pan, and M. White. Actor-expert: A framework for using action-value methods in continuous action spaces. arXiv preprint arXiv:1810.091003, 2018.
[32] J. d. R. Millan, D. Posenato, and E. Dedieu. Continuous-action q-learning. Machine Learning, 49, 2002.
[33] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5644 2013.
[34] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
[35] A. Pourchot and O. Sigaud, Cem-rl: Combining evolutionary and gradient-based methods for policy search. arXiv preprint 180.0012220002
[36] A. Raffin. RI basclines3 zoo, 2020.
12

---

[37] A. Rafin, A. Hill, A. Gleave, A. Kanervisto, M. Emestus, and N. Dormann. 3table-baselines: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268), 2021.
[38] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems (RSS), 2018.
[39] M. Ryu, Y. Chow, R. Anderson, C. Tjandraatmadja, and C. Boutilier. Caql: Continuous action q-learning, In International Conference on Learning Representations (ICLR), 2020.
[40] T. Schaul, J. Quan, I. Antonogiou, and D. Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016.
[41] Schulman, Wolski, Dhariwal, A. Radford, and Klimov. Proximal policy optimization algorithms. arXiv preprint 3.0.006777.03477 2017.
[42] T. Seyde, I. Gilitschenski, W. Schwarting. B. Stellato, M. Riedmiller, M. Wulfmeier, and D. Rus. bang-bang control all you need? solving continuous control with bernoulli policies. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[43] L. Shao, Y. You, M. Yan, S. Yuan, Q Sun, and Bohg. Grac: Self-guided and self-regularized actor-critic. In Conference on Robot Learning (CoRL), 2022.
[44] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning (ICML), 2014.
[45] D. Silver, J. Schrittweeser, Simonyan, I. Antonoglou, Huang, A. Guez, T. Hubert, Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550
(7676), 2017.
[46] R. Simmons- Edlerr, B. Eisner, E. Mitchell, S. Seung, and D. Lee. Q-learning for continuous actions with cross-entropy guided policies. arXiv preprint arXiv:1903. 2019.
[47] M. Srinivas and L. M. Patnaik. Genetic algorithms: A survey. computer, 27(6), 1994.
[48] R. S. Sutton and G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998.
[49] E. Todorov, T. Frez, and Tassa. Mujoco: physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
[50] M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goulão, A. Kallinteris, M. Krimmel, A. KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17022 2024.
[51] H. Van Hasselt and M. A. Wiering, Reinforcement learning in continuous action spaces. In IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, 2007.
[52] Wang, H. Li, and C.-Y. Chan. Quadratic q-network for learning continuous control for autonomous vehicles. arXiv preprint arXiv: 19/2.0004//0044 2019.
[53] M. Yan, A. Li, M. Kalakrishnan, and P. Pastor. Learning probabilistic multi-modal actor models for vision-based robotic grasping. International Conference on Robotics and Automation (ICRA), 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Paper's contributions and scope are summarized in detail in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The authors explain the limitations of the proposed work at the end of the paper, in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: Supplementary.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All the details required to reproduce the results provided in the main paper and the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The authors release their code with sufficient instructions reproduce the experiments.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cd (oglic/guides/Coeeeee
srriiissssooooliiio onPolicy iyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctppblicc////ccccccccccccc...lt wwwgc//guss///ee/ee /Cars//oddSubmmim ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details to understand and reproduce the results are provided in the main paper and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All of the results presented in the paper provide information about the statistical significance of the experiments with plots including standard deviation across runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The provides information about the type of compute workers CPU or GPU, internal cluster used for running the experiments the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http::///euuiseeeeeee http://p......pblceccccccceceeee
Answer: [Yes]
Justification:
The paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the supplementary.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The authors stated which version of the asset is used and cited the original papers that produced the code package.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo//aaooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htopp://nnerips... httpe//rppp.ccooooorrer/eee/e/// for what should or should not be described.
20

---

A Impact Statement
This paper introduces purely value-based reinforcement learning framework for continuous control, aimed at improving both stability and tractability in complex action spaces. By avoiding restrictive policy parameterizations and enabling direct action selection through structured Q-function, our method is particularly well-suited for tasks with constrained or safety-critical action spaces. While our approach improves robustness and computational efficiency over existing baselines, does not explicitly account for fairesss, interpretability, or real-world deployment risks. Care must be taken when applying this method safety-sensitive or high-stakes domains, where additional mechanisms susss-suchhaa as constraint-aware training or human oversight- -my be necessary to ensure reliable and ethical deployment.
B Proof for Proposition
We show that replacing a neural network Q-function with wire-fitting interpolation in the Q-function preserves its universal approximation ability.
Proposition. Let A be compact action set and s be a given state. For any continuous Q-function Q. == Q(s. a) and any > there exists a finite set of control-points {ā....., âN- C A with corresponding values y; = Qa((a)) such that the wire-fitting interpolator
Ex= wa(a) f(a)
wi(a) =
la a.l + C, max; (yxs y.) satisfies
EM (a)
I - = sup (()) - Q.(a)) <c aEA
Hence, the wire-fitting formulation can approximate any continuous (s, arbitrarily well.
Proof. We prove this proposition by following the classical convergence analysis of diiessne interpolation [10], which was adapted to the control-point formulation for reinforcement learning by Baird and Klopf [5].
Uniform-conee continuty radius. Because is compact and Qa is continuous, Qs uniformly contin- uous; hence there exists 8 >> such that (Q)(a) Qa (b) e/2 whenever la bj < 6.
Control-point 8-net. Choose finite 8-net J.V. A. i.c. for every A there is an index (a) with la ag-1 < Define the targets == (a:): and the positive offiets e, >> 0 (a constant choice a = suffices).
Interpolant as convex combination.
For each a E A set
Wi
(a) (a 0: + A,
A = max - yi), Aı(a) = Σn (a)
Because all (a)a) O, the (a) form partition of unity and the wire fitting interpolant is f(a) = >;>aa) y/-
Error decomposition.
Write f(a) Qa(a) Enear + Eia. where
Enear
Σ
ila-a,1<6
Eiar
Σ
((a)(Q.(a)-Q)) -
4: 120
Near indices. For the "near" set the uniform-continuity condition yields (Q.(a.) - QQ((a) < hence |neaar < e/2.
Far indices. Let R max A Q. min A Qe and Cmin min, cr For any j with la ayl 8,
Wi (a)
Gmin =: 0.
< 1 1
Wi+ (a)
21

---

It follows that E 0,,00-,,226 (2 - 0-1 Choosing 8 < Cuin Cuin Re/(2R -c) makes 0-1 < <c(22R) so that
|Earl R < c/2.
Uniform bound. Combining the two parts gives [f(a) - Qx(a)) < e for every E A; hence Il Qalloc <.
C
Implementation Details and Ablations for Q3C
C.1 Control-point Diversification Functions
In addition the separation loss defined in Eq. (6), experiment with an alternative loss formulation that explicitly encourages each control-point to maintain a minimum distance from its nearest neighbor:
N
Cmin N (0) =
Σ
min SJEN N
Iz
2,112
N
This variant is used for subset of tasks specifically, Hopper and BipedalWalker-wwe -where we empirically observed improved performance. These similar objectives aim to enhance the expressivity and robustness of the Q-function approximation by avoiding clustered or redundant control-points.
C.2 Smoothing Coefficient
We experimented with three primary methods of setting the smoothing parameter c: tuning the constant value of smoothing as a hyperparameter, learning the smoothing coefficient, and scheduling decay for smoothing.
find Figure that an exponentially decaying smoothing scheduler typically works better than learning smoothing parameter. Utilizing a scheduler induces greater stability in the smoothing parameter that allows the model to learn the Q-function and distribution of control-points more effectively. The learning process becomes less susceptible erratic shifts in the smoothing parameter.
Hopper-v4
BipedaWWallerr-3
Walker2d-v4
HaliCheetah-v4
Average Retum
2000
Xo
1:40
200
3000
300
2500
5000
10c0
1000
2000
-000
Global Step
Global Step
Global Step
Step
Learnable Smoothing
Fixed Smoothing
Decnying Smoothing
Figure Ablations of Q3C with different strategies for smoothing parameter: The method of decaying smoothing is consistently more stable as justified in section C.2.
C.3 Learning Rate Schedulers
In traditional actor-critic architectures like TD3, there is often in-built policy delay with the critic being updated more frequently than the actor. However, since Q3C employs the same network to serve as both actor and critic, our implicit policy is updated at the same rate as our critic. Thus, Q3C is more sensitive to various learning schemes to stabilize training.
We tried six different learning rate schedulers: constant, linear decay, exponential decay, inverse exponential decay, cosine decay, and one-cycle learning rate. For each scheduler except for the constant learning schedule, kept the maximum learning rate the learning rate specified in the hyperparameters and the minimum learning rate as 10% of the maximum learning rate.
Among these, we found that inverse exponential decay and cosine decay were the optimal learning schedulers across environments. For sake consistency, we apply delayed exponential decay learning scheduler for all results with the final learning rate set to 10% of the initial learning rate.
22

---

C.4 Control-points with K-Nearest Neighbors
We experimented across various control-point combinations of N (total of control-points) and (top k rankings), and found that setting N = 20 and A: 10 consistently worked best for nearly all environments. We utilize this setting as starting point for hyperparameter tuning, as balances coverage and computational cost. We then progressively increase the value of N, keeping & relatively constant, until learning optimal. We employ N 30 and & = 15 for Ant-v4, and N 70 and & = 10 for Adroit environment. However, control-point combinations, as shown in Figure 8.
algorithm is reasonably robust across different
As general rule, increasing the number of control-points N allows for greater expressivity but involves learning a more complex Q-function, which may slow down learning. Due this tradeoff, the optimal number of control-points N tends to scale more conservatively than proportional increase with action dimensionality (e.g., Adroit-and//ammer- requires 70 control-points). Increasing the value of l typically leads to a smoother and continuous Q-function across the action space, whereas a smaller value of k allows for more granular and piecewise control of the Q-function.
action space is 26-dim but only
Additionally, the conditional Q architecture enables parallclization across control-points during Q-value generation, ensuring that the parameter count does not scale linearly with N. This parallel structure supports Q3C's scalability nigh-dimensional and non-convex Q-function landscapes.
Hopper-v4
BipedalWallker-v3
Walker2d-v4
HalfCheetah-va
Average Retum
3000
3004
9020
10000
200
7500
3000
2x00
100
3000
3000
Inco
2530
-100
0000
0.25
0.50
025
1.00
0.75
0.00
0.25
0.50
0.75
1.00
0.00
0.75
1.0
Glabal Step
16
Glubal Step
1eb
Step
10:5
10:10
20:20
30:10
30:15
20:10 (Default)
Figure 8: Ablations of Q3C with different configurations for # of control-points (N), and top k ranking. Different configurations are represented as N:k the legend.
D Further Experiment Results
AdroithlandHammeervv
In addition to ablations of the core method modifications, we have provided extra results to demonstrate the effect of the specific implementation details listed above.
0.8
Success Rafe
0.4
D.1 Q3C's
TD3
Performance in High-Dimensional Action Spaces
QBC
We have conducted preliminary experiments on Adroit Hand Hammer environment [38] to evaluate Q3C's performance in high-dimensional action spaces (26-dim). As seen in Figure 9, Q3C able to match TD3, while surpassing SAC, which shows that Q3C is able to scale its learning to high-dimensional action sptdigh-dimensions environments.
0.0
Globaa Step
Figure 9: Success rate plot of
Q3C against TD3 and SAC on
D.2 Comparison Against SAC & PPO
In addition to the baselines reported in the main paper, we also compare Q3C with two widely used algorithmss, SAC and PPO [41, 16]. The results are shown Figure 10 and Figure 11, alongside TD3. PPO is an on-policy method and is therefore typically less sample-efficien than off aff-policy approaches. SAC, on the other hand, employs a different exploration scheme based on soft Q-value maximization, which differs fundamentally from the poise-based exploration used by the baselines in the main paper. Because of these differences, SAC and PPO are not directly comparable for evaluating the specific contribution of Q3C "structurally maximizable Q-function". We therefore do not expect, nor
23

---

claim, that Q3C should outperform these methods across all settings. Nevertheless, Q3C achieves competitive results in all standard environments and consistently matches or surpasses state-of-the-art performance in the restricted environments.
Pendulum-vi
Swimmer-v4
Hopper-v4
BipediJWaller--33
400-
Avenge (eeum
-500
500-
200
-100
0.0
06
Global Step
Global Step
Global Step
Global Step
Walker2d-vd
HaliCheetah-vr
Ant-v4
Averago Raturn
10000
6000
2000
2000
0
08
1.0
0.0
02
0.1
06
10
Clobal
Global
3:6
Global
TD3
SAC
PPO
QBC
Figure 10: Comparison against TD3, SAC, and PPO Standard Environments.
InvertedPPendulumBBxxoo
HoliChheetahBox-vd
HopperBux-v4
1000
3000
6000
Average
500
2000
2000
1000
0.0
1.0
Global Step
Global Step
Global Step
TD3
SAC
PPO
Q3C
Figure 11: Comparison against TD3, SAC, and PPO on Restricted Environments.
D.3 Wall-clock Time and Memory Footprint Q3C
We conducted detailed comparisons of wall-clock time and memory usage between Q3C, SAC, PPO, and TD3 on subset of evaluation environments. Experiments were run with random seeds each: restricted environments Tesla P100 GPUs and unrestricted enviromments on A100 GPUs. To ensure fair comparisons, we defined a reward threshold for each environment (Hopper: 3000, BipedalWalker: 300, HalfCheetah: 9000, InvertedPendulumBoxx 1000, HalfCheetahBox: 4000. HopperBox: 2000) corresponding to what generally considered optimal performance. We then recorded the wall-clock time required by each algorithm reach and stabilize around the threshold. Algorithms marked as "N/A" did not reach or sustain the target reward within the full training duration. Results are summarized in Table 4.
Table 4: Wall-clock time comparison of different algorithms across environments.
Algorithm
Hopper
Bipeda/Walker
HalfCheetah
InvertedPendulummox
HalfCheetahBox
HopperBox
Q3C
58.3 min
44.6 min
61.5 min
12.43 min
4
0.7 hr
32.9 min
45.6 min
39.2 min
N/A
N/A
76.3 min
50.5 min
69.2 min
16.5 min
6.6
hr
PPO
N/A
N/A
N/A
N/A
1.0
N/A
NAF
N/A
N/A
N/A
N/A
7.2
N/A

---

Table 5: Comparison of PyTorch memory allocation across algorithms.
Algorithm
PyTorch Allocated (MB)
Q3C
26.58
TD3
23.51
SAC
20.01
PPO
18.38
In HopperBox and InvertedPendulumBoox environments, Q3C consistently achieves the target reward in significantly less time than all baselines. For instance, in InvertedPendulumBox,, it reaches optimal performance in about 60% of the time required by SAC, while other baselines fail to converge. In terms of memory usage, although Q3C utilizes a parallelized evaluation of Q-values of all the control-points the increase in memory footprint is marginal and does not hinder training.
From runtime perspective, Q3C removes the actor network, which reduces computational overhead. However, this advantage is partially offset by the additional cost introduced by the interpolation mechanism. Since have used simple interpolation module, its overhead cancels out the time savings from removing the actor. We consider this an engineering limitation rather than a conceptual one: with better parallelization and optimized implementations, Q3C could realize significant runtime advantages.
More importantly. from memory perspective strates clear advantages over TD3. We analyzed memory usage across increasing network sizes while ensuring that the critic parameter count remained equivalent between Q3C and TD3, isolating the effect of the actor. As shown in Table 6, we observe increasing memory savings for Q3C as network size scales.
censpetive-ppartiullaay particularly in large-scale settings Q3C demon-
Table 6: Parameter counts and memory usage of Q3C and TD3 across model scales.
Model scale*
Q3C params
TD33 params
Q3C memory (MB)
TD3 memory (MB)
MeMorr saved
Small
37M
52M
341
400
14.8%
Medium
135M
226M
707
1,057
33. 1%
Large
538M
906M
2,245
3,675
38.9%
X-Large
1.57B
2.65B
6,212
10,348
40.0%
Mapping of Layer Sizes: Small + 1,024 control-points, Medium + 2,048, Large + 4,096, X-Large - 7,000.
D.4 Augmenting Q3C with Cross-Q
While Q3C generally converges to competitive final performance, observe that it occasionally lags behind other baselines in terms of sample efficiency. Recent work by Bhatt et al. [7] ad- dresses this issue in actor-critic settings by removing the target network and introducing batch normalization in the critic (and optionally in the actor), resulting in improved sample efficiency. Motivated by this, we also tried adopting a similar modification into our framework. Specifically, we removed the target network and applied batch normalization within the Q-value generation network. We evaluated this variant on the Walker2d task. Our preliminary results indicate that this approach improves sample efficiency during the early stages of training; however, the final performance remains comparable to the original Q3C. Improving sample efficiency remains an important direction for future work. We plan to further explore architectural and algorithmic modifi- cations to Q3C that can accelerate learning while preserving the stability and performance of the full model.
Walker2d-v4
4100
Average Returnt
3000
1000
Q3C CrossQ
Q3C
0.0
0.2
04
0.s
os
1.
Global Step
Figure 12: Comparison of Q3C with Q3C + Cross Q style modifi- cations
25

---

E
Environment Details
We test Q3C and our baselines across 7 Gymnasium environments [50, 49] and custom restricted environments [24]. The restricted environments are variations of their corresponding standard versions where the action spaces are limited to a series of hyperspheres within the region to create complex Q-function landscapes. The traditional control environments are described in Table and Figure 13.
Pendulum Bipedal Walker
Hopper
Swimmer
Walker2d
HalfCheetah
Ant
Figure 13: Visualizations of the testing environments.
Table 7: Environment Summaries with Observation and Action Spaces
Environment
Summary
Observation Action Space
Pendulum-vl
pendulum starts at random angle must
3D obs ID cont act
swung up and halanced upright using torque.
Hoopper-v4
A one-legged robol must learn hop forward without falling.
11D obs 3D act
HalfCheetah-vd
A 2D hipedal rohot with a flexible spine must learn to
17D obs 6D cont act
run forward.
Bipkda/Walker-v4
two-legged robot must learn to walk across rough
24D obs 4D cont act
Swimmer-v4
2D snake-like robot must swim forward in viscous fluid.
8D obs 2D cont act
Walker2d-vd
A 2D rohot with two legs must learn to walk forward
17D obs 6D cont act
while keeping balance.
Ant-v4
four-legged robot (quadruped) must learn move
11ID obs cont act
forward stably 3D.
E.1
Restricted Environments
The restricted environments are derived directly from their standard MuJoCo counterparts. The observation and action dimensions, environment goals, and transition dynamics remain identical between the standard and restricted versions. The only modification lies in the admissible action space. However, as demonstrated Figure 14, this modification alone is sufficient to introduce significant irregularities into the optimal Q-function.
Following prior work, the restricted action space constructed as a union of hyperspheres embedded in the original action space. The volume of these hyperspheres is scaled by coefficient k, which determines the degree restriction. A larger и corresponds to larger hyperspheres and thus a less constrained action space, while smaller induces tighter restrictions by shrinking the valid regions. For some environments, we lower the coefficient from the original implementation achieve an more restricted action space, detailed in Table 8.
26

---

Table 8: Action Hypershere Volume Multiplier
InvertedPendulumBox HopperBox
HalfCheetahBox
Walker2dBox
Default Q3C K
3.0
1.65
1.0
1.0
3.0
1.25
0.25
0.25
Standard
TD3 Standard
Loppee
05.40
Expected Q.eedQQVVi.. Value
Expected Q Valuue
11.25
64.30
50.00
01.33
69:75
69.00
09:35
65.00
-1000
-1.50
0.30
1.00
48.0
0.50
Action Range
Action Range
Q3C Restricted
TD3 Restricted
95.90
70.00
Expested Q Value
95.45
Exprcted Q Value
68.00
66.00
33.00
62.00
85336
40.00
58.00
31300
56.00
0.30
Action Range
Action Range
Figure 14: The Learned Q-Functions for TD3 and Q3C are notably complex and have more local optima on the restricted version InvertedPendulum as compared to the standard Mujoco version.
F
Implementation Details and Hyperparameters
F.1
Baselines
For TD3, SAC, and PPO we have used the official implementations from Stable-Baselines3" with the optimized hyperparameters taken from RI. Zoo". For NAF and RBF-DQN, we have followed the original publications and used the official codebase and hyperparameters, where available. If the hyperparameters are not available, we performed our own hyperparameter tuning. While made a strong effort to identify competitive configurations, acknowledge that better parameter combinations may exist.
F.2
Q3C Hyperparameters
Q3C adopts its hyperparameters from its underlying implementation of TD3, but we tune certain important hyperparameters such as learning rate and learning starts. Furthermore, we tune the hyperparameters specific to Q3C such as the number of control-points, the number of nearest neighbors k, separation loss weight, and initial smoothing value.
27

---

F221
Classic Environments
Table 9: Q3C Hyperparameters (Part 1 of
Parameter
Pendulum
Hopper
BipedalWalker
Ant
Total timesteps
le5
le6
1e6
3e6
Discount factor (?)
0.98
0.99
0.98
0.99
Batch size
64
256
256
256
Buffer size
2e5
le5
2e5
5e5
Learning starts
1000
30000
10000
10000
Target network update (r)
0.005
0.005
0.005
0.005
Noise type
Gaussian
Gaussian
Gaussian
Gaussian
Noise std
0.1
0.1
0.1
0.1
Learning rate
Ic-3
le-3
3e-4
5e-4
Network dim
[400,300]
[400,300]
[400,300]
[400,300]
Target update interval
1
Gradient clipping
10.0
10.0
1.0
5.0
of control-points
20
k
3
10
10
15
Separation loss weight
0.01
0.01
1.0
0.1
Initial smoothing value
0.1
0.01
0.001
0.001
Table 10: Q3C Hyperparameters (Part 2 of 2)
Parameter
HalfCheetah
Swimmer
Walker2d
Total timesteps
le6
le6
Discount factor (7)
0.99
0.9999
0.99
Batch size
256
256
256
Buffer size
1e5
5c4
1e5
Learning starts
30000
10000
10000
Target network update (r)
0.005
0.005
0.005
Noise type
Gaussian
Gaussian
Gaussian
Noise std
0.1
0.1
0.1
Learning rate
1e-3
1e-3
1e-3
Network dim
[400,300]
[400,300]
[400,300]
Target update interval
4
4
4
Gradient clipping
10.0
10.0
10.0
of control control-points
20
20
20
10
10
Separation loss weight
1.0
1.0
0.1
Initial smoothing value
0.01
0.1
28

---

F.2.2
Restricted Environments
Table 11: Q3C Hyperparameters for Restricted Environments
Parameter
InvertedPendulumBox
HopperBox
HalfCheetahBox
Total timesteps
le6
2e6
3e6
Discount factor (7)
0.99
0.99
0.99
Batch size
256
256
256
Buffer size
2e5
le5
le6
Learning starts
1000
30000
30000
Target network update (r)
0.005
0.005
0.005
Noise type
Gaussian
Gaussian
Gaussian
Noise std
0.1
0.1
0.1
Learning rate
le-3
1c-3
3c-4
Network dim
[400,300]
[400,300]
[400,300]
Target update interval
4
4
4
Gradient clipping # of control-points
10.0
10.0
10.0
3
20
30
3
10
10
Separation loss weight Initial smoothing value
0.01
0.1
1.0
0.1º 1.
1.0
0.000001*
Smoothing value is fixed throughout training
29


Reviews:
[{"review_id": "cnpWTipHcc", "reviewer": "Reviewer_U2v4", "summary": "This paper introduces Q3C, a novel actor-free Q-learning approach for continuous control, aiming to overcome the stability issues associated with actor-critic methods. Traditional actor-critic methods suffer from instability during training due to the coupling of policy and value function updates. The authors propose a solution using a structurally maximizable Q-function, optimized by learning control points. By leveraging deep learning and a set of architectural and algorithmic improvements, Q3C is designed to be stable and sample-efficient.", "questions": "1. The paper suggests that actor-critic methods are generally stable, but the proposed method also involves training two models: one for the policy and another for the value function. This dual-model structure could introduce potential instability, as the training of both models might become misaligned. Could the authors elaborate on how stability is maintained in their method despite the dual-model setup? What specific mechanisms or architectural choices help ensure training stability, and how does Q3C compare with traditional actor-critic methods in this regard?\n\n2. The paper compares Q3C primarily with value-based methods. To gain a better understanding of its performance and robustness, would the authors consider adding comparisons to stronger policy-gradient methods like PPO? How do the proposed method’s strengths and weaknesses compare with these baselines, particularly in terms of exploration, sample efficiency, and policy robustness?\nGeneralization to High-Dimensional Action Spaces:\nHow does Q3C scale to environments with high-dimensional action spaces? Are there any performance bottlenecks in optimizing the Q-function in such spaces? Additionally, how well does the method handle partial observability or environments where the action space is not fully observable?\n\n3. Given that Q3C does not use an explicit actor for exploration, how does the method ensure sufficient exploration during training? Would additional exploration mechanisms, such as Boltzmann over the control-point values, improve performance in environments that require more extensive exploration?\nReal-World Applicability and Sample Efficiency:\nWhile the paper provides strong empirical results in simulation environments, has there been any consideration of the method's applicability in real-world settings, especially for high-dimensional or physical tasks? Additionally, Q3C’s sample efficiency could be compared in greater detail with methods such as SAC or TD3, especially in tasks with sparse rewards or large state spaces.", "limitations": "Yes. The paper acknowledges that Q3C’s performance may suffer in certain high-dimensional or sparse-reward environments, and it notes that further exploration of the method’s sample efficiency and real-world scalability would be useful. However, a more detailed discussion of the computational limitations or potential scalability challenges in real-world applications would provide a more comprehensive understanding.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The method proposed is technically robust, with clear theoretical motivation and sound empirical validation.\nThe integration of control points for Q-function maximization is a novel approach, ensuring that the maximum value can be found efficiently in continuous action spaces, a known challenge in Q-learning.\n\n2. The variance reduction techniques, such as exponential moving averages (EMA), effectively address training instability, ensuring better convergence.\n\n3. Empirical results show Q3C is competitive with leading actor-critic methods such as TD3, SAC, and others, performing well across multiple environments, including complex and constrained settings.\n\n**Weaknesses:**\n\n1. While the concept of using control points is not entirely new, the combination of this approach with deep learning and the specific modifications introduced by the authors is what makes this work original.\nHowever, it would be useful to see a deeper comparison of how Q3C outperforms or offers advantages over other actor-free methods, such as Implicit Q-learning or Policy-conditioned Q-learning, which also aim to address similar problems in continuous control.\n\n2. The paper suggests that actor-critic methods are generally stable, but the proposed method also involves training two models: one for the policy and another for the value function. This dual-model structure could introduce potential instability, as the training of both models might become misaligned. Could the authors elaborate on how stability is maintained in their method despite the dual-model setup? What specific mechanisms or architectural choices help ensure training stability, and how does Q3C compare with traditional actor-critic methods in this regard?\n\n3. The paper compares Q3C primarily with value-based methods. To gain a better understanding of its performance and robustness, would the authors consider adding comparisons to stronger policy-gradient methods like PPO? How do the proposed method’s strengths and weaknesses compare with these baselines, particularly in terms of exploration, sample efficiency, and policy robustness?\nGeneralization to High-Dimensional Action Spaces:\nHow does Q3C scale to environments with high-dimensional action spaces? Are there any performance bottlenecks in optimizing the Q-function in such spaces? Additionally, how well does the method handle partial observability or environments where the action space is not fully observable?\n\n4. Given that Q3C does not use an explicit actor for exploration, how does the method ensure sufficient exploration during training? Would additional exploration mechanisms, such as Boltzmann over the control-point values, improve performance in environments that require more extensive exploration?\nReal-World Applicability and Sample Efficiency:\nWhile the paper provides strong empirical results in simulation environments, has there been any consideration of the method's applicability in real-world settings, especially for high-dimensional or physical tasks? Additionally, Q3C’s sample efficiency could be compared in greater detail with methods such as SAC or TD3, especially in tasks with sparse rewards or large state spaces.", "was_revised": true, "final_justification": "The proposed method is technically robust, supported by clear theoretical motivation and solid empirical validation. The integration of control points for Q-function maximization is a novel and effective approach, enabling efficient identification of the maximum in continuous action spaces. The authors also clearly articulate the scope and contributions of the paper.", "rebuttal": "We appreciate the reviewer’s thoughtful comments and positive assessment of our method’s originality, as well as the value of our theoretical and empirical validations. Below, we address the all the concerns and questions, with added experiments wherever possible.\n\n## Comparison of Q3C with IQL\n\nOur work focuses on employing control-point-based Q-function interpolation in the context of online reinforcement learning, and we compare against other online RL methods accordingly. While Implicit Q-Learning (IQL) is a strong offline RL algorithm, it is not directly applicable to online settings and thus not included as a baseline. However, future work could explore the utility of our Q-function architecture in offline RL settings.\n\nRegarding policy-conditioned Q-learning, we would appreciate clarification or a reference from the reviewer, as we have not been able to find a method by this name in the online RL literature.\n\n## Difference between Q3C’s framework and Actor-Critic Methods\nQ3C’s control points enable training a single end-to-end network instead of a separate actor and critic, because the control points function as just another layer of the neural network. While Q3C has two modules, for proposing and evaluating the control points, a **crucial difference from actor-critic** approaches is that these modules are trained end-to-end with a single Bellman objective.\n\nAlthough this setup resembles actor-critic methods at first, it differs significantly: Q3C uses a single, unified loss function to train both components end-to-end, without the alternating optimization of separate actor and critic models. Thus, it retains a purely value-based formulation.\n\nThus, it removes the complexities seen in prior works such as DDPG, TD3, and SAC, of specifying the gradient objective for training the actor and hyperparameter tuning of entropy, policy delay, or dual optimizers. Fundamentally, the only architectural addition in Q3C is the parameter-free inverse-distance interpolator. We have revised the manuscript to clearly establish this distinction.\n## Comparison with policy-based methods: SAC, PPO\nAs the reviewer recommended, we have added SAC and PPO, as well as RBF-DQN suggested by Reviewer UQVA, as new baselines. These were trained for an equal number of timesteps as Q3C and TD3. For SAC and PPO, we use the Stable-Baselines3 (SB3) implementation and hyperparameters from RL Zoo3. For RBF-DQN, we rely on the official implementation and recommended hyperparameters provided by the authors. We report the final performance over 10 trials.\n\n| Environment | Q3C | TD3 | SAC | PPO | RBF-DQN |\n|-------------|-----|-----|-----|-----|---------|\n| Pendulum-v1 | -159.53 ± 16.46 | -144.64 ± 95.28 | -140.31 ± 31.14 | -176.85 ± 23.70 | 159.68 ± 15.80 |\n| Hopper-v4 | 3206.14 ± 407.23 | 3113.41 ± 888.17 | 2922.56 ± 590.18 | 882.85 ± 286.17 | 1912.34 ± 1203.35 |\n| Walker2d-v4 | 3977.39 ± 879.70 | 4770.82 ± 560.16 | 4091.98 ± 405.00 | 2917.58 ± 895.72 | 903.62 ± 308.31 |\n| HalfCheetah-v4 | 9468.66 ± 949.01 | 9984.74 ± 1076.58 | 10909.02 ± 416.07 | 5397.83 ± 746.90 | 7267.01 ± 1546.27 |\n| InvertedPendulumBox-v4 | 1000 ± 0 | 782.76 ± 348.92 | 993.40 ± 7.15 | 839.41 ± 372.97 | 734.83 ± 560.40 |\n| HalfCheetahBox-v4 | 4357.82 ± 1503.33 | 2276.70 ± 2036.59 | 4248.83 ± 2184.33 | 4637.60 ± 686.19 | 3023.26 ± 1980.03 |\n| HopperBox-v4 | 1974.28 ± 1170.05 | 1406.83 ± 1162.72 | 3330.64 ± 140.25 | 637.12 ± 449.74 | 1225.27 ± 793.74 |\n\nAs shown, Q3C achieves competitive performance in all standard environments and matches or surpasses the state-of-the-art in all restricted environments. In contrast, the newly added baselines exhibit varying performance across different environments.\n## High-dimensional Action Spaces and Partial Observability\nWe have conducted preliminary experiments on two environments with high-dimensional action spaces. Without any hyperparameter optimization, runs of Q3C are able to match the final performance of TD3 in AdroitHandHammer (26-dim action space) and AdroitHandDoor (28-dim action space) environments, which shows that Q3C is able to scale its learning to high-dimensional actions.\n**Final Eval Success Rate on Adroit Envs**\n\n| Environment | Q3C | TD3 |\n|-------------|-----|-----|\n| AdroitHandHammer-v1 | 0.9 | 0.9 |\n| AdroitHandDoor-v1 | 0.92 | 0.84 |\n\n\n**Regarding the reviewer’s question about non-observable action spaces:** if this refers to partial observability in the state space, our algorithm remains unaffected, as our method does not impose any assumptions requiring full observability. If the concern is about action spaces with unavailable or invalid actions, our restricted environments explicitly model such scenarios, and we demonstrate that Q3C performs robustly under these conditions (Section 5.2). If the reviewer meant something else, we would appreciate further clarification.\n\n## Exploration strategy of Q3C\n\nFor exploration, we refer the reviewer to Algorithm 1 in the main paper. We use Gaussian noise added to the maximizing action, consistent with TD3, to isolate the effect of the learning algorithm from the exploration strategy. This ensures a fair comparison. We also experimented with Boltzmann exploration over the control points, but found that Gaussian noise resulted in more stable and consistent performance. We will add those comparisons to the final version of our paper.\n\n## Sample Efficiency of Q3C\n\nIn terms of sample efficiency, Q3C performs comparably to other baselines. As a hybrid between DQN-style and actor-critic methods, it offers the flexibility to incorporate improvements from both paradigms. In Appendix 5.3, we explored one such strategy of Cross-Q [1] and observed promising preliminary results. Future work can build on this direction to further improve sample efficiency. Additionally, our initial experiments on Adroit dexterous manipulation tasks indicate that Q3C can handle high-dimensional action spaces, suggesting potential for future applications in real-world scenarios that combine both scalability and efficiency.\n\n**References**\n\n[1] Bhatt, Aditya, et al. \"Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity.\" arXiv preprint arXiv:1902.05605 (2019).\n\n---\nWe thank you for your feedback and hope we've addressed your comments adequately. We are happy to answer any further questions and incorporate any further suggestions."}, {"review_id": "vRk5khYMM4", "reviewer": "Reviewer_UQVA", "summary": "This paper proposes Q-learning for Continuous Control with Control-points (Q3C), a value-based method for control in domains with continuous action spaces. This approach structures the Q-function approximator such that it covers the entire continuous space, but the maximizing action can still be determined efficiently. The authors prove that this method retains the universal approximator property of the underlying neural network models and that it compares favorably to some other value-based methods in several continuous control domains.", "questions": "How would you present Q3C in a context including RBF-DQN?\n\nWhat would you say are the signfiicant differences between Q3C and RBF-DQN and how do you think they improve on RBF-DQN?\n\nCan you add RBF-DQN as a baseline?\n\nIs there a principled (or inspired-by-principle) way of selecting the number of control points used for a problem?", "limitations": "I would like to see more discussion about how to select the number of control points to use. While the theory contribution shows that there exists an N such that any arbitrary maximum error can be guaranteed, such an N could be quite large for domains with complex value functions.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\nThe paper is well-written and easy to follow.\n\nThe problem of value-based continuous control is an important one that I believe is relevant and of interest to the community.\n\nQ3C's control point diversity and control point ranking components appear novel and useful additions to the algorithm.\n\nQ3C performs well compared to the baselines the authors use and the experiments thoroughly cover a large number of interesting domains, though key baselines are missing (see Weaknesses).\n\n\n# Weaknesses\n\nQ3C is very similar to RBF-DQN [1], but it the latter is neither mentioned in either the text nor used as a baseline for the experiments. RBF-DQN uses the same kind of structure to generate candidate actions (centroids there) and values as functions of the state. These are then interpolated, but using radial basis functions explicitly. Equation 4 for Q3C is essentially the same as Equation 5 in the RBF-DQN paper, but uses something like an inverse quadratic RBF where RBF-DQN uses negative exponential RBF. RBF-DQN similarly retains the universal approximator property and the approximator is maximized (with bounded error) by selecting the maximizing centroid.\n\nQ3C is not compared to policy-based methods (policy gradient or actor-critic) in the experiments.\n\n# References\n[1]  Deep Radial-Basis Value Functions for Continuous Control, Asadi et al., AAAI 2021", "was_revised": true, "final_justification": "The authors addressed my concerns related to the lack of comparison/discussion regarding RBF-DQN, which is a very similar approach which was published in 2021. However, with the authors adding this discussion and adding RBF-DQN as a baseline to their experimental results, I believe that the paper provides novel, useful additions to such approaches.", "rebuttal": "We thank the reviewer for recognizing the novelty of our additions to the control-point-based Q-function approximator and for appreciating the wide variety of our evaluation environments.\n\n----\n\n## Added comparison to RBF-DQN\n\nWe appreciate the pointer to RBF-DQN (Asadi et al., AAAI’21) and agree it should be appropriately cited, discussed, and compared. Thus, we (i) implement and evaluate RBF-DQN across all environments using its official code with the same exploration mechanism used for all our reported results, and (ii) include a discussion to situate Q3C among continuous-control, value-based interpolation methods.\n\n**(i) Q3C empirically outperforms RBF-DQN**\n| Environment              | Q3C                     | RBF-DQN                |\n|--------------------------|--------------------------|-------------------------|\n| Pendulum-v1              | -159.53 ± 16.46          | -159.68 ± 15.80          |\n| Hopper-v4                | 3206.14 ± 407.23         | 1912.34 ± 1203.35       |\n| Walker2d-v4              | 3977.39 ± 879.70         | 903.62 ± 308.31         |\n| HalfCheetah-v4           | 9468.66 ± 949.01         | 7267.01 ± 1546.27       |\n| InvertedPendulumBox-v4   | 1000 ± 0                 | 734.83 ± 560.40         |\n| HalfCheetahBox-v4        | 4357.82 ± 1503.33        | 3023.26 ± 1980.03       |\n| HopperBox-v4             | 1974.28 ± 1170.05        | 1225.27 ± 793.74        |\n\n**(ii) Added discussion in paper to contextualize Q3C among value-based interpolation methods like RBF-DQN**\n\n### Similarities between Q3C and RBF-DQN\n- Both approaches address the problem of value-based RL for a continuous action space without an actor.\n- Both generate a finite set of action anchor points, which help construct an interpolation over Q-values.\n- The maximizing action is chosen easily from among the action anchor points.\n\nTo assign appropriate credit to RBF-DQN, we have revised the manuscript to include it alongside the body of work on wire-fitting interpolators, which is the key inspiration and baseline of Q3C. A key benefit of RBF-DQN over Q3C could be its stronger approximation ability due to the use of explicit radial basis functions. However, since reinforcement learning only requires the Q-function to be maximal for the optimal action, the benefit of a more accurate Q-function may not necessarily result in sample-efficient learning.\n\nWe clearly contextualize the structural maximization in Q3C and contrast it to approximate maximization (under bounded error) of RBF-DQN, along with other key differences below.\n\n### Differences between Q3C and RBF-DQN\n\nWhile both Q3C and RBF-DQN use interpolation techniques, there are key differences in design and practical implications:\nInterpolation Scheme: RBF-DQN employs radial basis functions with negative exponential kernels. In their standard form, these do not guarantee that the interpolated function attains its maximum at one of the basis centroids. To mitigate this, RBF-DQN introduces an inverse smoothing parameter to push the maximum toward a centroid, albeit with bounded approximation error. However, this smoothing also spreads high values to nearby actions, smoothing the curve and making it less suitable in scenarios with multiple local optima.\n\nIn contrast, Q3C uses inverse-distance weighting, which guarantees that the maximum Q-value occurs at one of the control points. The smoothing parameter in our formulation affects the function's value only at other control points: with a low smoothing parameter, the approximated function passes through all control points, while with a large smoothing parameter, the function's value may differ slightly at some control points. However, in all cases, it is still guaranteed that the maximum of the function lies at a control point—an advantage in highly non-convex Q landscapes.\n\nThese theoretical differences are also reflected in our empirical findings:\n- Q3C consistently outperforms RBF-DQN in standard environments and shows even stronger gains in restricted settings, where the Q-function is discontinuous or non-smooth. RBF-DQN struggles in these cases, which supports our claim that smoothing-based interpolation may be a poor fit for such landscapes.\n- Additionally, **Q3C is more parameter-efficient**. Most of our experiments use only 20 control points, whereas RBF-DQN uses at least 100 centroids. \n\nBeyond the core interpolation strategy, Q3C introduces several novel architectural and algorithmic enhancements, including:\n- Action-conditioned Q-value generation, which improves representational capacity;\n- Relevance-based top-K control point filtering;\n- Q-value normalization for scale-awareness;\n- A light entropy-inspired regularization term for training stability.\n\nThese enhancements are validated through ablations in the paper, and several of them could potentially benefit RBF-DQN as well, which adds to the significance of our paper.\n\n----\n\n## Added policy-based comparisons: SAC, PPO\nWe also included SAC and PPO as additional baselines, and re-report TD3 for comparison. The results are summarized below:\n| Environment | Q3C | TD3 | SAC | PPO | RBF-DQN |\n|-------------|-----|-----|-----|-----|---------|\n| Pendulum-v1 | -159.53 ± 16.46 | -144.64 ± 95.28 | -140.31 ± 31.14 | -176.85 ± 23.70 | 159.68 ± 15.80 |\n| Hopper-v4 | 3206.14 ± 407.23 | 3113.41 ± 888.17 | 2922.56 ± 590.18 | 882.85 ± 286.17 | 1912.34 ± 1203.35 |\n| Walker2d-v4 | 3977.39 ± 879.70 | 4770.82 ± 560.16 | 4091.98 ± 405.00 | 2917.58 ± 895.72 | 903.62 ± 308.31 |\n| HalfCheetah-v4 | 9468.66 ± 949.01 | 9984.74 ± 1076.58 | 10909.02 ± 416.07 | 5397.83 ± 746.90 | 7267.01 ± 1546.27 |\n| InvertedPendulumBox-v4 | 1000 ± 0 | 782.76 ± 348.92 | 993.40 ± 7.15 | 839.41 ± 372.97 | 734.83 ± 560.40 |\n| HalfCheetahBox-v4 | 4357.82 ± 1503.33 | 2276.70 ± 2036.59 | 4248.83 ± 2184.33 | 4637.60 ± 686.19 | 3023.26 ± 1980.03 |\n| HopperBox-v4 | 1974.28 ± 1170.05 | 1406.83 ± 1162.72 | 3330.64 ± 140.25 | 637.12 ± 449.74 | 1225.27 ± 793.74 |\n\nPPO is on-policy, so it is usually sample-inefficient. SAC employs a different exploration scheme based on soft Q-value maximization in contrast to the noise-added exploration used in the baselines in the paper: TD3, NAF, wire-fitting, Q3C, and now RBF-DQN. Thus, these methods are not directly comparable to evaluate the key contribution of Q3C as a “structurally maximizable Q-function”, and we do NOT expect or claim that Q3C should outperform these methods.\n\n----\n\n## Selecting number of control points\n\nRegarding control point count: while it is a hyperparameter, our ablation studies (Appendix D.4) show that simply increasing the number of control points n does not always improve performance. A too large n runs the risk of overfitting or unnecessarily increasing the problem complexity which slows down convergence. A typical recipe is to start with the default n=20 and try larger numbers of control points with a fixed top-k filter (k=10), which balances coverage and computational cost, leading to robust and stable performance. \n\nWe find that the optimal number of control points tends to scale more conservatively than a proportional increase with action dimensionality due to the constant tradeoff between increasing representational potential and increasing the learning difficulty for the model. However, a limited number of additional control points are beneficial for environments with increased general task complexity and a considerable higher action dimensionality. With relatively complicated environments such as Ant, HalfCheetahBox, and Adroit, we found an optimal control point combination to be between n=30,40,50 and k=10. \n\nFor environments with an even higher dimensionality, we can utilize a larger n with relatively minimal increases to computational cost and problem complexity as the backpropagation still only needs to optimize across the top k control points in a single backward pass. A key benefit of the conditional Q architecture is that even with a large number of control points, the number of parameters or the compute required does not increase because it is applied parallel over all the control points. This is also a major representational benefit over RBF-DQN.\n\n---\n\nWe thank the reviewer once again for their valuable comments, which helped us improve our work. We will incorporate these details and the supporting experimental results into the final version of our paper. As the comparison with RBF-DQN was the reviewer’s primary concern, we have now included this baseline along with a detailed discussion and new experimental results. Given these additions and clarifications, we kindly ask the reviewer to consider revisiting their score, and we are happy to answer any further questions."}, {"review_id": "ygxcIQGtta", "reviewer": "Reviewer_SR2K", "summary": "The authors in this submission proposed a value-based framework for reinforcement learning with continuous actions. In contrast to the actor-critic methods, they first constructed the control points and subsequently learned policies with respect to Q-functions over those points. In order to overcome the issues reported in previous work, they proposed a few techniques to address them. The experiments showed that the proposed method is on par with TD3 in standard domains, and achieves better performance on constricted environments.", "questions": "* Could you comment on the complexity for Q3C, when compared against relevant value or policy based methods, given the additional steps w.r.t. control points?\n* Why is the variance for Q3C in Table 2 on InvertedPendulum 0?", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\nThe revisit of the Q-function approximation using the control-points is an interesting direction. The authors also showed that the proposed method, when combined with neural networks, works reasonablly well in some standard domains\n\nWeakness:\nGiven the additional complexity involving the control points, the performance of the proposed Q3C algorithm is mostly similar to that of TD3. Note that TD3 is far behind some other policy gradient methods, e.g., SAC. Subsequently, its significance is a bit limited.\n\nSome steps in the proposed method need justification and the presentation can be improved. A few of them are listed as follows. \n* In L175-L179, the authors proposed to keep top k from N candidates, so why not just generate k control-points?\n* In Eq. (6), should there be $N (N - 1) / 2$ pairs only?", "was_revised": true, "final_justification": "I've read the authors' response and some of my comments and questions have been addressed. I increased my score slightly. The memory improvement looks valid, while I am not confident about the time saving, which may depend on the exact implementation.", "rebuttal": "We thank the reviewer for their comments and suggestions. We’re glad to hear they found our method interesting and appreciated the broad range of environments we evaluated. Below, we address the concerns about the introduction of control points in Q3C, significance in comparison to TD3 and SAC, and justifications for algorithm design.\n\n----\n\n## Complexity: Control point architecture reduces, not adds to, learning complexity\n\n### 1. Control points enable training a single end-to-end network instead of a separate actor and critic.\nQ3C reduces the learning complexity to simply learning a single Q-network by eliminating the actor network and its training objective completely. This single Q-network can be trained end-to-end because the control points function as just another layer of the neural network.\nWhile Q3C has two modules, for proposing and evaluating the control points, a **crucial difference from actor-critic** approaches is that these modules are trained end-to-end with a single Bellman objective. Thus, it removes the complexities seen in prior works such as DDPG, TD3, and SAC, of specifying the gradient objective for training the actor and hyperparameter tuning of entropy, policy delay, or dual optimizers. Fundamentally, the only architectural addition in Q3C is the parameter-free inverse-distance interpolator.\n### 2. Q3C has a lightweight wall-time and memory footprint\nContrary to concerns about additional complexity, Q3C simplifies training by eliminating the need for a separate actor network and its corresponding backward pass. We report wall-clock and PyTorch memory used versus baselines (5 seeds each) to show Q3C remains lightweight in practice.\n\n**Wall time until convergence**\n\n| Algorithm | Hopper | BipedalWalker | HalfCheetah | InvertedPendulumBox | HalfCheetahBox | HopperBox |\n|-----------|--------|---------------|-------------|---------------------|----------------|-----------|\n| Q3C | 58.3 min | **44.6 min** | 61.5 min | **12.43 min** | 4 hr | **0.7 hr** |\n| TD3 | **32.9 min** | 45.6 min | **39.2 min** | N/A | N/A | N/A |\n| SAC | 76.3 min | 50.5 min | 69.2 min | 16.5 min | 6.6 hr | 1.3 hr |\n| PPO | N/A | N/A | N/A | N/A | **1.0 hr** | N/A |\n| NAF | N/A | N/A | N/A | N/A | 7.2 hr | N/A |\n\n\n**Pytorch memory**\n\n| Algorithm | Pytorch Allocated (MB) |\n|-----------|------------------------|\n| Q3C       | 26.58                  |\n| TD3       | 23.51                  |\n| SAC       | 20.01                  |\n| PPO       | 18.38                  |\n\n----\n\n## Significance compared to TD3 and SAC\n\n### 1. Note: TD3 is NOT far behind SAC\nWe politely point out that TD3 is actually proven to be better than SAC in certain environments, as observed in prior work [1, 4, 5, 7], and there is **no general consensus** that SAC is the superior algorithm against TD3 [1, 2, 3, 8]. These two algorithms employ different exploration mechanisms and thus outperform each other often because of which style of exploration is more efficient in a given environment and which hyperparameters were tuned better [6, 9].\n\n### 2. Clarify: Q3C is primarily compared to TD3 because it uses TD3’s fairly comparable noise-added exploration.\nSince the exploration mechanism is known to conflate the outcomes of an algorithm in different environments [10, 11], all the baselines in the paper: TD3, NAF, Wire-fitting, and Q3C were run with the same noise-added exploration scheme. This helped us delineate the effect of the learning algorithm fairly. \nNevertheless, as suggested by the reviewer, we have added the results on SAC and PPO, which are not directly comparable due to a significantly different experiment backbone, i.e., PPO is on-policy and SAC has learned exploration.\n\n### 3. Added experiments on SAC, PPO, RBF-DQN\nAs suggested by the reviewer, we have added SAC and PPO, as well as RBF-DQN recommended by Reviewer UQVA, as baselines. These methods were trained for the same number of timesteps as Q3C and TD3, which we also include for reference. SAC and PPO were implemented using Stable-Baselines3 (SB3) and trained with hyperparameters from RL Zoo3. For RBF-DQN, we used the official implementation and hyperparameters provided by the authors.\n\n| Environment | Q3C | TD3 | SAC | PPO | RBF-DQN |\n|-------------|-----|-----|-----|-----|---------|\n| Pendulum-v1 | -159.53 ± 16.46 | -144.64 ± 95.28 | -140.31 ± 31.14 | -176.85 ± 23.70 | 159.68 ± 15.80 |\n| Hopper-v4 | 3206.14 ± 407.23 | 3113.41 ± 888.17 | 2922.56 ± 590.18 | 882.85 ± 286.17 | 1912.34 ± 1203.35 |\n| Walker2d-v4 | 3977.39 ± 879.70 | 4770.82 ± 560.16 | 4091.98 ± 405.00 | 2917.58 ± 895.72 | 903.62 ± 308.31 |\n| HalfCheetah-v4 | 9468.66 ± 949.01 | 9984.74 ± 1076.58 | 10909.02 ± 416.07 | 5397.83 ± 746.90 | 7267.01 ± 1546.27 |\n| InvertedPendulumBox-v4 | 1000 ± 0 | 782.76 ± 348.92 | 993.40 ± 7.15 | 839.41 ± 372.97 | 734.83 ± 560.40 |\n| HalfCheetahBox-v4 | 4357.82 ± 1503.33 | 2276.70 ± 2036.59 | 4248.83 ± 2184.33 | 4637.60 ± 686.19 | 3023.26 ± 1980.03 |\n| HopperBox-v4 | 1974.28 ± 1170.05 | 1406.83 ± 1162.72 | 3330.64 ± 140.25 | 637.12 ± 449.74 | 1225.27 ± 793.74 |\n\nWhile SAC and PPO perform well in certain environments, we observe that TD3 outperforms both in others—a finding consistent with prior benchmarking studies [1, 4, 5, 7]. Q3C achieves competitive performance in all standard environments and matches or surpasses the state-of-the-art in all restricted environments. In contrast, the newly added baselines exhibit varying performance across different environments. This supports the claim that Q3C is a robust and effective alternative to actor-critic methods.\n\n----\n\n## Questions on top-k, Eq. 6, results variance.\n\n### 1. Ablation experiment to validate top-k: Fig. 6, Appendix D.4\n\nWe refer the reviewer to the results on Q3C - Ranking ablation in Figure 6 of the main paper and a general control point ablation in Appendix D.4. Here, we show that keeping top-k from N control-points outperforms generating k control points directly (this is equivalent to setting n=k).\n\nDiscussion added to paper revision: This is because the number of control points n and the top-k value k serve key, different purposes when constructing the Q-function. Increasing the value of n gives us more coverage across high-dim action spaces and provides more representational potential as a larger number of control points can generally model more complex and vast Q-landscapes. The hyperparameter k is not tuned for overarching expressivity of the function, but for enforcing locality and allowing for piecewise approximation of the Q-function. By utilizing only the k closest points to the state-action pair we are approximating, we ensure that only control points that are relevant to that subset of action space contribute to the estimated Q-value, improving accuracy. This also ensures that control point values that lie very far in the action space are not distorted by training experiences on a very different action subset. While some of this distance weighting is already performed by our interpolator, selecting the top k control points enforces this more firmly and yields better results in terms of both performance and computational cost.\n\n### 2. Equation 6 Clarification.\n\nFor the separation loss used in Equation (6), minimizing the sum of pairwise inverse distances encourages the control points to spread out across the action space. In fact, this formulation effectively minimizes the maximum pairwise overlap, and the optimal solution corresponds to a uniform distribution of control points.\n\n### 3. Variance in InvertedPendulumBox.\n\nThe zero variance was incidental: Table 2 is consistent with the converged performances in Figure 4. Thus, since Q3C consistently achieved a score of 1000 across all 10 random seeds, the variance of the result is zero. \n\n----\n\n**References**\n\n[1] Huang, Shengyi, et al. \"Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning.\" arXiv preprint arXiv:2402.03046 (2024).  \n[2] Liu, Shijie. \"An Evaluation of DDPG, TD3, SAC, and PPO: Deep Reinforcement Learning Algorithms for Controlling Continuous Systems.\" DAI 2023 (2024).  \n[3] Mock, James W., and Suresh S. Muknahallipatna. \"A Comparison of PPO, TD3 and SAC Reinforcement Algorithms for Quadruped Walking Gait Generation.\" Journal of Intelligent Learning Systems and Applications (2023).  \n[4] Jain, Ayush, et al. \"Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-Functions.\" Reinforcement Learning Conference.  \n[5] Moosa, Farhana, et al. \"Benchmarking Reinforcement Learning Algorithms for Autonomous Mechanical Thrombectomy.\" International Journal of Computer Assisted Radiology and Surgery (2025).  \n[6] Wan, Yi, Dmytro Korenkevych, and Zheqing Zhu. \"An Empirical Study of Deep Reinforcement Learning in Continuing Tasks.\" arXiv preprint arXiv:2501.06937 (2025).  \n[7] Rzayev, Altun, and Vahid Tavakol Aghaei. \"Off-Policy Deep Reinforcement Learning Algorithms for Handling Various Robotic Manipulator Tasks.\" arXiv preprint arXiv:2212.05572 (2022).  \n[8] Achiam, Joshua. \"Spinning Up in Deep Reinforcement Learning.\" OpenAI (2018).  \n[9] Haarnoja, Tuomas, et al. \"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.\" ICML (2018).  \n[10] Balloch, Jonathan C., et al. \"Is Exploration All You Need? Effective Exploration Characteristics for Transfer in Reinforcement Learning.\" arXiv preprint arXiv:2404.02235 (2024).  \n[11] Su, Yi, et al. \"Long-term value of exploration: measurements, findings and algorithms.\" Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 2024.\n\n----\n\nWe hope the clarification on reduced learning complexity and new results adequately address your concerns by demonstrating the theoretical and practical significance of Q3C. We are happy to answer any further questions and incorporate any further suggestions."}, {"review_id": "w2iuOgcGoH", "reviewer": "Reviewer_mqYt", "summary": "This manuscript introduces Q3C, which is an actor-free RL algorithm designed for continuous control tasks. The key idea is to approximate the Q-function with a set of learned control points via a wire-fitting interpolator, so that the maximum Q-value for any state is achieved at one of these actions. By turning the maximization problem into a comparison over a finite set of candidate actions, Q3C sidesteps the need for a separately trained actor to navigate the continuous action space. To make this idea suitable with deep RL, several architectural and algorithmic enhancements are proposed on top of the TD3 algorithm to stabilize training and improve performance. These enhancements include action-conditioned Q-value generation, relevance-based control-point filtering, a diversity loss to encourage control points to cover the action space, and control point scaling. Empirical evaluations across standard Gymnasium environments demonstrate that Q3C achieves performance comparable to TD3 on most tasks. However, in tasks with constrained action spaces where the valid actions are limited to certain region and therefore Q-functions tends to be highly non-convex, Q3C significantly outperforms TD3.", "questions": "1. Could the authors provide runtime comparisons or memory usage statistics relative to TD3 and NAF?\n2. There are some notational inconsistencies throughout the manuscript. For instance, equation 4 is quite understandable but it is not clear how action-conditioned Q-value generation fits within equation 4. Do authors mean to say $Q_i(s) = h_\\psi(s, a_i(s))$? \n3. What is the typical number of control points that is needed? Does it have to be the same for all state-action pairs? Does it grow with action dimensionality or task complexity? Please discuss the trade-offs involved with the choice of the number of control points.\n4. How robust is Q3C to the choice of the number of control points and other new hyperparameters (top-$k$, $\\gamma$, etc.)? The method introduces several knobs – it would help to know if these were fixed across all experiments or tuned per environment.  \n5. The manuscript mentions that future work could extend Q3C to stochastic settings. Could the authors elaborate on the challenges and potential approaches for modelling soft-Q functions with control points?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* Strengths\n\n1. The manuscript presents a simple yet effective solution via control-points-based wire-fitting interpolators to the long-standing “max-Q” problem in continuous action spaces. This is an interesting solution that departs from conventional actor-critic methods.\n2. Empirical results demonstrate that Q3C matches the performance of the TD3 algorithm on most Gym environments, and significantly outperforms it in challenging settings with constrained action ranges. The fact that Q3C can find better policies where gradient-based actors get stuck in local optima demonstrates a clear advantage of the proposed method.  Additionally, the manuscript includes comprehensive ablations to pinpoint the contribution of each component in the design of Q3C. \n3. The proposed method is also well-motivated theoretically since the wire-fitting formulation can approximate any continuous $Q(s,.)$ arbitrarily well based on proposition 3.\n\n* Weaknesses\n1. Given that the performance of Q3C is largely on par with TD3, the main benefit of is in the method being actor-free, which potentially means smaller memory footprint and faster training. However, there are no discussion on the computational complexity or experiments to evaluate the computational speed ups.\n\n2. Q3C introduces new hyperparameters such as the number of control points and $\\gamma$, in addition to the control point generator. It is not clear how different choices for these hyperparameters impact the performance of Q3C, and the manuscript does not explore how sensitive the results are to these settings\n\n3. The evaluation could be more complete if other baselines are included such as SAC. Also, while the benchmark suite is diverse, the manuscript does not explore real-world tasks where control-point scalability could be more challenging.", "was_revised": true, "final_justification": "The majority of my concerns have been addressed in the rebuttal, and the authors have conducted extensive additional evaluations. Accordingly, I have raised my score.", "rebuttal": "We thank the reviewer for their insightful comments. We are glad that they found our approach interesting and appreciated both the theoretical and empirical validations of our work. Below, we provide responses to the questions and clarifications requested.\n\n## Wall-clock Time and Memory Efficiency\n\nWe conducted detailed comparisons of wall-clock time and memory usage between Q3C, SAC, PPO, and TD3 in the restricted environments (5 seeds each, restricted environments on Tesla P100 and unrestricted on A100). The results are summarized below:\n\n**Wall Time until convergence**\n\n| Algorithm | Hopper | BipedalWalker | HalfCheetah | InvertedPendulumBox | HalfCheetahBox | HopperBox |\n|-----------|--------|---------------|-------------|---------------------|----------------|-----------|\n| Q3C | 58.3 min | **44.6 min** | 61.5 min | **12.43 min** | 4 hr | **0.7 hr** |\n| TD3 | **32.9 min** | 45.6 min | **39.2 min** | N/A | N/A | N/A |\n| SAC | 76.3 min | 50.5 min | 69.2 min | 16.5 min | 6.6 hr | 1.3 hr |\n| PPO | N/A | N/A | N/A | N/A | **1.0 hr** | N/A |\n| NAF | N/A | N/A | N/A | N/A | 7.2 hr | N/A |\n\n\n**Pytorch Memory**\n\n| Algorithm | Pytorch Allocated (MB) |\n|-----------|------------------------|\n| Q3C       | 26.58                  |\n| TD3       | 23.51                  |\n| SAC       | 20.01                  |\n| PPO       | 18.38                  |\n\nIn HopperBox and InvertedPendulumBox environments, Q3C consistently achieves the target reward in significantly less time than all baselines. For instance, it requires approximately 60% of the time taken by SAC to reach the same performance, while other baselines often fail to reach that level at all. In terms of memory usage, although Q3C utilizes a parallelized evaluation of Q-values of all the control points, the increase in memory footprint is marginal and does not hinder training.\n\n## New baselines: SAC, PPO, RBF-DQN\n\nAs suggested by the reviewer, we have added SAC and PPO, as well as the RBF-DQN method recommended by Reviewer UQVA, as baselines. These methods were trained for the same number of timesteps as Q3C and TD3, which we also include for reference. SAC and PPO were implemented using Stable-Baselines3 (SB3) and trained with hyperparameters from RL Zoo3. For RBF-DQN, we used the official implementation and hyperparameters provided by the authors.\n\n| Environment | Q3C | TD3 | SAC | PPO | RBF-DQN |\n|-------------|-----|-----|-----|-----|---------|\n| Pendulum-v1 | -159.53 ± 16.46 | -144.64 ± 95.28 | -140.31 ± 31.14 | -176.85 ± 23.70 | 159.68 ± 15.80 |\n| Hopper-v4 | 3206.14 ± 407.23 | 3113.41 ± 888.17 | 2922.56 ± 590.18 | 882.85 ± 286.17 | 1912.34 ± 1203.35 |\n| Walker2d-v4 | 3977.39 ± 879.70 | 4770.82 ± 560.16 | 4091.98 ± 405.00 | 2917.58 ± 895.72 | 903.62 ± 308.31 |\n| HalfCheetah-v4 | 9468.66 ± 949.01 | 9984.74 ± 1076.58 | 10909.02 ± 416.07 | 5397.83 ± 746.90 | 7267.01 ± 1546.27 |\n| InvertedPendulumBox-v4 | 1000 ± 0 | 782.76 ± 348.92 | 993.40 ± 7.15 | 839.41 ± 372.97 | 734.83 ± 560.40 |\n| HalfCheetahBox-v4 | 4357.82 ± 1503.33 | 2276.70 ± 2036.59 | 4248.83 ± 2184.33 | 4637.60 ± 686.19 | 3023.26 ± 1980.03 |\n| HopperBox-v4 | 1974.28 ± 1170.05 | 1406.83 ± 1162.72 | 3330.64 ± 140.25 | 637.12 ± 449.74 | 1225.27 ± 793.74 |\n\nQ3C achieves competitive performance in all standard environments and matches or surpasses the state-of-the-art in all restricted environments. In contrast, the newly added baselines exhibit varying performance across different environments. This supports the claim that Q3C is a robust and effective alternative to actor-critic methods.\n\nFurthermore, we have conducted preliminary experiments on two environments with high-dimensional action spaces. Without any hyperparameter optimization, Q3C is able to match the final performance of TD3 in AdroitHandHammer (26-dim action space) and AdroitHandDoor (28-dim action space) environments, which shows that Q3C is able to scale its learning to high-dimensional actions.\n\n**Final Eval Success Rate on Adroit Envs**\n\n| Environment | Q3C | TD3 |\n|-------------|-----|-----|\n| AdroitHandHammer-v1 | 0.9 | 0.9 |\n| AdroitHandDoor-v1 | 0.92 | 0.84 |\n\n## Equation 4 Clarification\n\nWhen conditional Q-value generation is used, the formulation in Equation 4 remains structurally the same. Specifically, the control-point actions are defined as $\\hat{a}\\_i = g\\_{\\phi}(s)\\_i$ and the corresponding Q-values are given by $\\hat{Q}\\_i = h\\_{\\psi}(s, g\\_{\\phi}(s)\\_i)$, where $g\\_{\\phi}$ is the control point generator network and $h\\_{\\psi}$ is the Q-estimator network for those control points. The full expression is provided below for completeness:\n\n$$\nQ(s,a) = \\frac{\\sum\\_i \\hat{Q}\\_i(s) \\,w\\_i(s, a)}{\\sum\\_i w\\_i(s, a)}\n$$\nwhere\n$$\nw\\_i(s, a) = \\frac{1}{\\lvert a -  g\\_\\phi(s)\\_i\\rvert^2 + c\\_i \\,\\max\\_k\\bigl(\\hat Q\\_{\\max}- h\\_\\psi(s, g\\_\\phi(s)\\_i)\\bigr)}.\n$$\n\n## Selecting number of control points and other hyperparameter ablations\n\nRegarding the hyperparameter ablations, we refer the reader to the supplementary material (Appendix D.4). We also report all of the hyperparameter configurations used in different environments in (Appendix G.3). In addition to the ablations in the main paper, we did extra ablation studies on the number of control points, top-k ranking and different smoothing strategies. Notably, we found that simply increasing the number of control points n does not always improve performance. A too large n runs the risk of overfitting or unnecessarily increasing the problem complexity which slows down convergence. A typical recipe is to start with the default n=20 and try a larger number of control points with a fixed top-k filter (k=10), which balances coverage and computational cost, leading to stable and expedited convergence. \n\nWe find that the optimal number of control points tends to scale more conservatively than a proportional increase with action dimensionality due to the constant tradeoff between increasing representational potential and increasing the learning difficulty for the model. However, a limited number of additional control points is beneficial for environments with increased general task complexity and considerably higher action dimensionality. With relatively complicated environments such as Ant, HalfCheetahBox, and Adroit, we found an optimal control point combination to be between n=30,40,50 and k=10. For environments with an even higher dimensionality, we can utilize a larger n with relatively minimal increases to computational cost and problem complexity, as the backpropagation still only needs to optimize across the top k control points in a single backward pass.\n\n## Q3C for soft Q-functions\nThe basic idea to extend Q3C to stochastic settings is to replace control points with control distributions. This would require at least the following changes to the algorithm:\n1. Instead of control points $\\hat{a}\\_i$, we generate control distribution mean and std: $\\mu\\_i, \\sigma\\_i$.\n2. A Q-value is not associated with each control point distribution, and its estimator would take $s, \\mu\\_i, \\sigma\\_i$ as an input.\n3. Finally, to evaluate the Q-value of a given action, an interpolator is still used. However, the weights of the interpolator would change from inverse-distance to the probability of the action under the control distribution: $w_i = p(a | \\mathcal{N}(\\mu\\_i, \\sigma\\_i^2))$\n4. At inference, instead of computing the $\\arg\\max$ over all control points, one can sample an action from the Gaussian Mixture Model (GMM) defined by the N control distributions $N(\\mu\\_i, \\sigma\\_i^2)$.\n\n----\n\nWe thank you for your insightful feedback and hope we've addressed your comments adequately. We are happy to answer any further questions and incorporate any further suggestions."}]

Author rebuttal:

Meta-review: The paper proposes Q3C, an actor-free Q-learning framework for reinforcement learning (RL) in continuous action space settings. Unlike traditional actor-critic methods, which can be prone to training instability due to the tight coupling of the actor and critic updates, Q3C relies purely on value-based learning by structurally maximizing Q-functions. The core idea behind Q3C is to approximate the Q-function using a set of learned control points, implemented via a wire-fitting interpolator. This allows the problem of action maximization to be reduced to selecting among these candidates, which then avoids the need for a separately trained actor network. The paper proposes several architectural/algorithmic modifications suited to deep RL settings that improve the stability and efficiency of training. Empirical results across a range of continuous control benchmarks show that Q3C achieves performance and sample efficiency comparable to state-of-the-art actor-critic methods such as TD3, while significantly outperforming them in constrained action-space environments where traditional gradient-based maximization struggles.

The paper was reviewed by four referees who are in agreement regarding the key strengths and weaknesses of the paper as originally submitted. Among the strengths, the reviewers agree that the paper addresses a long-standing problem in Q-learning---Q-function maximization---and the means by which it does so through the integration of control points is both novel and interesting. Meanwhile, at least two reviewers note their appreciation for the theoretical motivation behind the proposed Q3C framework. While the reviewers acknowledge that the experimental evaluation provided in the submission demonstrates that Q3C is competitive with leading actor-critic baselines, several reviewers found that the paper would benefit from comparisons to other methods including PPO, SAC, and the related RBF-DQN method, which was neither cited nor compared against. Additionally, some reviewers raised questions about the computational cost associated with Q3C, suspecting that the use of control points introduces added complexity. The authors address these concerns in their rebuttal by providing comparisons to these baselines, addressing the concern about computational complexity, and introducing complexity results. In their Final Justifications, three reviewers acknowledge their appreciation for the authors' response and comment that it addresses their primary concerns.
Decision: Accept (poster)