Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow
Authors: Yunyue Wei
Chenhui Zuo
Yanan Sui
Venue: ICLR 2026 Poster (2026)
Keywords: Scalable exploration
high-dimensional continuous control
Primary area: reinforcement learning

Abstract: Controlling high-dimensional biological and robotic systems is challenging due to expansive state–action spaces, where effective exploration is critical. Commonly used exploration strategies in reinforcement learning are largely undirected with sharp degradation as action dimensionality grows. Many existing methods resort to dimensionality reduction, which constrains policy expressiveness and forfeits system flexibility. We introduce Q-guided Flow Exploration (Qflex), a scalable reinforcement learning method that conducts exploration directly in the native high-dimensional action space. During training, Qflex traverses actions from a learnable source distribution along a probability flow induced by the learned value function, aligning exploration with task-relevant gradients rather than isotropic noise. Our proposed method substantially outperforms representative online reinforcement learning baselines across diverse high-dimensional continuous-control benchmarks. Qflex also successfully controls a whole-body human musculoskeletal model to perform agile, complex movements, demonstrating superior scalability and sample efficiency in very high-dimensional settings. Our results indicate that value-guided flows offer a principled and practical route to exploration at scale.

Published as a conference paper at ICLR 2026
SCALABLE EXPLORATION FOR HIGH-DIMENSIONAL CONTINUOUS CONTROL VIA VALUE-GUIDED FLOW
Yunyue Wei
Chenhui Zuo
Yanan Sui
Tsinghua University
yunyuewei@mall Berpiii..sssngggg tsinghua. edu. zuoch22emails ttsinghuaa edu. cn ysui@tsinghua enu.cn cn
ABSTRACT
Controlling high-dimensional biological and robotic systems is challenging due to expansive state-action spaces, where effective exploration is critical. Com- monly used exploration strategies in reinforcement learning are largely undirected with sharp degradation as action dimensionality grows. Many existing methods resort to dimensionality reduction, which constrains policy expressiveness and forfeits system flexibility. We introduce Q-guided Flow Exploration (QFLEX). scalable reinforcement learning method that conducts exploration directly in the native high-dimensional action space. During training. QFLEX traverses ac- tions from a learnable source distribution along a probability flow induced by the learned value function, aligning exploration with task-relevant gradients rather than isotropic noise. Our proposed method substantially outperforms represen- tative online reinforcement learning baselines across diverse high-dimensional continuous-control benchmarks. QFLEX also successfully controls a whole-body human musculoskeletal model to perform agile, complex movements, demonstrat- ing superior scalability and sample efficiency in very high-dimensional settings. Our results indicate that value-guided flows offer principled and practical route to exploration at scale.
INTRODUCTION
Controlling over high-dimensional dynamical systems underpins a broad range of applications in robotics, sports, and embodied intelligence from legged locomotion to full-body musculoskeletal control. Complex sensorimotor coordination and over-actuation common in such systems. As the number sensors and actuators grows, these systems gain flexibility and robustness, enabling agile and precise movements. However, increasing dimensionality also amplifies the challenges of coordination and learning efficiency due to rapidly expanding state-action spaces, making effective exploration strategies essential.
A widespread practice in online deep reinforcement learning (RL) is to inject undirected stochastic- ity (e.g., Gaussian noise) into policy outputs for exploration (Haarnoja al., 2018). While simple and effective in moderate dimensions, such isotropic perturbations rapidly lose coverage and be- come sample-inefficient action dimensionality and actuator redundancy grow, yielding vanishing signal for discovering task-relevant actions. Dimensionality reduction-based learning constrains control within low-rank subspaces to make search tractable (Berg et al., 2024). Such complemen- tary strategy may forfeit the flexibility and redundancy that high-dimensional dynamical systems designed to provide.
Iterated sampling techniques, most notably diffusion models and flow-based transports, have achieved striking success in high- high-dimensional generative modeling, providing robust procedures for sampling in thousands of dimensions (Song al., 2021; Lipman al., 2023). Motivated by these advances, several works have adapted iterated seraaed-ssmpling ideas to control and decision making (Janner al., 2022; Yang al., 2023). Despite promising results in moderate-dimensional set-

---

Published as a conference paper at ICLR 2026
tings, these methods have not demonstrated success for nigh-dimensional continuous control with substantial over-actuation.
In this paper, we introduce Q-guided Flow Exploration (QFLEX), scalable exploration mechanism that operates in the native high-dimensional action space. QFLEX achieves directed exploration for policy improvement by sampling from probability flow induced by learned state-action value function Q, proposing informative actions aligned with task-relevant direction. Our design preserves flexibility of the complex systems, and integrates cleanly into actor-critic loop, yielding efficient learning across diverse high-dimensional continuous-control benchmarks. QFLEX also succeeds in controlling full-body musculoskeletal system to perform complex movements, highlighting its scalable and efficient exploration.
Our contributions: (I) We propose QFLEX, scalable RL method which achieves value- aligned directed exploration with policy-improvement validity, enabling efficient learning over high- dimensional state-action spaces. (2) We present an actor-critic implementation QFLEX that con- sistently outperforms representative Gaussian-based and diffusion-based RL baselines range of high-dimensional continuous-control benchmarks. (3) We demonstrate QFLEX on a full- body human musculoskeletal model with 700 actuators, achieving agile, complex movements and efficient exploration without dimension reduction.
wide
2 RELATED WORK
High-dimensional over-actuated control. The control of nigh-dimensional dynamical system is challenging due to its high-dimensionnlity and over-actuation. With few model-based strategies (Hansen et al., 2024; Wei et al., 2025), model-free deep reinforcement learning is the mainstream solution for solving complex control tasks (Kidziński al., 2018; Geißß et al., 2024; Caggiano al., 2024). Hierarchical RL decomposes decision making into high-level planning and low-level control, reducing exploration burden by restricting search to joint- or skill-level choices (Lee et al., 2019; Park et al., 2022; Feng et al., 2023). Curriculum- based learning iterates over sub-tasks to smooth the learning curve for diverse skill learning over nigh-dimensional embodiment (Caggiano et al., 2023; Park al., 2025). DEP-RL employs bio-inspired sampling for coordinated exploration (Schumacher et al., 2023b;a). Lattice generated correlated noise for exploration by injecting stochasticity into la- tent embeddings of the policy network (Chiappa al., 2023b; Simos al., 2025). Synergy-basee approaches such as DynSyn (He et al., 2024) learn or impose low-dimensional control subspaces derived from morphology or task structure, enabling more stable training on systems with high de- grees of freedom. These methods primarily mitigate undirected exploration issue by explicit or implicit dimensionality reduction, which can constrain policy expressiveness and underutilize re- pondaancy-potentiin limiting the flexibility required for agile, task-diverse movements.
Iterated sampling-basee based online reinforcement learning. Inspired by early successes of iterated sampling for offline and imitation decision making (Janner et al., 2022: Chi al., 2023), many works have adapted diffusion-batte based policy parameterization to online reinforcement learning to encourage diverse action distribution (Yang al., 2023; Li al., 2024; Ishfaq et al., 2025; Celik et al., 2025). DACER introduces diffusion actor-critic with an entropy regulator to stabilize policy learning and maintain exploration (Wang et al., 2024). Given unknown target distributions, several studies utilize the learned value function to regularize the policy learning (Ding et al., 2024; Dong et al., 2025; Jain et al., 2025). QSM matches the score of diffusion policy with the gradient of the Q-function (Psenka et al., 2024). SDAC introduces Q-reweighted score matching function to avoid unstable training of backpropagating gradients through the diffusion chain (Ma et al., 2025). Recent works also employ flow-based policy in KL-constrained policy optimization (Ly et al., 2025; McAllister et al., 2025). These methods typically use standard Gaussian as a general initial distribution for the primary goal of multi multi-modal policy learning. The uninformative, isotropic bases can hinder scalability of policy learning in high- digh-dimensional continuous control.
3 PRELIMINARIES
3.1 HIGH--DIMENSINAL CONTINUOUS CONTROL
In this paper, we formalize the control of high sigh-dimensional dynamical system as a infinite horizon Markov decision process (MDP) defined by the tuple M (S, 7, f.r, p), where S c [R|() is
2

---

Published as a conference paper at ICLR 2026
the state space, A C |-1, is the action space, 7/ is the discount factor, f = P(S) is the transition probability to s' E S when being in 8 E S and executing a E A, r= x is the reward function, and == S + P(S) is the initial state distribution. The MDP starts from an initial state 8y sampled from p. and proceeds with actions sampled from a policy л := P(A).
Our goal is to optimize the policy parameters to maximize the discounted cumulative reward:
J(x) R T (r(sh;
Compared with low-dimensional dynamical systems, controlling high-dimensional dynamics is sub- stantially more challenging. We use the human musculoskeletal system as motivating example to illustrate the difficulties of high-dimensional continuous control.
High dimensionality. Full-body human locomotion integrates rich sensory feedback with more than 600 muscles. Unlike robotic arms or quadrupeds, which typically operate within state and action spaces the order of tens, the human musculoskeletal system features state and action spaces that are orders of magnitude larger. The size of the state-action space grows rapidly with dimension, leading pronounced "curse-of-dimensioonllll
effects (Köppen, 2000). This demands expres- sive models and substantial informative data to reliably map high-dimensional states and actions to control performance.
Over-actuation. The number of biological actuators far exceeds the system's degrees of freedom (DoFs): many joints be actuated by multiple muscles, and identical joint torques can arise from numerous activation patterns. This redundancy enlarges the feasible action set and complicates ex- ploration and credit assignment, multiple action sequences can yield indistinguishable kinematics but different internal forces and costs (Valero-Cuevas et al., 2015).
3.2 ACTOR-CRITIC ONLINE REINFORCEMENT LEARNING
Online reinforcement learning typically employs actor-critic framework, where the Q-function (= a) represents the value of state-action pair (s, a) under policy
(r(Sn, an) = 8, ag a
The value function and the policy can be iteratively learned via a two-step scheme: policy evaluation and policy improvement. During policy evaluation, the Q-function updated by Bellman equation operator Tx from any function Q which converges to Q" when the operation number goes to infinity:
T"Q( an) A r(8n, as) + 7Baa [Q(s4+1, 11)]
where the transition data tuples (8h, (h, r(sh, an), 88+1) are collected by interacting with the en- vironment and stored in the replay buffer B. To enhance the stability of Q-function update, two or more Q-functions are learned with separate parameters, where we use the minimum estimation to compute the regression target. In high-dimensional continuous control setting, the transition tuples are often collected from large number of parallel environments for better time efficiency.
In policy improvement, the policy parameters 0 can be updated via optimizing the Q-function: TTa (8, a).
new arg max Ex.ax
In practice, the Q-function and the policy are typically parameterized by neural network Qs and me. and optimized by minimizing the following loss functions with gradient descent:
Lo(Ø)= B,s.a, J-8 [(r(s, a) + 7Eer [Qo(s', a')l - (csssaa))]]
(5)
L (0) = Eg (8.
(6)
3.3 FLOW MATCHING
Flow matching is a simulation free method for generative modeling that learns a probability flow directly by matching velocity fields along continuous-time probability paths. Let (00) (mo) and
3

---

Published as a conference paper at ICLR 2026
coussian-based Exploration
Q-Guided Flow Exploration
[ 5
A 10
| 50
[ 100
Figure 1: Exploration behavior across increasing action dimensionality. The gray polyline de- picts planar kinematic chain with .A| degrees of freedom. The orange background (darker is higher) visualizes the state-action value Q. Green contours show the end-effector distribution in- duced by undirected Gaussian proposal over joint angles, whose exploratory reach collapses as increases. Red streambines/contours depict Q-guided probability flows that transport probability mass from the Gaussian proposal toward high-value modes, sustaining directed exploration high dimensions.
|A|
(x1) denote the source and target distributions over IRd respectively. Flow matching considers continuous- time probability path (x))oo))))) that evolves smoothly from p (xx) to (x1).
This evolution is governed by velocity field (x)) such that the density p(t) satisfies the conti- nuity equation:
dpl (x)
pp (x)x = 0.
dt
Assuming the path p is regular enough, can define flow map Φ00) through the following Ordinary Differential Equation (ODE):
(x) D (0) dl
(x) x.
(8)
We denote (x) as (+))) Flow matching seeks to Iearn an approximate velocity field u (x, t), parameterized by neural network, that induces flow transporting Pa to p1. To train the flow matching minimizes the expected squared error between the model and a reference (or target) veloc- ity field, which tractable when conditioned on samples from the target distribution (Lipman al., 2023):
CCFM E
(((((011) (.) (1()) mp(h) (1)m(11)
VANISHING EFFECTIVENESS OF UNDIRECTED EXPLORATION
To enable exploration during interaction with the environment, the policy æ is often designed to be a stochastic distribution (Haarnoja et al., 2018) or perturbed with undirected, isotropic noises (Schulman al., 2017). Gaussian distribution a commonly used choice for policy and noise pa- rameterization for its simplicity and tractable likelihood computation. The policy that uses Gaussian for exploration follows the generalized form as:
<Tals) s) (N(p(s)
(10)
+ RIAI are mean and (typically) diagonal standard deviation functions of each Gaussian action distribution. In this section, demonstrate that the undirected stochasticity leads to vanishing exploration in high-dimensional continuous control by the following case analysis:
where == RI R and o = R S
4

---

Published as a conference paper at ICLR 2026
Case analysis: vanishing exploration in high DoF settings. Consider planar kinematic chain with |A| sfgrees-of-freedone (i.e. |A| revolute joints and terminal link) in 2D, where each link has length 4 L/ Under i..d. zero-mean joint-angle perturbations with fixed variance, the end-effector position variance scales as O ((TT)): equivalently, it decays proportionally to Al as [A] grows. (See Appendix A.1 for the proof.)
We visualize the vanishing exploration in Figure 1. When the action dimension is moderate (A < 10), Gaussian-based exploration suffices to find informative samplings with high values. However, the diversity of the Gauusian-bbsse exploration collapses as the system complexity grows, leading to uninformative sampling behavior. Related observations of vanishing exploration for over-actuated systems have also been reported in previous works (Schumacher et al., 2023b), These findings motivate directed exploration mechanisms rather than relying on isotropic perturbations.
Algorithm 1 Q-guided Flow Exploration (QFLEX)
Input: Initialized parameters 0, o, gradient step number N, initial gradient step size learning
rates AØ. Ao, Aw
1: for h = 1,2, do
2: an "8 (118)
(1)
3:
8/+1 ~ p(-lap-- s1))
4:
B B ( ah. Ph: $h+1)
5:
Φ - 0 do (o
Eq. (5) Eq. (6)
6:
0 r Ao (o)
7:
a (0) ~ (-1a)
8:
for to N do
9:
a (x) < a + Vo ((xx)))
Q-guided flow construction
end for
11: w < w Aw V (w)
Eq. (17)
end for
FLOW-BASED BASED POLICY FOR SCALABLE EXPLORATION
In this section, first demonstrate how policy improvement can be achieved by sampling from probability flow guided by the learned state-action value function. Then we introduce QFLEX, an efficient online RL method for scalable exploration in nigh-dimensional continuous control.
5.1 POLICY IMPROVEMENT VIA VALUE-GUIDED FLOW
Since undirected exploration vanishes in high-dimensional action space, a suitable strategy to ex- plore is to take the "best" action under current experience. Given policy vold and the Q-function QTid, the policy improvement procedure in Eq. (4) seeks new policy Tnew that maximizes the ex- pectation of FFuu, We denote the policy learned by minimizing Eq. (6) as π x()), , In practice, x often deviates from neew due factors such as restrictive parameterization or insufficient optimiza- tion. To bridge this gap, we construct a Q-guided velocity field that transports (()) towards Teewi
(1) (4) di UQ
(a MV QTT (8,a),
(0)
π (0) (-1s),
(11)
where M is any positive definite preconditioner that that rescales and reorients the raw action- gradient. Defining the advantage of (() over π (U) as:
F(t: s) = (a-r()(1ee) (QTTH(s,
(12)
Under mild regularity assumptions, the transformed policy n (1 thth (TT(-11)) constitutes valid policy-improvemmnt flow which increases the expected state-action value.
Proposition 1. Assuming QTaI is once continuously differentiable with locally Lipschitz V VaQTT
M has bounded operator norm [| and (Tall is intergrable under (:)(s) (- for relevant
f. Then the map t + F(t; s) monotone nondecreasing, i.e., &. F((: s) 2 0. (See Appendix A.2 for the proof.)
5

---

Published as a conference paper at ICLR 2026
Figure demonstrates that the Q-guided flow consistently steers actions toward high-value regions across action dimensionalities, enabling directed exploration and yielding more informative samples.
5.2 Q-GUIDED FLOW EXPLORATION
As summarized in Algorithm I, we embed the above results into actor-critic online RL routine and introduce Q O-guided Flow Exploration (QFLEX), which explores tigh-dimensional action spaces via sampling from the Q-guided conditional normalizing flow in Eq. (11). We parameterize the flow- based policy via a Gaussian initializer To Starting from initial samples drawn from Gaussian policy, QFLEX transform actions following the learned vector field by solving the ODE:
(0) (als) and state-dependent velocity field (waat, 8, (
(1) T0 (als, ( a (0) + S'" (1, a dd,, a (0) K (-18).
(0)
(13)
The training of QFLEX proceeds as follows:
Update of Q-function and Gaussian policy. At each training iteration, QFLEX collects trajectories into replay buffer B by sampling from the flow-induced policy =g,w
(1) (line 2-4). The Q-function and the Gaussian policy are updated according the standard policy iteration and policy improvement steps (line 5-6). Since the sample efficiency of QFLEX hinges on the quality of the learned Q- function, we employ batch normalization within the Q-network to normalize state-action batches and stabilize optimization (Bhatt et al., 2024). This stabilization allows to dispense with target Q-network and to train with a low update-to-date ratio, yielding more efficient Q-learning.
Q-guided flow construction. Starting from samples of (()) we adopt identity matrix I as the preconditioner of the Q-guided velocity field, which corresponds to Euclidean steepest ascent in action space. We then construct the Q-guided flow by taking N finite gradient-ascent steps on the differentiable Q-function, where the transported actions (1) are treated as samples from the target distribution π (1) (line 7-9). Because the Q-network's gradients can be poorly behaved outside the admissible action domain, updates near the boundary may push actions outside [-1, 1]|A1, Thus a fixed step size li can destabilize learning. To mitigate this, cap each update using the la-diameter of the action space:
2A
( t- a ( + +iVaao((sn a (5) n = min
7) (
(14)
V VaQa a w
The truncated step size bounds the per-iteration displacement, enabling stable, valid exploration within the action space.
Update of Q-guided velocity field. Given target a (2) and source sample a (0) from Gaussian policy (0) specify the optimal transport conditional probability path and its target velocity field as:
Tg
( a (0) (1) 5 (*) [1 ()) + +la("))
(15)
(1)
(a ( (0) (1) = a a (0)
(16)
where d(-) denotes the Dirac distribution. The velocity field v.. can be updated by optimizing the state-dependent conditional flow matching loss (line 11):
st (0 s (0) a w>)
(ww E
(17)
s.a 10 (+(s)
(m (-bx.a)))
(0 (-1e, (0)
a
Compared with ciffusiin-based online RL methods that initialize from a fixed standard Gaussian, QFLEX maintains learnable source distribution. This yields informative initialization points for transport toward the target distribution and substantially easing the learning of high- perforerforing flow- based policies. In contrast approaches that rely dimensionality reduction, QFLEX preserves

---

Published as a conference paper at ICLR 2026
(a)
SMPL Humannidd
Uniree
Myolland
Myokig
Ostrich
MS Human- 700
[S] 358
[S] 51
[S] 83
S 403
|S] = 596
[S] = 3148
[A]
|A| 19
|AI
LAI
80
LA 120
[AI 700
(b)
Qfiex
CrossQ
SAC
SDAC
DACER
QSM
DynSyn
Lattice
DEP-RL
SMPL Humenold-Jump
Unitree H1-Run
Unitree HI-Ballnce
Episode Return
40
Episode Return
Episode Return
600
600
400
200
0
1.00
Stops
1e6
Steps
1e6
Stops
1e7
fand Twirl
Myoling Walk
Ostrich Run
Episade Return
1500
Episode Return
15000
Episode Return
15000
1000
10000
5000
5000
0
0.0
0.5
1.0
1.5
2.0
0.25
0.50
0.75
1.00
Steps
1e6
Steps
1e7
Stops
1e7
Figure 2: Control over high-dimensional control benchmarks. (a) Morphologies and state-action dimensions of evaluated benchmarks. (b) Learning curve of algorithms. Results show mean perfor- mances with one standard deviation of 5 independent runs. Baselines in the second row are run only on musculoskeletal benchmarks.
the full flexibility of high-dimensional dynamical systems by exploring the original action space, thereby facilitating agile, complex motor control.
Although Algorithm presents minimalist instantiation, QFLEX readily extends to various RI. frameworks and exploration regimes. The flow-based policy parameterization permits direct sim- ulation of policy likelihoods via the instantancous change of variables (Chen et al., 2018), making QFLEX naturally compatible with KL-constrained policy optimization (Schulman al., 2015; 2017) and maximum-entron entropy RL (Haarnoja et al., 2017). Moreover, geometry-aware or aurvature-adaption choices of the preconditioner M (e.g. natural-gradien or Newton-type updates) can induce more structured exploration to accelerate search. We leave a systematic study of these design choices to future work.
EXPERIMENT
In this section, we present a comprehensive evaluation QFLEX for high-dimensional continuous control. We first compare QFLEX against extensive online RI. baselines on simulated benchmarks. Then we demonstrate its control performance on a 700-actuator human musculoskeletal model ex- ecuting agile, full-body movements. Finally, we analyze QFLEX's behavior to assess its scalability in exploration. For all experiments, we construct the Q-guided flow by N = 20 gradient steps with initial step size 7 0.01. The ODE in Eq. (13) is solved with naive Euler integrator by 20 dis- crete steps with timestep At 0.05. Our code and video results can be found in our project page: https: 1nsgroup. ec/ research/gfi ex.

---

Published as a conference paper at ICLR 2026
(a)
ME-humaa--7BB-Wal
(b)
20000
15000
SAC
Dyndym
Epiced Retum
10000
Lattice
DEIF-LL
5000
-5000
0.0
0.5
1.0
2.0
Stwps
307
Figure 3: Control over full-body human musculoskeletal system. (a) Learning efficiency over walking control of MS-Human AS-Human--700 Results show mean performances with one standard deviation of independent runs. (b) Learned behavior of whole-body running. (c) Learned behavior of ballet dancing.
6.1 CONTROL OVER IIGHI-IIENSIONAL SIMULATED BENCIMARRS
We evaluate on a diverse suite of simulated high-dimensional continuous-contre benchmarks: (1) SMPL Humanoid-Jump (Tirinzoni et al., 2025), which controls humanoid agent based on the SMPL skeleton (Loper et al., 2023) to execute jumps; (2) Unitree H1-Run/Balance (Sferrazza al., 2024), which controls Unitree H1 humanoid'
forward maintain balance stable platform; (3) MyoHand-Pen Twirl Myoleeg eag-Walk (Caggiano et al., 2022), which controls hand musculoskeletal system to twirl pen and lower-body musculoskcletal system to walk; and (4) Ostrich-Run (La Barbera et al., 2021), I), which controls an ostrich musculoskeletal system to run. The state and action spaces for all tasks are summarized Figure (a).
We compare QFLEX to representative online RL baselines: (1) Gaussian-basedd CrossQ (Bhatt al., 2024), SAC (Haarnoja et al., 2018); (2) Diffusion-baseds SDAC (Ma al., 2025). DACER (Wang et al., 2024), QSM (Psenka et al., 2024); and (3) figh-dimensiooal musculoskeletal con- trol: DynSyn (He al., 2024), Lattice (Chiappa et al., 2023a), DEP-RI. (Schumacher et al., 2023b).
As shown in Figure 2 (b), QFLEX demonstrates consistently superior learning efficiency across all benchmarks. The performance gap widens with increasing action dimensionality and over-actuation, indicating scalable exploration behavior.
6.2 CONTROL OVER FULL-BODY HUMAN MUSCULOSKELETAL SYSTEM
We employ QFLEX for locomotion control of MS-Human-700 (Zuo et al., 2024), a full-body musculoskeietal system with 206 joints and 700 muscle-tendon units. Its state-action dimen- sionality is more than five times that of the most complex benchmark in the previous sub-
1.0
Flow Superiority
0.8
section (Ostrich-Run). As shown Figure (a), QFLEX exhibits high-learning efficiency and strong scalability over whole-bodd body walking control, outperforms existing high-dimensional musculoskeletal control baselines by large margin without dimension reduction.
SMPL llumenodd-jumo
0.7
Unitree HI-Run
Myoriands-RenTTrtt
0.6
Myoleg Walk
Ostrich-Run
0.5
MS Human 700 Walk
20%
40%
100%
We further deploy QFLEX on two challeng- ing skills-running and ballet dancing that, to our knowledge, have not previously been demonstrated 700-actuator full-body sys- tem. In Figure 3 (b), QFLEX enables rapid high-dimensional sensorimotor coordination,
Training Procedure
Figure 4: Sample quality between QFLEX and source Gaussian policy during training. Over- actuated musculoskeletal control tasks de- noted as dash-dotted lines.
8

---

Published as a conference paper at ICLR 2026
achieving stable rumning gait. In Figure 3 (c), QFLEX successfully imitates ballet routine featur- ing complex whole-body sequences with single-foot spins and balance. By exploring in the original action space, QFLEX fully leverages the flexibility of high-dimensional dynamical systems, enabling agile and complex motion control.
6.3 ALGORITHM ANALYSIS
We examine QFLEX's efficiency by analyzing its learning dynamics. To compare sampling quality between the flow-based policy and the Gaussian reference, we track the flow superiority ratio during training, which is the proportion of states in minibatch for which
(ss, f8 (+ss)
((s To (-1s)).
(0)
(18)
As shown in Figure 4, the flow- based policy consistently yields higher action values than
17500
Gaussian exploration, and this advantage strengthens over the course of training. Notably, the superiority ratio is substan- tially higher musculoskele- tal control tasks than on torque- controlled benchmarks, under- scoring the importance of value- aligned exploration in high- dimensional, over-actuated set- tings.
15000
15000
15000
Fosede tatura
12500
12500
10000
10000
7500
Episode
7500
5000
2500
61-0.0
50
10.2
0.02
0
14 100
0-
20-2
A1 0.03
Steps
Steps
Steps
(8)
(b)
(c)
Figure 5: Ablation study over hyperparameters of QFLEX. (a)
Gradient steps N. (b) Step size 77. (c) Euler solving timestep At.
On MyoLeg- Myokeg-Walk Walk task, we fur- ther perform sensitivity study over QFLEX's hyperparameters: number of gradient steps N, step size and Euler solving timestep At. Figure shows broadly comparable learning performance across a reasonable range of these choices.
7 CONCLUSION
In this paper, we introduce QFLEX, scalable online RL method for efficient exploration in high- dimensional continuous control. Our method conducts directed exploration by sampling from Q- guided probability flow with policy-improvement guarantees, yielding superior learning efficiency over representative online RL baselines across benchmarks characterized by high dimensionality and over-actuation. QFLEX further demonstrates agile, complex motion control musculoskeletal model with 700 actuators, achieving high efficiency and strong scalability in truly high-dimensional settings. Our analysis shows that aalue-dligned exploration in QFLEX surpasses undirected sampling strategies in high-dimensional regimes, which readily extensible to a variety of online RL frameworks and exploration settings.
full-body
ACKNOWLEDGMENTS
This work is supported by STI 2030-Major Projects 2022ZD0209400, Beijing Academy of Artificial Intelligence and Beijing Municipal Science & Technology Commission 7251100008125022 and AMD Research. Correspondence to: Yanan Sui (ysui @tsinghuu..duuucm)
Ethics statement. This work follows the ICLR Code of Ethics. We considered the potential ethical and socictal impacts of this work. No human or animal subjects were directly involved. We report limitations and assumptions transparently and strive to promote beneficial and responsible use of this work.
Reproducibility statement. We provide anonymous codebase, full hyperparameters, and exact evaluation protocols to enable faithful replication.
LLM usage statement. We used large language model solely for language polishing.

---

Published as a conference paper at ICLR 2026
REFERENCES
Cameron Berg, Vittorio Caggiano, and Vikash Kumar. Sar: Generalization of physiological agility and dexterity via synergistic action representation. Autonomous Robots, 48(8):28, 2024.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, 2024. URL https: wwppopeeeeiiiieee net/ forum? d=p.cdtTTTTT......C
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milln and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com/ jai-maa ml/jax.
Vittorio Caggiano, Huawei Wang. Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite-a contact-rich simulation suite for musculoskeletal motor control. arXiv preprint arXiv:2205. 2022.
Vittorio Caggiano, Sudeep Dasari, and Vikash Kumar. Myodex: a generalizable prior for dexterous manipulation. In International Conference on Machine Learning, pp. 3327-3346. PMLR, 2023.
Vittorio Caggiano, Guillaume Durandau, Seungmoon Song, Chun Kwang Tan, Huiyi Wang, Balint Hodossy, Pierre Schumacher, Letizia Gionfrida, Massimo Sartori, and Vikash Kumar. Myochal- lenge 2024: Physiological dexterity and agility bionic humans. In NeurIPS 2024 Competition Track, 2024.
Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion- maximum entropy reinforcement learning. arXiv preprint ar arXiv:2502.02316, 2025.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273666 2023.
Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Huang, and Alexander Mathis. Latent ex- ploration for reinforcement learning. Advances Neural Information Processing Systems, 36: 56508-56530, 2023a.
Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Huang, and Alexander Mathis. Latent ex- ploration for reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023b.
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 37:53945-53965 2024.
Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy. In Forty-second International Conference on Machine Learning, 2025.
Yusen Feng, Xiyan Xu, and Libin Liu. Musclevae: Model-based controllers of muscle-actuated characters. In SIGGRAPH Asia 2023 Conference Papers, pp. 1-11, 2023.
Henri-Jacques Gei,3, Firas Al-Hafez, Andre Seyfarth, Jan Peters, and Davide Tateo. Exciting ac- tion: investigating efficient exploration for learning musculoskeletal humanoid locomotion. In 2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids), pp. 205-212. IEEE, 2024.
Tuomas Haarojj,, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control. In The Twelfth International Conference on Learning Representattinss 2024.
Kaibo He, Chenhui Zuo, Chengtian Ma, and Yanan Sui. Dynsyn: Dynamical synergistic represen- tation for efficient learning and control in overactuated embodied systems. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Haque Ishfaq, Guangyuan Wang, Sami Nur Islam, and Doina Precup. Langevin soft actor-critic: Efficient exploration through uncertainty- critic learning. In The Thirleenth International Conference on Learning Representations, 2025.
Vineet Jain, Tara Akhound Saaheh, and Siamak Ravanbakhsh. Sampling from energy-based -bsed policies using diffusion. In Reinforcement Learning Conference, 2025.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
Eukasz. Kidziñski, Sharada P Mohanty, Carmichael F Ong, Jennifer I. Hicks, Sean F Carroll, Sergey Levine, Marcel Salathé, and Scott L Delp. Learning to run challenge: Synthesizing physiologi- cally accurate motion using deep reinforcement learning. In The NIPS'17 Competition: Building Intelligent Systems, pp. 101-120. Springer, 2018
Mario Köppen. The curse of dimensionality. In 5th online world conference on soft compuling in industrial applications ( (WSC5), volume 1, pp. 4-8, 2000.
Vittorio La Barbera, Fabio Pardo, Yuval Tassa, Monica Daley, Christopher Richards, Petar Kor- mushev, and John Hutchinson. Ostrichrl: A musculoskeletal ostrich simulation study bio- mechanical locomotion. In Deep RL Workshop NeurIPS 2021, 2021.
Seunghwan Lee, Moonscok Park, Kyoungmin Lee, and Jehee Lee. Scalable muscle-actuated human simulation and control. ACM Transactions On Graphics (TOG), 38(4):1-13, 2019.
Steven Li, Rickmer Krohn, Tao Chen, Anurag Ajay. Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neu- ral Information Processing Systems, 37:38456-38479 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow match- ing for generative modeling. In The Eleventh International Conference on Learning Representa- tions, 2023.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. pp. 851-866. 2023.
Lei Ly, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma. Flow-based policy for online reinforcement learning. arXiv preprint arXiv:2506. /28/1, 2025.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. In Forty-second International Conference on Machine Learning, 2025.
David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients. arXiv preprint arXiv:2507.21053. 2025.
Jungnam Park, Sehee Min, Phil Sik Chang, Jaedong Lee, Moon Seok Park, and Jehee Lee. Genera- tive gaitnet. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-9, 2022.
Jungnam Park. Euikyun Jung, Jehee Lee, and Jungdam Won. Magnet: Muscle activation generation networks for diverse human movement. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pp. 1-11, 2025.
11

---

Published as a conference paper at ICLR 2026
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. In International Conference on Machine Learning, pp. 41163-41182 PMLR, 2024.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Pierre Schumacher, Thomas Geijtenbeek, Vittorio Caggiano, Vikash Kumar, Syn Schmitt, Georg Martius, and Daniel FB Haeufle. Natural and robust walking using reinforcement learning without demonstrations in high- sigh-dimensional museuloskeletal models. arXiv preprint arXhv:2309.02976 2023a.
Pierre Schumacher, Daniel Haeufle, Dieter Büchler, Syn Schmitt, and Georg Martius. Dep-rl: Em- bodied exploration for reinforcement learning in overactuated and musculoskeletal systems. In The Eleventh International Conference on Learning Reprexentations, 2023b.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv Preprint 4xxiv:2403.105500 2024.
Merkourios Simos, Alberto Silvio Chiappa, and Alexander Mathis. Reinforcement learning- based motion imitation for physiologically plausible musculoskeletal motor control. arXiv preprint 2rXi: 2025.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. Interna- lional Conference on Learning Representations, 2021.
Bhavya Sukhija, Stelian Coros, Andreas Krause, Pieter Abbeel, and Carmelo Sferrazza. Maxinforl: Boosting exploration in reinforcement learning through information gain maximization. arXiv preprint arXiv:2412.12009 2024.
Andrea Tirinzoni, Ahmed Touati, Jesse Farebrother, Mateusz Guzek, Anssi Kanervisto, Yingchen Xu, Alessandro Lazaric, and Matteo Pirotta. Zero-shot whole-body humanoid control via behav- ioral foundation models. arXiv preprint / 1054, 2025.
Francisco Valero Cuevas, BA Cohn, HF Yngvason, and Emily L. Lawrence. Exploring the high- dimensional structure of muscle redundancy via subject-specific and generic musculoskeletal models. Journal of biomechanics, 48(1 (888))22887,,,6, 2015.
Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang. Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-54200 2024.
Yunyue Wei, Shanning Zhuang, Vincent Zhuang, and Yanan Sui. Motion control of high- dimensional musculoskeletal systems with hierarchical masel-based planning. In The Thirteenth International Conference Learning Representations, 2025.
Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for rein- forcement learning. arXiv preprint a3Xiv:2305. 2023.
Chenhui Zuo Kaibo He, Jing Shao, and Yanan Sui. Self model for embodied intelligence: Mod- eling full-body human musculoskeletal system and locomotion control with hierarchical low- dimensional representation. In 2024 IEEE International Conference on Robotics and Automation (ICRA). pp. 13062- 13069. IEEE, 2024.
12

---

Published as a conference paper at ICLR 2026
A THEORETICAL PROOFS
A.1 PROOF OF CASE AIS
Case analysis: vanishing exploration in high DoF settings. Consider a planar kinematic chain with |A|] begrees-of-freedon (i.e. |A| revolute joints and terminal link) in 2D, where each link has length 4 L/) Under i.id. zero-mean joint-angle perturbations with fixed variance, the end-effector position variance scales as O( [A equivalently, it decays proportionally to Al as (Al grows.
Proof. We denote the position of the end-effector as æ (I, v). The forward kinematics of the system can be expressed as a function:
f(+),
(19)
where - (.0),... *PIA)) is the system joint positions. For small noise, we can use a first-order Taylor expansion of f(()) around the current joint position yo:
x f(() + (15)64p
(20)
where J(Ø) is the Jacobian matrix of the forward kinematics with respect to S. and öe (ōp),... .. , )AILA)) with Var(ōp;) =a. The covariance matrix of dep Sup = I.I. Therefore the covariance of the end position is given by:
LA Σ II max Σ
Σn = J(
(21)
where || the norm of the i-th column of the Jacobian matrix J, and d max; 0; Where we can extract the end position variance as the trace of Eg:
Var(x) = Tr(2),)
(22)
For planar [Al-link system where each link 1, Al, the Jacobian entries influenced by these link lengths, and the trace term can be approximated as:
L
L
(.ll |A|
(23)
AAp
TAi
which leads to the total variance of the end-effector as:
022 L
Var(x)
(24)
|AA
A.2 PROOF OF PROPOSITION 1
Proposition 1. Assuming QT is once continuously differentiable with locally Lipschitz V Ti
M has bounded operator norm [| and (Faelll II is intergrable under 7(() (-1s) for relevant 1. Then the map + F(1; 8) is monotone nondecreusing, ie., F1; (1; 2 0.
Proof. Let !!) -bb Then can reparameterize F
F(t:s) s) = Ba)r((((a)) [Q (s, 0 (f) (a)) 1,) (s.
(25)
13

---

Published as a conference paper at ICLR 2026
The differentiate under the expectation is
d
F(;
B (a)r)))(1a))
-Ba
-a (1a)
(26)
di
di
B
(1)
(s,
(27)
dt
E (0
V
VOQNu T
dt c (a)
(28)
E (-)7)
VaQ
(29)
E (0
V.Q MV M (s,
(30)
E (-11)
V.Q a) M 20,
(31)
where Eq. (28) follows the derivative chain rule, and Eq. (30) is derived by reparameterization.
B EXPERIMENTAL DETAILS
B.1 ALGORITHM IMPLEMENTATION
We implement QFLEX on the JAX platform (Bradbury et al., 2018). Specifically, the neural networks implemented using Haiku? with parameters optimized with Optax'.
For the implementation of SAC, DACER and QSM, we refer to Ditf--usffiioniiihhheeee implementation
RL4 in the official code repository of DACER, which provide efficient of SAC and diffusion based online RL baselines.
For the implementation of SDAC, we directly use the official repository which provides JAX-based implement based on DACER repository.
For the implementation of CrossQ, we refer the official repository". and reproduce JAX-based implementation to improve the time efficiency of training.
For the implementation of DynSyn, we directly use the official repository and use SAC as the RL backbone.
For the implementation of Lattice, we directly use the official repository and use SAC as the backbone.
For the implementation of DEP-RL, we directly use the official repository?: and use SAC as the RL. backbone.
For all algorithms, we align the network parameters and learning rate with gradient steps after each parallel sampling step. For the training of Lattice, we follow the default training setting of 8 gradient steps after parallel sampling steps, which we consider comparable to other baselines on average. Otherwise the default hyperparameter in the original implementation. The full experimental details is listed Table and
'https: httteer//hhhtiicccmmmooooeeeeeed
httpes///gghbbbcccm///dtyyaassss
hittes//gstaaabcccmm//ems////y/s
htt.es//ggthbbbbccom//mmttiiiiis
14

---

Published as a conference paper at ICLR 2026
Table 1: Training details of each environments.
System
Unitree Hl
SMPL Humanoid
MyoHand
MyoLeg
Ostrich
MS-Human-7700
Parallel number
70
80
80
80
80
224
Critic hidden layer
3
3
3
3
3
Critic hidden size
256
256
256
256
256
1024
Policy hidden layer
3
3
3
3
3
Policy hidden size
256
256
256
256
256
1024
Diffusion/Blow hidden layer
3
3
3
3
Diffusion/flow hidden size
256
256
256
256
256
1024
Table Hyperparameter settings.
QFLEX
SDAC
DACER
QSM
CrossQ
SAC
DynSyn
DEP-RL
Lattice
Gradient steps
8/8
Discount
0.99
0.99
0.99
0.99
0.99
Batch sixe
256
256
256
256
256
256
256
256
256
Buffer size
le6
lc6
1e6
le6
lc6
1c6
lc6
1c6
1c6
Leaminng
3c-4
3e-4
3e-4
3e-4
3o-4
3e-4
3e-4
3o-4
3o-4
Optimizer
Adam
Adam
Adam
Adam
Adam
Adum
Adum
Adam
Adam
Diffusion/llow steps
20
20
20
20
Batch normalization decay
0.99
0.99
Batch normalization
le-5
lc-5
Target policy entropy
0.9 [A]
-0.9 [A]
[A]
-|A]
-|
[A
-|
B.2 BENCHMARK IMPLEMENTATION
SMPL-Humanoid-JJmp We implement the benchmark using the jump-2 task in the official Humenv repository 10 with provided reward function. The environment is wrapped to be compatible with Gymnasium environment make function.
Unitree HI-Run/Balance. We use h1-run-v0 and hi-balance simple-vo tasks in the official HumanoidBench repository"' to implement the benchmark with provided reward function.
We use myoHandPenTwizz www.ir11aaddmvv00 and myoLegwa k-vv tasks in the official MyoSuite repository to implement the benchmarks with provided reward function.
Ostrich-Run. We implement the benchmarks using the ostri ch-run task in the official Os- trichRL repository! with provided reward function. The environment is wrapped to be compatible with Gymnasium environment make function.
MS-Human-7000 Walk. We develop task environments with MS-Human-7700 under Gymnasium. The full states (observations) and dimensions listed in Table 3. We design the following walk reward functions to make the 700-actuator full-body model to walk forward based on a reference walking trajectory from motion capture data:
Wwalk 50 rapos 0.1 Tqvel + 50 Fax 5 Fvvl + 100 Phealthy
(32)
where ypps penalizes the squared error of between model and reference joint position; Typel penalizes the squared error of between model and reference joint velocity: ract penalizes the norm of the total actuator forces; Tvel penalizes the squared error of between model and reference center-of-mass velocity; Thealthy encourages the model not to fall down and deviate from the reference trajectory.
15

---

Published as a conference paper at ICLR 2026
Table States the MS-Human-700-Wainn environments.
State
Dimension
Joint position
85
Joint velocity
855
Joint acceleration
85
Actuator activation
700
Actuator force
700
Actuator length
700
Actuator velocity
700
Simulation time
Phase walking period
Pelvis position
3
Sternum position
3
Joint position error
85
We design the following walk reward functions to make the 700-actuator full-body model to run forward based on a reference trajectory from CMU Graphics Lab Motion Capture Database (Subject #2, Trial #3):
MS-Human-700-RRa
Trn 10 apos + 100 Thealthy
(33)
with reward terms defined same MS Human 700tmmann700---- 700--000--00
Wa k under different reference trajectory.
MS- ASs-Human-Pee Sorimann-70--Deen 700-Dance. We design the following walk reward functions to make the 700-actuator full-body model to perform ballet dancing based on clip of reference trajectory from CMU Graph- ics Lab Motion Capture Database (Subject #5, Trial #9):
dance r rapss
100 Expos + 100 Thealthy:
(34)
where xpss penalizes the squared error of between model and reference body position. The remain- ing reward terms are defined same as M 700
- -aakk under different reference trajectory.
C ADDITIONAL EXPERIMENTS
Comparison with PPO. We follow the official HumanoidBench repository and evaluate PPO on the Unitree H1 Run/Balance task using Babll-Bass We observe that PPO exhibits limited reward improvement, which is consistent with the findings reported in the HumanoidBench paper. QFLEX substantially outperforms this widely used baseline.
800
Qflex
Qflex
Episode Return
Episode Return
600
600
PPO
PPO
400
400
200
200
2
0.00
0.25
0.50
0.75
1.00
Steps
1e6
Steps
1e7
(a) Unitree HI-Run
(b) Unitree H1-Balance
Figure 6: Learning curve of of QFLEX and PPO on Unitree HI tasks
Comparison with flow-based online RL. We compare QFLEX with FlowRL (Lv et al., 2025) on the Unitree HI-Balance task, which also evaluated in the FlowRL paper. Since the official FlowRL
es cmu edu
16

---

Published as a conference paper at ICLR 2026
Method
Myol.eg-War Walk (80 actuators)
MS-Humaa-770-NW 700-Walk (700 actuators)
QFLEX
34.26 2.84
307.51 20.37
CrossQ
38.47 3.49
356.87 + 22.57
Table 4: Energy efficiency measured by total actuator activation (lower is better). Results shows mean performances with standard deviation.
implementation" 15 supports only single-environment training, we run all algorithms in single envi- ronment and align network architectures and training hyperparameters. We successfully reproduce the FlowRL performance reported in the original paper, and QFLEX substantially outperforms this baseline, highlighting its systematic advantages over FlowRL in high-dimensional continuous con- trol (see Figure 7).
Comparison with intrinsic motivation-based RL. We additionally compare QFLEX against Max- InfoRL (Sukhija et al., 2024), mated information gain and includes evaluations on Unitree-Hi robots. We refer to the official im- plementation and use the MaxInfoSAC variant, which is the primary version evaluated in the orig- inal paper. We observe QFLEX significantly outperforms MaxInfoRL on the Unitree HI H1-Balance task (see Figure 7). We consider intrinsic motivation-based RL methods encourage exploration by modifying the learning objective, but they do not directly address the challenge of inefficient sam- pling in high- sigh--imensional continuous control.
method that promotes exploration via esti-
Qflex
300
FlowRL
MaxInfoRL
Episode Return
250
200
150
100
50
0.0
0.5
1.0
1.5
Steps
le6
Figure 7: Learning curve of algorithms on Unitree H1-Balance task. Algorithms are trained on single environment. Results show mean performances with one standard deviation of 5 independent runs.
Ablation over exploration strategy. the Human-700-va task, we directly ablate the flow-based exploration strategy against a Gaussian-based alternative. QFLEX significantly outper- forms the ploration in high-dimensional continuous control (see Figure 8).
variant, demonstrating the systematic advantage of flow-based ex-
Energy efficiency. In the MyoLeg-Walk and Ho--Humm-77----ot tasks, we compare the total muscle activation of QFLEX with CrossQ (the strongest Gaussian-basee policy). QFLEX achieves substantially lower total muscle activation, demonstrating the superior energy efficiency enabled by flow-based exploration (see Table 4).
Runtime analysis. On the MyoLeg-Wall Walk task, we compare the runtime of QFLEX with Gaussian- based and diffusion-basad baselines NVIDIA GeForce RTX 4090 D GPU. QFLEX achieves
cowRi
"nhtps::/gii hhttee://ggthhb...o/////iiiaaaaa
17

---

Published as a conference paper at ICLR 2026
Qfiex
17500
Dolation-Gaussian
15000
Episode Return
12500
10000
7500
5000
2500
0.0
0.5
1.0
1.5
2.0
Steps
1e7
Figure 8: Ablation over exploration strategy on MS-Human-7700 700- Walk task. Results show mean performances with one standard deviation of independent runs.
Method
Training time
Per-step deployment time
QFLEX
52.94 0.29min
0.49 = 0.19ms
SDAC
82.26 0.71min
4.07 + 0.58ms
DACER
75.06 0.08min
0.54 + 0.28ms
QSM
48.36 0.24min
2.95 = 0.39ms
SAC
29.4 = 0.12min
0.11 + 0.05ms
CrossQ
34.83 0.04min
0.13 0.05ms
Table 5: Runtime comparison of algorithms on MyoLeg-Wall task. Results shows mean perfor- mances with one standard error.
comparable lower runtime relative all evaluated diffusion-basee methods (see Table 5,). Al- though its runtime is higher than that of Gaussian-based baselines, this overhead is acceptable given the substantial performance gains and remains well within real-time control requirements.
Correlation of QFLEX exploration. We conduct the exploration analysis on the MyoLeg-Walk task. Because the flow-based distribution difficult visualize directly, we approximate QFLEX's exploration noise by computing the standard deviation of 1,000 sampled actions at each timestep. QFLEX exhibits structured correlations across action dimensions, in contrast isotropic Gaus- sian noise (see Figure 9). We further observe strong correlations among actuators within the same anatomical groups, for example, the gluteus maximus (glmax 1, glmax2, glmax3) and the gastrocne- mius (gaslat, gasmed). These patterns provide evidence that QFLEX performs directed exploration informed by both task structure and system morphology.
18

---

Published as a conference paper at ICLR 2026
Action Std Correlation Matrix
addirev
widlong
addmagDis
addmaglsch
acdmaghid
addnnag ox
0.75
edi
ehi
gestet
glmax2
0.2
glmax3
gimeds
gimin1
0.00
gimin3
itacus
perlong
pri
-0.50 50
sart
semimem
soleus
vasitt
onlat
Smare amaz mmne cone
Figure 9: Correlation matrix of QFLEX exploration right lowerbody muscles in MyoLeg-Wai Walk.
19


Reviews:
[{"review_id": "pmrQGWOubf", "reviewer": "Reviewer_s5BZ", "summary": "This paper proposes a scalable online reinforcement learning method （QFLEX）to efficiently explore high-dimensional continuous control systems. Through a case analysis, the paper demonstrates that the undirected stochasticity leads to vanishing exploration in high-dimensional continuous control. The proposed method, QFLEX, outperforms six other baselines across multiple high-dimensional control benchmarks. Additionally, QFLEX successfully controls a full-body musculoskeletal model with 700 actuators to perform complex tasks, such as walking and ballet dancing.", "questions": "In the Unitree H1–Run/Balance task, the results presented in the paper indicate that QFLEX performs notably better than the other methods. Whether Proximal Policy Optimization (PPO) was tested as well ? It may strengthen the paper's contributions to include a comparison with PPO.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe paper presents clear and concise expression.\n2.\tThe paper presents thorough comparative experiments, testing QFLEX on six benchmark tasks and comparing it with eight baseline methods.\n3.\tThe proposed QFLEX method efficiently addresses exploration in high-dimensional action spaces, overcoming the inefficiencies of undirected exploration.", "weaknesses": "1. Although QFLEX performs well in simulated tasks, the paper does not provide sufficient validation on real-world systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TPTcaNYCeO", "reviewer": "Reviewer_i8Yj", "summary": "The work addresses a critical challenge in high-dimensional continuous control: achieving efficient exploration in complex state–action spaces. Common practices such as adding Gaussian noise to policy actions often fail to promote meaningful exploration—an issue well recognized in the community and clearly demonstrated by the authors. In contrast, the proposed method leverages action samples guided by estimated Q-value improvements, enabling more directed and effective exploration.\n\nThe method outperforms Gaussian-based and diffusion-based RL baselines across several high-dimensional locomotion tasks and even achieves control of a 700-actuator full-body human musculoskeletal model, which is particularly impressive and highlights the practical advantages of the proposed approach.", "questions": "The experiments are primarily conducted on tasks with dense reward signals, which are common in locomotion benchmarks. It might be unclear how the method would perform in sparse-reward environments, where accurate Q-function learning is more challenging. In such settings, would Q-guided flows still yield effective exploration, or would their reliance on precise Q-gradients diminish their advantages?\n\nFrom Algorithm 2, the policy appears to always execute the final transported action from the Q-guided flow when interacting with the environment. This design seems heavily biased toward exploitation, as each exploratory step moves strictly in the direction of higher estimated value. Could this cause the exploration to collapse prematurely and converge to sub-optimal policies, especially when the Q-gradient is inaccurate early in training? Have you considered sampling actions from intermediate states along the flow trajectory—rather than only at the terminal point—to preserve exploration breadth? If so, what empirical effects and trade-offs did you observe?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is well-motivated, clearly established, and novel. The problem it addresses is central to scaling reinforcement learning policies to more complex and realistic control tasks. Leveraging Q-guided flows to generate actions for directed exploration is both natural and insightful, with its benefits demonstrated across several high-dimensional tasks that are typically challenging for RL policies to learn from scratch. Moreover, the method introduces only minimal modifications to a standard actor–critic framework, further enhancing its practicality and applicability.", "weaknesses": "The primary weakness of the proposed method lies in its strong reliance on accurate Q-gradient estimation. During early training stages, or in scenarios where Q-gradients are unreliable, the resulting action updates may misguide exploration and ultimately hinder policy improvement. Although the paper discusses the use of batch normalization to stabilize Q-learning, a more thorough analysis of potential failure cases or mitigation strategies would further strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PA8AdAVrWa", "reviewer": "Reviewer_987M", "summary": "The paper introduces Qflex, a reinforcement learning algorithm designed for efficient exploration in high-dimensional continuous control. Exploration in such systems is challenging because Gaussian noise fails to explore efficiently, particularly in over-actuated settings (e.g. musculoskeletal control). Qflex addresses this by aligning exploration with value-function gradients through a probability flow formulation. The method directs exploration toward high-value regions by integrating a value-guided velocity field. Experiments on several high-dimensional benchmarks demonstrate improved sample efficiency and control performance over Gaussian and other exploration baselines.", "questions": "1. How does Qflex differ theoretically and empirically from [1] and [2]? Are the reported improvements due to the use of value-guided flows? A side-by-side comparison on shared benchmarks would clarify novelty.\n2. Can the authors provide a small-scale toy controlled example (e.g., a 50-100 DOF) comparing Gaussian vs. Qflex noise? Visualizing trajectories and action-space evolution would concretely illustrate how Qflex produces more directed exploration. Additionally, can the authors discuss how much Qflex depends on the value function at the beginning of the training (as the critic is inaccurate) and when it transitions from random (Gaussian-like) to value-guided? How far into the future (short vs. long-term value) does the velocity field effectively look?    \n3. Does Qflex noise exhibit the same correlation structure as the learned action covariance? Is it aligned with principal action subspaces or dependent on state geometry? This would help understanding whether Qflex adaptively focuses exploration on effective actuator subsets.\n4. How does solving the flow ODE scale with action dimension |A|? Does QFlex incur additional runtime or convergence cost compared to SAC, or are gains due to more efficient sampling?\n5. In over-actuated systems, exploration efficiency is often correlated with energy-efficient policies (as shown in [3]). Does Qflex yield more energy-efficient or smoother action trajectories?\n6. Are all baselines trained under identical regimes (fully online)? How are comparisons to DEP-RL (which is an offline method) made? Would Qflex still outperform other exploration baseline if trained with PPO instead of SAC?\n\n[1]: McAllister, David, et al. \"Flow matching policy gradients.\" *arXiv preprint arXiv:2507.21053* (2025).\n\n[2]: Lv, Lei, et al. \"Flow-Based Policy for Online Reinforcement Learning.\" *arXiv preprint arXiv:2506.12811* (2025).\n\n[3]: Chiappa, Alberto Silvio, et al. \"Latent exploration for reinforcement learning.\" *Advances in Neural Information Processing Systems* 36 (2023): 56508-56530.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of aligning exploration with the value function through a flow-based transformation is well grounded and connects flow matching with reinforcement learning in a principled way.\n- The method achieves state-of-the-art performance on several high-dimensional control benchmarks, including tasks with >700 actuators.\n- Qflex integrates easily into standard actor-critic frameworks such as SAC, without architectural modifications.", "weaknesses": "- While applying flow matching to online RL for exploration is interesting, closely related ideas appear in [1] and [2]. The main novelty lies in scaling to high-dimensional systems rather than a fundamentally new formulation.\n- The paper does not analyze how value-guided flows improve exploration or in which regimes they outperform Gaussian or latent-space baselines.\n- The qualitative figure contrasting Gaussian vs. Qflex exploration is illustrative but lacks quantitative evaluation or a controlled ablation demonstrating the behavioral difference.\n- Metrics such as energy efficiency would help interpret whether exploration leads to more structured or efficient control.\n\n[1]: McAllister, David, et al. \"Flow matching policy gradients.\" *arXiv preprint arXiv:2507.21053* (2025).\n\n[2]: Lv, Lei, et al. \"Flow-Based Policy for Online Reinforcement Learning.\" *arXiv preprint arXiv:2506.12811* (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fIoE5G0IwE", "reviewer": "Reviewer_cSUp", "summary": "This paper introduces QFLEX, a novel reinforcement learning method that uses a Q-guided probability flow for directed exploration in high-dimensional action spaces. The approach is motivated by the well-justified problem of vanishing exploration with undirected noise in over-actuated systems. The ideas are clearly presented, and the experimental evaluation is comprehensive, spanning from typical humanoids like Unitree H1 to a challenging 700-actuator musculoskeletal model.", "questions": "1. For the case analysis of vanishing exploration in high DoF settings, the result seems to depend on the choice of the link length $L/|\\mathcal{A}|$. If we choose the length as $L/\\sqrt{|\\mathcal{A}|}$, we can not get the conclusion. Could the authors comment on the choice of $l_i=L/|\\mathcal{A}|$.\n2. The algorithm simultaneously trains the Q-function, the initial Gaussian policy $\\pi^{(0)}$, and the flow model $v_w$. Since both the source and target of this flow are non-stationary, could the author provide more insight or empirical evidence on why this joint training is stable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea of using a learned probability flow for value-directed exploration is novel and well-motivated. It effectively addresses a key limitation (vanishing signal) of isotropic exploration in high-dimensional spaces.\n2. The paper is generally well-written. The main ideas are clear and easy to follow. \n3. The empirical validation is a major strength. The benchmarks are diverse and highly relevant, and the successful application to the full-body musculoskeletal model (MS-Human-700) provides strong evidence for the method's scalability and practical potential.", "weaknesses": "1. The method incurs a non-trivial computational cost. Obtaining an exploratory action $a^{(1)}$ requires calculating the gradient of $Q$ with respect to $a^{(\\frac{n-1}{N})}$ for N times (Eq. 14).\nQuantifying the wall-clock time compared to baselines would strengthen this analysis.\n2. The comparison focuses on Gaussian-based and diffusion-based RL baselines. Including a comparison with other advanced exploration strategies for high-dimensional spaces (e.g., those based on intrinsic motivation or curiosity) could better situate QFLEX within the broader RL exploration literature and highlight its specific advantages.\n3. The intuition behind learning a flow model, as opposed to simply taking a fixed number of gradient ascent steps from the current policy $\\pi^{(0)}$, could be more thoroughly justified. An ablation study directly comparing the learned flow against a simple multi-step gradient ascent baseline would help isolate the benefit of the flow matching component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)