GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies
Authors: He Zhang
Ying Sun
Hui Xiong
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning; Flow-matching policies;
Primary area: reinforcement learning

Abstract: Flow-matching policies hold great promise for reinforcement learning (RL) by capturing complex, multi-modal action distributions. However, their practical application is often hindered by prohibitive inference latency and ineffective online exploration.  Although recent works have employed one-step distillation for fast inference, the structure of the initial noise distribution remains an overlooked factor that presents significant untapped potential. This overlooked factor, along with the challenge of controlling policy stochasticity, constitutes two critical areas for advancing distilled flow-matching policies. To overcome these limitations, we propose GoldenStart (GS-flow), a policy distillation method with Q-guided priors and explicit entropy control. Instead of initializing generation from uninformed noise, we introduce a Q-guided prior modeled by a conditional VAE. This state-conditioned prior repositions the starting points of the one-step generation process into high-Q regions, effectively providing a ``golden start'' that shortcuts the policy to promising actions. Furthermore, for effective online exploration, we enable our distilled actor to output a stochastic distribution instead of a deterministic point. This is governed by entropy regularization, allowing the policy to shift from pure exploitation to principled exploration. Our integrated framework demonstrates that by designing the generative startpoint and explicitly controlling policy entropy, it is possible to achieve efficient and exploratory policies, bridging the generative models and the practical actor-critic methods. We conduct extensive experiments on offline and online continuous control benchmarks, where our method significantly outperforms prior state-of-the-art approaches.

Published as a conference paper at ICLR 2026
GOLDENSTART: Q-GUIDED PRIORS AND ENTROPY CONTROL FOR DISTILLING FLOW POLICIES
He Zhang'.. Ying Sun't, Hui Xiong!
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)
2 Department of Computer Science and Engineering.
The Hong Kong University of Science and Technology, Hong Kong SAR 3 AI Robotics
h 77 connect. hkust gz .ww.gz...uuuu.. edu. cn, yingsähkust edu.
xionghui @ust.hk
ABSTRACT
Flow-matching policies hold great promise for reinforcement learning (RL) by capturing complex, multi-modal action distributions. However, their practical ap- plication is often hindered by prohibitive inference latency and ineffective on- line exploration. Although recent works have employed one-step distillation for fast inference, the structure of the initial noise distribution remains an overlooked factor that presents significant untapped potential. This overlooked factor, along with the challenge of controlling policy stochasticity, constitutes two critical ar- eas for advancing distilled flow-matching policies. 'o overcome these limitations, we propose GoldenStart (GSFlow), policy distillation method with Q-guided priors and explicit entropy control. Instead of initializing generation from unin- formed noise, introduce Q-guided prior modeled by conditional VAE. This state-conditioned prior repositions the starting points of the one-step generation process into high-Q regions, effectively providing "golden start" that shortcuts the policy to promising actions. Furthermore, for effective online exploration, we enable our distilled actor output stochastic distribution instead of deter- ministic point. This is governed by entropy regularization, allowing the policy to shift from pure exploitation to principled exploration. Our integrated framework demonstrates that by designing the generative startpoint and explicitly controlling policy entropy, it possible to achieve efficient and exploratory policies, bridging the generative models and the practical actor-critic methods. We conduct exten- sive experiments on offline and online continuous control benchmarks, where our method significantly outperforms prior ofthe-af-the-art approaches. Code will be available at https:: / //github com/
INTRODUCTION
Recent advances in policy learning have increasingly leveraged generative models to capture com- plex and multimodal policies (Chi et al. 2023; Ghugare & Eysenbach| 2025 Black al.) 2024a). Unlike traditional methods that assume a unimodal Gaussian distribution, these approaches model the rich action distributions required for sophisticated control tasks, demonstrating potential across broader range of scenarios. However, this expressive power comes at cost: The iterative nature of the generation process, which requires multiple steps to produce a single action, leads to prohibitive inference latency. This bottleneck makes such models impractical for real-time scenarios, such as Vision-Language-Actti (VLA) models (Zhai et al. 2024 Black et al.) 2025).
Flow matching has recently emerged more efficient alternative to diffusion models (Lipman) ct al.) 2023; Liu et al.) [Albergo & Handen-Eijndeni 2023), Geng et al. 2025). This has spurred research into the acceleration of generative policies using flow matching (Braun et al.) 2024 Agrawalla et al.] 2025 Espinosa-Dice et al., 2025). although these approaches often still require multiple denoising steps the inference stage. To address this, more aggressive solution using
Corresponding authors.

---

Published as a conference paper at ICLR 2026
one-step distillation proves particularly effective by training student network to emulate the en- tire multi-step transformation in a single forward pass (Park et al. 2025b). Although effective in reducing latency, these methods overlook two critical opportumities to improve policies.
First, their generative process begins from a fixed, uninformed prior, typically a standard Gaus- sian distribution. However, emerging perspective in generative modeling suggests that initial noise is a critical component that can guide generation (Zhou et al. 2025 Ma et al. 20236
We posit that an optimized starting point (a "golden start") can create a powerful learning shortcut to high-value actions. As illustrated in Figure [ an informed prior (yellow) strategically shifted towards high-value regions provides more direct path to op- timal actions, compared to an uninformed Gaussian distribution (gray). The second opportunity stems from the deterministic mapping inherent in the dis- tilled policies. Given a specific prior noise, the gen- erator learns "point-to-poine mapping, transform- ing a single noise vector into a single determinis- tic action. This architecture inherently lacks explicit control over policy stochasticity, which is crucial for effective online exploration (Ma et al. 2025a).
from Normal (b) Demolsing Value-guulde Prior Distriaution
Distriburtin
Q-quided Distrilution
Normal
An illustration of denoising from an uninformed Gaussian prior (a) versus an informed, value-guided prior (b). Deeper blue indicates higher value.
Figure I:
To overcome these challenges, we introduce GoldenStart (GS-flow), a novel distillation framework that unifies high-speed inference with precise exploitation and adaptive exploration. Our work is built upon two key innovations: (1) First, we propose a Q-Guided Generative Prior, learned via a lightweight conditional VAE. This prior replaces the uninformative Gaussian noise with a state- aware distribution biased toward high-value actions, as identified by the critic. This provides the "golden start", effectively shortcutting the policy learning to optimal modes with negligible latency overhead. (2) Second, we introduce
Distillation, where the student policy learns a full distribution over actions, not just deterministic ones. This transforms the conventional "point-to to-poott"" mapping into a more expressive coont-to-distributionn paradigm. During the online RL stage, an entropy regularization mechanism is activated, allowing the policy to dynamically modulate its stochasticity for robust exploration.
By co-optimizing the generative starting point and the output distribution, our framework improves the policy's ability represent high-value actions while merging flow-based distillation models with adaptive exploration control. To this end, our approach, GS-flow, is extensively evaluated on continuous control benchmarks, including OGBench and D4RL. (Park et al. 2025a Fu al.. 2020)_ The results demonstrate that our method establishes a new state-of-the-arr in overall performance. It particularly excels on complex tasks requiring multi-modal action representations and principled exploration, where it significantly outperforms prior methods.
2 PRELIMINARY
2.1 PROBLEM DEFINITION
A reinforcement learning problem formulated as a Markov Decision Process (MDP) (Sutton ct al. 1998). defined by the tuple (S, P,r, S is the state space, A is the action space, P. S A + [0, is the state transition probability function, ward function, and > E [0, 1) is the discount factor. A policy Λ (a)s) is a distribution over actions given a state. The objective to learn an optimal policy π' that maximizes the expected discounted cumulative reward, J(T)) Erw, 1Er "r((st, ac)]. where - = (50, an, 51, ..,...) )) is trajectory sampled by executing the policy म. Offline RL involves learning from a static transition dataset D {(s,, 4g, Ft, $1+1) }.N. 1.y without environmental interaction, where N is number of steps in the dataset (Levine et al. 2020). The Offline-to Moineettt-nniiae Online RL setting extends this problem by introducing subsequent online interaction phase, also with the aim of maximizing the return function J J(T).
S A -> RR the re-
2

---

Published as a conference paper at ICLR 2026
2.2 DISTILLATION FROM SLOW-MATCHING POLICY
The significant inference cost iterative flow-matching policies has motivated researchers to distill them into single-step, fast student policies (Park et al.. (025b). This approach, named FQL., operates within an actor-critic structure and trains the student actor with hybrid objective: concurrently minimizing distillation loss against the flow-matching teacher while maximizing the Q value. The framework utilizes two distinct models:
Teacher Policy (T²): The flow-matching teacher policy is trained on the offline dataset D using behavioral cloning (BC) objective. For a given state-action pair (s, a) sampled from the dataset and noise sample 2y AO(0, I). the training objective is to learn conditional velocity field ug ( t)- This field parameterized by a time variable E [0, 1] and defines a straight path between the noise zo and the action (Lipman al. 2023 Liu al.). Assuming & is sampled uniformly from this interval (l ~ (0, 1)), the interpolated action along this path is v; = (1- l) + la. The Conditional Flow Matching (CFM) loss then trains the network to match the constant velocity of this path:
LCFM (6) =
[1 8, - (a xo)ll
During inference, the teacher policy Th generates final action ateacher by using the trained vs to iteratively denoise an initial noise sample over multiple steps.
Student Policy (T²): The separate student network is trained for fast inference. It takes state and a noise vector ng as input and produces an action in a single forward pass. The student policy Th is trained to concurrently maximize the Q-value while staying close the teacher's output. This achieved by minimizing a compound loss function that combines a Q-learning objective with a distillation loss:
(P) = EEND.RO-NN.... [- (s, (s, + (l) mo) - (s, =0]
where is the critic function learned within an actor-critic framework (Haarnoja al. 2018 and the hyperparameter a controls the strength of the behavioral cloning (BC) regularization (Tarsoov al. 2023). In particular, The requires iterative denoising at inference, it is trained directly approximate the multi-step denoising action in a single step.
2.3 MULTTICRESSENT TASK
To demonstrate our insight, we design the Multi-Crescent environment, shown in Figure 2 The environment consists of six separate, nonconvex, crescent-shaped regions high reward, designed to challenge agents that prone Q-value overestimation. The reward is structured into three levels: the top-leftbbottom-rige rrght crescents provide a moderate reward, the hiddlee-eet/miiddll--igge crescents provide higher reward, and the globally optimal top-right/bottom-left crescents offer the maximum reward. All other areas yield zero reward. This setup emulates a complex environment with multiple levels of local optima.
When constructing the offline dataset, we deliberately excluded all samples from the two highest-reward crescent regions (top- left/bottom--riht)) as shown by the blue scatter points in Figure This environment poses two challenges the algorithms: 1) During the offline learning phase: The algorithm needs to identify and con- verge to the higher-reward mode present within the dataset (middle- left/middle-right while suppressing Q-value overestimation for un- seen regions. 2) During the online exploration phase: The algo- rithm must demonstrate efficient exploration to discover and exploit the globally optimal modes (top-leftboottom-rig) -right) that were never present in the initial dataset. This environment allows us to assess whether an algorithm can escape the pull of a suboptimal data dis- tribution find the globally optimal policy. More details can be found in the Appendix D
Reward Landscape
6
50
Reward
Figure 2: The visualization of the multi-crescent task.
3

---

Published as a conference paper at ICLR 2026
GoldenStart
Gostiy Froas Buller (...
OHBY Nenwork ?
Afanngge Natue Seici
Seacher
Eyera
The Failyy
Fvors
3000
Teader
Ausie
Phose 1: (-Gasddd Price Learing
Phore Eatrpy Regulurized Distilution
Figure 3: Overview of our algorithm. During training, we first learn structured prior for the initial noise, which is then used to distill the teacher policy. For online exploration, actions are sampled from the student's entropy-regularian distribution. During evaluation, the deterministic mean of the policy's ss output is used. The critic update steps are omitted for clarity, detailed in Appendix B
METHODOLOGY
3.1
OVERVIEW OF THE ALGORITHM
Our method, GS-flow, is designed mitigate the two challenges of imprecise exploitation and in- effective exploration common in existing distilled policies through a two-phase training process, as illustrated in Figure The first phase. Q-Guided Prior Learning, focuses solving the suboptimal starting point problem. Instead of beginning the generation process from a standard, uninformed Gaussian noise, we use the Advantage Noise Selection module actively identify advantage ini- tial noises, which lead to high-value actions. We then train conditional Variational Autoencoder (CVAE) to model the distribution of these advantage noises, effectively learning an informed, state- conditioned prior. The second phase, train highly capable student policy. Both the teacher and student policies are provided with ini- stochastic
Distillation, uses the learned prior to
tial noise sampled from our learned prior. Furthermore, the student model is designed policy and trained with a hybrid objective that combines distillation with an entropy regularization term. This endows the final actor with controllable stochasticity, allowing it to explore intelligently during online fine-tuning. The complete training pipeline, which integrates these two phases with standard actor-critic updates, is detailed in Algorithm
At inference time, GS-flow operates with high efficiency using only the VAE decoder and the student policy, which are highlighted in yellow in Figure 3 Given the current state, the VAE decoder generates an advantage prior. This prior is then fed into the student actor to produce an action distribution. For online exploration, action is sampled from this distribution with its learned mean and variance. For evaluation, we only use the mean to maximize exploitation.
Algorithm GS-flow
Initialize: Critic Qe. VAE (BET Dea), Teacher Policy TG. Student Policy Tip.
2: for each training step do
3: Sample a batch (s, r.)rss))) s')} from dataset D.
# #- 1. Update Critic
4: Update critic parameters 0 using Temporal Difference (TD) learning.
#22. 2. Update Prior Learning Network
For each state generate Neand candidate actions Acand {a,} ) Nad using To. Find the prior noise Tadv corresponding to the highest-Q action (Eq.
5:
6:
7:
Update VAE parameters ε & by minimizing the CVAE loss (Eq.
3. Update Student Policy
8:
Update teacher policy parameters using the flow matching loss (Eq. T)
9: Generate a sampled prior for the current state: Tady <- Deg (s,
Generate the teacher's target action: ageacer +- (s, @adv).
11: Update student policy parameters by minimizing the actor loss LActor (Eq.
end for
13: return Trained student policy Tt..
4

---

Published as a conference paper at ICLR 2026
3.2 Q- D-GUIDED PRIOR LEARNING
To realize our first insight of initiating the denoising process from golden starting points, we propose learning Q-Guided Prior to model the distribution of what we named "advantage noises", denoted as Tadv- For this purpose, we employ a conditional Variational Autoencoder (CVAE) due to its flexibility in learning arbitrary multi-modal distributions. To achieve this, we first need construct samples Bady of these advantage noises for model training. A formal theoretical analysis is provided in Appendix C where we analyze our method from the perspective of amortized optimization.
Advantage Noise Selection. We introduce a data collection module named Advantage Noise Se- lection, shown in Phase of Figure 3 Given the state 8, we first collect a set of Naand candidate actions, denoted as Ocand E Acand, generated by the teacher policy T with Neand different initial noises To. which is sampled from a normal distribution:
(3) Although these candidate actions are all feasible behaviors learned from the dataset, they are not necessarily optimal. To identify the most promising starting point, we leverage the critic Q to evaluate all candidate actions. The initial noise that generates the action with the highest Q-value is designated as the advantage noise for s:
Acand = {a; = (s, 2r)) z; N/(, I), Nar.
Tadv (s) = arg max Qss, 2s))).
This selection process is applied on-the-fly within each training step, using the most up-to-date teacher policy generate new batch of pairings, Badv = {(s, [s)( This batch then serves as the target distribution for the CVAE update.
State Conditional VAE. With the data collected before, we then train Conditional Varia- tional Autoencoder (CVAE) (Kingma & Welling, 2013) to model the state-conditioned distribu- tion Pe, (Nadv The CVAE consists of conditional encoder Fe,, (x, 8) and a conditional decoder De, (), where z is the latent vector. The encoder maps prior-state pair to a latent distribution, while the decoder reconstructs the prior from a latent sample. The model is trained by minimizing the weighted sum of a reconstruction loss and KL-divergence regularization term:
LVAE {2)
Lreen + CKL
where ΛKl. is the scalar weight. The KL--divergence term LuI regularizes the latent space by encour- aging the encoded distribution be close to a standard normal distribution (0, I):
LKL= (. [DEL (9s, (2 Iadv, 8) || (0, T))],
(6)
where ge. is the approximate posterior distribution, diagonal Gaussian parameterized by the en- coder Eç, g5, (2 Tadv, s) = N (H&, (Fatv; s), Σş, (Fait, s)). Assuming Laiv denotes the prior predicted by (2, s), the loss of reconstruction Crecon can calculated as follows:
Lreen E B t:: (=)man..s) [|| 13]
(7)
Notably, CVAE is capable of approximating arbitrarily potentially multimodal prior distribution, offering an advantage over methods that learn a Gaussian distribution.
Validation. We validate the effectiveness of Q- guided prior in the MultiCrescent environment. Fig- ure 4 visualizes the distribution generated by the VAE decoder during inference. The red points represent Ladv. and the red region generated via KDE (Silver- man 2018) represents the predicted prior distribu- tion. After the offline phase (left panel), the prior cap- tures the high-value modes sonddde-left/middlle-e within the static dataset. After online fine-tuning (right panel), the prior adapts its density to focus on the newly discovered, globally optimal action modes (top- left/bottom-righe right).. This demonstrates that learned prior captures the distribution of advantage noises. Furthermore, Figure 5c shows that the actions generated from andy yield higher Q values compared to the baseline shown in Figure 5b
Reward Landscape
Reward Landscape
(a) Learned Prior (Oflme)) (b) Learned Priur (Online)
Figure 4: Visualization of the learned prior distribution after different training stages.
5

---

Published as a conference paper at ICLR 2026
Lantsage
Revel Landergg
Beaot Lobegs
Laddea
Rewer Tanlegs
(a) Dataset (Offline) (b) FQL (Offline)
(c) Ours (Offline)
(d) FQL (Online)
(c) Ours (Online)
Figure 5: Results on the multi-crescent task. Blue crosses denote samples from the offline dataset, while yellow stars represent the actions produced by the policies. (a): shows the offline dataset, which excludes the two globally optimal modes. (b, c): shows action distributions after the offline phase. Our method captures the higher-value modes within the dataset, while the baseline shows less focused distribution. (d, e): shows action distributions after the online fine-tuning phase. Our method quickly discovers and converges to both highest-reward modes. In contrast, the bascline only finds one. More results on this task can be found in Appendix g
3.3
DISTILLATION
Previous flow-matching policy distillation methods produce a deterministic actor. Although efficient for exploitation, is ill-suited for online exploration due to its lack of inherent stochasticity. This can be viewed woint-to-point generation process, where starting noise is mapped to single target action. Inspired by recent approaches that augment generative models with distributional models (Dong et al. 2025), we propose an regula--egulr distillation method. This transforms the distillation from a point-to-point mapping into a poin--o-adlptive--disiibbu the agent with a principled method for balancing the explorationeexploiittoon trade-off.
process, providing
To achieve this, we parameterize the student policy (a]s, fadv) as a Gaussian distribution using a dual-headed architecture that outputs both mean (upp (s, Tadv) and a standard deviation (s @adv). The action app for exploration is computed as:
(8, Tadve <) = (1 fadv) + ap(8, (ndv) O 6, where 6 6~ C0, T).
The actor policy is trained by minimizing composite objective that balances three key components: imitation of the teacher, value maximization, and entropy regularization. The training objective for catrropyrreullriin
actor is composite loss function designed to balance three key objectives: (1) imitating the high-qu quality teacher policy, (2) maximizing expected return according to the critic, and (3) maintaining sufficient policy entropy to encourage exploration. Therefore, with the advan- tage noise Fady =
Deo (2, 8), the total actor loss is defined as follows:
(9) The distillation term LL2-Diaull anchors the mean behavior of the student policy to the high high-quality teacher actions, and a1 is the scalar weight to control BC behavior. Two details are critical to ensure that this process has a low-variance and stable training signal. First, both teacher and student policies are conditioned on identical advantage noise advv. Second, the loss is computed using only the student's 's deterministic mean μ4p (s, adv rather than stochastic sample. These design choices reduce the variance of the loss signal and improve training stability. The loss is defined as follows:
CActor Ez swD [ar L2--iiiil + Lq - (ss,
Lisi E [1] (8, fndv) To fadv) )|3]-
(10)
The value maximization term Lo encourages the policy to seek actions that the critic evaluates as having a high value (Fujimoto & Gul (2021). Following the standard approach in Soft Actor-Critic (SAC) (Haarnoja et al. 2018), we use a sampled action ag from the policy: ae ~ TT (-1s, Fadv). The loss is then calculated as its negative Q-value:
= (sp.).).
(11)
The third entropy bonus term H(( @adv)) is the entropy of the policy under aadv To automate the trade-off between reward and entropy, the temperature parameter ag is learned by minimizing a separate loss function that aims to match the entropy to a predefined target entropy Hunget- This allows the agent to dynamically adjust its stochasticity, exploring more when the entropy is below the target and exploiting more when it is sufficient. These two components are calculated as follows:
H.(TTpp (-|s, )adv))
Eegrn [log (ppapl (s, Radv)]
(12)
Loz EynD (az ( (at)) Hurzet ))]
(13)

---

Published as a conference paper at ICLR 2026
Table 1: Offline performance on OGBench and D4RL benchmarks, averaged over 5 seeds (3 for Visual Environments due to computational cost). Best results are in bold. The performance of baseline methods reported from Park et (2025b).
Gaussian Policies
Dilfasion Policies
Policies
BC
IQL
ReBRAC
IQQ
SRPO
CAC
FAWAC
FBRAC
IFQL
FQL
Ours
OGBencc
AntMare Large Navigate
0
48
91
0::00
0::00
7
1
20
24: 17
80
88.4 2.7
AntMaze Giant Navigato
HumanoddMlze Medicm
11:
82 7
16:29
12
0::00
38
6.2:2
25
19
19:1
45.0
Muaue Lange
3=1
2=1
0=:0
0::00
1+1
0
0
6-22
74:
4.6.2:4
AnSeccer Arena
110
3
01:0
010
1243
24
1
46,0 10.5
Cahe Double Play
0::0
1
7=3
16 =
0::00
21:2
2
925
38 1
$1.3
Scene Play
111
12::3
50 13
33 14
211:2
50 M0
18 1
46 10
04:
70 1
88.0 1 8.6
Puzzle-3a3 Play
111
2
2 1
0:1:0
0:0:0
1
2
0::0
16 1
25.2 10.7
Puzzle-4xd Play
Average
0.7
19.1
29.7
17.2
9.1
21.4
12.1
27.7
21.8
33.5
47.1
D4RL AntMaze
AntMare U-Maze
98
94
97
5
1.
94
92
96
99.6
AntMaze
+
AntMaze Medium Play
90
84
81.
49 1:
52
77 4:7
78 4:
11.2 9.0
AntMare Medium Diverse
744
84
85
75
0++11
44 15
77 +-6
(0 25
71 13
75.3
11.0
AntMaze Large
52
64
54
21
55
+
86.5
Large
0:1:0
Average
17.2
57.2
78.7
79.2
73.8
30.2
44.5
63.7
64.8
83.3
86.1
Visual Environments
Visual Cahe
70
85
40
92.7
Visual Cube Double Play
34
6::.2
8:1.6
21
42.0
11.8
Visual Scene Play
97 4:22
98 4:44
46-+-4
86 = 10
98 +:3
100.0
0.8
Visual Puzzle-3x3
7
88 +-4
7
100 +-0
94 +
88.67
9.0
Visual Puzze-4ad
Average
59.8
22.8
50.2
63.4
70.9
Validation. The online fine-tuning results, shown in Figures 5d and 5e_ highlight the superiority
of our exploration mechanism. Our method effectively explores the action space and identifies both of the highest Q-value peaks (lop-lef/boottm--rign) In contrast, the baseline lacks exploration strategy and finds just one the peaks. Even better, our method finds both modes using significantly fewer samples. This demonstrates the clear efficiency of its entropy hatropy-regulariied exploration.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We test method across the OGBench (Park et al. 2025a) tasks, the standard D4RL AntMaze (Fu et al. 2020) tasks, and a set of challenging Visual Environments (Park er al. 2025a). The baselines range from standard Gaussian policies (BC, IQL., ReBRAC) (Kostrikov al. 2022 Tarasov al., 2023). to more expressive Diffusion Policies (IDQL, SRPO, CAC) (Hansen Estrucch et al. 2023 Chen al. 2024 Ding & Jin 2024), and finally to Flow Policies (FAWAC, FBRAC, IFQL) (Nair et al. 202 Wang et 2023 Park et al., 20255). And state-of-the-ar flo--machhing distilla- tion model FQL (Park et (025b). For offline-to-online we also compared with Cal-QL and
RLPD (Nakamoto et al. 2023; Ball et al. (023).
Notably, RLPD is an advanced reinforcement learning algorithm explicitly built upon and designed to improve the standard Soft Actor-Crités (SAC) for effectively utilizing offline data Ball et al. 2023 Haarnoja et al. (2018). While RLPD shares the same maximum entropy objective as SAC, incorporates architectural enhancements to achieve more stable and efficient performance in offline- to-online transitions. To provide more direct understanding of how our framework compares with standard continuous control methods, provide comparison between Ours and the original SAC in Appendix H.5
We validate our model in both offline and offline-to-onlin fine-tuning settings demonstrate the effectiveness our two contributions. More details on environments and baselines are shown in A Appendices and

---

Published as a conference paper at ICLR 2026
Table Offline-to-oning performance comparison. Similar to Table seeds. The best online results are highlighted in bold.
| we report the results over
IQL.
ReBRAC
Cai-QL
IFQL
FQL
Ours
dlaze Mediam
1
16
1=
0:0 0:0
82 20
12
AntSoceer
2=1-+0=0
0::03 020
0:0-0:::
0::0
19 10
28
Cube Domble Play
0-+11 -000
6.+55 28
0+0-0000
0 0
12
10
10:1111-
92-3
51 51:4-0-
Scmre Play
11
1
16+110- 100
1 50--5
0-:-0 100
0- 00.+-9
81=11
100
68 0:-00-0
0:0
Puzzle and Play
8:4
0000
1000
20:66-1
19:42
17:.4
100:2:0
Avcruge
5.4 5.4
17.0 25.6
0.2 10.0
0.0 41.6
21.4 45.0
31.0
07.6
49.4
S3.6
AntMaze U-Mase
00
0
06
AntMaze Diverse
74
50:27
Pay
Amtaae
88
04
AmtMze
67
04
Arerage
66.3 y 78.3
50,0 > 73.0
50.2 97.3
0,0
96.7
65.3 8B.
748 n 90.2
86.2
Pan Closed
84 102
74 158
-3-4
8:22
120 10
17 =
-
107 10
61 30::8
6
71 11
146-2:6
Door Cloaed
1-2
0-4 102
-0-3-00
0:2
102 7
0=2
0::0
102 #5
14:1-
185 4:4
Hommer Chaned
7-
0++0
0:+:0--
4
00
0-+0- 1:
10:033
Relaste Choncl
0++0
-0
0#00+2+2
-0=0-543
0
62
0
Averggo
4.2 RESULTS AND ANALYSIS
The Impact of the Learned Prior on Offline Performance. The results in Table I demonstrate the effectiveness of our proposed method, GS-flow. GS-flow achieves new state-of-the -frt perfor- mance on average, outperforming all baselines. This advantage is particularly pronounced on several tasks with multi-moda action spaces, where our method shows significant gains over the strong FQL baseline. According to the results in OGBench, the strength can be illustrated by the contrasting re- sults on the Cube tasks. On Cube Single Play, a task with relatively unimodal optimal policy, GS-flow performs comparably to the strong FQL baseline. In contrast, on the more complex Cube Double Play, which requires coordinating two objects and therefore presents a significantly more multimodal Q landscape, the offline score of GS-flow (51.3%) dramatically outperforms all com- peting methods. The contrast underscores our algorithm's 's specialized capability in multi- challenges. The advantage is further substantiated in other complex manipulation tasks such as Puzzle-3x3 and Puzzle-4x4, and in challenging locomotion environments such as HumanoidMaze Medium Navigate, where GS-flow achieves more than double the score of FQL.. Furthermore, GS- flow achieves the highest average scores on the remaining two benchmarks, D4RL AntMaze and Visual Environments, demonstrating the effectiveness of our algorithm in the offline setting.
Effective Online Exploration via Controllable Entropy. The second key advantage of GS- flow, its capacity for effective online exploration, is enabled by its controllable entropy mechanism based on the output of the distribution actor. The performance improvements shown in Table 2 are comparable, especially in tasks that require extensive exploration. The Puzzle- 4x4 environment serves as powerful case study. As noted by the authors of FQL (Park et al.) 2025b), this task is particularly challenging for methods with limited exploration capabilities. The baseline FQL reflects this, improving from 8% to 38% after online training. In the contrast, GS- flow leverages its entropy-regularized stochastic policy to achieve score from 17% to 100%, matching the performance of specialized online methods like RLPD (Ball et al. 2023). Furthermore, our method significantly outperforms RLPD on other complex tasks such as AntSoccer and Cube distillation successfully combines
online finesetuning
Double. These results demonstrate that our the high performance of the teacher model with the advantage of principled, controllable exploration found in traditional Gaussian policies (Haarnoja et al. (0188). We believe the idea of moving beyond "point-to toointtt--pint" mapping "point-to-distributae process is simple yet valuable, allowing GS- flow to effectively balance exploitation and exploration.
4.3 FURTHER ANALYSIS
The Importance of the Learned Prior. To analyze the impact of our proposed prior learning mechanism, we evaluate its performance while varying the number of candidate actions, Nennd used in the Advantage Noise Selection module. As depicted in Figure 6a there is a clear trend: increasing the number of candidates improves both sample efficiency and final performance. The red curve (Neand = 15) achieves the highest return, while the green curve (Neand = 5) learns more slowly. However, even with only five candidates, our method significantly outperforms the FQL (the
8

---

Published as a conference paper at ICLR 2026
Nalh Crecor Itak
Walt Cascern Toak
Pazzl 4ed Vork
15.0
Of
12.
100-
14-
CE (Wrer7hl Thasker
50
230
Trauug Supo Ed)
Thininng Steps (k)
Tranting Sheps
(a) Effect of the candidate number.
(b) Ablation on Multi-Crescent.
(c) Ablation on Puzzle-4x4.
Figure 6: Ablation studies on the oflline-to-online transition. (a): The plot analyzes the impact of the candidate number on learning efficiency. (b, c): The plots demonstrate the effectiveness of our controllable entropy, showing significant performance gains of our full method over a deterministic variant in both the Multi-Crescent environment and the Puzzle-4x4 task.
gray curve), which can be viewed as degenerate case of our approach without the Q-guided prior learning module(Neen eand = 0). This strongly validates the effectiveness of learning a structured prior. Given the trade-off between performance and computational overhead of the selection module, we chose Neand = 10 (the blue curve) as a balanced setting for all main experiments.
The Importance of the Controllable Entropy. To isolate the contribution of our controllable entropy, we conduct ablation studies in the Multi-Crescent and Puzzle-4x4 environments. As shown in Figures 6b) and [6c] our full method (the blue curve), which uses dual-headed architecture with an hatropy-regularizer loss, demonstrates significantly higher learning efficiency during the online deterministic variant that uses only our learned prior module (the gray curve, denoted "Ours w/o CE"). In particular, the online performance of the gray curve in the Puzzle- 4x4 task is similar that of the FQL (as seen in Table T. This similarity in performance strongly suggests that our controllable entropy mechanism is the key component that provides superior online exploration, effectively addressing this known limitation in prior work.
phase compared
Computational Cost Analysis. We demonstrate that significant performance gains of our method do not come at a prohibitive computational cost, particularly during the critical inference phase. We compare the wall-clock time for single training step and single inference step against FQL and IFQL.. As presented in Figure [ the inference time of GS-1low(0.51 ms) is only marginally higher than that of FQL (0.42 ms), which is caused by the VAE Decoder model (De,) and remains significantly faster than the multi-step IFQL (0.97 ms). This confirms that our method preserves the single-step efficiency of the distillation paradigm. Although the training time for GS-fflo(33110 ms) is higher due to the addi- tional inference the Advantage Noise Selection module under candidate number Neand = 10, this one-time training cost is a well-justified trade-off for the substantial improvements in both policy quality and online adaptability.
Step Time Compatison
Mechod
P21
0hm
Time
(ms)
Figure 7: Average step time required on cube-double task.
RELATED WORKS
Efficient Inference for Generative Policies. Generative policies, including diffusion (Ho et al. 2020) and fow-matching models (Liu et Alberrgo & Handen-Eijndee 2023 Geng et al.][
excel in representing multimodal action distributions in RL. (Chi et al.) [003] Estruch al.,
2023 Ding & Jin 2024, Nair et al. 2021). However, their practical adoption is hindered by high inference latency (Shi& & Zhang) Zhai et al.) (022)). While one-step distillation methods (Park et al.)
2025b) have improved inference speed, they often overlook the impact of the noise prior opti- mization, a factor shown to be promising in the image generation field (Zhou et al., 2025). DSRL Wagenmaker et al., 2025) takes advantage of this idea by learning Gaussian prior distribution for online adaptation, without optimizing for inference latency. In contrast, our method integrates a more flexible prior while introducing negligible inference overhead.
9

---

Published as a conference paper at ICLR 2026
Online Exploration for Generative Policies. Another key challenge for generative policies is prin- cipled online exploration (Fan et al. 2025). One line of research focuses on introducing stochasticity
into the inference denoising process (Yang al., 2023 Black et al. 2024b Chen et al. 2025). An- other line focuses on the training phase, using techniques such as reweighted score matching (Ma al. (0225) and entropy estimation with Gaussian Mixture Models, which can be computationally
expensive (Wang et al. 2024). Recently, EXPO (Dong et al., (025) enhances sample efficiency by training an additional Gaussian edit policy with entropy regularization. In contrast these methods, our approach is more lightweight, integrating entropy control directly into the distillation process.
CONCLUSION
In this work, we introduced GS-flow, novel framework for distilling flow-matching policies. Our method makes two key contributions: it learns a Q-Guided Generative Prior to provide a "golden start" that shortcuts the policy to high-value actions, and it uses endow the policy with controllable, principled exploration. Extensive experiments show that GS- flow establishes a new state-of-the-an
Distillation to
overall performance on challenging benchmarks, partic- ularly excelling on complex tasks that require multi-modal actions and effective exploration. Our framework successfully bridges the gap between expressive generative models and practical actor- critic methods, delivering potent combination of inference speed, precision, and exploratory power.
ACKNOWLEDGMENT
This work supported in part by the National Key R&D Program of China (Grant N0.2023YFFO725001,, in part by the National Natural Science Foundation of China (Grant N0.92370204, 62306255), part by the guangdong Basic and Applied Basic Research Foundation (Grant 02222315111120055 in part by the Key-Area Special Project of Guangdong Provincial Ordinary Univeritis<(22-42DZX00000))
REFERENCES
Bhavya Agrawalla, Michal Nauman, Khush Agarwal, and Aviral Kumar. floq: Training critics via fow-matching for scaling compute in value-based rl. arXiv preprint arXiv:2509.0688 2025.
Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Kleventh International Conference on Learning Representations, 2023. URL.
https: /ttppi:///axvv../// org/ abs 2209. 15571
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi_O: A vision- assioonlanguage-actee flow model for general robot control. arXiv preprint arXiv:2410.24166 2024a.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Rep- resentations, 2024b. URL http::tt/////t
/ww.revieeeeeee /attps://openzevient
net/
Kevin Black, Manuel Y Galliker, and Sergey Levine. Real-time execution of action chunking flow policies. arXiv preprini arXiv:2506.07333 2025.
Max Braun, Noémie Jaquier, Leonel Rozo, and Tamim Asfour. Riemannian flow matching policy for robot motion learning. In 2024 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 5144-5151. IEEE, 2024.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimiza- tion through diffusion behavior. In The Tweifth International Conference on Learning Represen- fations, 2024. URL htt.hhttsss//ppeeeviieeeee http://wweeeeeeeee.ee.ooono/nann
10

---

Published as a conference paper at ICLR 2026
Tianyi Chen, Haitong Ma, Na Li, Kai Wang, and Bo Dai. One-step flow policy mirror descent. arXiv preprint arXiv:2507.23677. 2025.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273665 2023.
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL
net http://orrmmidddvivemmm ottp://eee.....t.eoooooooooooooo
Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn. Expo: Stable reinforcement learning with expressive policies. arXiv preprint arXiv:2507.07990 2025.
Nicolas Espinosa-Dice Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kiante Brantley, and Wen Sun. Scaling offline rl via efficient and expressive shortcut models. arXiv preprini arXiv:2505. 22866, 2025.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: urXiv:/802.0607) 2018.
Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu. Online rewihd-weighted fine-tuning of flow matching with wasserstein regularization. In The Thirteenth International Conference Learning Representations, 2025. httl:tttts///pereee///e
met/forum 2itnnnddd=IIexeeee
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning (ICML), pp. 1587-1596. PMLR, 2018
Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505. /3447, 2025.
Raj Ghugare and Benjamin Eysenbach. Normalizing flows are capable models for rl, 2025, URL https::/aaxxi> wttp://wwwwrg/bb/25.3222222 http:ss//arxxuu.rrra////222522/3
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, 2018.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10511 105733 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlane, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg. Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang. Lili Yu. and Ury Zhilinsky, No.s a vision-language-un action model with open- generalization, 2025. URL /tttps://arxiv.. org/abs 2504. 10004...05550 16054
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.6114 2013.
11

---

Published as a conference paper at ICLR 2026
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022. URL https: // openreview net
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprini arXiv:2005.0164 2020.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL /
net Btt.://rrmmm.// htte:///ooorrr..oocoooo
Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy, 2025a. URL wttptttss:// /ttts://arriiue v. orgg abs / 2502. 2250200361111
Nanye Ma. Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang. Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Scaling inference time compute for diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2523- 2534, 2025b.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online re- inforcement learning with offline datasets, 2021. URL
09359
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36:62244-62269, 2023.
Scohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised with metric-aware abstraction. arXiv preprint arXiv:2310.0883 2023.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine OGBench: Benchmarking offline goal-conditioned RL. In The Thirteenth International Conference on Learning Represen- tations, 2025a. URL httphh::/ http:///ppppeeeeeeeeeeeee.eeeeem net forter/ormmmmmm MdddddMM99mmmi..
Scohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In Forty-second International Conference on Machine Learning, 2025b. URL
net. formm
KVf2SFL1p1
Chang Shi and Amy Zhang. Fastdp: Deployable diffusion policy for fast inference speed. In RLC 2025 Workshop on Practical Insights into Reinforcement Learning for Real Systems.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 2023.
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Scohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint S..tv::500.119990 2025.
Yinuo Wang, Likun Wang. Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-54204, 2024.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL wtt.sttt::t.. http://ppp/evvveeeeeeee net htt.rrrwweeeeeeeefffoouee//aaaaa
12

---

Published as a conference paper at ICLR 2026
Yifan Xu, Lu Dai, Udaikaran Singh, Kening Zhang, and Zhuowen Tu. Neural program synthesis by self-learning. arXiv preprint 1rX.v:111000585,, 2019.
Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for rein- forcement learning. arXiv preprint arXiv:2305.13122 2023.
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Sain- ing Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. In The Thirty-eighth Annual Conference Neurul Information Processing Systems, 2024. /RLLittps///ppeeeiviie nwt. forum? id=nB JMMF 2I2U
He Zhang, Ying Sun, Weiyu Guo, Yafei Liu, Haonan Lu, Xiaodong Lin, and Hui Xiong. Inter- active interior design recommendation via coarse-to-fine multimodal reinforcement learning. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 6472-6480, 2023.
He Zhang, Ming Zhou, Shaopeng Zhai, Ying Sun, and Hui Xiong. Efficient skill discovery via aegret-aware optimization. arXiv preprint arXiv:2506. 2004, 2025.
Zikai Zhou, Shitong Shao, Lichen Bai, Shufci Zhang. Zhiqiang Xu, Bo Han, and Zeke Xie. Golden noise for diffusion models: A learning framework. International Conference on Computer Vision, 2025.
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
We utilized Large Language Models as tool to assist in the preparation of this paper. Specifically, LLMs were used for polishing the language, correcting grammar, and providing suggestions for LaTeX formatting to improve the manuscript's presentation. Our use of LLMs is in full compliance with the ICLR 2026 policy. We reviewed all LLM-generated outputs and take full responsibility for the scientific claims and all content in this work.
B CRITIC UPDATE DETAILS
Our critic network, denoted Qo(s, a), is trained to estimate the expected discounted cumulative reward (the Q-value) for taking action a state s. To improve training stability and mitigate the overestimation of Q-values, employ standard techniques from modern actor-critic methods (Fuji- moto et al. 2018 Haarnoja et al. 2018). Specifically, we use twin-critic architecture, maintaining two separate Q-networks (Qo,: Qn,)) and slowly-updated target networks (Qæ; Qe;) ) to struct the Bellman target. The critic parameters are optimized by minimizing the Mean Squared Bellman Error (MSBE). For given transition (s, a, r,)') from the replay buffer, we first compute the target value, The next action, a', is sampled from our stochastic student policy, Tip and the target value includes an entropy term to maintain consistency with the actor's objective:
y ツrrty min Qø (8', (s's,'') d') i=1,2
log (p(''ss'))
where a' (-1s')
(14)
The total loss for the critic networks is the sum of the MSBE for each critic with respect to this common target value:
Cerise (01, E,,a,,ne''
Σ (Qo y)² -y) 1=122
(15)
The target network parameters 0' are updated via Polyak averaging with the main critic parameters at each training step: 0' + 70 + (1 T)0', where T a small interpolation factor.
C THEORETICAL ANALYSIS: AMORTIZED OPTIMIZATION BOUND
In this section, we provide formal theoretical analysis for the performance improvement of our method. frame our approach as an instance Amortized Optimization, where the computa- tionally expensive test-time selection tized" into learned generative model (CVAE).
for the optimal noise vector-is "amor-
C.1 ORACLE POLICY VIA REJECTION SAMPLING
Consider idcalized oracle policy that performs Best-of-N sampling test time. For given state s, it draws N noise vectors {æ; 1N ~ No,,1 I) and selects the one that maximizes the predicted value:
NN = arg max Q(s, (8)
(16)
By standard order-statistics results, the expected return of this oracle is strictly superior single random sample (the baseline policy):
Joraele E[Q(s, (s,,+;)))) 2 E[Q(s, (T8 (1)x)))] Jhuse-
(17)
We define this non-negative gain as the Selection Gain: Agelect = Jaracte Jouse
C.2 AMORTIZING THE ORACLE VIA CVAE
To bypass the prohibitive computational cost of running the oracle during online inference, train a CVAE to approximate the distribution of the oracle-selected noise vectors wyy- The training objective minimizes:
LVAE =
Lrewon + ALKL.
(18)
where the reconstruction error is defined as 6vae = E1|x" 211 with & ~ Pe(x (s) sampled from the learned CVAE prior.
14

---

Published as a conference paper at ICLR 2026
C.3 PERFORMANCE BOUND
Let ((x) (1s, æ)) be the value associated with a noise vector I. The expected return of amortized policy is amortized Ba-pe (&)]. We assume (a) is L-Lipschitz continuous with
respect to the noise vector i.c., (v V(y) L| a|1. The performance gap between the
oracle and our policy can be bounded as:
Joracle Jamoeiized = (xV V - L El| - -||-
(19)
Applying Jensen's inequality (E|X) [[ we obtain:
Joracle J amootized S LV Bll 2|12 L /
(20)
Substituting Jonacle Jass Aselect we derive the final lower bound for our method:
J Jamottized
Jnase Baseline
A select
T
(21)
Sclection Gain
Amortization Gap
C.4 INTERPRETATION AND EMPIRICAL ALIGNMENT
The derived bound offers a theoretical explanation for the empirical phenomena observed in our experiments:
Effect of N (Aselect:: As the number candidates N increases, the expected maximum of the samples increases, raising Ayket- This aligns with findings in Figure 6(a), where performance improves consistently with larger N used during prior training.
Robustness Q-landscape (L): The Lipschitz constant L reflects the smoothness of the value landscape. In complex environments like the Multi Culti-Crescent toy task, the selection gain Agele remains sufficient to maintain performance gains despite the challenges of non- convex landscapes.
Importance of Reconstruction (Evae): The bound explicitly shows that minimizing the CVAE reconstruction error maximizes the return's lower bound. This is corroborated by results Table 3, where lower VAE loss directly translates to higher returns.
D ADDITIONAL STUDIES IN THE MULTI-CRESCENT ENVIRONMENT
To further validate the effectiveness of our custom Multi-Crescent Environment at highlighting key algorithmic challenges, we conducted a broader set of experiments with different baseline settings, as shown in Figure 8 in the main text. this analysis, the gray bars represent the final offline training performance, while the blue bars show the performance after the subsequent online fine- tuning phase. The hyperparameter a corresponds to the weight the behavioral cloning (BC) term in the FQL loss function (Equation 2 A smaller places relatively larger emphasis the Q- maximization term. The primary results in the main body compare our method against FQL with a high BC weight (a 100).
By lowering a, we test the hypothesis that our non-convex environment can induce Q-value over- estimation in the baseline. The experimental results confirm this hypothesis. When 1, FQL's offline performance drops sharply. The policy learns to target points overestimated by the critic- locations between the tips of the crescent shapes but outside the actual high-reward regions-leading to significant decrease in return. In the extreme case where a = 0 (i.e., pure Q-maximization), the offline return predictably falls to zero, further demonstrating our environment's 's ability to challenge methods susceptible to Q-value overestimation. The final two columns in the figure are designed to evaluate the performance of pure imitation learning. The "FQL w/o Q" baseline isolates the effect of imitation learning via flow-matching, while "BC (MSE)" represents standard behavioral cloning approach. Our method outperforms both of these baselines. Notably, the standard FQL provides only marginal improvement over "FQL w/o Q." This is because the reward modes in our environ- ment are disconnected; when the Q-guidance is too weak (high a), the policy struggles to jump from mode to another, and when it is too strong (low a), the policy is misled by critic overestimation. In contrast, our algorithm learns an initial noise distribution that directly fits the inherently high-Q actions from the dataset, making significantly more robust the effects of Q-value overestimation.
15

---

Published as a conference paper at ICLR 2026
Comparison of Different Settings
14
Offiine
Online
22
764
2-
(MSE)
Figure 8: Additional Studies in the Environment
E BENCHMARK DESCRIPTIONS
E.I OGBENCH
OGBench (Offline Goal-Conditioned RL Benchmark) Park al. (2025a) is a high-quality bench- mark designed for offline goal-conditioned reinforcement learning. It aims to systematically evalu- the capabilities of algorithms across several key dimensions, such as trajectory stitching, long- horizon reasoning. handling high-dimensional inputs (e.g., pixels), and coping with environmen- tal stochasticity. We utilize a variety of environments from OGBench in our experiments, span- ning locomotion, manipulation, and visual tasks. Notably, we evaluate on the default task for each environment, For instance, in the the cube double sasooleeplly-a pplay-singlese singletask seattask--ass task2-v0 task, which can be found in Park et al. b).
play environment, we exclusively use
The specific environments used in work include:
AntMaze and HumanoidMaze: These are maze navigation tasks requiring an agent to control complex quadruped robot (Ant) or 21-DoF humanoid robot (Humanoid), spectively, to reach a target location. We employ various maze layouts, including "Large" and "Giant", with the "Navigate" dataset type test long-horizon planning and control capabilities.
AntSoccer:
This is more challenging locomotion task that requires the Ant agent to dribble a soccer ball while navigating. We use the "Arena" (open-field) version of this environment.
This a robotic manipulation task involving multi-block pick-and-place operations. The agent must move, stack, swap single multiple cubes according to goal config- uration. We use the "Single Play" and "Double Play" versions to test the agent's 's ability to learn generalizable multi-object manipulation skills from unstructured, random trajectories.
Cube:
This complex sequential manipulation task requiring the robot arm to interact with various household objects, including drawer, a window, button locks, and a cube. It is designed to challenge the agent's 's sequential and long-horizon reasoning abilities.
Scene:
Puzzle: In this task, robot arm must solve "Lights Out" puzzle. The agent presses buttons on a grid to toggle the color of the pressed button and its neighbors to match goal configuration. We use the 3x3 and 4x4 grid versions to specifically test for combinatorial generalization.
Visual Environments: Many tasks in OGBench, particularly the manipulation suite, sup- port both state-based and pixel-based inputs. We evaluate our methods in the correspond- ing visual environments (e.g., Visual Cube, Visual Scene, Visual Puzzle), which require the agent to learn control policies directly from 64x64 RGB images.
16

---

Published as a conference paper at ICLR 2026
B
hunanoi
cube
ecene
puzzle
Figure 9: Visualization the OGBench tasks.
dfst
Figure 10: Visualization of the D4RL tasks.
E.2 D4RL
D4RL (Datasets for Deep Data-Driven Reinforcement Learning) Fu et al. (2020) a public bench- mark focused on offline reinforcement learning. It is designed provide datasets that reflect chal- lenges present in real-world applications, such as narrow data distributions, undirected multi-task data, sparse rewards, and suboptimal data. These characteristics make D4RL an essential bench- mark for evaluating the robustness and generalization of offline RI algorithms.
Our experiments primarily make use of the AntMaze environment from D4RL. This is a popular navigation task that requires an 8-DoF 'Ant' quadruped robot to reach specified goal in maze. The task features sparse rewards (a reward is only given upon reaching the goal), and the datasets generated by non-Markovian controller: This setup is designed to test an algorithm's 's ability to stitch effective trajectories from undirected data to solve long-horizon horizon, sparse-reward tasks. We also implement the Adroit domain, which involves controlling 24-DoF robotic hand. Task examples are shown Figure TO
F DETAILS ON ADDITIONAL BASELINE METHODS
In this section, provide additional details on the bascline methods used in the paper (except FQL, which is introduced in Preliminary). categorized by their underlying policy structure and learning paradigm. The settings for all baseline methods are adopted directly from the original paper (Park et al. 2025b) for comparison. You can find more implement details in their paper.
F.1 OFFLINE RL BASELINES
For the offline RL experiments, we compare with 10 recent and representative methods to demon- strate contributions.
17

---

Published as a conference paper at ICLR 2026
Gaussian Policies. For standard offline RL methods that Gaussian policies, consider BC, IQL (Kostrikov et al. (022). and ReBRAC (Taasovv et al. 2023). In particular, ReBRAC is known to perform well many D4RL tasks (Fu et al. 2020h, which are based on a behavior-reguularized actor-critic framework.
Diffusion Policies. For methods based on diffusion policies, we compare against IDQL ( (Hansen- Estruch al. 2023). SRPO (Chen al. 2024). and Consistency-AC (CAC) (Ding Jin (024). These methods employ different policy extraction techniques; IDQL is based on rejection sampling, whereas SRPO and CAC utilize policy distillation. CAC trains the distillation policy within the behavior-regularized actor-critic framework and is based on consistency models.
Flow Policies. We also consider several flow-based variants of existing algorithms cover dif- ferent policy extraction schemes. Flow Advantage- advantage-Weighted Weighted Actor-Critic (FAWAC) is a flow-based variant of AWAC (Nar et al., (02II), which uses the Advantage- Weithhe Regression (AWR) objec- tive for policy learning. Flow Behavior Rehavior-Regularizeed Actor-Critic (FBRAC) is the flow counterpart to Diffusion-QL. (DQL) (Wang al., 2023). which is based on the original Q-loss with backpropa- gation through time. Implicit Flow Q-Learning (IFQL) is the flow counterpart to IDQL, based on rejection sampling scheme.
F.2 OFFLINE-TO-ONIING FINETUNING BASELINES
The offline methods include IQL, which Iearns a policy implicitly through ddvantaag--wiigheee re- gression over learned Q and Value functions; ReBRAC, stable behavior-regularizeed actor-critic algorithm; and IFQL, a flow-based policy utilizing rejection sampling. We also include two meth- ods designed for data-driven online RL: Cal-QL (Nakamoto al., 2023), which calibrates the Q- function with the offline dataset to enable safer online exploration, and RLPD (Ball et al.) 2023 which employs balanced sampling strategy from both offline and online data buffers accelerate fine-tuning.
G HYPERPARAMETERS
G.1 HYPERPARAMETERS SETTINGS
Table lists the hyperparameters used for the cube-double experiment, based on the provided exe- cution command.
The Offline Alpha and Online Alpha refer the pre-set values of the hyperparameter 0₁ in Equa- tion 9 for the offline-to-online transition. This distinction is made because the confidence in the critic's estimates differs between the offline and online phases. offline RL that the critic often overestimates Q-values, necessitating regulation of the Behavior Cloning (BC) weight. However, during the online phase, excessive reliance on the BC term can stifle exploration, which is why these values are set in advance.
common phenomenon in
Additionally, when running the online phase for the puzzle environment, we utilized the balanced sampling technique from Ball et al. (2023). To ensure fair comparison, we also applied this tech- nique to our FQL agent. We found that only our method showed performance improvements with this technique.
G.2 THE EFFECT OF LATENT DIMENSION.
We investigate the sensitivity of our learned prior the VAE's latent dimension. Figure IT illus- trates the results. We observe that low-dimensional, compact latent space is optimal for this task, with performance peaking at dimension latent dimension increases, performance gradually degrades, particularly during the online phase. This suggests that a higher-dimensional space may increase the difficulty of learning a meaningful prior, potentially introducing noise leading to overfitting. However, method still outperforms the FQL across different tested dimensions in both settings. This demonstrates the fundamental robustness and benefit of our learned prior, even when its key hyperparameter perfectly tuned.
or for both offline and online settings. As the
18

---

Published as a conference paper at ICLR 2026
Table 3: Hyperparameters for the cube-double experiment.
Hyperparameter
Value
Offline Steps
1,000,000
Online Steps
1,000,000
Seed
0,2,4,8,16
Latent Dimension
8 (default)
KL Weight
0.1 (default)
Reconstruction Weight
(default)
Number of Candidates
10 (default)
Offline Alphal
300
Online Alphal
50
Offline Temperature
0 (default)
Target Entropy Multiplier
0.5 (default)
Table 4: Hyperparameters for the Puzzle 3x3 experiment.
Hyperparameter
Value
Offline Steps
1,000,000
Online Steps
1,000,000
Seed
0,2,4,8, 16 8 (default)
Latent Dimension
KL Weight
0.1 (default)
Reconstruction Weight
(default)
Number of Candidates
10 (default)
Offline Alphal
1000
Online Alphal
10
Offline Temperature
0 (default)
Target Entropy Multiplier
0.5 (default)
Balanced Sampling( Ball et al. (2023)
True
H ADDITIONAL EXPERIMENTS
H.1 COMPARISON TO REJECTION SAMPLING (RS)
We demonstrate that Ours is better, not merely faster, than Rejection Sampling (RS). While RS relics on stochastic filtering at inference time, Ours actively optimizes the generative process dur- ing training. We highlight the superiority of our approach regarding stability, robustness against overestimation, and safer distillation.
1. Superior Stability and Precision. Ours achieves significantly lower variance compared to IFQL. As shown in 55 IFQL relies on probabilistic sampling from a standard Gaussian distri- bution, which makes difficutt to guarantee high-quality actions in every iteration, leading to higher variance. In contrast, Ours learns a structured Q-guided prior via a CVAE, As the CVAE loss min- imizes, the prior fits the distribution of high-value actions, allowing for stable generation without redundant sampling.
Table 5: Comparison of stability and precision between IFQL and Ours (Reward Avg/Std).
Method
IFQL (N = 24)
IFQL (N 28)
IFQL (N 32)
Ours
Reward (Avg) 7.6
8.1
8.0
9.2
Reward (Std)
1.5
1.6
1.5
0.2
The data Table shows that for IFQL, increasing N from 28 to 32 does not yield further improvements (8.1 8.0). In RS, increasing N increases
Robustness to Q- Valuu Overestimation.
19

---

Published as a conference paper at ICLR 2026
Effect of Varying Latent Dimension on Return
- Our coftinn)
Our (Otine)
14
12
10
Latent Dimension
Figure 11: The effect of the VAE's latent dimension the final return in both offline (black) and online (blue) settings.
the probability of selecting Out-of- DistribtDitiitti (OOD) actions where the critic network erroneously predicts high Q-values (overestimation), In contrast, our Advantage Noise Selection is anchored to the data distribution. The target "advantage noises" derived exclusively from the teacher policy, which is strictly constrained to the offline dataset's support.
3. Further Analysis on Robustness. To isolate the benefit of our distillation framework, we compared Ours against an FQL agent enhanced with Rejection Sampling (N 32) on the Cube- Double task, where Q Q-overestimation is severe. As shown in Table even with RS added, FQL fails to match Ours (37 vs. 51).
Table 6: Success Rate (%) on the Cube-Double task, comparing Ours with FQL and FQL-RS.
Method
IFQL.
FQL
FQL-RS (N = 32)
Ours
Success Rate (Avg)
9
36
37
51
Success Rate (Std)
6
Ours structurally superior from two perspectives:
Gradient Alignment: In FQL, the distillation loss pulls the policy towards the teacher's average behavior, while the Q-loss pulls it towards the mode (highest value). These direc- tions often conflict. Since RS merely filters outputs at inference time, it cannot mitigate this training conflict. In Ours, the distillation target pre-selected for high value, ensuring the distillation and Q-maximization objectives are aligned.
FQL allows the optimizer to warp the mapping from any random noise to high-Q actions, risking overfitting to noise vectors that trigger critic over- estimation. Ours constrains the student's input to the learned prior (CVAE), restricting the student to a "safe" latent manifold known to generate valid, in-distribution actions.
Input Manifold Constraint:
H.2 SENSITIVITY ANALYSIS an AND REWARD HACKING
To further examine whether our Q-guided prior amplifies overestimation artifacts, we conducted sensitivity analysis the BC weight a1 We measured the bias between the predicted Q and true return r (Bias Q - r) on the Multi-Crescent environment.
The results in Table 7 confirm that reducing the BC constraint (a, - 0) leads to severe overestima- tion. However, with reasonable setting (a1 = 100), Ours exhibits only marginal overestimation compared to the baseline (0.4 vs. 0.3) while achieving significant performance gains. This indicates that our Q-guided prior directs the agent to legitimate high-value regions rather than hacking the reward function.
20

---

Published as a conference paper at ICLR 2026
Table 7: Analysis of Q-Overestimation and Robustness.
Corresponding trends and performance
comparisons are also illustrated in Figure 12
Model Setting
Ours 0)
Ours (a) 1)
Ours (a) 10)
Ours (a)
100)
Baseline (a: 100)
Bias (Q r)
7.0
0,4
Abs Bias (Q-
10.0
7.0
1.1
0.4
0.3
Return
9.3
7.0
Q-Bias Suppression Trend
Performance a, 100
Ours Bass
Bisx
Average Return
7.0
Bins
Standard
Weight d.
Figure 12: Q-bias suppression and performance gain.
H.3 SYNERGY BETWEEN ENTROPY REGULARIZATION AND PRIOR LEARNING
To investigate the interaction between the entropy temperature 02 and the learned Q-guided prior during online fine-tuning, conducted ablation study in the Multi-Crescent environment. We varied the hyperparameter target_egnee _ent ropy lmmmmtttt maultig multiplier selier (denoted as mulf), where the target entropy is defined Htarget -mult x action dim. A larger mult results in lower target entropy, thereby reducing the incentive for exploration.
We initialized all variants from the same offline checkpoint and performed online fine-tuning for 100k steps. The results, including Return, final a2 and VAE Loss, are summarized in Table
Table 8: Interaction analysis between entropy and Q-guided prior on Multi- Crescent Environment. Note that variations in VAE Loss primarily driven by the KI. divergence component.
Setting
Target Entropy
Return
Final 0:
VAE Loss
Ours (High Exp.)
High (mall 0.1)
14.0
3.6
0.090
Ours (Med Exp.)
Medium (mult = 0.5)
12.8
2.3
0.093
Ours (Low Exp.)
Low (malt 0.75)
9.3
1.9
0.117
Ours w/o Q-Prior
High (malt 0.1)
5.9
2.1
Analysis of Mutual Benefit. The experimental results yield three key insights regarding the syn- ergistic relationship between these two components:
Q-Prior as the Foundation: The variant without the Q-guided prior suffers a drastic per- formance drop (from 14.0 to 5.9). This confirms that the Q-prior provides a robust distri- bution that ensures a valid performance lower bound and protects the student policy from Out-of-Distributte (OOD) inputs during early fine-tuning.
Entropy Drives Local Optimization: While the Q-prior successfully points the agent to- ward general high-value regions, the entropy term (a²) is essential for driving the local exploration required to find the optimal solution within those regions. Increasing the ex- ploration incentive (mult 0.75 + 0.1) leads to a substantial increase in Return (9.3 to 14.0).
Entropy Assists Prior Fitting: We observe strong correlation where higher entropy leads to lower VAE loss. When exploration is constrained (Low Exploration). the policy tends to
21

---

Published as a conference paper at ICLR 2026
collapse into sharp, potentially suboptimal local modes that are difficult for the VAE prior to model, resulting in higher KL loss (0.117). Conversely, when exploration sufficient, dg smooths the policy distribution. This smoothing effect prevents mode collapse and makes high-value actions easier for the prior to fit, resulting in the lowest VAE loss (0.090).
In conclusion, entropy regularization actively assists the prior learning process by maintaining learnable distribution landscape, proving that the two modules are synergistic rather than competing during online adaptation.
H.4 QUANTIFICATION OF EXPLORATION CAPABILITY
Quantitative Validation Puzzle-4x4. We further evaluate exploration efficiency on the Puzzle-4x0 task. This environment consists of a 4 x 4 matrix where the objective is to turn off all lights (set the matrix values 0) by toggling states. To measure the extent of state-space coverage during exploration, we define the wax.button.stars metric as the maximum value of the matrix mean observed during the entire online exploration process. A higher value indicates that the agent has reached states that are significantly different from the initial or goal states, reflecting broader exploration scope. The results, averaged over 3 random seeds, are presented in Figure [3
Exploration Capability (Puzzle-4x4)
0.53
0.4
0.3
0.2
0.1
Figure 13: Exploration capability the Puzzle-4x4 task. Ours reaches states with significantly higher max_ max_button_state, indicating a wider exploration of the state space.
Interpretation of Results. As shown in Figure [3] Ours achieves max.button.state of 0.53, which is significantly higher than FQL's 0.34. These maximum values primarily occur during the early stages of online interaction. This quantitative gap confirms that our algorithm indeed visits diverse and novel states that the baseline fails to reach.
H.5 DIRECT COMPARISON WITH SOFT ACTOR-CRITIE (SAC)
To further evaluate the effectiveness of our entropy--regularized distillation framework, we conducted a direct comparison against the original Soft Actor-Critic (SAC) and its offline-to-online variant, RLPD, on OGBench and D4RL AntMaze tasks. SAC is standard baseline for continuous control, while RLPD a high-performance algorithm designed to improve SAC for utilizing offline data.
As shown in Table 9 SAC struggles significantly in most OGBench tasks, often failing to make progress (0% success rate). This is primarily due to the multi-modal action spaces and sparse re- wards in these environments, where standard Gaussian actor cannot effectively leverage the pro- vided offline demonstrations.
Analysis. The results demonstrate that Ours achieves far superior performance on the most com- plex tasks;
Handling Multi-modality: In tasks like AntSoccer Arena and Cube Double Play, both SAC and RLPD fail completely (0% success rate). This highlights a critical limitation of Gaussian actors in tasks requiring complex, multi-modal coordination. Ours, by distilling
22

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison (Success Rate %) during the Offline + Online transition. All
results are averaged over 3 random seeds.
Task
SAC (Off - On) RLPD (Off + On)
Ours (Off + On)
HumanoidMaze Medium
0 + 2 0 -
0 + 84
- 67
AntSoccer Arena
0 +
+
Cube Double Play
0 - 0
0 - 0
51 99 88 + 100
Scene Play
- 100
- 100
Puzzle-4x4 Play
0 - O
0 100
17 - 100
AntMaze U-Maze
0 - 100
0 + 98
100 - 100
AntMaze U-Maze Diverse
0 100
0 94
93 98
AntMaze Medium Play
0 - 98
0 + 98
77 + 98
AntMaze Medium Diverse
0 98
0 97
98
AntMaze Large Play
0 - 98
0 + 93
86 - 91
AntMaze Large Diverse
0 96
0 94
85 96
flow-matching policy with Q-guided prior, maintains the expressivity needed to represent diverse modes while achieving high sample efficiency during online adaptation.
Ottinnett-OOnlnee Efficiency: Ours starts with a significantly higher offline performance in most tasks (e.g., 51% in Cube Double Play) compared to the zero-start of SAC. This confirms that our "Golden Start" effectively anchors the policy in high-value regions from the beginning, while the entropy regularization ensures stable and principled exploration to reach near-pennt perfeetecctttt success rates (99%-100%) during fine-tuning.
I
FUTURE WORK
While our method demonstrates improvements in continuous control tasks through Q-guided priors and entropy control, several promising directions remain for future research:
Unsupervised Skill Discovery: We plan to extend the "Golden Start" concept to the do- main of unsupervised skill discovery Eysenbach et al. (2018); Park et al. (2023); Zhang et al. (205)) By leveraging the expressive power of flow-matching policies, discover diverse and reusable primitive skills from unlabeled offline data without manual reward engineering. The learned priors could potentially serve structured latent space for skill induction and autonomous exploration.
aim to
VLA Models: Integrating our framework into high-dimensional, multimodal VLA archi- tectures (such as Black et al. (2024a); Intelligence et al.) (2025)) is priority. Preliminary investigations suggest that while data curation remains bottleneck in VLA settings, the ability of our method to model complex, multi-modal distributions could significantly hance the performance of embodied agents in tasks involving long-horizon reasoning and language instructions.
Generalization to Discrete Action Spaces: Although the current implementation focuses on continuous control, the underlying principle of value-aligned generative priors is also applicable discrete domains. We intend to explore the adaptation of flow-matching dis- tillation to discrete action spaces, which could broaden the applicability of our method to a wider range of decision-making tasks, such strategic games or combinatorial optimiza- tion Xu et al. (2019) Zhang et al. (2023).
23


Reviews:
[{"review_id": "qSSEkJXIKZ", "reviewer": "Reviewer_frzm", "summary": "This paper proposes GoldenStart (GS-flow), a novel framework for distilling flow-matching policies in reinforcement learning. The authors identify two key limitations of existing one-step distilled flow policies: \n\n(1) the uninformed Gaussian prior that ignores the value landscape, and (2) the deterministic mapping that prevents controllable exploration.\n\nTo address these issues, GS-flow introduces two main innovations:\n\n(1) Q-Guided Prior Learning, where a conditional VAE learns a state-conditioned prior that biases the initial noise toward high-Q regions, providing a “golden start” for generation; and\n(2) Entropy-Regularized Distillation, which replaces deterministic action generation with a stochastic actor trained under an entropy-regularized loss, enabling adaptive exploration during online fine-tuning.\nExtensive experiments on OGBench, D4RL AntMaze, and Visual RL benchmarks demonstrate that GS-flow consistently outperforms prior state-of-the-art methods (e.g., FQL, IFQL), particularly in multi-modal and exploration-heavy tasks such as Cube Double and Puzzle-4×4. The method preserves single-step inference efficiency while achieving significant gains in both offline and online performance.", "questions": "1. The Multi-Crescent environment is an interesting design to expose Q-value overestimation, but given its continuous nonconvex reward surface, there might be potential for spurious high-Q regions or reward hacking when the critic is imperfect. Could the authors examine whether GS-flow’s Q-guided prior might amplify such artifacts? For instance, visualizing true reward vs. predicted Q over the state-action space or adding a calibration/consistency check (e.g., reward clipping or Q–r alignment) would strengthen the claim that the method is robust to reward misestimation.\n\n2. The actor is trained with a composite objective combining distillation, Q-value maximization, and entropy regularization. Given the potentially conflicting gradients among these components, could the authors clarify whether they observed optimization instability or gradient interference during training?\n\n3. Have the authors explored combining GS-flow with language- or vision-conditioned policies (e.g., VLA settings) to test generality beyond low-dimensional control?\n\n4. Could the authors provide empirical insights into the interaction between entropy temperature α₂ and Q-guided prior learning — are they synergistic or competing during online fine-tuning?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper precisely identifies two previously overlooked weaknesses in existing one-step distilled flow policies — the use of an uninformed Gaussian prior and the absence of controllable stochasticity — and builds upon them to formulate a coherent research question and motivation.\n\nThe work introduces a conditional VAE that learns a state-conditioned high-Q prior, transforming the random starting point in generative inference into a structured, value-aligned initialization, thereby providing an effective “shortcut” for policy optimization.\n\nThe authors reformulate the student policy from a deterministic mapping into a distributional generator with entropy control, achieving a principled balance between exploration and exploitation and significantly enhancing online adaptability.\n\nAcross benchmarks such as OGBench, D4RL, and Visual RL, GS-flow consistently achieves state-of-the-art performance, demonstrating superior sample efficiency and exploration capability while maintaining single-step inference speed. Moreover, the ablation studies clearly disentangle and validate the independent contributions of the prior learning and entropy regularization modules.", "weaknesses": "1. The Advantage Noise Selection module relies on a hard argmax over Q-values to identify the optimal noise per state. While simple, this approach may amplify critic bias and reduce diversity in the learned prior. More robust alternatives such as soft advantage weighting or top-k filtering could potentially mitigate this brittleness.\n\n2. While GS-flow demonstrates impressive efficiency within the family of flow-matching methods (notably compared to FQL and IFQL), it remains unclear whether this efficiency translates to practical advantages over lightweight Gaussian actor-critic approaches such as SAC or IQL. Given the additional complexity of the teacher–student architecture and prior learning modules, the overall time–performance trade-off might still be less favorable in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdgE6RKyD5", "reviewer": "Reviewer_hfGV", "summary": "This paper introduces GoldenStart, an improvement of previous existing methods for offline and offline-to-online reinforcement learning tasks, that requires to model and learn complex distributions. \n\nMethod:\n\nThis work builds on Flow Q-learning (FQL), a method based on the actor-critic structure. FQL is based on flow matching: the goal is to train a distilled (one-step) state-conditional flow student network to output actions that (i) maximize the value function (ii) while minimizing a distillation loss against a flow teacher. The flow teacher is trained to do behavioral cloning on an offline dataset D using multiple timesteps (unlike the student), akin to the standard flow matching method. (i) encourages exploitation and (ii) encourages staying close to the collected high reward dataset D.\nOn top of this, GoldenStart introduces 2 main contributions: \n-  First, instead of using gaussian noise as prior,  they learn a Q value guided prior model as a state-conditional VAE. \n     Such a conditional VAE is trained in 2 steps: \n          - First, they generate \"advantage noises\". Given a state and N_cand initial gaussian noises, they generate actions using the student flow network, and they keep the action with the highest value function,\n           - Second, they train a state-conditional VAE that learns to generate the advantage the distribution of advatange noises from a gaussian noise.\n- Second, they train the student flow to predict a mean and a variance. Why? Because in addition to the value maximization term and the distillation term of FQL, that were already present in FQL, they add an entropy term. This encourages exploration in addition to exploitation.\n\nExperiments: \n\nThey introduce a new toy task called crescent task,  and they do ablation studies to illustrate the added value of each of their contributions in terms of exptected return. \nThey also use the same experimental setups as in FQL (i.e. OGBenchmark, AntMaze, Visual envs ), and the average expected return (over different subtasks) is higher with GoldenStart than with FQL and other baseline methods .", "questions": "- The teacher flow network trained on the offline dataset using gaussian noise as a starting point, but in the text you do inference from it using the distribution of x_advantage as a starting distribution. Could you explain how the teacher adapts to the distributional shift of this initial distribution? \n- Correct me if I am wrong, but it seems to me that the conditional VAE is learns a \"higher return actions distribution\" than the student flow network (because of the argmax in the Q value over N_candidates, and the absence of the entropy regularization term) ? If yes, could you show the expected return over some tasks in the OGBench using the advantage noise selection module only? \n- Why do you need to predefine an entropy target H_{target}? How do you choose it in your exps? Does it depend on the task?\n- Do you have some results that could show that GoldenStart is doing principled exploration that FQL is not doing? E.g. discovering some actions that FQL does not discover, and that are not in the offline dataset D ? (It is difficult to infer from the expected return only that GoldenStart allows for better exploration compared to FQL).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The clarity of the paper, the explanations, the contributions and evidence supporting them\n- The introduction of the crescent toy task with the 2D visualisations, as well as the ablation studies in the experiments for crescent for each of the contributions (prior / entropy), which helps in understanding,\n- The simplicity of the introduced changes\n- The exhaustive set of experiments (as in FQL)", "weaknesses": "- The additional computational cost for learning the VAE prior distribution", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFtoMy45d6", "reviewer": "Reviewer_NLQk", "summary": "The main contribution of this paper is a new method to learn a “golden prior” noise distribution for flow matching, that produces higher-valued actions than the initial Gaussian distribution. This prior is learned by sampling many noise vectors, mapping them to actions through flow matching, and then training a conditional VAE to generate only the highest-valued of the noise vectors. This transformed distribution is then used to seed the flow-matching policy for action selection. The starting point for the broader algorithm is FQL, and this modification, along with adding stochasticity and entropy-regularization to the policy, improve the performance of the final algorithm.", "questions": "Mainly, I would like to better understand the comparison of IFQL and other rejection-sampling methods with yours. First, what “N” was used for rejection sampling in IFQL? Second, should we expect Advantage Noise Estimation to be better than rejection sampling, or just faster?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of Golden priors is an interesting one, and there are not many instantiations of it. As this paper points out, directly optimizing flow-matching policies using RL has many challenges that this method avoids. Advantage Noise Selection is a reasonable method for creating higher-valued noise targets. The experimental results are thorough, and the crescent experiment is a very good demonstration of the method doing what it claims. The method achieves quite good performance on the extensive benchmark suite.", "weaknesses": "I would describe the instantiation of Advantage Noise Selection as amortizing the rejection-sampling process. I think a stronger argument could have been made for, and more time spent on, why we expect an advantage of this method over rejection sampling (IFQL is the included reference for this) — why does this lead to better performance when it is doing something pretty similar? The 2x speedup I think is not strength enough alone given the added complexity.\n\nThe entropy regularization is empirically helpful, and good to have tested, but I would argue not very surprising.\n\nI think the theory section has some weaknesses. First, it relies on the prior actually producing better initializations, and Lemma 2 is not quite rigorous. And second, is that implicit in Theorem 1 is that sampling a higher-Q-estimate action is always than sampling a lower-Q-estimate action. When this in fact depends on whether your Q function can be trusted, which in off-policy RL is one of the main difficulties, and why we encourage sticking to the prior. So I don’t think this proof is wrong, but also doesn’t provide much supporting evidence for the model’s effectiveness. As a minor thing, I think the theory should be mentioned in the main text, as I was surprised to see it when I scrolled down.\n\nAnother minor thing, L_distill is confusingly included twice meaning two different things, Eq 10 and Eq 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJ5Cz4LFHT", "reviewer": "Reviewer_ndGM", "summary": "The authors introduce a golden start method for policy distillation with Q-guided priors. In addition, they allow the distiller to output a stochastic distribution based on entropy regularization, that allows the policy to explore beyond what has been learnt from offline data. The authors claim that they provide a new state-of-the-art.", "questions": "What does the math in the Appendix reveal that is novel?\n\nWhat would be the performance of the new distillation method against SAC, let us say, in locomotion and maze tasks?\n\nTypos: Line 366 “achieves”, and Line 483 “establishes”.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is nicely written and easy to follow. There are many comparisons with baselines (but see below). Combining a golden start with entropy regularization seems to be an important move forward.", "weaknesses": "It looks like there is an important typo in Table 1. The results claimed are not as good as I initially thought: In the 4 first tasks, the new method outperforms the previous ones only in 1 case. But a closer look reveals that this is a typo, as ReBRAC has a performance of 91, above the new method. Therefore, the new method does not show any benefit in the locomotion tasks. \n\nFurther, a closer look at the D4RL environments reveals that the improvement in performance is only marginal with respect to FQL, despite having a higher computational cost. \n\nSecond, the mathematics presented in the appendix are kind of trivial. Showing that under an updated policy that considers the highest values of a learned Q value there is going to be an improvement in reward is a trivial fact. I am not sure what the scope of Theorem 1 is beyond showing an expected result, which is so by construction. \n\nFinally, the introduction of the entropy bonus term follows standard approaches in previous RL literature (Haarnoja, icml, 2018, quoted by the authors). From this addition it is totally expected that exploration would be improved. However, no comparison with continuous optimal controllers, such as SAC, are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)