Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
Authors: Yongjae Shin
Jongseong Chae
Jongeui Park
Youngchul Sung
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Offline-to-Online Reinforcement Learning
Flow Matching
Noise Injection
Primary area: reinforcement learning
TL;DR: We propose FINO, a flow matching–based policy that injects noise and uses entropy-guided sampling to enhance exploration and consistently outperform baselines in offline-to-online RL.

Abstract: Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL).
While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed.
In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL.
FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset.
In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning.
Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Published as a conference paper at ICLR 2026
FLOW MATCHING WITH INJECTED NOISE FOR OFFLINE-TO-ONLING REINFORCEMENT LEARNING
Yongjac Shin
Jongseong Chae
Jongeui Park
Youngchul Sung'
KAIST (yongjae.shis shin, ycsung)0kaist..o ac.kr
ABSTRACT
Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learn- ing (RL). While they have shown strong performance in offline RI., particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with In- jected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy train- ing, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine entropy-guided sampling mechanism to balance exploration and ploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.
INTRODUCTION
Generative models have recently demonstrated substantial success across diverse domains, produc- ing high-quality outputs in areas such as text and image (Brown et al. 2020 Rombach et al. 2022). By leveraging their expressive capacity, these models can capture complex multimodal distribu- tions present in the underlying datasets, beyond the reach of conventional parametric models. This opens up new opportunities in reinforcement learning (RL), particularly for policy design.
Since policy in RL can be regarded as a generative model conditioned on states, there has been increasing interest in applying generative modeling to policy design, such as denoising diffusion
(Sohl-Dickstein et al.] 20155 Ho et al. (020) and flow matching (Lipman et al., 2023 Albergo & Vanden-Bijnden, 2023). While Gaussian policies have been the conventional choice, they often struggle to represent multimodal or -igeeniinnii action distributions (Park et al. 2024). By contrast, generative policies provide the expressivity required to handle complex RL tasks and have demonstrated superior performance, particularly in offline RL where the target distribution is explic-
itly defined (Wang et al. 2023b Maansen-Estruch et al. 2023 Kang et al.) 2023
Zhang et al. 2025
Fang et al. 2025
Park al. (0256)).
Despite such expressivity, offline RL inherently suffers from a fundamental limitation in that the performance of the policy is constrained by the quality of the offline dataset. Accordingly, offline- to-online RL has been proposed to address this issue, enabling pre-trained policy to further improve its performance through short-term direct interaction with the environment (Lee al., 2022 Zhang et 2023) Nakamoto et al. 2023; Zhang et al., 2024) Zhou et al., (2224). While some studies (Hansen-Estruch et al. 2023] Park et al.) (025b) have incorporated additional online fine-tuning of
generative policies. they merely treat it as a continuation of offline pre-training rather than designing
approaches specialized to the online fine-tuning.
As offline-to-online RL encompasses both offline and online stages, it naturally introduces chal- lenges beyond those faced a purely offline RL framework. Unlike offline RL, which relies solely
"Corresponding author.
Our code is available at/ttps:////t thu.. com www/CT/D22230

---

Published as a conference paper at ICLR 2026
on pre-collected datasets, offline-to-onling RL incorporates an online fine-tuning phase, making it beneficial to design the offline pre-training with this subsequent stage in mind from the beginning. At the same time, the framework raises the practical question of how to best exploit the pre-trained policy during online fine-tuning. Thus, framing offline-to-online RI. merely as an extension of of- fline RI. can limit the extent to which its potential is realized.
In this work, we propose Flow Matching with Injected Noise for Offline- tonestttooooo to-Online RL (FINO), a novel policy learning approach for the oflinneto-online RL framework. Motivated by recent findings that maintaining diversity facilitates more effective fine-tuning (an et al. 2025 Li et al. 2025 Zhai
al. 2025 Sorokin al.) 2025), we introduce a training strategy that injects noise into the flow matching to explicitly promote diversity in the policy from the beginning of offline pre- pre-training This injection encourages the policy to learn a broader range of action space than that present in the offline dataset, thereby establishing a strong foundation for exploration during online fine-tuning, To effectively leverage this during online fine-tuning, we introduce an entropy- -guided sampling mecha- nism that exploits the acquired diversity for exploration while balancing exploration and exploitation by adapting to the evolving behavior of the policy. We experiment on 45 diverse and challenging tasks from OGBench (Park et al.) 2025a) and D4RL (Fu et al., 2020) under limited online fine- tuning budget. The results show that FINO achieves consistently strong performance across tasks, even in complex environments, thereby demonstrating FINO as effective and reliable approach for offine--to-onlne RI.
2 PRELIMINARIES
Offiine-to-nnie Onlline Reinforcement Learning. In this paper, we consider Markov Decision Process (MDP) (Sutton et al. 1998) M = (S, where S denotes the state space, A the action space, the reward function, P the transition probability distribution, and y the discount factor. The objective of RL is to train policy that maximizes the expected cumulative return Ej 1E,,, ''r(s,, a. Offline-to-online RL is a two-stage learning framework consisting of offline
pre-training and online fine-tuning <ee et al. 2022 Zhang et al. 2023 Nakamoto al., 2023
Zhang et al.] 2024; Zhou et al.] 2024). This framework is designed to combine the strengths of
offline and online RL: the stability gained from pre-collected datasets without interactions and the adaptability that comes from environment interaction. In the offiine pre-training, a policy is trained on static dataset D {(s o, r, s')), providing reliable initialization. Subsequently, during the online fine-tuning, the pre-trained policy directly interacts with the environment, allowing it to refine its behavior and correct limitations inherited from the offline dataset.
Flow Matching, Flow matching (Lipman et al. 2023 Albergo & Vanden-Eijnden 2023) is a gen-
erative modeling framework that constructs transformation between two probability distributions via ordinary differential equations (ODEs). Unlike diffusion models Dicks-Dickstein et al.) 2015) Ho et al., 2020), which rely on stochastic differential equations (SDEs), flow matching is based on deterministic ODEs. This design not only simplifies training but also enables faster inference.
The central component of flow matching a time-dependent vector field (t, x:) that defines a flow e., mapping a base distribution pa into a target data distribution pI:
d dt Φ((x) vo (t, (x)),
(o(()) x.
(1)
A widely used formulation of flow matching is based on Optimal Transport (OT) (Lipman et al.) 2023), where transformations are constructed by linearly interpolating between samples from the base and target distributions (xo Po and PI). with the interpolation time t sampled uni- formly:
2t (1 t) + tæ t~ Unif((o, 1)).
(2)
The vector field is trained align its prediction with the direction of this linear path: [| 2.) - (x1 (1 (min) [12]
min 2xp [0.
(3)
where Jmin is sufficiently small constant. Once the vector field vo is trained, generation is per- formed by sampling 2o ~ Pn and solving the learned ODE until t = 1 to obtain @ (xp) ~ PI. In this work, we use the Fuler method to solve the ODE for sample generation.
2

---

Published as a conference paper at ICLR 2026
Flow Q-Learning. Flow Q-Learning (FQL) (Park al. 2025b) applies flow matching to policy design for offline RL. formulates the policy as a state-conditioned flow model and trains it by adapting flow matching to behavior cloning:
(0)
B No..) (1, - (x1 =o)1l
s,a= t-Unni([ [0.1])
Integrating the trained vector field ty induces mapping ag(s, 2) from state and noise = to action, which defines policy Sg, linking the flow formulation policy representation.
To enable efficient training, FQL further introduces a one-step policy ww. which jointly optimized by distillation from the flow policy and oction-value maximization:
L (w) E
sruD, D. (0. aw (e,5)raT
[ Qo(s, (s) (s, as as.
(5)
where a is hyperparameter. In practice, the one-step policy provides direct mapping from noise to actions without sequential ODE integration, enabling efficient action selection while inheriting the expressiveness of the flow model.
3 MOTIVATION
witmasz--iant--aavie
FOL
FINO (ours)
10000
300
FEAT
TAC
Performence
100
FINO(Ours)
FOL
Figure 1: Comparison of FQL and FINO (ours) in terms of performance and exploration patterns on The green circle and red star indicate the initial
the environment antma www.eee-ggtttiiiggee and goal states, respectively.
Our motivation lies in better leveraging the expressivity of generative policy, flow policy in par- ticular, to address the challenges of offline-to-online RL. There exist prior studies in offline RL. (Hansen-Estrucd et al. 2023 Park et al. (2025b) employing generative policies and extending them online fine-tuning tunettunig. To examine their behavior during online fine-tuning we conducted an experi- ment with the challenging task antma ze iilant-navis aaigatt with FQL (Park et al. (025b). The second plot of Figure Toaureeilllaa illustrates the maze, where the gray region marks the feasible paths, while the third plot shows the visitation frequency of the FQL agent during the first 100k interaction steps. is seen that the agent stays mostly near the starting point and reaches the goal only via the upper path, ignoring other possible routes, yielding degraded performance [ This behavior reflects an offline pre-trained policy that is overly confined to the dataset, which mainly contains the upper success route, resulting limited exploration during online fine-tuning. In strictly offline setting, such confinement to the data distribution is a primary design objective to ensure stability. But, considering the subsequent online learning, such confinement may not be the best strategy.
shown in the first plot in Fig-
One could address this limitation by constructing a larger dataset with diverse data, but this incurs additional cost and time. Then, how can one induce diverse behavior from a given dataset without increasing the dataset size? To answer this question, we propose perturbed cloning during the offline pre-training phase, especially suited to flow-based policy learning. In our training scheme, we inject noise into flow matching, thereby driving the flow behavior model to extend its support beyond the coverage of the dataset to some reasonable extent. The policy is then distilled from this perturbed behavior model, which allows it to acquire behaviors spanning a broader action space. So learned policies leverage this broader coverage during subsequent online fine-tuning, facilitating effective exploration to yield better performance, as shown in the rightmost plot of Figure Details of the proposed method follow in the next section.
3

---

Published as a conference paper at ICLR 2026
4 METHOD
In this section, we present Flow Matching with Injected Noise for Offline-to-nlline RL (FINO), a novel method that utilizes flow policies within the offline-to-online RL framework. Our approach consists of two main components: (1) from the beginning of offline pre-training, we inject controlled noise into flow matching, encouraging the policy to explore a broader range of actions beyond those in the dataset; (2) in the online fine-tuning, we leverage this expanded action space for exploration, while introducing an entropy-guided sampling mechanism that dynamically balances exploration and exploitation according to the behavior of the policy.
4.1 NOISE INJECTION FOR FLOW MATCHING
Since our method builds directly on the flow matching formulation, we begin by presenting its conditional probability path. The rationale behind training flow matching as in Equation lies in the design of its conditional probability path (Lipman et al. 2023):
pt (x(21) = =>(xxtz1,, (1 (1 F It)
(6)
where mmin is a sufficiently small constant ensuring that pPM()) (x/m1) concentrates around the given data point.
In FQL (Park et al. 2025b), the variance is set to min = in Equation which reduces the training objective Equation With 7min 0, the distribution collapses onto individual data points, leaving little coverage beyond the dataset itself. This narrow formulation shows clear limitations as shown in the previous section, as it restricts effective exploration during online fine-tuning. To overcome this limitation, we propose noise-injecter training scheme that retains the core objective of flow matching while enabling the model to learn a wider action space than point-wise matching:
(i) E S,aa,y DD [1)vo(t, 8, 84 + (e) (x1 - (1 17) 6 ~ ~00, T² (7)
(0.I). t-Unni( 0.1])
where of (17² 2m) t + (2n) is the scheduled variance for some 17 00 1], and t is the
interpolation time. Equation 7 reduces to the standard flow matching (Equation 4 when r = 0. The variance of the injected noise non-negative for all 17 2 0 and t E [0, 1]. Note that at t 0, ag 0, and at = 1, of = 2/2 > O.
Proposition For notational simplicity, we denote (si,xt), as It. Given a dataset D (x: | the proposed time dene-dependent noise injection ~ N(0, ((0,afT) induces the following conditional probability paths of flow Φr:
FINO (x|2,) - - p((z)) = tx,, -1 ps
in which the mean tx; is equal to the mean induced from flow matching, and the variance (1 (1- n)) is greater than or equal to the variance induced from flow matching.
Theorem 1. Given data z4 from dataset distribution and noise zo from the base distribution, the conditional probability paths in Proposition induce the unique conditional vector field that has the following form:
ve(x(1x)) x, 11 -1--11n))
Then, for any dataset distribution, the marginal vector field vt (x(x) generates the marginal probability path pr))), in other words, both vx(x)) and (z() satisfy the continuity equation.
Theorem shows that FINO (Equation yields valid continuous normalizing flow, which means that the flow model trained by Equation 7 can generate samples close to those obtained by the behavior policy of the training dataset.
Theorem 2. Suppose the cardinality of the dataset is finite. D 12,,,2,......................... AND for some N, and data are independently and identically distributed (i.id.) sampled. The variance of the marginal probability path induced by FINO (Equation 7 is greater than equal that of the marginal probability path induced by the flow matching (FM) objective (Equation 3 For any time E [0, 1}.
Var (XF1NO) 2 Var (X(M),
X.INO pr FINO (x), XFM Pt FM (x).
4

---

Published as a conference paper at ICLR 2026
Algorithm FINO: Flow Matching with Injected Noise for Offline-to-Online RL
1: Inputs: flow matching policy $s. one-step policy Tw. value function Qş. candidate action samples Nample entropy update steps Ne
2: while in offline pre-training do
3: Update w, based on Equation and update o via TD loss
4: end while
5: while in online fine-tuning do
6: Sample sample candidate actions {a1, 02: ". ONaam } (s)
7: Compute sampling probability p(t) using Equation 8
8:
Select a from categorical distribution p
9:
Update w, based on Equation and update di via TD loss
10: if step mod Ng == then
Estimate the entropy of policy H 12: Update & using Equation 9
11:
end if'
14:
end while
Theorem states that at time 1, the marginal probability path induced by FINO (p) FINO (x)) ex-
hibits larger variance than flow matching (v/ (z)). This means that the model trained by Equation represents wider action regions than the flow matching model (Equation Bp, making more suitable
for exploration. The proofs of Proposition I and Theorems I 2 are provided in Appendix C
To illustrate the effect of our design, we conduct simple toy experiment. We consider a setting with fixed state and two-dimensional action space, where the dataset is generated by sampling points inside four circular regions. Both flow match- ing and our proposed method are trained the same dataset. As shown in Figure 2 flow matching predominantly focuses points themselves, leading the trained actions to remain almost entirely within the dataset dis- tribution. In contrast, our method with noise injection learns to cover a wider region of the action space. Notably, this expansion occurs in reliable manner, remaining centered around the dataset and thereby providing a broader yet plausible coverage of the action space.
FLOW
FINO
the data
Figure 2: Toy example: blue contours represent the log-density of model samples; red circles de- note the dataset.
The expanded flow model then guides the training of the one-step policy that interacts with the environment. the one-step policy trained under Equation the expanded flow model enables action-value maximization over broader region of the action space, which is then utilized for exploration during online fine-tuning. We provide a detailed explanation in Appendix D
4.2 ENTROPY-GUIDED SAMPLING
After offline pre-training, the next step to leverage the policy effectively during online fine-tuning, where the agent continues improving through direct interaction with the environment. With noise injection, the policy is trained to yield more diverse actions, each reflecting slightly different be- haviors. To exploit this action diversity, the agent first samples multiple candidate actions for given state using multiple base noises for the flow model. Now, we do simply choose the action that maximizes action-value, which corresponds to exploitation. Instead, we construct a sampling distribution over the candidate actions based on their action-values as
exp(E (0(),a))) Σ, exp(E Qs(s,))))*
Psampling
Vi E
samplee]
(8)
where & is a temperature parameter. An actual action is drawn from Psampling, so that even lower- value actions can be sampled for exploration purposes. A smaller produces flatter distribution that promotes more uniform sampling and encourages exploration, whereas a larger sharpens the distribution and prioritizes greedy actions for exploitation.
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance of FINO and baselines across OGBench and D4RL tasks. Results show scores after offline pre-training and after online fine-tuning, averaged over 10 seeds with mean and 95% confidence intervals. D4RL. antmaze and adroit aggregate six and four tasks, respectively, while OGBench reports results over five tasks (task names abbreviated by omitting the singletask suffix). Full results are presented in Table 4
Task
ReBRAC
Cal-QL
RLPD
IFQL
FQL
FINO
OGBench humanoiddmaze medium navigute
21as-+31s
Out 0.0
0za->
59 70
53:6-+611
50a7-> 97
OGRench huranoichaze- large-aggaa navigaa
251-3
-0_e
010
11a3 10g
521
1023
63_-
OGBench antmaze large navigate
85:3 99
12:4-> 12
Oso 80st
32s4->72n: 72
81a2 92ss
S1a2 99.0
OGBench antmaz-- giante--gan-nnnn navinat
38g7 > 9661
2 10_0
Oie 147
111 >010
16.0
7711
14.6 79.
OGBench antsoccer arena navigate
0-e
33
61a2-4
57
OGBench cube double play
911 > 2812
211 >0_0
Dio 211
1411 140
31,3 >73
34.1 799..
OGBench puzele-4x4-play
14e1 2945
2-3 20e-
Deo St-
2642-442ed
15-2-+455:
19-5-556-3
D4RL antmaze
80 891s
10
D1o 01g-
79u5
80.4-3 95ar
79
D4RL adroit
21 83+2
-0=z -0
0se 73
18+1-4 42+1
14:3-4 100
13+2-+122
Since sample-efficient learning under a limited interaction budget is the primary objective of online fine-tuning, maintaining an appropriate balance between exploration and exploitation remains crit- ical challenge. However, relying on fixed value of cannot adequately address the dynamics of the learning process. So, we adapt the sampling strategy to the behavior of the policy, using entropy of the policy (H) an indicator and adjusting accordingly:
Snew { ag lg H.),
where is the target entropy, and ag denotes the learning rate. By adapting its behavior according to the policy entropy, it properly controls the balance between exploration and exploitation through- out online fine-tuning. At inference time, the agent deterministically selects the action with the highest action-value, ensuring stable performance. The overall training pipeline is summarized in Algorithm
4.3 PRACTICAL IMPLEMENTATION
We use FQL (Park et al. 2025b) the backbone model, and accordingly the one-step policy, trained with Equation is employed for environment interaction. Since this policy is obtained through dis- tillation and action-value maximization, its distribution intractable, making direct entropy com- putation infeasible. To address this, we follow prior work (Wang et al.) 2024) and estimate entropy by sampling multiple actions from the same state and fitting them with a Gaussian Mixture Model (GMM). A detailed description of the computation procedure is provided in Appendix E.
Regarding hyperparameters, FINO involves two key parameters. For 7), which determines the vari- ance of the injected noise, we set its value based on the action range. Since all experimental environ- ments use actions bounded within [-1, 1]. we fix = 0.1. For Nsample as the volume the action space explore grows significantly with the dimension, more samples ciently diverse set of candidates for effective exploration. Therefore, we set the number of sampled actions to half of the action dimension. In implementation, we adopt smooth shifted exponential schedule for or that satisfies the same boundary conditions, i.e., og Rs 0 and a = ry² > 0, and we simply use the target vector 21 To. empirically observed difference in performance. We note that these core hyperparameters remain fixed throughout the training process. Further imple- mentation details and additional hyperparameter settings are provided in Appendix and G
required to obtain suffi-
EXPERIMENTS
In this section, empirically demonstrate the effectiveness of FINO. To this end, evaluate the proposed method across a range of challenging environments, comparing its performance against several baselines.
Environments. We primarily evaluate the performance of FINO on OGBench (Park et al. 2025a), recently proposed benchmark that extends beyond the commonly used D4RL (Fu et al. (020) by incorporating tasks with greater diversity and complexity. Although OGBench is originally intro- duced for benchmarking offline goal-conditioned RI, we adapt it to setting by employing its

---

Published as a conference paper at ICLR 2026
ReBRAC
Cal-OL
RLPD
- IFQL
OGBench
D4RL
100
100
Perforaance
50-
0-
(x100k)
Steps (x100K)
Figure 3: Aggregate performance across two benchmark domains. Each figure reports the averaged learning curves over the common environments within the respective domain. Full results are pre- sented in Figures 9 and I0]
single-task variant, where each goal is treated as an independent task. We also include results on the widely adopted D4RL. benchmark, which remains a common benchmark in offline-to-online RL.
Baselines. We consider the following baselines for comparison: (1) ReBRAC (Taasovv et al.
2023) is Gaussian policy-based method, has demonstrated strong performance across offline RI. and the offline-to-online RL setting. (2) Cal-QL (Nakamoto et al. 2023) is a representative offline- to-online RL algorithm that extends CQL (Kumar et al. 2020) to the offline-to-online setting. (3) RLPD (Ball et al. 2023) is an online RL algorithm initialized with an offline dataset, which achieves superior performance despite relying solely online training. Since prior work has provided lim- ited investigation of flow matching and denoising diffusion in the offlne-to--nling RL setting, we additionally construct flow matching variants of existing algorithms to provide meaningful point of comparison.
(4) IFQL, introduced in the FQL. paper (Park et al. al.] (0255)) is an adaptation of IDQI. (Hansen-Estruce et al., 2023) the flow matching setting. Similar to our approach, samples mul- tiple actions from a single state and selects one for execution. (5) FQL (Park et al. (025b), described in Section 2 serves the backbone algorithm upon which our method built.
Evaluation. For all baselines, we report results using the same experimental protocol, consisting of 1M offline pre-training steps followed by 500K online fine-tuning steps. To assess the performance gain during online fine-tuning, we present both the results immediately after offline pre-training and those obtained at the end of online fine-tuning. All experiments are averaged over 10 random seeds, and we report the mean and 95% confidence intervals. The best-performing results are highlighted in bold when they fall within 95% the best performance.
Results. Table summarizes the results across total of 45 tasks, aggregated by task category. Overall, FINO consistently achieves strong performance across a diverse range of environments. Crucially, this is achieved without degrading offline performance, that preserves the mean of the probability path while increasing variance from the offline dataset, supported by Theorem 2 When compared ReBRAC (Tarasov et al. 2023), observe that al- though ReBRAC exhibits strong performance on environments such as antmaze, it struggles to effectively learn in more complex and challenging humanoi amaze environments due the in- herent limitations of its conventional policy. The comparison with IFQL underscores that action candidate sampling alone is insufficient to explain the performance improvement. In addition, when compared to the backbone algorithm FQL (Park et al., 2025b). the results highlight the effective- ness of our method during online fine-tuning, where FINO demonstrates both efficient exploration and balanced trade-off between exploration and exploitation. The improvements observed in the navi gate environments further suggest that FINO well suited to environments where effective exploration is critical.
our method learns the model
In addition to the tabular summary, we provide aggregate learning curves by benchmark Figure to visualize the progression of performance over training steps. Consistent with the results in Table the figure shows that FINO consistently outperforms the baselines throughout training. In particular, the experiments OGBench demonstrate that, despite starting from the performance as the backbone algorithm, our method achieves stronger improvements, underscoring its high sample efficiency.
7

---

Published as a conference paper at ICLR 2026
humannsidaazz--rre-nvvvggge
puzzle-4x4-play
door-cloned
100
100
-Q-
FINO
-Q-
Action Noise
Performance
50
50
Steps (x100k)
Steps (x100k)
Steps (x100k)
Figure 4: Comparison between FINO and the direct action noise injection baseline. Each plot shows results aggregated over five tasks and averaged across 10 seeds, with shaded regions indicating 95% confidence intervals.
DISCUSSION
6.1
IMPACT OF NOISE INJECTION POINT
One of the key components of the proposed method is the injection noise into the flow matching objective, which expands the action space and enables more efficient exploration during online fine- tuning. To evaluate this design choice, we compare our approach with a simpler alternative that promotes exploration by injecting Gaussian noise into the actions generated by the policy rather than into the flow matching objective (denoted as Action Noise). For a fair comparison, we retain other components such the action-candidate mechanism and entropy guidance in this baseline as well.
The results in demuontraee that the noise injection strategy of the proposed method yields notable performance gain. Consistent improvement is observed across both navigation and manip- ulation environments, indicating that the proposed noise injection scheme remains effective across task categories. This difference arises because, as mentioned in Section 4.1 the proposed method enables the one-step policy to maximize action-value over a broader action space, rather than simply adding noise to the action. Notably, the results on door-cl oned show that, when compared with backbone algorithm FQL. (whose performance approximately 100), simply adding noise to exploratioo
the action fails facilitate exploration and even degrade performance. These findings highlight that the proposed noise injection method serves as an effective approach for promoting exploration during online fine-tuning. Additional analyses of various noise injection strategies provided in Appendix D
6.2 COMPARISON WITH ENTROPY-REGULATED NOISE SCALING
In Section 4.2 we introduce an entropy-basee guidance method for action sampling. This approach en- ables the policy achieve balanced trade-off between exploration and ex- ploitation during online fine-tuning. which critical under a limited on- line budget. Since previous studies (Haarnoja et al.] 2018 WWang et al.] 2024) have also employed entropy to regulate this balance, we com- pare our method with an alternative entropy-driven strategy (denoted as ER-Noise) to evaluate its effectiveness. sampling. the baseline replaces it with on the entropy. The noise is then directly added to the action, allowing the action to be adjusted according to the entropy of the policy.
humanodmaae-meeduu--nnvvee
humanoodmaar--ang--awvvee
100
FINO
ER-Noise
Performance
50
Steps
Steps
Figure 5: Comparison between FINO and the entropy- regulated noise scaling baseline. Full results are presented in Table
Specifically, instead of using entropy to guide the action simpler approach that scales the Gaussian noise based
8

---

Published as a conference paper at ICLR 2026
hurnannddaaze-meeuu--vvvieeee
Training time (4)
Inference time (4)
100
15
Performance
[ms]
10
[ms]
50
FIND
w/o Noise
Guidance
ReBRAC Cal- Cal-Ol
RLPD
IFOL
FO
FINO
ReBRAC Cal Cal-Ol
RLPD
IFOL
FOL FINO
Steps (x100k)
Figure 6: Comparison of performance and computational efficiency. The left figure shows the learn- ing curve the humanol dma e--med um-navi gate task, averaged over five tasks with 10 random seeds, with shaded regions denoting 95% confidence intervals. The middle and right figures report the training and inference time per step of each baseline. Full results are presented in Table
Figure 5 shows that the proposed method significantly outperforms entropy-based noise scaling by effectively balancing exploration and exploitation through sampling aligned with the entropy of the policy. In particular, despite the inherent difficulty of finding relevant actions high-dimensional action spaces such humanoidmaze, the proposed method successfully identifies appropriate actions in such settings. However, the performance of ER-Noise indicates that mere noise scaling guided by entropy is insufficient for selecting actions consistent with the policy. These findings thus confirm that the proposed method effectively leverages entropy to enable sample-efficient learning during online fine-tuning.
6.3 ANALYSIS OF NOISE INJECTION AND ENTROPY-GUIDED SAMPLING
In our proposed method, we incorporate two key components, namely noise injection during offline pre-training and entropy-guidee sampling during online fine-tuning, To clarify the contribution of each element, we design controlled experiments that reflect its intended role. In particular, we first examine the case without injected noise (w/o No ie),, where the procedure reduces to the same formulation as Equation 4 while still retaining entropy-guides sampling. We then consider the case without entropy guidance (w/o Guidance), in which the sampling process still produces action candidates, but the selection restricted to the one with the highest action-value, thereby excluding the entropy-based balancing mechanism.
The left plot of Figurr/ddemonstratee that both components are indispensable to the effectiveness of method. Noise injection, in particular, proves to be especially critical. This is because, without it, the offline pre-training is limited to actions contained in the offline dataset. Even when action candidates are generated, this restriction makes them lack diversity, which in turn leads insuf- ficient exploration and thereby reduces overall performance. In the absence of entropy guidance, the performance deteriorates later stages, as the training process fails maintain an appropriate balance between exploration and exploitation. These observations suggest that both noise injection and entropy-guide guided sampling play an important role in enabling sample-efficient learning in the offline-to to to-lne RL setting.
6.4 TRAINING AND INFERENCE EFFICIENCY
Since computational cost is also an important factor in methods employing generative models, we compare our algorithm with the baselines on humanoi dwwaze-ment um, where it achieves the largest performance improvement. The middle and right plots of Figure present the training time and inference time, respectively. The results show that although additional components such as en- tropy estimation and action candidate sampling slightly increase training time relative to the back- bone algorithm, this increase negligible when compared to algorithms such as Cal-QL, leaving overall training efficiency largely unaffected. Regarding inference time, method requires fewer samples than baselines such as IFQL, demonstrating that the additional computation does not impose significant overhead.

---

Published as a conference paper at ICLR 2026
6.5 EFFECT OF ACTION SAMPLE SIZE (NsAMPLE)
The proposed method injects noise into the flow matching objec- tive and samples action from a of action candidates to utilize the expanded policy. Since the size of the exploration space in- creases with the action dimension of the task, we set the hyper- parameter N ssmpll.. which determines the number of action can- didates, half of the action dimension. To examine the impact of this hyperparameter, we evaluate performance across 6 envi- ronments with action dimensions greater than 10 while varying Nample
100
Performance
95
80
1/8
(Ours)
As shown in Figure 7 performance generally improves as the number of action candidates increases, but the marginal gains diminish beyond a certain point. Since sample directly affects inference cost, adopt the choice of setting it half of the action dimension, achieving a practical balance between performance and computational overhead.
Moompe diml,A))
Figure 7: Comparison of perfor-
mance with varying Nample-
7 RELATED WORK
RL is a framework that learns through two stages: offline pre-training and online fine-tuning (Lee et al. 2022: Zhang al. 2023 Wang et al.] 2023a Nakamoto et al.] 2023: Zhou et al.) (022). The goal is to improve an agent, initially trained offline data, by allowing
Offine-to-OOninee Reinforcement Learning.
further refine through environment interaction. The simplest way to train within this framework is to employ the same offline RL algorithm for both offline pre-training and online fine-tuning (Lyu et al.) 2022 Wu al. 2022) Tarasov et al. 2023). However, this strategy inherits the conservative nature of offline RL methods, which restricts explo- ration during online fine-tuning (Yu Zhang 2023 Luo al. 2024 Zhang et al. 2024, Kim al,) 2025). Several prior studies have attempted to ease this conservatism and support more effective on- line fine-tuning (Wang et al. 2023a Nakamoto et al. 2023). Still, because these approaches remain conservative, they fail to provide sample efficiency through effective exploration (Shin et al. (025). In contrast, our method leverages the expressivity of generative models to encourage exploration and regulates it with entropy-guue -ugided sampling, thereby achieving high sample efficiency.
Reinforcement Learning with Generative Models. Recent advances in generative models such as denoising diffusion (Sohl-Dickstein al. 2015 Ho al. 2020) and flow matching (Lipman al.,
2023 Albergo & Vanden-Eijnden 2023) have spurred growing interest in applying these techniques
RL. Among these efforts, research that employs generative models policies has shown strong
results in both offline (Wang et al.] 2023b; Hansen-Estruch et al.) 2023 Kang et al.] 2023 Zhang
al. 2025 Kim al. 2024a by Fang al. [2025] Park al. 2025b] Chae | (0026 and online (Wag et al.) [0224] Psenka et al. 2024 Ding et al. (2024) settings by framing the policy as a state-
conditioned generative model. Within the offline-to-oninn RL framework, there have been studies
that exploit the expressive capacity of diffusion models for data augmentation (Liu et al., 2024 Huang et al.) (025). However, no prior work has leveraged such expressivity directly as policy in this setting. In contrast, our work harnesses the generative model for exploration, demonstrating method that achieves strong sample efficiency in offline-to-online RL.
CONCLUSION
In this work, we propose FINO, a novel approach that leverages the expressivity of flow matching through noise injection and enhances online fine-tuning via entropy-guided sampling. Noise injec- tion, applied to the offline pre- traitrainngg broadens the action space and yields a stronger initialization for exploration, while entropy a workable verse and challenging benchmarks while maintaining modest computational cost. Beyond empirical gains, our study highlights how flow matching can be effectively utilized to address the challenges of oflline-to-online RL, and we believe this line of work opens new directions for harnessing generative policies in advancing the broader offline--oo to-oline RL paradigm.
sampling adapts to the policy's 's evolving behavior to maintain balance. FINO achieves sample-efficient learning across di-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported in part by the Institute of Information & Communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. RS-2022- I1220469, Development of Core Technologies for Task-oriented Reinforcement Learning for Com- mercialization of Autonomous Drones, 50%) and part by the National Research Foundation of Generative
Korea (NRF) grant funded by the Korea government (MSIT) (No. Model Based Efficient Reinforcement Learning Algorithms for Multi-mode Expansion in General- ized Environments, 50%). We would like thank Woohyeon Byeon for providing valuable insights into Theorem
REFERENCES
Michael Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023.
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jongseong Chae, Jongeui Park, Yongjae Shin, Gyeongmin Kim, Seungyul Han, and Youngchul Sung. Flow actor-critic for offline reinforcement learning. The Fourteenth International Con- ference Learning Representations, 2026. URL https:/ owwnreview....e.... net forum?
id W.wwwwcwNN20
Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. Advances in Neural Information Processing Systems, 97:53945-53966 2024.
Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu. Online reward-weighted fine-tuning flow matching with wasserstein regularization. In The Thirteenth International Conference Learning Representations, 2025.
Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bingyi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming, pp. 1861-1870. Pmir, 2018.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint N0w:230-410573.. 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). 1rXiv:1606.08411 2016.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Xiao Huang, Xu Liu, Enze Zhang. Tong Yu, and Shuai Li. Offline-to-ollin reinforcement learning with classifier-free diffusion generation. arXiv preprint arXiv:2508.0668 06806, 2025.
11

---

Published as a conference paper at ICLR 2026
Marco F Huber, Tim Bailey, Hugh Durrant- Whyte, and Uwe D Hanebeck. On entropy approxima- tion for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp. 181-188. IEEE, 2008
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:67195- 67212, 2023.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. In International Conference on Learning Representations, 2024a.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Adaptive q-aid for conditional supervised learning in offline reinforcement learning. In The thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Jeonghye Kim, Yongjae Shin, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Youngchul Sung, Kanghoon Lee, and Woohyung Lim. Penalizing infeasible actions and reward scaling in reinforce- ment learning with offline data. In Forty-second International Conference on Machine Learning. 2025.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179-110 2020.
Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic g-ensemble. In Conference on Robot Learning, pp. 1702-1712. PMLR, 2022.
Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Zhi-Quan Luo, and Ruoyu Sun. Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow match- ing for generative modeling. In The Eleventh International Conference on Learning Representa- tions, 2023.
Hui Liu, KannShuolla Liu, Shengyi Jiang, Ruifeng Chen, Zhilong Zhang, Xinwei Chen, and Yang Yu. Energy-guided diffusion sampling for offline-toonnlnee reinforcement learning. In Proceed- ings of the 41st International Conference on Machine Learning, pp. 31541-31565, 2024.
Qin-Wer Wen Luo, Ming- Kun Xie, Yewen Wang. and Sheng-Jun Jun Huang. Optimistic critic reconstruc- tion and constrained fine-tuning for general offline-to-online rl. Advances in Neural Information 2024.
Processing Systems, 37:108167-108200
Jiafei Lyu. Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma. Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. Advances in Neural Information Processing Systems, 36:62244 62269, 2023.
Jongeui Park, Myungsik Cho, and Youngchul Sung. Empo: A clustering based on-policy algorithm for offline reinforcement learing. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In The Thirteenth International Conference on Learning Representa- fions, 2025a.
12

---

Published as a conference paper at ICLR 2026
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference Machine Learning (ICML), 2025b.
Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. In Proceedings of the 41st International Conference on Machine Learning, pp. 41 41166-41182, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Yongjae Shin, Jeonghye Kim, Whiyoung Jung, Sunghoon Hong, Deunsol Yoon. Youngsoo Jang, Seon-Hyeong Kim. Jongseong Chae, Youngchul Sung, Kanghoon Lee, and Woohyung Lim. Online pre-training for offline-to-online reinforcement learning. In Forty-second International Conference on Machine Learning, 2025.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, pmlr, 2015.
Dmitrii Sorokin, Maksim Nakhodnov, Andrey Kuznetsov, and Aibek Alanov. Imagerefl: Balancing quality and diversity in human-aligned diffusion models. arXiv preprint 2rXiv:2505.22569 2025.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 36:1 1592-11620, 2023.
Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2008
Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthicu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get family: State-adaptive balances for offline-to-onliee reinforce- ment learning. Advances in Neural Information Processing Systems, 36:47081-47104, 2023a.
Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 37:54183-5453 2024.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023b.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278-31291, 2022.
Zishun Yu and Xinhua Zhang. Actor-critic alignment for offline-to-online reinforcement learning. In International Conference on Machine Learning, pp. 40452-40474, PMLR, 2023.
Shuangfei Zhai, Ruixiang ZHANG, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Ángel Bautista, Navdeep Jaitly, and Joshua M Susskind. Normaliz- ing flows capable generative models. In Forty-second International Conference Machine Learning, 2025.
Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-toonlinn reinforce- ment learning. In The Eleventh International Conference on Learning Representations, 2023.
Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
13

---

Published as a conference paper at ICLR 2026
Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang. Yu Liu, and Wanli Ouyang. A perspective of q-value estimation on offline-to-onling reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16908-16916, 2024.
Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforce- ment learning fine-tuning need not retain offline data. In The Thirteenth International Conference on Learning Representations. 2024.
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
The entropy of the policy approximated using a Gaussian Mixture Model (GMM) (Huber et al,, 2008), which incurs computational overhead and remains an approximation rather than an exact cal- culation (Wang et al. 2024). Future work could focus on developing entropy estimation methods that are both computationally efficient and more precise, or on exploring alternative metries that capture policy behavior beyond entropy. In addition, while our method directly addresses the chal- lenges of exploration and the euploration-exploieee of distribution shift remains. Addressing this challenge constitutes another promising direction for future research.
trade-off in offline-to-onling RL, the issue
B THE USE OF LARGE LANGUAGE MODELS (LLMs)
Large Language Models are used solely to aid and polish the writing. They are involved in research ideation, methodological design, experimental analysis.
15

---

Published as a conference paper at ICLR 2026
C THEORETICAL PROOFS
Proposition 1. For notational simplicity, we denote x.), as x. Given a dataset 'D = {r,) i=
the proposed time deeeddependent noise injection et ~ N(0,, 1) induces the following conditional probability paths of flow dr:
pt FINO (z) (=)=:))-N( N
(((e)( = tr,, (1-
in which the mean tx; is equal to the mean induced from flow matching, and the variance (1 n)) is greater than or equal the variance induced from flow matching.
Proof. We consider time-dependent conditional probability paths of flow (x(x)) as follows:
(r(x),))) = / (1/)e (
(10)
Following the flow matching (Lipman et al. 2023), we set the time-dependent mean and variance as
M4 (z,)) tx
oe (x.))
(1 min )t,
where min is negligible small constant. The conditional probability paths provide the following canonical transformation for Gaussian distribution, i.e., the flow conditioned on dy²
(x) o1(x;) pe = (1 (1 T )t)r
miin)t) + tæ;
(1 t)x tæ.
By injecting the introduced dimendependent noise ct, we obtain the perturbed flow
(x(x) tx, + (1-t)z t) + E2.
where æ is distributed normal distribution No, I).
Since the injected noise designed as a Gaussian distribution (0, T² and is independent over distribution 10, the perturbed flow can be expressed the sum of two Gaussian distributions. This leads to the following conditional probability paths:
P.. (x(ox; N (xllar, ((1 1)2 T))))
N (xlta,, ((1- 101-t 1)2 (n² -27)t² +ntt)
= N (xlto,, (1- -1-11)1) n)))²²
as desired.
Theorem 1. Given a data x; from a dataset distribution and a noise Do from the base distribution, the conditional probability paths in Proposition Iinduce the unique conditional vector field that has the following form:
(a(a)))))) x,
1)
Then, for any dataset distribution, the marginal vector field ve (z) generates the marginal probability path pr(x), in other words, both ve(x)) and pe(x) satisfy the continuity equation.
Proof. The conditional probability path pe (e(xe(az)) = pr (e provides the canonical transfor- mation of Gaussian distribution the perturbed flow (1:) conditioned æ;:
3 (20)
to;; + (1 (1 7))
its derivative the vector field that generates the flow Φe by the definition of vector fields of contin- nous normalizing flow (Lipman et al. 2023).
d d d1 (l())) di
(la: + (1 - (1 -))))o)) - -1 1)
16

---

Published as a conference paper at ICLR 2026
which is the same result one from Theorem of (Lipman al. 2023)
o(((4) (x444 (It Me((rz)) + (2s),
(1 - -1)
(It - tx;) + æi
1 (1 7)}6
(1 7)0 + (1 - ta) + (1 (1 - x))))x
1 - (1 n)]
(1 (tz; + (1
n))t)20) + 24
1 - (1 n))t
(1 n)t))2 (1 7) (1 1))))0
(1 7)
42 (1 17)200
where f' denotes the derivative w.r. interpolation time 6, and oxxx)) = Ex;r;) and p<(z;) are the covariance and the mean of (e(g))z)) respectively. This means that the Gaussian probability path (e())(2)) induces the unique conditional vector field (x(((22))
7))
Since the conditional probability path (=)])),, has the form of Gaussian probability distribution, and the unique vector field ey (y (x,) generates the perturbed flow di (x) conditioned on 24, can use the definition of the marginal probability paths pe and the marginal vector field e (24) in flow matching (Lipman al. 2023):
(z( = f
PI(x;) so g(x;)
vx(x)= = (x)aa)))) pe(x)x;)/(1) dri,
p,(x;)
where q is dataset distribution, both marginal probability paths pt (xr) and vector field oe (xx)) satisfy the continuity equation (Villani et al. 2008) (refer to Theorem of Lipman et al. (2023)
for some N, and data are independently and identically distributed (i.id.)) sampled. The variance of the marginal probability path induced by FINO (Equation 7 is greater than equal that of the marginal probability path induced by the flow matching (FM) objective (Equation 3). For any time E [0, 1].
Theorem 2. Suppose the cardinality of the dataset is finite, D = 1,,,, .NN. NN)
Var (X, 2 Var (X(M),
XFINO ~ pr
FINO (2)) FM Pt (x).
Proof. For notational simplicity, given a data x,, let p. FINO (z) and p. FINO [z] (2)) be the marginal and conditional probability paths induced by equation and p. by equation 3
FM (x) and FM (x|æ;) be them induced
Since data are i.i.d. sampled, we can write the marginal probability paths as follows:
N FINO
/
Pt (x)
PINO
(x(
Pr
N
1=1
p PM (x(x) = / (MMaaar))(a
1
p PM
(x|z.)
N
To simplify notation, we denote random variables as X, and Xi that follow a marginal distribution (z(x) and (=] respectively, given data 2;. Assume the distributions (a and (x(x,) have identity covariances, then, for fixed t, the random variable X, from the marginal probability path pe(x:) has the variance as follows (the following equation can apply both FINO (2) and p (x)
17

---

Published as a conference paper at ICLR 2026
since their conditional probability paths are isotropic Gaussian distributions):
- pi(x))addr - (fpilt)zzu p)(())add
Var (X,) = Ep, [
Eyp. [x,]²
2d (1
(1
2
=
1 N Ex(cs) p((())
N Σ p((x)
xdx
2
Σ /
pi(z) 2 dx (+ /
pt p((l))
(N)
Σ
E pp [x
Σ Bag [x,]]
N
N
Σ NE [X
B [x,] [x,]
N
i= 3=1
Σ NE NEW [X - Σ 1x
ΣΣ
BBg (Xo [Xc]
N
MEP [X+] Σ NE N + Σ (N 1)
Σ Σ
(x
N2
3:
Σ \Var(X;)+ + 1) Σ Σ
Ep 18 [x,]
N2
Ji
Using the equation above, the variance of the marginal probability path p.
can be rewritten as FINO
Var Var(X
FINO
FINO No.
d Σ(N
1)44
F.FINO
ΣΣ
P
FINO FINO
(11)
N22
where d is the dimension of data Vt. given data v.. o, bility path pT By the same argument, the variance of the marginal probability path of FM P.
FINO is the variance of the conditional proba-
(x (z)), and p. FNNO the mean of the path.
N
Var(X PM
1
Σ
FM d + > 1) No.
8.FM
L.FM ΣΣ
(12)
N2
where d is the dimension of data vi. given data x;, of path p. (zz|2;). and y.C is the mean of the path.
I. the variance of the conditional probability
From Proposition we already have 14 tion from equation then we obtain
i.FINO
1.FM and of
FINO
2 o,
i.FM
by subtracting equa-
Var(X!INo) Var(X.M
1
Nd 0e
.INO
o. (.FFM) 2
N22
18

---

Published as a conference paper at ICLR 2026
D ANALYSIS OF NOISE INJECTION
we describe a training approach that injects noise into the flow matching objective, enabling the model to learn over broader action space. this section, discuss alternative noise injection strategies that can be applied to the flow matching objective and illustrate their effects through a toy example. Throughout this section, we assume the use of zero-mean noise c Pnoise (e.g., Gaussian Noise).
In Section 4.1
D.1 CASE I: INJECTING NOISE TO VELOCITY TARGET
Adding noise to the target velocity in Equation 4 is the simplest form of noise injection. The flow matching objective with the added noise can be written follows:
(0) E (0,,) [11v0(t,8,244) (x1 ro) =c1
Unin
enePaise
= E (1)vo 8, x4) (x1 - ro)ll2 2 (vo (t, 8, - (x1 xa)) ||<1|]]
Since the noise has zero mean (Re--eannic [c] = 0). the second term becomes zero in expectation. Furthermore, because is independent of 0, the last term || is a constant with respect to 0, so does not contribute to the gradient during optimization. As result, the total gradient of the objective identical to that of the original flow matching objective in Equation 4 which means that training proceeds in exactly the same way in expectation.
19

---

Published as a conference paper at ICLR 2026
D.2 CASE 2: INJECTING NOISE TO POLICY ACTION
A straightforward way to encourage exploration is to inject noise directly to the actions generated by the policy. However, as shown in Section 6..] this approach leads to limited improvement dur- ing online fine-tuning. This difference stems from the training process of the one-step policy, as described in Section 4.1 which is the component that directly interacts with the environment.
The one-step policy is trained with the following objective:
(w) = E
[- Qo(s, (s, + a (s, as (s.
(0, (8,2))mTT
distills the flow model trained from the offline dataset while simultaneously maximizing action- value through the value function. Since our algorithm employs the flow model trained with the expanded action space from Equation the resulting one-step policy learns to explore regions that are more informative for improving action-values In contrast, simply adding noise to the action ignores action-value information, making an inherently less efficient exploration strategy.
To illustrate this difference, we conduct comparative experiment in toy example. In this setting, the state is fixed, and the action is two-dimensional, corresponding to the x- and y- axes the figure. The dataset is sampled from Gaussian distribution centered at the origin, and the reward increases monotonically toward the right. We train two flow models using Equations and respectively, and each flow model is then used to train separate one-step policy via Equation 5 The baseline that uses the flow model trained with Equation and injects Gaussian noise directly into the action output referred to the Action Noise. The approach based the flow model trained with Equation
corresponds to our proposed method, which does not apply any modification to the action output.
Action Noise
Reward
Reward 1
Figure 8: Action samples of two one-step policies: blue contours indicate the log-density of the sampled actions, and red x marks denote the centers of the dataset.
Figure 8 shows that, since the reward increases toward the right, the action samples generated by both policies shift rightward relative to the dataset. While adding noise directly the actions causes the samples to spread in random directions, FINO guides the samples toward regions with higher action-values. This is because the method leverages both the expanded action space and the action- value maximization objective. As result, the one-step policy is guided toward more informed learning direction, leading to more effective exploration during online fine-tuning and ultimately explaining the superior performance of our approach.
20

---

Published as a conference paper at ICLR 2026
E IMPLEMENTATION DETAILS
We implement our proposed method, FINO, based on the official implementation of FOL In FINO, the entropy estimation module is adapted from the official DACER implementationf For all base- lines except Cal-QL, we adopt the components provided in the FQL implementation, while for Cal-QL we rely on its official implementation.
E.1 ENTROPY ESTIMATION
In our proposed algorithm, entropy-guided sampling requires estimate of policy entropy. How- ever, since the one-step policy training objective combines behavior cloning from flow policy with action-value maximization, the entropy cannot be derived in closed form. To address this, adopt an estimation strategy introduced in prior work (Wang et al. 2024).
To compute the policy entropy, we employ Gaussian Mixture Model (GMM) approximation of the action distribution. A GMM represents complex data distributions by combining multiple Gaussian components. Formally, the likelihood of sample under GMM is defined mixture of Gaussian densities:
K
p(a) MaNaaa
(13)
where K denotes the number of Gaussian components and Tu [0, 1] is the mixing coefficient that specifies the probability of selecting the k-th Gaussian, satisfying Σ Tr =
Training a GMM corresponds to estimating its parameters such that the likelihood of the given data maximized. To approximate the action distribution of the policy using a GMM, we first sample multiple actions (a (a ..)) from the policy for each state. We then fit the GMM to these samples using the Expectation-Maxite Maximizaionnn (EM) algorithm. The EM algorithm alternates between two iterative phases, namely the expectation step and the maximization step. In the expectation step, the latent probabilities required to compute the likelihood are estimated:
T&N pk, Ex)
(14)
Σ T:N(a> (p,, Ex)
where (=(²) denotes that under the current parameter estimates, the observed data a" come from the k-tth component of the probability. In the maximization step, the GMM parameters are updated based on these probabilities:
En 7(2)) az N Σn 7(2))
Le En (2)) ma)(a" 1a)
mTk
(=f),
(15)
By repeating these two steps until convergence, we obtain GMM that approximates the action distribution of the policy.
The entropy the fitted GMM is then computed following the approach established in prior work (Huber et al.) 2008):
K Tk (- log Th + 2 ( [{())
H R
(16)
k=1
where d denotes the dimensionality of the action space. The entropy estimate of the policy is ob- tained by averaging this quantity across a batch of sampled states. In practice, we set the number of mixture components to K = 3, which we found sufficient across all tasks. The number of states used for entropy estimation is determined by the batch size, and the number of actions sampled per state is fixed at 200, following prior work (Wang et al. 2024).
bbatss//gituubbcc/////oonnggggg hhtgs://gtthhh...mmmmaaamttoCooo
21

---

Published as a conference paper at ICLR 2026
F EXPERIMENTAL DETAILS
F.1 BENCHMARKS
We conduct experiments on 35 tasks from OGBench (Park al. 2025a) and 10 tasks from D4RL (Fu et al. 2020). For OGBench, we adopt single-task variants (s1 singgetaskk) provided in the benchmark and configure them to fit the offline-to-online RL framework. Each environment consists of five distinct tasks, each defined by different goal. In our experiments, use the following environments and datasets:
humanniddmaz---eddduun-nrri----
bumnnodnnazeelaeee-nvaggaatvv
antmaze- lataaz--lrge-navigate--
antmaze-gianl-nnaviaee---
antsoceer-arenn--avviaaat--
cube-double-play-vv)
puzzle-4x4pplay-v0
In humanoi dmaze, the objective is to control humanoid robot with 21 2i-dimensional action space to reach the designated goal. In antmaze and ant soccer, the agent controls quadrupedal robot with an 8-dimensional action space to navigate the goal; in antsoccer, the robot is required to move a ball to the goal. For cube and puzzle, the agent manipulates a robotic arm with a 5- dimensional action space. The cube task requires pick-and-place, while the puzzle task involves pressing the buttons to solve the puzzle.
For D4RL, evaluate the following environments and datasets:
antmaze-umaze-v2
antmaze-umaze-diivreee-22
antmaze- untaze-mediumpplay..
autmsz--mddum-diverse-ww
antmaze-large-play-22
antmaze-arge-diierse--22
pen-cloned-vl
door-cloned-vi
hammer-cloned-vi
relocate-cloned-vl
The ant amze tasks share the same 8-dimensional robot in OGBench but differ in their environ- ment and dataset settings. The adroit suite (pen, door, hammer, rel ocate) involves high-dimensional dexterous manipulation tasks, with action spaces exceeding 24 dimensions.
F.2 EVALUATION
We evaluate all baselines during online fine-tuning by reporting the average return over 50 episodes every 50,000 environment steps. For OGBench and the antamze tasks in D4RL., we follow the original evaluation protocol and use the success rate the performance metric, while for the adroit suite D4RL, we adopt the normalized score. All experiments are conducted with 10 random seeds.
22

---

Published as a conference paper at ICLR 2026
G HYPERPARAMETER SETTINGS
G.1 FINO
Since our method builds on FQL (Park et al. 2025b) the backbone algorithm, retain all hyper- parameters from FQL without modification. The hyperparameters introduced in our method, namely Og for noise injection and sample and H. for entropy-gun -upiyyg sampling, are configured depending on the environment settings. The entropy update interval (Nç) is aligned with the evaluation frequency and set 50,000 steps. complete list hyperparameters is provided in Table
Table 2: Hyperparameters
Hyperparameter
Value
Noise constant
0.05 [.4]
Candidate samples Norgy
0.5 -d)
Target estropy H
dim(.A)
Entropy update steps
50,000
Standard deviation of noise
expp 13)
Hyperparamcter (from PQL.)
Value
Learning
0,003
Optimizer
Adam Kingma & Ba 2015
Minibatch size
256
MLP dimensions
[512, $12.512 512]
Nonlinearity
GELU (Hendrycks Gimpel, 2016
Target network smoothing cocfficient
0.005
Discount factor
(default), 0,995 (antmaze-giant, humanolemaze, antsocees)
Flow sleps
Flow time sampling distribution
UniA(0, II)
Clipped double Q Q-learning
Falke (definlt), Thue (adroit, antmaze-large, giant-navigate)
BC cocfficient
Table
G.2
OTHER BASELINES
For the other baselines, we retain the hyperparameters used in FQI. (Park et al. 2025b). For Re- BRAC (Tarasov et al. 2023), we treat the actor be coefficient (a1) and the critic be coefficient (a2) tunable hyperparameters, while keeping all other settings their default values. For Cal-QL, the eql regularizer coefficient (ax) and the target action gap (3) are used as hyperparameters. Regarding the network size, set [256, 256, 256, 256] for manipulation tasks and [512, 512, 512, 512] for locomotion tasks of OGBench, with all other parameters kept at their default values. For RLPD, re-implementation of RLPD from the codebase of FQL and adopt the same configuration as FQL, including setting the update-to-data ratio to and using two value functions. For IFQL, the only hyperparameter is the number of action samples (N). For FQL., the behavior cloning coefficient (a) is the sole hyperparameter. The task-specific hyperparameters are summarized in Table 3
Table 3: Task specifci hyperparameters for each baseline.
Task
ReBRAC (a)
Cal-QL (a, 9)
(FQ
FQL (a)
munaanoidaazz-mmmcii----aaaveee mediimn-navigat---00
(0.01.0.01)
(5, (5.0.8)
32
100
humanaidmaze- wargeeee------aaaeeeee naviggatte---)
(0.01, 0.01)
(5, 0.8)
32
antmaze-lan mammazz-large-nne large-navigate--
(0.003, 0.01)
(5,
uutmaze-giant-nate glant-navigato-v0
(0.003,
(5.0.8)
32
antsoceer-arenal aeeeeerr-rnn--aaeeeeeeee arena-navigate--aa...a navigate-v0
(0.01.0.01)
(5.0.2)
64
30
cube-double play--pay--OO
(0.1.
(0.01. 1)
32
300
puzzle 4x4 play vD
0.01)
(0.003, 1)
32
1000
untmaze- Hamaazz-unazev22
(0.003, 0.002)
0.8)
32
antinaze umaze
(0.003, 0.001)
(5,)
antmare- medium play-v2
(0.001, 0.0005)
(5.0.8)
32
(0.001. 0)
(.,0.8)
32
10
antmazo-ange-plaay-22
(0.002, 0.001)
(5.
32
laimazz-lagge-diveee--22
(0.002, 0.002)
(..
32
pen-eloned-vl
(0.05. 0.5)
(1,088)
128
1000
door-cloned-v1
(0.01.0.1) 0.1)
(1.0.8)
128
cammee--loned--]
(0.1.0.5)
(1.0.)
128
1000
relocate relocate-cloned cloned-vi
(0.1,
(1.08)
128
10000
23

---

Published as a conference paper at ICLR 2026
Table 4: Full results for main experiments (corresponding to Table 1 and Fig. 3 Scores show offline pre-training - online fine-tuning, averaged over 10 seeds (mean + 95% CT). For OGBench, the s singletase suffix is omitted.
Fnvinonment
ReBRAC
Cal-ur
RL
IFQI.
FQL.
FINO
OCBRench humanoidhnuzemmeeleee saapp-----iimma-ppeee (gsto-nskk
14ga- lar
0gg
0ga-+0a4
11
17
OGBench bumanoidmaze medium navigate lask2
18:9- lat
0ze Dag
0:a Oze
92.4 83
89:10- 90
77 99
OGBench tumanodmazz meedium savigate task3
20 lai
0so-00a
Ose E43
38 74
56-2 8942
52:31-199.
OGBench bumanoddmaze medium savigate task!
17:11 lat
0zo-+020
Osa-+ 01e
0sc-+ 56ss
9:12 18
11:4- 94
OGBzzcc humancidmaze meddium-an savigate tasks
28 no 10
Ose 0se-10a 00g
-a 4
99s1-192a8
100 94
99s1-9 100
OCBeecch humanoidmaze cavigate- task1
Dso
0sa-s0a0
Oss O.e
Ose
4
5:4-0 S19
OCieench humannidnnuze-uuu weg.---------viitt---l-
Ogn 0gn
0gn
Ogr-4 Oge
0g
Ogr- 624
OGBancc tumanoidnaze large caviigate-taak task3
9s 250
Ose Oae
45 41
1756 36g
22 99
OCTRench humanoidhuuze hinuzeelrrr www.auon----go---iiiioooeee
0gn
OGBBench humanoddnaze waape--argg--giiiiin aage--aavigtte-msk
1ai -o--0aaa Osl
-0g
Ogt Ogg
Rgi -3 533
1al Sg7
Oal -3 8g.s
OCBench animace- harge navigste-sakk
94 100
20:2- 30
Ose- 93:s
32am 66:15
82:3 98a1
82.x 98:1
OCBench animure-lirge-nn sammmo--lrge-nvvigaee 2
8R M8
02e--027 02g
Out-4 54
17 70s1
63g 71
67 97.1
OGBench unimare large navigate tank?
65:12 - 10010
20:2- - 10:20
0: 2911
57ss + 88.14
94:2 100
92 100..
OCBench mmimsce-larp-naviin nage--aggit--sdd
86gs
13 75
9621
OGBench animaze lurg navigate task5
90 100
20:2x
20:2
0za-+ 09:35
39 59.13
85 95:2
85 99.11
OCBench animace giant navigate task
53 97
0:a-0020 -20
0ss-4 7a32
Oss-+ Oss
8s5 65:24
3:4-7 96:1
OCBencc antmass-gizat-nn ginat-navigste-
25
Das 48.34
17au 96
021 99
OGBench unimaze glant navigate task?
24 16
0:0-000
0:2-+44a 44:
0ss-+0
0si 201
010 0.0
OGBeauch antmes--iiatt nnviigtte-ian
0se-1
59
10 962
OCBench animare-giint -avi--tt---ssssss 1
61amt 99
10.0--0a3 0gn
Ogr 80
4g1-+0a5
41gm 99
M6 99.1
OCiBench peaeroor--rnnattvviiiii 1
1gi - Dan
0an 0g)
02e-4621
69g -4 642
R2gt 91
17 93,7
OGBench aanoocco--rren-tavviiaa task?
0go
0.e-10a
70g 00-22
88 97
84: 98.1
OGBench unisoccer--renee senneerr-neeettvviiiiee 3
0gn-+
6at - 26:s
60g 88
56 91a111117
OGBench anisoceer savigate taski
lsi Oso
0:0-020
0:s-+0:1 0.s
20:5-+ 17at
32:4- 70
34x 70..
OGBench unisucece-uneeet www.-nunvi-----eeeeeeeee task5
02m-0
Oza-+0ar Oze
lat-+443
43 22
32a1- 33.1
OCBencc nube--ouble-plyy-ee
27 100g0
02g
Ozr- 1ass
31 89
59 17
62.1 98.1
OGBench cube double play task2
8:5-
0:0-+020
0:a-+0:1
13s2-+2::4
8617
40:1-190 90:1
OGBeznch cuhe-doalte-play-- -lay-
Sgg 1657
0.e-10a 02g
Oga-a Oae
643-13157
29 89
35 90.4
OGBench cube double play taski
Dai
0:0-0a0 0za
0sa-+011
111-+0is
4112
11:4-> 21
OCBench cuhe--Sullle-plly-aas
Sgo 3g0
0ze 02a
Daa 0as
20 88
23 96-1
OGBench 4ozzlo-444-gaay-o <ask1
25 100
10 - 70
Oat 100
M8ar -3 100
32.0 100
33 -3 100
OCBench puzilo-444-plan 4u44-lay--asse
OGBeench suzzlo-4ck-play-eee -askk
16g -a 42g/
Oge - 30g
Og 80
18 70gl
22 100g
OCBench purdle-k4-gjle ply--ll--aaaeee 4ask4
Rez 3gs
020
Oct 40
20.4+ 19a77
T S3ar
20 80
OCBBench suzzlo-4x--gla--ask
9g0 0gn
02n
Oaa 20
Agg
8--- Oge
DARL zntmaze umaze v2
90 100
81:3-99911
0: 100:1
94:2-+9::
98 99
96 100::
D4RL. matmav-omave-dee diverse-12
75
36am-
71att+
78
DARL mimaae medium play x2
B 91:11
0ss- 97
S4 19
78: 94
19:1-97111
D4RL antmaze-man medium- diverse-22
1as- 98
61 95
Oct 98
41 86.1
66 95
60:n 97.1
DARL anmaae large play
42
35:5-4 7416
6411-478.1
33a-- 95
75 95
D4RI. anmon--lngg-divvveee
77 93
29 - 80gg
Oga 83
Tha -344)
81gn 1lnn-912
82.1 -3 97
DARL. pen-clonad-vl
77ai 129g
-Ig--
140g)
DAKL cooncdon
0so 8316
0ia-10a
Osa 101
141-+2345
0ie-100 101a
01c-> 194
DARI, hammer-connd-vl
Sga 119ge
020
Ogt 0ca-19920 99.
1gi-344g
0se- 110gs
Dar-3 134g)
DARL relccat-skkee--
02n-+0an 0gn
021+021
Oas
0at-+2a1
1a--44as
lat 72st
24

---

Published as a conference paper at ICLR 2026
Table 5: Full results for ablation studies (Fig. fine-tuning, averaged over 10 seeds (mean 95% CT). omitted.
Fig. 6
Scores show offline pre-training - online For OGBench, the singletask suffix is
Environment
FINO
Direct Noise
w/o Noise
w/o Guidance
OGBench humansiddmaze medium nuvigate task
13 9117
141:-+0. 011
1815 1
1615 50
OGBench humanoidmaze medeo--eeem-------- navigate-takk
77+23 - 99+1
88-16
96
80+3 + 99-1
71+2 - 84++2
OGBench humanoidmazo-mng mat-s--eddd----- tavi
52 99
54:1
60
45 49
OGBench humanoidmaze-nne dicuuaa-eeeee-----e-e-- inatteaagttt-
11+- 94+3
7=10+ 31=24
2+4 - 20=2s
0#o 74eis
OGBench humanoidmaze-- medium-navae nari--aaaiiittteteeee task5
99 100
99 100
9911 -+ 99
9821 -+ 99
OGBench humanoidmaze-lkie lagaaa---r----iiiiiii
5e4- See
4s4 Des
364 8:144
4+4 0=1
OGBench humanoidmazz-lan wawpre-aar----igitt----oo
020- 626
02e-
02t
OGBench humansidmaze large navigate task3
22 19
1916 41
18.7 9951
25
OGBench humaniidnaze-lrrge nnvigate task4
0gn-> 48
2 8
01a 5u4
0gn 18,27
OGBench humanoidmazee large navigate task5
112
OGBench antmaze-large navigate-tuskl
82a6 98.'
82g 98
68 98
7845 35
OGBench animaze large navigate tusk?
62:6 9711
591: - 69.x
6211 1411-994.3
75:6 21.1
OGBench catinaze-larg--ia hagge--avvigatt-aaaaaa. -igate-
92+3 100
96=1 99
95+2 100
95+3 99
OGBench antmaze- large navigute- lusk4
72-16 96
-0::-+98- 98-1
81+4 98
OGBench antmaze-large large--tgeet harge-aavigan naviggat---ask
85es 99*1
83 95
834) 98=1
8245 83
OGBench animaze giant navigate taskl
3:4 > 96
9 64::2
9::0 85:13
4:5-- 79
OGBench antmaaz-gginnt-ee sia-a--ggin---viitt---eee
0=1 - 99*1
18=1
96=1
1+2-
1+2 99
OGBench animaze- weree--gnnnnnnnnnneeeee
0.0
010
0gn 7
OGBench antmaze-giant giant-nnvigate-tase
29 99
10=12 85
547-4
28 97
OGBench antmaze giant-nan navigate task5
36.r 99.1
43 98
20,111 (:111->99..
31 99
OGBench aotsocer-arcaa-nnvv navigate-task!
77 93#2
81 93
69s7-490 94
OGBench anisoccer-an navigaii task2
84 98.1
9011 97.3
8314 97
8311 93
OGBench antsoccer arena navigate task3
56.5 9112
58 87:4
54 :1
57:4-4 78:4
OGBench anisoceer arena navigate task4
34.6 70.l
3311 71.x
4311 3111->62.1 62.
OGBench antsoccer-ae aeceeer--eeaaaee navigate- task5
32+1 33+22
43+1-+ 14-n
3348
OGBench cube- double play- task!
62.5 98.1
64.. 97
74 96.1
OOBench cube-doute cube-double-pla saale--pay--siiee task2
40 90++
40 + 86:3
60+7 80=>
61 89
OGBench heae--oobb--pll--eeee
35 90+4
26a5 + 88
5745 1 88=s
5247 96-2
OGBench cabe-double-plee sauubl--play--seeee task1
Se1- 3-1
-++1+2+1
OGBench cabe-double-plee saee-play-ttttete task5
23 9623
21 88
43 862s
2629 91
OBBench puzzle-4xf tal--paa--tskk
33 - 100
31 + 100
56+7 100
61+7 100
OGBench puzzle-4xt-pp play-ay--tkkkk
11 02e
12 + 0e6
1545 Ose
10 0=0
OGBench puzzle 4x4 4xy--lay-ttanea t
22 100
2013 80
531 88
59 97.14
OGBench 4uzzle-4x4-pp pozzee4x4-play-lee play-task4
20 801M
9g1 61
18
1926 10
OGBench puzzle 4x4 play-tnsk5 task5
812 0.e
713 1-3--00 0.o
613 1a1-+011 010
914-+0.m 0.1
D4RL antmazze-mazz-v2
98 100
9622 99
98 -4 99
97 99
D4RL animare divers
78. 9911
851s
97
8517
8216 100_0
DIRL cacittaa--mmiii-llly-----.
79:5 - 97+1
74 93-5
79:4- 96+2
D4RL unimaze medium divium-
60an + 97st
62sx-199 95
61au-197::
D4RL aattmaa--aare--llay--2
72-19- 90-5
67+2
73+1-
D4RL, hoamanze-lrgg-diieree-e-
8224 97
83g9-> 90
722/6-1 94
8123 94
D4RL pen-cloned-vl
51 - 140
$7 137-4
605s+13354
60ss 135=4
57 136-4
D4RL door-cloned-vl
02n 104
0ge- 102
0.e 102
020- 101
D4RL hammcc--loned-v]
0 134s2
120
0=o- 112=14
D4RI, Slacate-cloned--v
1go 72
19
119-161
121-
25

---

Published as a conference paper at ICLR 2026
humancidmaze med-emm-mmuuuiaaatt
Performance
humanoldmazze-lare lar.rmmmaa--arrr--iiittttt task3
task4
task5
100
Perforance
antmaae
tack2
task2
task3
task4
task5
100
Perfomence
antmanz--gaant catrmare-gian-nnvvigga task3
caski
tack2
task4
task5
100
Perfomance
antsoccer-arom novigaaee task3
ta5k1
ta0k2
tock4
t06k5
100
Perfrmance
50
cube-double play
task4
100
Performance
50
9-9-9-9-----0
puzzle-xxd-po
Retomanca
Steps (x100k)
Steps (x100k)
Steps (x100k)
Steps (x100k)
Steps (x100k)
Figure 9: Full results OGBench environments. Each row corresponds to environment, with
five single-task variants shown side by side. Shaded areas denote 95% confidence intervals over 10 seeds.
26

---

Published as a conference paper at ICLR 2026
REBRAC
CalQL
RUD
POL
FIND
diversev2
medium v2
large play
Iarge-diersew22
pat-clonad-vl
doce-lonedavl
hammer-coredal
rwora/e-clonedel
100
150
Performance
EDG
50
50
0-
Steps (x300R)
Steps (x100k)
Staps (x1001)
Steps (x100k)
Steps (x100k)
Figure 10: Full results on D4RL environments. For AntMaze tasks, the prefix "antmaze-" is omitted for clarity. Shaded areas denote 95% confidence intervals over 10 seeds.
27


Reviews:
[{"review_id": "u4MuU2vs1A", "reviewer": "Reviewer_czZj", "summary": "This paper presents Flow Matching with Injected Noise for Offline-to-Online RL (FINO) that leverages flow matching policy for offline-to-online RL. It injects noise into policy training to facilitate exploration, which encourages action diversity. It balances exploration and exploitation by choosing the softmax of value functions over the action candidates. FINO is evaluated on a wide range of environments.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. FINO explicitly integrates exploration into the learning process and balances between exploration and exploitation. The empirical result demonstrates superior performance over a wide range of tasks.\n2. The paper is overall well-written and clear.", "weaknesses": "1. Does the noise injection step damage the performance in the offline learning stage?\n2. Does FINO reduce policy randomness during evaluation?\n3. The benefit of injecting noise during training (Eq. 7) rather than directly adding Gaussian noise to the output action is unclear. \n4. The role of entropy guidance in improving performance is also not well explained. Does it enhance policy learning, or does it serve as a technique to improve test-time behavior?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3xX1VAWqnp", "reviewer": "Reviewer_HJNR", "summary": "This paper proposes FINO, a novel approach for offline-online RL. Centered on a Flow Matching generative model, it addresses the insufficient exploration capability and exploration-exploitation imbalance of existing methods through two key designs: in the offline pre-training phase, time-dependent Gaussian noise is injected to expand the coverage of the action space; in the online fine-tuning phase, an entropy-guided sampling mechanism is introduced, which dynamically adjusts temperature parameters based on policy entropy to balance exploration and exploitation. Across 45 complex scenarios from OGBench and D4RL, FINO achieves significantly better performance than baseline methods.", "questions": "1. Why $N_{sample}$ relates to the dimension of action space?\n2. Do the baseline methods employ candidate action sampling? And would candidate action sampling significantly improve the performance of these baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper exhibits a clear logical flow and high readability. The proposed algorithm is simple yet effective, supported by solid theoretical foundations and validated by striking experimental results.", "weaknesses": "Injecting noise helps enhance exploration capability, which is beneficial for the offline-to-online transition. However, why does this not compromise offline performance? After all, offline RL is inherently conservative and discourages exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfKbNh43fg", "reviewer": "Reviewer_QgFC", "summary": "This paper introduces a new approach for offline-to-online RL via injecting noise into the flow matching objective for improved exploration.", "questions": "What does the y-axis scale on the right-hand side of Figure 2 represent? Also see questions above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "To my knowledge, this is the first paper to introduce the idea of noise injection into flow matching for offline-to-online RL, addressing a key limitation of prior work in offline-to-online RL. The empirical results demonstrate strong performance over the existing baselines (Tables 1 and 4), and attempts to justify the benefit of their method through ablations, though I provide comments about the ablations below.", "weaknesses": "The weaknesses I identify can be grouped broadly into three categories: exposition/justification of FINO’s objective, baselines and ablations, and smaller questions. \n\nI believe greater explanation and exposition of the noise injection for flow matching objective is needed. For example, why is this objective better than simply adding noise to the velocity target, $x_1 - x_0$? Additionally, is $\\eta$ fixed throughout training? \n\nDo any of the baselines considered employ a similar policy extraction scheme, specifically a Q-value weighted softmax? The authors argue that directly injecting noise into action selection is less effective than FINO (Figure 4), but the direct noise injection seemed to happen only at inference time. \n\nA relevant baseline may be adding noise to the offline dataset’s actions during training, in addition to at inference-time in the online stage. \n\nThe authors do not clearly define what type of noise is added in the Direct Noise ablation (Figure 4), only referring to it as “random perturbations.” I would suggest the authors investigate adding Gaussian noise to the actions, with varying standard deviations, similar to what they do for FINO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)