Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data
Authors: Lingkai Kong
Haichuan Wang
Tonghan Wang
GUOJUN XIONG
Milind Tambe
Venue: NeurIPS 2025 spotlight ()
Keywords: Reinforcement Learning
Flow Matching
Dynamics Shift
Primary area: reinforcement_learning
TL;DR: We propose a flow matching method for reinforcement learning with shifted-dynamics data.

Abstract: Incorporating pre-collected offline data can substantially improve the sample efficiency of reinforcement learning (RL), but its benefits can break down when the transition dynamics in the offline dataset differ from those encountered online. Existing approaches typically mitigate this issue by penalizing or filtering offline transitions in regions with large dynamics gap. However, their dynamics-gap estimators often rely on KL divergence or mutual information, which can be ill-defined when offline and online dynamics have mismatched support. To address this challenge, we propose CompFlow, a principled framework built on the theoretical connection between flow matching and optimal transport. Specifically, we model the online dynamics as a conditional flow built upon the output distribution of a pretrained offline flow, rather than learning it directly from a Gaussian prior. This composite structure provides two advantages: (1) improved generalization when learning online dynamics under limited interaction data, and (2) a well-defined and stable estimate of the dynamics gap via the Wasserstein distance between offline and online transitions. Building on this dynamics-gap estimator, we further develop an optimistic active data collection strategy that prioritizes exploration in high-gap regions, and show theoretically that it reduces the performance gap to the optimal policy. Empirically, CompFlow consistently outperforms strong baselines across a range of RL benchmarks with shifted-dynamics data.

Composite Flow Matching for Reinforcement Learning with Shifted-Dynamies Data
Lingkai Kong" Haichuan Wang"
Tonghan Wang" Guojun Xiong Milind Tambe
School of Engineering and Applied Sciences Harvard University
Abstract
Incorporating pre-collected offline data can substantially improve the sample effi- ciency of reinforcement learning (RL), but its benefits can break down when the transition dynamics the offline dataset differ from those encountered online. Existing approaches typically mitigate this issue by penalizing filtering offline transitions in regions with large dynamics gap. However, their dynamics-gar esti- mators often rely on KL divergence mutual information, which can be ill ill-defind when offline and online dynamics have mismatched support. To address this chal- lenge, we propose COMPFL.Ww principled framework built the theoretical connection between flow matching and optimal transport. Specifically, we model the online dynamics conditional flow built upon the output distribution of pretrained offline flow, rather than learning it directly from a Gaussian prior. This composite structure provides two advantages: (I) improved generalization when learning online dynamics under limited interaction data, and (2) a well-defined and stable estimate of the dynamics gap via the Wasserstein distance between offline and online transitions. Building on this dynamics gap estimator, we further develop an optimistic active data collection strategy that prioritizes exploration in high-gat regions, and show theoretically that it reduces the performance gap to the optimal policy. Empirically, COMPFI.OW consistently outperforms strong baselines across range of RI. benchmarks with shifted-dynamics data.
Introduction
Reinforcement Learning (RL) has demonstrated remarkable performance in complex sequential decision-making tasks such as playing Go and Atari games (53 (55), supported by access to large amounts of online interactions with the environment. However, in many real-world domains such as robotics [26] [60], healthcare [69]. and wildlife conservation [29] 30 66 67]. access to such interac- tions is often prohibitively expensive, unsafe, infeasible. The limited availability of interactions presents major challenge for learning effective and reliable policies. To address this challenge and improve sample efficiency during online training, promising strategy is to incorporate a pre-collected offline dataset generated by previous policy [4 [9]. This approach enables the agent to learn from a broader set of experiences, which can help accelerate learning and improve performance (59]
critical challenge online RI. with offline data arises when the transition dynamics in the offline dataset differ from those in the online environment where the agent actively interacts and learns [49] This issue, commonly referred to as shifted dynamics, can create severe distribution mismatch, bias policy updates, destabilize learning, and ultimately degrade performance. For instance, in robotics, the transition dynamics specify how the robot's 's state (c.g., position and velocity) evolves after executing an action. During deployment, changes in physical parameters such surface friction can alter this state evolution, causing the true next-state distribution to deviate from that implied by the historical data. Similarly, in conservation planning, the transition dynamics describe how the spatial risk of
* Equal contribution. Corresponding author: lingkaikong@g harvard.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

poaching evolves in response to patrol actions. Data collected in one region may not transfer to another because differences in terrain, accessibilityy and human activity patterns lead to different state transitions and behavioral responses under the same patrol strategy [66
To address these challenges, existing methods either penalize the rewards value estimates of offline transitions with high dynamics gap [37 46). or filter out such transitions entirely [64]. However, these approaches face key limitations. Most notably, the estimation of the dynamies gap typically relies on KL divergence or mutual information, both of which can be ill-defined when the offline and online transition dynamics have different supports
In this paper, we propose COMPFLOW, a new method for RL with shifted-dynamite data that leverages the theoretical connection between flow matching and optimal transport. We model the transition dynamics of the online environ- ment using a composite flow architecture, where the online flow is defined on top of the output distribution of a learned offline flow rather than being initialized from a Gaussian prior. This design enables a principled estimation of the dynamics gap using the Wasserstein distance be- tween the offline and online transition dynamics. On the theoretical side, we show that the com- posite flow formulation reduces generalization error compared standard flow matching by reusing structural knowledge embedded in the offline data, particularly when online interactions are limited.
Direct Flow
Composite Flow
Damamics Gap WWosorraain
Onióne (rg)
Officee (ec)
Latent kL
Latent
Figure 1: Comparison between direct and compos- ite flow matching. Composite flow first transports from a Gaussian latent variable to the offline tran- sition distribution, then adapts to the online distri- bution via optimal transport flow matching.
Building on the dynamics gap estimated by composite flow matching using the Wasserstein distance, we go beyond selectively merging offlin transitions with low dynamics gap and further propose an active data collection strategy that targets regions in the online environment where the dynamics gap relative to the offline data is high. Such regions are often underrepresented in the replay buffer due to the dominance of low-gap samples. Our theoretical analysis shows that targeted exploration in these high-gap regions can further close the performance gap with respect to the optimal policy.
Our contributions summarized follows: (1) We introduce composite flow model that estimates the dynamics gap by computing the Wasserstein distance between conditional transition distributions. We provide theoretical analysis showing that this approach achieves lower generalization error compared to learning the online dynamics from scratch. (2) Leveraging this principled estimation, we propose new data collection strategy that encourages the policy actively explore regions with high dynamics gap. We also provide a theoretical analysis of its performance benefits. (3) We empirically validate our method on various RL benchmarks with shifted dynamics and demonstrate that COMPFLOW outperforms or matches state-of- of-the-art baselines across these tasks.
2 Problem Statement and Background
2.1 Problem Definition
We consider two infinite-horizon MDPs: Moir == (S, A. Pur,r, r, and Mon (S,A, Pon: r; 7) sharing the same state/action spaces, reward function r : S x A -,, and discount factor y E (0, 1), but differing in transition dynamics:
Por(s a) # Pon(s'1ss,) (s,a) for some (s,
We assume rewards are bounded, i.e.. |r(s, a)l I Tmax for all s, a. For any policy n, let (sts at) be the trajectory generated by M and т. We define the discounted state-action visitation (occupancy) measure as pT (s, a) = (1 - -7) E 7 1{ = 8, a, = a}}, and the discounted state visitation as (s) := E. (s, a). The expected return is (T)) :- (s,a)>>p7, [r(s,))].
Definition 2.1 (Online Policy Learning with Shifted-Dymnn Dynamics Offline Data), Given an offline dataset D.N = {(s,, a, s/, r2) from Mai and limited online access to Man: the objective is learn a policy π maximizing 1. (T)), ideally approaching MM (MT"), (TT*), where n² == arg max, MMa. (T).

---

Lemma 2.2 (Return Bound between Two Environments (37). Let the empirical behavior policy in Doft be TDa (a 8). Define C₁
2rmaz Then for any policy n,
(1-7)2.
MMa (T) 'MMur (TT) 2 201 E
Poll ppop (DTV(T(()') TP (+s'))]
(x.a)rPM
- -C I Dot Drv (PolPPoff (Pou (18,, a) || (-1s a))]
(s.a)~PM
This bound highlights two key sources of return gap between domains: (1) the mismatch between the learned policy and the behavior policy in the offline dataset, and (2) the shift in environment
dynamics. The former can be mitigated through behavior cloning [37 65 while the latter can be addressed by filtering out source transitions with large dynamics gaps (39 64]. A central challenge lies accurately estimating this gap. Existing methods often rely on KL divergence or mutual
information 37 46 which can be ill-defined when the two dynamics have different supports.
2.2
Flow Matching
In this paper, we will adopt flow matching [ 36 to model the transition dynamics, owing to its ability capture complex distributions. Flow Matching (FM) offers simpler alternative to denoising diffusion models [21 177. which are typically formulated using stochastic differential equations (SDEs). In contrast, FM is based on deterministic ordinary differential equations (ODEs), providing advantages such faster inference, and often improved sample quality.
The goal of FM is to learn time-dependent velocity field vo(2, t): R x [0, 1| + Rd, parameterized by 0, which defines a flow map the (2o; (). This map is the solution to the ODE
d (zo t) va (zo. t), t), (e 0) 2o. dt
and transports samples from simple source distribution Po (z (e.g., an isotropic Gaussian) at time
t = O to a target distribution pI (x) at time = In practice, generating a sample from the target distribution involves drawing Zo ~ (a() and integrating the learned ODE to obtain Iι (zo. 1).
commonly used training objective in flow matching is the linear path matching loss
(())= = Eu-M((o, (o -1m [[ t) (x1 - m0))1
(1)
where I, = (1 t)æg + tæ, denotes the linear interpolation between Hg and r₁. Using linear interpolation paths encourages the learned flow to follow nearly straight-line trajectories, which reduces discretization error and improves the computational efficiency of ODE solvers during sampling 36).
2.3 Optimal Transport Flow Matching and Wasserstein Distance
Optimal Transport Flow Matching (OT-FM) establishes direct connection between flow-based modeling and Optimal Transport (OT), providing principled framework for quantifying the discrep- ancy between two distributions. This connection is especially valuable in our setting, where key challenge to measure the distance between two conditional transition distributions.
Let : Rd x Rdd + IR be cost function. Optimal transport aims to find a coupling g E I1(po, (1)0,P1)-o joint distribution with marginals Po and pi-that minimizes the expected transport cost:
/ c(zo, x1) dg(xo,
inf pEII(po.P:).
This minimum defines the Wasserstein distance W (Po. P1) between the two distributions under the cost function When c( = ||zo x1| 2| the resulting distance is known as the squared Wasserstein distance.
In the training objective of Eq. 1 when sample pairs (ro, #1) are drawn from the optimal coupling q², the flow model trained with the linear path matching loss learns a vector field that approximates the optimal transport plan. The transport cost of the learned flow approximates the Wasserstein distance
Eizo--p (1oo (No. 1) xall2
W3 (Po. p1).
For theoretical justification, see Theorem 4.2 in
3

---

Composite Flow Dynamics Gap Estimator Á(s, a) (2 a), Pon a))
off
Data Filtering Á a) S
8is.
I
Dynamics Shift
Latent X
Offline (xa)
Online (sa)
Exploration Bonus A (s,
Don
Replay Buffer
Behavioral Cloning
Interact
Online Environment
Agent
Figure 2: Overall Framework of COMPFLOW. To estimate the dynamics gap, we propose composite flow matching, which computes the Wasserstein distance between offline and online transition dynamics. Guided by the estimated dynamics gap, augment policy training with offline transitions that exhibit low discrepancy from the online dynamies, and incorporate a behavior cloning objective to stabilize learning. To enhance data diversity and facilitate adaptation, we further encourage exploration in regions with high dynamics gap.
In practice, the optimal coupling æ¹ is approximated using mini-batches by solving discrete OT problem between empirical samples from Po and P1- The use of mini-batch OT has also been shown implicitly regularize the transport plan [13] [4], the stochasticity from independently sampled batches induces behavior similar to entropic regularization
9 Further details of the OT-FM training
procedure are provided in Appendix C
Proposed Method
In this section, we introduce our method, COMPFL.OW, We begin by presenting a composite flow matching approach to estimate the dynamics gap via Wasserstein distance. Next, we propose data collection strategy that actively explores high dynamics-gap regions and provide theoretical analysis of its benefits. Finally, we describe the practical implementation details our method.
3.1 Estimating Dynamics Gap via Composite Flow
To estimate the gap between dynamics, we first learn the transition models for both the offline dataset and the online environment. Flow matching provides a flexible framework for modeling complex transition dynamics; however, the limited number of samples available in the online environment presents a significant challenge. Training separate flow models for each environment can result poor generalization the online environment, as the model may overfit to the small amount of available data.
To mitigate this, we propose composite flow formulation. Instead of learning the online transition model Poa(8'1s a) from scratch, we leverage structural knowledge from well-trained offline model Potr(s') a). This enables incorporate prior knowledge to improve the generalization.
Offline flow, We begin by learning a conditional flow model for the offline data. Let zo (O, I) be the initial latent variable. The offline flow map whell (zo, t|s, a) is defined as the solution to the following ODE:
dt Gi (Io, 118, a) rggit (1/2 Loff (wo, 4ss, a), 4),4, a)).
(2o. O|s, a) = Zo-
Solving this ODE from t 0 to 1 produces intermediate representation: 21 (/ 1 a).
Online flow. Instead of learning online flow directly from a Gaussian prior, we initialize it from the intermediate representation 2 produced by the offline flow. The target flow map ui (z, t|s, a) is defined as:
d (x, tlss a) e (z, tts. (2 ([s, =x
on (x,
dt

---

Solving this ODE from t = to 2 yields the final prediction for the online environment: s' 22 4/ (I1, 2|s, a).
We now show that when the offline flow induces a distribution (s'l,s' a) that closer to Pon (s'1s, ss a) than the standard Gaussian distribution, the proposed composite flow method enjoys a smaller generalization error bound.
Theorem 3.1 (Conditions for Composite Flow Yielding Smaller Errors). Assume that composite flow and direct flow share the same hypothesis class H of (measurable) vector fields v : R x [0. 1] - Rd. and that there exists B E (0, oo) such that suPvEM SuPze3<, ve(o,1; [|o[ 0) S B. Also, assume that max Tr(Eon), Tr(Eott) < CTR.JO some CTR. The composite flow enjoys a strictly tighter sigh-probbbiiitty generalization bound than the direct flow if and only if
W2(Pa, > W2lPoni.Poi
Here, PG := (0, La). Mon the covariance of Pon and Eost is the covariance of Poft Remark 3.2. In our setting, we assume that the offline data and the online environment share meaningful similarities. Given the abundance of offline data available to accurately learn the offline flow, this assumption is expected to hold well.
To compute the Wasserstein distance between the offline and online transition dynamics, we can use the OT-FM objective to train the online flow, initialized from the offline flow distribution. However, when the state-action pair (s, a) lies in continuous space, it is not feasible to obtain batch of samples corresponding to a fixed (s, a) from the online environment.
To address this issue, we follow prior works [8 23 and incorporate the conditioning variables directly into the transport cost. Specifically, consider pairs of transition samples (Saff, Goff <ax) and s. We define the transport cost as
(Son, Gon.
c((Sutt, Ooff sot) (Son; Gon: snn)) || s'nn 17 (1]soit Son 11 + |laoff - aon KonlB).
where 1) > 0 controls the strength of alignment between conditioning variables.
Let 9" denote the optimal coupling between the empirical distributions of (soff, aoft stf)) and (Sons Gon: ) under the above cost. The online flow is trained using the objective
Lon (Ø) Bi((eat (01))(Pan log (01211 Ila (at, t,, Sun; aan) (8on -
S'a )|?)
where I, is the linear interpolation between 1 and x2.
Proposition 3.3 (Informal: shared- conditionni coupling is W2-optimal). Assume that, when training the online flow in Eq. ①.. the marginals over the conditioning variables are matched, i.e., Pott (s, a) = Pou (s, a). Then, for any fixed (s, a) E S x A, as 1 - сo and the minibatch size used to compute the Of coupling tends to infinity, the expected squared displacement induced by the composite flow recovers the squared 2- Wassserstein distance between the offline and online transition distributions:
E Ezo-)(o0))
[1vg (zo, 11s,a) v.8 (v:8 (ro, 1|s,a) a). 2/s, a)
-
W3 (pon (-|s, a),Pon (-au((ss,00 a)).
Monte Carlo Estimator. This result yields practical Monte Carlo estimator of the dynamics gap Ass,a::
1/2
ā(s,
v (x) 11s,a) - 6000 (vg (a (=()), 1/s, a), 21s, 20, 11,,
l|s,0),,
20 (
NO, I).
M
Remark 3.4 (Enforcing the shared marginal condition). The marginal condition Por a) = P a)
can be enforced by construction. In practice, we construct the empirical distribution of offline triples (Soff, Gofl, s'a)) by first sampling (Soff, dolt from the online dataset Don. and then generating Hol ~ Poft (-1Sooff aott) using the pretrained offline flow. Separately, we sample online transitions (Son; Oonn )om) from D.n- As result, the two empirical measures used in the OT coupling have matched (8, marginaas in expectation, while their conditional transitions differ. The complete training algorithms of the offline flow and the online flow are in Appendix C
5

---

3.2 Data Collection at High Dynamics Gap Region
As discussed in Section 2.1 using behavior cloning and augmenting the replay buffer with low dynamics gap offline data can alleviate performance drop from distribution shift. However, relying solely on such data may limit state-action coverage and hinder policy learning. To address this, we propose new data collection strategy.
At each training iteration, we construct the replay buffer by selectively incorporating offline transitions with small estimated dynamics gap:
{ (s, Dott ((s. S UDn.
B
where r a predefined threshold. To improve data diversity and encourage better generalization, we actively explore regions with high dynamics gap areas likely underrepresented in the buffer due to the dominance of low-gap samples. We adopt an optimistic exploration policy that selects actions by
arg max ((s.aa) + Bā(s,))
(6)
where / is hyperparameter that trades off return and exploration of underexplored dynamics. To ensure sufficient coverage during training. we can further incorporate stochasticity by adding small perturbations to the selected actions [17 or following stochastic policy [19] consistent with widely used deep RL frameworks.
Theorem 3.5 (Large Dynamics Gap Exploration Reduces Performance Gap). Compared to behavior cloning policy The on the offline dataset. training a policy ā by replacing all offline samples with a dynamics gap exceeding K (as estimated by the composite flow) with online environment samples can reduce the performance gap to the optimal online policy Jnn with high probability by
(1 71 ()pp) (Aw: - K - -V + C1 W2ofott Pon)) 6)..
(1+(1+
Here Ly and Lp are the Lipschitz constants for the reward and transition function, respectively. Lp < 1. Aw, == sup,,a Wa(Pott (-|s,a) a), Pon(-Is, a)) is the largest dynamics gap. Co and C₁ are
two constants. I'N...6 = NN. (H) + V 16) where is the same as in Theorem 3.1 and Non is the number of samples used to train the online flow.
Non
exploration in regions with high dynamics gap can reduce the performance gap relative to the optimal policy. The parameter k serves as threshold: the bound holds when the policy is trained without offline samples whose dynamics gap exceeds k. As increases, more samples are collected from high-gap gap regions, which decreases k. This increases performance gap reduction, resulting in policy with higher expected return.
From Theorem 3.5
3.3 Practical Implementation
Our method can be instantiated using standard actor-critic algorithms with a critic Qs a) and a policy (a(s))) To incorporate the dynamics gap, apply rejection sampling retain fixed percentage of offline transitions with the lowest estimated gap in each iteration. The critic is trained by minimizing
(8) where Âen is the Equantile of the estimated dynamics gap in the offline minibatch. The target value is y + (s', ,a') (8 a), with a' ~ (-ls')
LQ= (ts.a.))-,.l(00((( - -) + (( < (xx)(Q,(s,a) =
The policy is updated by combining policy improvement with behavior cloning regularizer, using the objective
C. = BE-DDDD a (1)) (s. a)] w (s.)) ã~z, (-1s) [lla a|]3]
(9)
where w 0 controls the trade-off between maximizing the estimated Q-value and staying close to the offline behavior. The behavior cloning term encourages the policy's sampled action to match the offline action a, motivated by Lemma 2.2
The pseudocode of COMPFLOW, instantiated with Soft Actor-Critic (SAC) 19 is presented in Appendix D

---

4 Related Work
Online RL with offline dataset. Online RL often requires extensive environment interactions
15 688. which can be costly or impractical in real-world settings. To improve sample efficiency,
Offline-to-Online RL leverages pre-collected offline data to bootstrap online learning
[43] A typical
two-phase approach trains an initial policy offline, then fine-tunes online [32
43
44 However,
conservative strategies used to mitigate distributional shift, such as pessimistic value estimation [31]
can result in suboptimal initial policies and limit effective exploration (40]
44 To resolve this
tension, recent methods propose ensemble-based pessimism 132 value calibration 44]. optimistic
action selection [32]. and policy expansion [70]. Others directly incorporate offline data into the replay buffer of off-policy algorithms [3, improving stability via ensemble distillation and layer normalization. However, these approaches typically assume that the offline dataset generated under the same transition dynamics as the online environment In contrast, our work explicitly accounts for dynamics shift between the offline data and the online environment.
RL with dynamics shift. Our work is related to cross-domain RL, where the source and target domains share the same observation and action spaces but differ in transition dynamics. Prior work has addressed such discrepancies via system identification [5] g [00, domain randomization [41
56 [I, imitation learning 20 24 and meta-RL 42
50 often assuming shared environment
distributions [38] or requiring expert demonstrations. More recent work has relaxed these assumptions.
One line of research focuses on reward modification, which adjusts the reward function penalize source transitions that are unlikely under the target dynamics [12 [5]. or down-weights value estimates in regions with high dynamics gap [6). Another line of work explores data filtering.
which selects only source transitions with low estimated dynamics gap (65] using metrics such as transition probability ratios [2, mutual information [64], value inconsistency (65), or representation- based KL divergence [37]. However, these metrics can become unstable or ill-defined when the transition dynamics have different supports, and value-based methods are prone to instability caused by bootstrapping bias.
In contrast, our approach leverages the theoretical connection between optimal transport and flow matching to estimate the dynamics gap in a principled manner. A closely related work by [39 also applies optimal transport quantify the dynamics gap. However, their setting assumes both the source and target domains are offline, and their method estimates the gap based on the concatenated tuple (s, a, s') observed in offline data, rather than comparing conditional transition distributions. As result, their metric does not accurately capture the gap in transition dynamics and cannot be used to compute exploration bonuses, which require gap estimation conditioned on given state-action pair.
RL with diffusion and flow models. Diffusion models [2128] 28 57 and flow matching M 34 have
emerged powerful generative tools capable of modeling complex, high-dimensional distributions. Their application in RL is consequently expanding. Researchers have employed these generative models for various tasks, including planning and trajectory synthesis [22]. representing expressive multimodal policies [1]. providing behavior regularization 66) or augmenting training datasets with synthesized experiences. While these works often focus on policy learning or modeling dynamics within single environment, our approach targets the transfer learning setting. Specifically, we address scenarios characterized by a dynamics gap between the offline data and the online environment. We utilize Flow Matching, leveraging its connection to optimal transport, to estimate this gap.
5 Experiments
In this section we first evaluate our approach across a range of environments in Gym-MuJoCo that exhibit different types of dynamics shifts. We then conduct ablation and hyperparameter studies to better understand the design choices and behavior of COMPFLOW. Finally, we assess the effectiveness of our method in real-worke inspired wildlife conservation task.
5.1 Gym-MuJoCo
5.1.1
Experimental Setup
Tasks and datasets. We evaluate our algorithm under three types of dynamics shifts, namely mor- phology, kinematic and friction changes, across three OpenAl Gym locomotion tasks: HalfCheetah,
"Our code is available at httt::::/ggthubb..oo/////ccaaaan

---

Dataset
Task
SAC
BC-VGDI
BC-PAR
Ours
setting. The results are HalfChoetah (Morphology)
1457 89
2495 43
1430 + 408
2705 +
1790 =
3119 = 107
HallCheetah (Kinematic)
2255
197
186
4257 600
4192
4179
5189
HallCheetah (Friction)
184
7799 157
6397 673
621
8056 1.512
8241
Hopper (Morphology)
3611
361
354 3:1:::55
335
Hopper (Kinematic)
547
1024:::0
10125
1024
1024 1
1024
Hopper (Friction)
234
228 2
229 +
230
232
27
Walker2D (Morhooogy))
253
598
1011 193
672 576
158 151
1091
MR
Walker2D (Kinematic)
152 4
2973 + 185
1907 + 851
1586 923
948 131
1568 + 1315
MR
Walker2D (Priction)
301
+
296 +
302
321 +
20
HallCheetah (Morphology)
1467
1522 72
273
1427
2282
HalfChoctah (Kinematic)
2316 92
5/151 195
+: 773
1972
5213
HallCheetah (Friction)
238
7108 1001
816
7800
7871
M
Hopper (Morphology)
396 60
436 +
410
406
13
604 173
M
Hopper (Kinematic)
724 635
1022
970 +
934 084:445
1020
1023
M
Hopper (Friction)
229 1.5
232 1.
228 1.
229 1.
233 1: 5
66
M
Walker2D (Momhology)
301 + 177
457
577 201
584 + 219
431 + 177
866 + 372
M
Walker2D (Kinematic)
258 : 174
1966 1155
1965 +: 568
1921 922
806 1: 278
21339 936
M
Walker2D (Friction)
301 1
286 1:
298 1:
289 1: 47
308 1:
3220 31
ME
HalfCheetah (Morphology)
1392 + 238
1195 + 241
1147 160
1072 + 102
1207 +
1485 + 67
ME
HalfCheetah (Kinematic)
2323
421 202
5143 + 330
4603
4309 164
5750 +
ME
HaliCheetah (Priction)
1950 1 312
4185 1: 732
21:00 1: 733
4078 1032
4989 500
5596 1: 1557
ME
Hopper (Morphology)
359 1. 75
349 1.
444 1
357 63
407 28
462 89
ME
Hopper (Kinematic)
721 535
1021
1031 3
1022 3:
1027
1022 +
ME
Hopper (Friction)
229 +:
230 =3
232 +
230 :
232 =
268 70
ME
Walker2D (Morpholoyy)
228 + 51
429 117
1103 + 444
502 = 301
380 + 231
648 + 180
ME
Walker2D (Kinematic)
386 + 184
850 963
1614 782
1204 734
756 + 268
1511 = 1206
ME
Walker2D (Friction)
266 1. 66
240
114
258 1.
245 1. 51
242 24
326 26
Average Return
878
1920
1783
1868
1803
2193
Table 1: Comparison of return under different dynamics shift scenarios and dataset types after 40K environment interactions. MR = Medium Replay, M Medium, ME. = Medium Expert. cell is green if the method has the highest mean and improves over the second best by at least 2%. Cells within 2% of the top mean are marked in yellow.
Hopper, and Walker2d 4 Each experiment involves an offline environment and an online envi- ronment with modified transition dynamics following (38). Morphology shifts alter the sizes of body parts, kinematic shifts impose constraints on joint angles, and friction shifts modify the static, dynamic, and rolling friction coefficients. For each task, we use three D4RL source datasets: medium, medium replay, and medium expert, which capture varying levels of data quality [16]. Additional environment details are in Appendix K
Baselines. We compare COMPFLOW against the following baselines: BC-SAC extends SAC by incorporating both offline and online data, with behavior cloning (BC) term for the offline data, H20 46 penalizes Q-values for state-action pairs with large dynamics gaps. BC-VGDF [65] selects offline transitions with value targets consistent with the online environment and adds a BC term. BC-PAR (37) applies reward penalty based on representation mismatch between offline and online transitions, also including a BC term. Implementation details are provided in Appendix K
5.1.2 Main Results
We evaluate each algorithm by measuring its return in the target environment after 40K environment interactions and 400K gradient updates, reflecting reported in Table I Our key findings are summarized as follows:
(1) COMPFLOW consistently outperforms recent off-dynamics RL baselines across diverse offiine dataset qualities and dynamics shifts. It achieves the best return on 19/27 tasks and ties for best on S more. On average, COMPFLOW reaches score of 2193, compared 1920 for the strongest baseline (BC-SAC), a 14.2% relative improvement. Additional results under more severe dynamics shifts in Appendix show the same trend.
(2) Leveraging offline data substantially improves performance: on average, COMPFLOW improves over SAC by 149.8%. Moreover, COMPFLOW outperforms BC-SAC, which directly uses all offline data, on 23/27 tasks and matches it on the remaining 3, highlighting the benefit of our dynamics-gap- aware design.
8

---

(3) We also find that many recent baselines perform similarly to BC-SAC, consistent with [38]. suggesting limited ability effectively exploit offline data under dynamics shift. One reason is that methods such as H2O and BC-PAR estimate the dynamics gap using KI. divergence or mutual information, which can be ill-defined under large shifts or mismatched support. BC-VGDF instead filters data based on value estimates, which is often harder due to bootstrapping bias and target non- stationarity. In contrast, COMPFLOW models transition dynamics with flow matching and estimates the dynamics gap via the Wasserstein distance through its connection to optimal transport, yielding more principled and robust criterion.
5.1.3 Ablation and Hyperparameter Analysis
Composite Flow. We first evaluate the effec- tiveness of the proposed composite flow match- ing design. For comparison, we train a direct flow model on the target environment initialized from Gaussian prior. We compute the mean squared error (MSE) on a held-out 10% vali- dation set and report the average MSE across different epochs during RL training. As shown in Figure 3 the composite flow significantly re- duces the MSE of the transition dynamics in the environment. This improvement stems from its ability to reuse structural knowledge learned from the abundant offline data. These empirical findings support the theoretical insight presented in Theorem 3.1 that our composite flow improve the generalization ability.
Siveet
Compacke
HaltChastah Modium
0.00
0.00
0:30
0.30
Figure 3: Comparison of MSE between direct flow and composite flow.
Impact of data selection ratio 5%. The data selection ratio decides how many offline data in a sampled batch can be shared for policy training. A larger & indicates that more offline data will be admitted. To examine its influence, We sweep & across 170, 50, 30, 20J. The results is shown in Figure 4 Although the optimal Ç seems task dependent, moderate values of & (e.g.. 30 or 50) generally yield good performance across tasks, striking balance between leveraging useful offline data and avoiding high dynamics mismatch. When & is too large (e.g.. 70), performance often degrades, particularly in Walker Medium (Morph) and Walker Medium Replay (Morph), likely due the inclusion of low-quality transitions with large dynamics gap. HalfCheetah consistently achieves higher returns compared other domains, suggesting that knowledge transfer and policy learning in this environment are easier. Consequently, overly conservative filtering (e.g.. < = 20) may exclude valuable offline data, leading slower learning. In this case, allowing more offline data (c.g.. & 70) appears beneficial.
selection ratio: 70%
selection ratio: 50%
selection ratio: 30%
solection ratio: 20%
Hal/Cheetah Medium (Morph)
Hopper Medium (Morph)
Walker2d Medium (Morph) Walker2d Medium Replay (Morph)
6:0
600
1500
2000
6.0
400
1000
4.00
1000
200
200
10
30
10
40
Eay Steps (X10f)
Staps (X10f)
Eoo Staps (x110f)
Eav Steps (x119)
Figure 4: Comparison of return under different data selection ratios across tasks.
Impact of exploration strength 3. 3 controls the strength of exploration toward regions with large dynamics gap. A large 3 indicates higher incentive explore such regions. As shown in Figure 5, the effect of 39 on return is task-dependen possibly due to differences in the underlying MDPs. In the Friction tasks, we observe that larger exploration bonuses lead higher asymptotic returns. This suggests that incentivizing exploration helps the algorithm discover high-reward regions more effectively. contrast, in the Morphology tasks, moderate values of 3 typically outperform both smaller and larger values. This indicates that excessive exploration may not be effective in some tasks. While the optimal 9 varies by task, one trend consistent: across all five experiments, the setting with exploration (8 0) consistently ranks as the lowest the second lowest performers. This

---

empirical finding confirms the importance of the exploration bonus and provides evidence supporting our algorithmic design.
# 0.001
#
0.01
0.1
Hopper Medium (Friction)
Hopper Medium Replay (Priction)
Walker2d Medium Replay (Friction)
300
300
250
Beturn
200
200
100
200
Env Steps (x10%)
Env Steps (x10')
Env Steps (x10')
HallChectah Medium (Marph)
Hoppor Medium (Morph)
600
2000
Ens (x10f)
Stops (x10f)
Figure 5: Comparison return under different exploration strengths across tasks.
5.2 Patrol Policy Learning for Wildlife Conservation
We evaluate COMPFLOW in a wildlife conservation setting, where the goal is to learn a patrol policy that allocates limited ranger effort to reduce poaching. We adopt the simulation environment of Xu et al. [67], which represents the park as a 1 1 km spatial grid. At each time step, the agent allocates a distribution of patrol effort across cells under fixed budget. The state, including wildlife density and poaching risk, evolves according to dynamics driven by both poacher behavior and patrol deployment, and the per-step reward reflects the expected number of animals preserved through deterrence and spatial coverage.
We assume access to an offline dataset collected under previous policy in Murchison Falls Na-
750
tional Park, and aim to leverage it to learn improved patrol policy for Queen Elizabeth Na- tional Park with only limited online interactions. Since ecological conditions and poacher behav- ior differ across parks, the transition dynam- ics are mismatched. As shown Figure 6 COMPFLOW achieves the highest reward, out- performing the best baseline, BC-PAR, by 8.8%. It also improves SAC by 20.8% % under the same interaction bud- Figure 6: Reward on the wildlife conservation task. get. These gains particularly valuable in large protected areas where ranger capacity is constrained: Murchison Falls spans roughly 3,900 km², while Queen Elizabeth covers about 1,980 km², making data-efficient learning essential for effective protection.
700
650
Return
550
500
400
training from scratch with
SAC
BC-SAC
H20
BC-VGDF BC-PAR Ours
6 Conclusion and Limitations
In this paper, we proposed COMPFLOW, new method for estimating the dynamics gap in reinforce- ment learning with shifted-dynamicn data. COMPFLOW leverages the theoretical connection between flow matching and optimal transport. To address data scarcity in the online environment, composite flow structure that builds the online flow model on top of the output distribution from the offline flow. This composite formulation improves generalization and enables the use of Wasserstein distance between offline and online transitions as robust measure of the dynamics gap. Using this pricinpled estimation, further encourage the policy to explore regions with high dynamics gap and provide a theoretical analysis of the benefits. Empirically, we demonstrate that COMPFLOW consistently outperforms matches state-of-the-arr baselines across diverse RL tasks with varying types of dynamics shift.
adopt
Due to the space limit, we discuss the limitations of COMPFLOW in Appendix A
10

---

Acknowledgement
We are thankful to the Uganda Wildlife Authority for granting us access to incident data from Murchison Falls and Queen Elizabeth National Park. We also thank the anonymous reviewers for their valuable feedback. This work was supported by ONR MURI N00014-24-1-2742.
References
[1] Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representationss 2023. URL /t///p/nrorooo...oo/ororrrrrrrrp
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. In International Conference on Machine Learning, pages 214-223, PMLR, 2017.
[3] Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learming, pages 1577-1594. PMLR, 2023.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01:3 2016.
[5] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pages 8973 8979. IEEE, 2019.
[6] Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In The Twelfth International Conference Learning Representations, 2024. /ttltttt///pppeeeiiiiee
[7] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[8] Ignasi Clavera, Anusha Nagabandi, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model-based control. arXiv preprint 18Xiv: 1 1347, 3:3, 2018
[9] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[10] Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1290-1296. IEEE, 2021.
[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[12] Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari. Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. arXiv preprint arXiv:2006.1338 13916, 2020.
[13] Kilian Fatras, Younes Zine, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Learning with minibatch wasserstein: asymptotic and gradient properties. arXiv preprint 2019.
[14] Kilian Fatras, Younes Zine, Szymon Majewski, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Minibatch optimal transport distances; analysis and applications. arXiv preprint axXiv:2101.01922.200 2021.

---

[15] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy. levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL htttttt:// wwtp://pg//////////////
451 html
[16] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[17] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[18] Adam Generale, Andreas Robertson, and Surya R Kalidindi. Conditional variable flow matching: Transforming conditional densities with amortized conditional optimal transport. arXiv preprint arXiv:2411.08311 2024.
[19] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. Pmlr, 2018.
[20] Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morpho- logical transfer. In International Conference on Machine Learning, pages 4159-4171. PMLR, 2020.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[22] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning, pages 9902-9915. PMLR, 2022.
[23] Gavin Kerrigan, Giosue Migliorini, and Padhraic Smyth. Dynamic conditional optimal transport through simulation-fia free flows. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL /ttp:tttp///ppeerririiii eew net/forrm?id-tkkualle
[24] Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. In International Conference on Machine Learning, pages 5286-5295. PMLR, 2020.
[25] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Representation Learning, 2015.
[26] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(1 I): 1238-1274, 2013.
[27] Lingkai Kong, Harshavardhan Kamarthi, Peng Chen, Aditya Prakash, and Chao Zhang. Un- certainty quantification in deep learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5809-5810, 2023.
[28] Lingkai Kong, Yuanqi Du, Wenhao Mu, Kirill Neklyudov, Valentin De Bortoli, Dongxia Wu, Haorui Wang, Aaron Ferber, Yi-An Ma, Carla P Gomes, et Diffusion models as constrained samplers for optimization with unknown constraints. arXiv preprini arXiv:2402.1801 18011, 2024.
[29] Lingkai Kong, Haichuan Wang, Charles A Emogor, Vincent Börsch-Supan, Lily Xu, and Milind Tambe. Generative ai against poaching: Latent composite flow matching for wildlife conservation. arXiv preprint 250iv:2508. 14342, 2025.
[30] Lingkai Kong, Haichuan Wang, Yuqi Pan, Cheol Woo Kim, Mingxiao Song, Alayna Nguyen, Tonghan Wang, Haifeng Xu, and Milind Tambe. Robust optimization with diffusion models for green security. arXiv preprint 2503.23503 2025.
12

---

[31] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33: 1179-1191, 2020.
[32] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
[33] Yinghao Li, Lingkai Kong, Yuanqi Du, Yue Yu, Yuchen Zhuang, Wenhao Mu, and Chao Zhang. MUBen: Benchmarking the uncertainty of molecular representation models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL atttps://openreviee net/ forumiid-gYceeeeHgnn
[34] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023. URL. htttpps://opnnrevieeee http://wwww....eeo/o//oo//////aa
[35] Jinxin Liu, Hongyin Zhang, and Donglin Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning. arXiv preprint arXiv:2203. 06662, 2022.
[36] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 22..vv:20990300000 2022.
[37] Jiafei Lyu, Chenjia Bai, Jingwen Yang, Zongqing Lu, and Xiu Li. Cross-domain policy adaptation by capturing representation mismatch. arXiv preprint arXiv:2405. 2024.
[38] Jiafei Lyu, Kang Xu, Jiacheng Xu, Jing-Wen Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu, Xiu Li, et al. Odrl: A benchmark for off-dynamics reinforcement learning. Advances in Neural Information Processing Systems, 37:59859 199911 1, 2024.
[39] Jiafei Lyu, Mengbei Yan, Zhongjian Qiao, Runze Liu, Xiaoteng Ma, Deheng Ye, Jing- Wenn Yang, Zongqing Lu, and Xiu Li. Cross-domain offline policy adaptation with optimal transport and dataset constraint. In The Thirteenth International Conference on Learning Representations, 2025.
[40] Max Sobol Mark, Ali Ghadirzadeh, Xi Chen, and Chelsea Finn. Fine-tuning offline policies with optimistic action selection. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
[41] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1176. PMLR, 2020.
[42] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta- reinforcement learning. arXiv preprint arXiv: 1347, 2018.
[43] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 09359, 2020.
[44] Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:62244-62269, 2023,
[45] Ariel Neufeld and Julian Sester. Bounding the difference between the values of robust and non-robust markov decision problems. Journal of Applied Probability, pages 1-14, 2023.
[46] Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When trust your simulator: Dynamics-aware hybrid offine-and-online reinforcement learning. Advances in Neural Information Processing Systems, 35:36599-36612. 2022.
[47] Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky TQ Chen. Multisample flow matching: Straightening flows with minibatch couplings. arXiv preprint
2023.
13

---

[48] Ben Poole. Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[49] Chengrui Qu, Laixi Shi, Kishan Panaganti, Pengcheng You, and Adam Wierman. Hybrid transfer reinforcement learning: Provable sample efficiency from shifted-dynamies data. In The 28th International Conference on Artificial Intelligence and Statisties, 2025. URL https
//openreviev kett/forum?A&-xadWLL@BBBB
[50] Roberta Raileanu, Max Goldstein, Arthur Szlam, and Rob Fergus. Fast adaptation to new environments via policy-dynamics value functions. In Proceedings of the 37th International Conference on Machine Learning, pages 7920-7931, 2020.
[51] Allen 7. Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wwwpiiewwweefoooooo wwweet/foormmiidmmpppppe
[52]
Filippo Santambrogio. Optimal transport for applied mathematicians. 2015.
[53] Julian Schrittweseer, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-66 609, 2020.
[54] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484-48) 2016.
[55] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354-3355 2017.
[56] Reda Bahi Slaoui, William R Clements, Jakob N Foerster, and Sébastien Toth. Robust domain randomization for reinforcement learning. 2019.
[57] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
[58] Yuda Song, Aditi Mavalankar, Wen Sun, and Sicun Gao. Provably efficient model-based policy adaptation. arXiv preprint arXiv:2006.08055 2020.
[59] Yuda Song, Yifei Zhou, Ayush Sekhari, Andrew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid Using both offline and online data can make efficient. arXiv preprint arXiv:22 10.00:220000671.. 2022.
[60] Chen Tang. Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martin-Martin, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28694-28698, 2025.
[61] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[62] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[63] Cédric Villani al. Optimal transport: old and new, volume 338. Springer, 2008
14

---

[64] Xiaoyu Wen, Chenjia Bai, Kang Xu, Xudong Yu, Yang Zhang, Xuelong Li, and Zhen Wang. Contrastive representation for data filtering in cross-domain offline reinforcement learning. arXiv preprini arXiv:2405.06119, 2024.
[65] Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering, Advances in Neural Information Processing Systems, 36:73395-73421, 2023.
[66] Lily Xu, Elizabeth Bondi, Fei Fang, Andrew Perrault, Kai Wang, and Milind Tambe. Dual- mandate patrols: Multi-armed bandits for green security. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 35, pages 14974-14982, 2021.
[67] Lily Xu, Andrew Perrault, Fei Fang, Haipeng Chen, and Milind Tambe. Robust reinforcement learning under minimax regret for green security. In Uncertainty in Artificial Intelligence, pages 257-267. PMLR, 2021.
[68] Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, et al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621-632, 2020.
[69] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
[70] Haichao Zhang, We Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. arXiv preprint arXiv:2302.00935, 2023.
[71] Yuchen Zhuang, Yue Yu, Lingkai Kong, Xiang Chen, and Chao Zhang. Dygen: Learning from noisy labels via dynameee-enhanced SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3674-3686, 2023.
enhanced generative modeling. In Proceedings of the 29th ACM
15

---

Appendix for Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data
Limitations
16
Broader Impacts
16
Algorithms of Training Optimal Transport Flow Matching
17
Algorithms of Training Offline and Online Flows
17
Algorithm of COMPFLOW built on Soft-Actor-Critic
18
Proof of Lemma 2.2
20
G Proof of Theorem 3.1
21
H Proof of Theorem 3.5
25
Proof of Proposition 3.3
28
I.1 Formal Proposition
28
1.2 Proof
29
J
Additional Experimental Results
31
K
Experimental Details of Gym-MuloCo
32
Environment Setting
32
2
Implementation Details
33
Experimental Details of Wildlife Conservation
35
M Discussion on Computational Cost
36
A Limitations
(1) COMPFLOW is currently limited to settings where the offline data and the online environment share the same state and action spaces. Future work could explore estimating the dynamics gap in a shared latent embedding space to relax this assumption 77.. (2) Our evaluation is conducted entirely in simulated environments; applying COMPFLOW to real-world scenarios remains an important direction for future work. (3) We have not yet incorporated the uncertainty in estimating the dynamics gap. Future work may explore how to quantify this uncertainty and use to improve both data
filtering and exploration strategies 27 33
B Broader Impacts
Our work aims make reinforcement learning more practical in real-world domains such as health- care, robotics, and conservation, where online interaction is often costly, limited, or unsafe. By addressing the dynamics shift between offline data and the online environment, our method enables more reliable and sample-efficient policy learning. This capability supports safer deployment in high-stakes applications, including clinical decision support and adaptive anti-poaching strategies. Nonetheless, we emphasize that policies trained on historical data should be applied with caution,
16

---

as misaligned dynamics or biased datasets may lead to unintended consequences if not carefully validated.
C Algorithms of Training Optimal Transport Flow Matching
The full training algorithm of Optimal Transport Flow Matching is given in Algorithm |
Algorithm Training OT Flow Matching (OT-FM)
Require: Dataset D; batch size k; learning rate Ir; vector field dg
Ensure: Trained parameters 0
I: while not converged do
2: (Sample) Draw latent codes (3) (x00)) ) (O(,
3: Draw data points (x(
D
4: (OT coupling) Set Ci f || (6) - 214
11 for all (i,j) E [|] x [
5: Compute optimal transport plan
+ arg min (A, C), AEBa
B₁ (Ae R A1 11 11 13
is a (scaled) doubly-stochastic coupling
6: Sample index pairs (ee; i.i.d from the categorical distribution on [A] [k] with probabilities As
7:
Sample times (tr) ~ U[0,1]
8:
for 1 to & do
(2)
(1 10) 4xx11 (fr)
(6x)
9:
Lie
1 t 11 - vo (6))
10:
Le + | te) A (0)
(1)
12:
end for
C
Σx le VoL
14:
0 0-
end while
16:
return 0
D Algorithms of Training Offline and Online Flows
The full training algorithms of the offline flow and online flow given in Algorithm 2 and Algo- rithm respectively.
Algorithm 2 Training the Offline Flow via Flow Matching
Require: Offline dataset Dor: batch size k; offline vector field uo: learning rate
Ensure: Trained offline flow parameters 0
1: while not converged do
2: Sample transitions (()),a())) s(0)) ~ Doif
3: Set Set + (+) for all i [k]
4: Sample latent codes (5) (1) y/ ~ I
5: Sample times (1,) ~ 10,
6:
for i 1 to & do
7:
Øi, < (1 -1-f8 (2) x
+ t, 17
8:
( + x x
4 ||10(1))) t. ( a ( - - 10
9:
end for
11: C- > e
0+1 eL
13:
end while
14:
return
17

---

Algorithm 3 Training the Online Flow via Optimal Transport
Require: Pre-trained offline flow sampler (,, 8, a,a); online replay buffer Doni batch size k;
regularization weight y > O; learning rate Ir; initialized online vector field
Ensure: Trained online flow parameters ø
1: while not converged do
(4) (8) (Oltline-synthetic batch) Sample (5off off Don (
2:
3: Sample latent codes (#/) y ~ (0,
4: for 1 to do
(() 21 < off
(1) + be (oo 1/ off Ooff
(t)
5:
6:
end for
7:
(Online batch) Sample (son' dou
(i) (i) j=1 Don
(1) Son for all [k]
8:
9:
(T coupling) For (i.j) E [k] x [k], set
Cjj - | 1 n soff
(8)
2 +Ila
-a
10:
Compute optimal transport plan
arg min (A, C)2 AEBa
BE (AE R4** A1 1 A 11
A a (scaled) doubly-stochastic coupling
Sample index pairs (e): i.d. from the categorical distribution [k] probabilities Aij 12: Sample times (1e) ~ 21112]
[k] with
for L 1 to k do
14:
Ltz - (2 te)
(2)
21
(6x) + (tc - 1) 22
(ic)
15:
A t x2
(jc) -21
le | le, 8oa
(f)
(1e)
dbie)) 10
17:
end for
18:
L- R
19:
- voL
end while
21:
return
E Algorithm of COMPFLOW built on Soft-Actor-Critic
The full training algorithm of COMPFLOW built on SAC is given in Algorithm g
18

---

Algorithm 4 COMPFLOW built on Soft Actor-Critic (SAC)
1: Input: Offline dataset Doff. online environment Mons max interaction steps max update frequency train_freq, offline selection ratio Ç E (0, 1l, gap reward scale 3, batch size B, behavior cloning weight warmup steps warmup_sted sermup_stepss, gradient steps per interaction K, learning rate
Ir. target update rate w, discount 7, entropy temperature a.
2: Initialize: Policy Tip critics {Qa, fim1,2 target crities (Qu 1im1,2 {Qec Dic 1,2, online replay
buffer Don f- 0.
3: Pretrain offline flow (o t s, a) on Dor via Algorithm 2
4: for = 1 to maxx do
5: Interact with Mon using Tip and store transition (s, (s,a,r,8)) into Don.
D ~ aa~Tpp11
Tp(:1s).
(r, MMn
6:
if warmup permup_steps and mod train_ fonn_ffeee-aa = 0 then
7:
Train online flow 4o (2, s, on Don via Algorithm
8:
end
9:
for to K do
10:
Sample offline minibatch boft = {(8 (2) (8) (() 1(0)
off: offs off 8
ooff ) ~
Doff-
Sample online minibatch bon {(son don ron Soa ] Don.
(f) (6) (6)
12:
Estimate dynamics gap (s off ooff for each (s off oof via Eq. (4)
(1) (1)
(6)
13:
Select the lowest-gap subset boff boff with [b = B]
Shape rewards for selected offline transitions: for each (s, (s,a,r,s) r') boet set F + ((s, a).
15:
# Critic update (use online batch + selected offline batch) Define critic batch bq t bon U ub with rewards r for online and 7 for selected offline.
16:
17:
for i = 1,2 do Compute Bellman targets for (s, r, bq:
18:
as,,arr,s) r + Bn min Q (1st (s',aa) log Tppaa 1s)] 1.2.
19:
Update critic by gradient descent:
Σ (Qu(5,,)- -(8,,,,)))
5 G - -
[bQ|
(e,e,r,"')Ebe
20:
end for
21:
# Soft update target critics
22:
for = 1,2 do
(1
24:
end for
25:
Actor update (policy improvement + behavior cloning) Define actor state batch by +- {s (8) bon bott)) Sample à T.p. () for each by. Set normalization weight
27:
28:
<>
b= Even, |min (s, +e
29:
Minimize actor loss:
Cx(1)
Σ sEh,
(alog
min Qu Qu ()) + X
Σ
à
[b
1=12
boal
where à ~ (: (.1 s) for s E br and ā ~ x. | 8) for (s, a) E boff.
30:
ip + - -- (ip).
31: end for
32: end for
33:
Output: Learned policy Tip
19

---

Proof of Lemma 2.2
Lemma 2.2 (Return Bound between Two Environments (37). Let the empirical behavior policy in Doft be TD. (a 8). Define C1 = (1-7)2. Then for any policy n,
Maa (T) MMMs (TT) 2 - 220 ( (s,0)~
woll sompout (Pr((T(-15') TDa (+ss)))
- C1 E TP [Drv (Dov(pon(iin (-0, Patt(- (-11,,, a)]]-
(s.a)~pM
Proof. Proof of Lemma 2.2 is already provided an intermediate step for Theorem A.4 in [7), the offline performance bound case. We include their proof here with slight modifications for completeness.
We first cite the following two necessary lemmas also used in [37
Lemma F.1 (Extended telescoping lemma). Denote Mi (8,,, and Ma (S,A, 7)) as two MDPs that only differ in their transition dynamics. Suppose we have two reach the following conclusion:
policies m T2,
1
MM, (1M)
(T2)
piny (s.a) [BerePi
[Mg, ('', - Eg' a'wma ( (8', ad)]]
Proof. This is Lemma C.2 in 1651.
Lemma F.2. Denote M = (S, A, P.r,)) as the underlying MDP. Suppose we have two policies TI, T2. then the performance difference of these policies in the MDP gives:
1
TM(TT) (M((T)) 1-7 (s)))))) snp Ba'nT (s' a')] EN'Nn (
Proof. This is Lemma B.3 in 137v.
Proof. We have access to the offline data Mott and the empirical behavioral policy TD so we bound the performance between 7MM. (T) and TTMa (TD.).. We have
Moc (It) Tr (T) M7 (T) MM (TDaa)
TTD (TD.a)
MT (TT)
(10)
(a)
(b)
To bound (a) term, we use Lemma F. in the second equality
7Me (T) TM (Duir
(1M (Doat) TMa (T))
B
(o Pall
Begaeppa woror [Qin (fott Beg ~p aloo [Q (8 a')]
1
1
E
Pall
y
Exg Exia al a'mat [OM, (S'mT - B upsa a'wu [M7,, (S a')]
(6)
(E a'nu QMnm ( d')] Es
MM EE
a'nut [QM, (s d')l)]
(d)
To bound term (c), use
Byg P.aa al a'ouEron [Q., (S a')]
Ey Balarraaan a' [Q (8 a')]
Dooriaal son)) Tr(aa sbaa) 1Q (shar a')l
2r 2rmax Edgermaa
[Drv (TDaa (oar)) ( (s))))
20

---

Zmax where the last inequality comes from the fact that 1QT, (c a')l S distance.
and the definition of TV
I-7
(d) Re' Spmppma a² rust [Q (5', Bgr Bemppaara a' [QMa (6'
SS (p (s' | - PMM (s' (sM ET(dl [(dd ds'
Tmax
(s (s Ps ds
7 S
2r max
[Drv (pmarl s, pP s, a))]
1-7
where the inequality comes from the fact that QM,,. (s', d') weighted by probability is bounded by the upper bound for all Q-values.
1-7y
Combine the bounds for (c) and (d), we obtain a bound for the (a) term:
2r max
M. (T) MMM (TD,r)
E
TPad s'mppe Drvv (- ss) Il x(. ss))]
(1 7)2
(
2r max
[Drv (p.iin (- s, pl (- (,,s,))) s.)))]
(1 - 7)² (a.a))-Maal
Paar
Now we try to bound term (b).
MM (TD)))
1M (T)
E Pun (s.0)~PMae
Ea'nTD [Q [6',,']] - Ea'na [Q (8', a')]
s'mpme (
E "Dal sf vemee (18,a)
Eu'nssoa [QMar (s', ,''] -uu'n [M (s', aa)]
1 ? (s.a))-pap
(Trarla' | s<a'= s')) Qiaa (s',
1 (s.a)~PMi
Dull aum (sa)
l'EA
Tmax
(Trar |8') =aaaa |s')
(1 7)2 (eaa)~pp
The of s'mpp (lea)
a'EA
2r 2mmax
71- (aaa)~ The mom (1aa)) [Drv (TDan
s || [| ss')]]
where the first equality is direct application of Lemma we conclude that
F22 Combine the bound for term (a) and (b),
M. (T) M.M (TT) 2 - 2C E Taat
Drv(c(-Is') TDA (+Is)))
(a.0)~PM
of paa
C1 E
Pan (Prv(Pon--ssd) a) |l (a((-ss,)))]
where C1 =
2rmaxx (1-7)2-
G Proof of Theorem 3.1
Definition G.I. For any distributions po. Pi on Rd with finite second moments, define D2(po, p1) =
F||X; - Xollž. Xo ~ Po, X1 ~ Pr independent.
Theorem 3.1 (Conditions for Composite Flow Yielding Smaller Errors). Assume that composite flow and direct share the same hypothesis class H. of (measurable) vector fields v Rd x [0, - Rd and that there exists E (0, oc) such that suPtEM supzerd, te[o,1] [|o[x, 1) S B. Also, assume that max (Tr(Eon)) Tr(E..:)) < CTn for some CTR- The composite flow enjoys a strictly tighter high--proubbility generalization bound than the direct flow if and only if
W2(pa, Pon) > Walpott (on)
(2)
Here, PG (O, Eon is the covariance of Pou and Eost is the covariance of Poft.
21

---

Proof. We consider linear-path flow matching (FM) trained with squared loss. A single training example is generated by: On 1]; Xg Po (the start distribution), independently of X1 Pon. independently of (t, Xo); The interpolation X2 = (1 t)Xo + 1X1 and the label Y = -1 - Xo E I
Let H. be hypothesis class of (measurable) vector fields Rd x [0, 1] R and define the population risk
R(v) =- E [l) 112
(11)
where the expectation is over the sampling mechanism above. Let
v² (0, == E =x, t|
(12)
denote the Bayes (population) minimizer. Given i..d. samples, let é be any empirical risk minimizer over H for the squared loss.
Assumptions. AI (Uniform boundedness) There exists B E 00) such that su rEjo.)) [|v(2, l)l B. This ensures integrability and controls the Lipschitz con- stants used below.
FM as regression; Bayes predictor and excess-risk identity.
By definition,
R(v) E [|v(X,, t) - 112, Y X1 - Xo.
For any measurable the Bayes (population) minimizer for the squared loss is the conditional mean v"(r,t) E|Y Xx = x, t].
A standard identity for squared loss (we derive it fully) is
R(v) R(v*)= E ([ () v² (Xe, 112. Derivation of 13 Expand and add/subtract 0" (X,,,, t):
(13)
R(v) = 1 (X,, t) (',,,, t) + 8z (X,, t) 112
E
||u 0" 112 + 2E (v 2", 8² -Y) E || Y
where all functions are evaluated at (X,, t) but notationally suppressed for readability. Condition on (X,,t): t): by definition, E|Y X,, t] o- (X,, t), hence E u* Y Xx, t and the cross term vanishes after taking expectations. Therefore
R(c)= = E || 1-1 R(v*),
which rearranges
In Lemma G.3 we prove that there exists an absolute constant e such that, for all E (0, 1), with probability at least - 8 (over the sample draw),
R(6) S R(v") c (B+
E 1111)) Tn.6.
Fn,s = RR, +
log(1/5) (14)
n
where RR, (H) is the (data-independent) Rademacher complexity of H. In short: symmetrization vector contraction + Lipschitzness the squared loss on the B-ball yields (14).
Subtract R(v*) both sides of [4] and apply 13) with 6:
0 S e B E 1111
T
(15)
Identify E ||Y||2 as D2(po, Pon) and linearize the square root. By independence of X, ~ Po and X1 ~ Pon.
E E E || Xoll
D2(po, pon)
(16)
By applying Lemma G.4 we have
E |
<
Tr(Eon)) + CTR + W2 Pon).
(17)
22

---

Therefore, we obtain
(B+ + Vr((on)) + Crn + W2(po. Tn. E F S (B
(18)
This gives the following master bound (19) with Co == c(B + v Tr(Con) + CTR and C1 := c.
Master bound.
Under A1, for any start Po (with fixed target Pon). with probability at least 6,
E 0 0 112 S
Co.n Po independent
+
C1 (p0 Pon) n, depends start Po
Co:= : co:=<(B-/TT( + CTR), C == c.
(19)
Composite vs. direct. Consider two trainings sharing the same online Pon and class H:
(Composite step) po Pott(: 8, a), (Direct step) po = PG = NO0, Ta).
(20)
(21)
Applying (99 to 20) and yields
comp W Vcomp
11 - Co n, + C1 Walpott: Pon 1n,6:
(22)
|80 o2
S Co n,s + +1 WaPP:: Pon) T n
(23)
When is the composite bound strictly tighter? Applying 19 to Po Poff (composite) and Po (direct) gives (22) and (23). Since Co and I 5 are identical across the two trainings (they depend on (B,c, c, Don CTR). n, o, and H. but not on po), the only difference in the right-hand sides is the factor W2(,, Pon).
Therefore the composite bound 22 is strictly smaller than the direct bound 23 if and only if
We(Pott Pou)
Walpe: Pon)
(24)
Remark G.2 (Optional D2 strengthening). One can analogously replace W2 Pon) by D2(Po, Pon)
when bounding 15 using the standard inequality trick vz S 1 (x + 1) for all 0. We use W2 metric because it more commonly adopted in the literature, but D2 leads to tighter bound since we no longer needs the constant CTR- We do not expand the D2 case here, as the argument mirrors the W2 case.
Lemma G.3 (Generalization gap for squared loss under uniform boundedness). Assume AI. Let (=) := [, i) for:.for: z = (x, l, y) generated in the theorem. Then there exists positive constant e > 0 such that, for any 8 E (0, 1). with probability at least - over an i.i.d. sample of size
R(0)= < R(ee) + (B+III In,o,
:=
: Rr, + log(1/8)
Proof. We sketch the standard steps and make constants explicit where needed:
(i) Symmetrization. Let R(v) denote the empirical risk. For ERM 0,
R(ό) R(w*) S sup (R(x) - f((w))
vEH
up to negligible terms; this is standard (e.g., by one-sided symmelrization plus a chaining step). Thus it suffices to bound sup.EH (R(u) (R())-R(e)).
(ii) Lipschitz envelope via contraction. For fixed (x, y) with [| f) < B and any u E Rdd with |ul]| B,
|| - y|1² -1|u(1,, 1) yll²
1), (u + v(a, 2n)) 2 + Ivll) u|u v(a, 1)
23

---

Thus, fumction of u, the map + 1lu- y1 is (2(B + Ilvvl)) on the B-ball. Applying the vector contraction inequality the class { H. gives
E sup (R(u) R(v)) x [ - [1X11) Ri (H.)
vEHt
By Cauchy-Sehware and Jensen, ||II / |
(iii) High-probability upgrade. A standard bounded differences (or bernstein-type argument for Lipschitz. sub-quadratic losses upgrades the expected sup-gap high-probabillty bound, adding
the usual / (log(//0// term. Collecting constants and using E Y| /VIII yields the claimed
form with some absolute >
sup (((V) - R(v)) & c cB++BBBBB (B++VBIYD (EII)) nn,5- NE
Since è an ERM, R(0)= < R(v*), and therefore R(ö) - R(v*) < sup, (R(v) R(v)) in the final step, absorbing the factor 2 into
0
Lemma G.4. Let μ. and Σ. denote mean and covariance, respectively. We show that if Tr(r))))) S
CTR for some OTR > 0, we have:
V D2(Po,Pi) S Tr(Ei) CTR + W2(po. PI)
Proof. Let No po and X1 pı be independent with means (4o, in and covariances Eo, Er. We compute
D2(Po. p1) E |X Xol12 = E ||x1| 2 + E X 2E (X1,
By independence, T. Xo) (,X,, E.oo) = (1r, no)- Moreover, for any random vector Z with mean f and covariance Σ, E | || + Tr(E). Thus
D2 P1) (1|m1 Tr()))) (11/o112 Tr(Eo)) - 2 (Mi Mo)
= || - po Tr + Eo),
Let [(po.1)) P1) be all couplings of po, For any π E I(Pu, with (Xo, X1) n, we have
E2 x - Xoll2 2 | [X Xoll |1 - roll
by Jensen's inequality since = + ||=1|2 is convex. Taking the infimum over all couplings gives
inf W (PT(
E Ex|Xi - Xoll3 2 ||a - -o
Therefore, we have || - moll W2 PI). Hence,
/ DD((o.PP)
V Ilm - moll2 Tr(E,, En)
$ Tr(Eo + En) + v 11 moll2 $ Tr(E,) + CrR + W2(po.P1)
This completes the proof.
The assumption that Tr(Eo) S Crn makes sense because we assume bounded state space for all settings considered in paper.
24

---

H Proof of Theorem 3.5
Theorem 3.5 (Large Dynamics Gap Exploration Reduces Performance Gap). Compared to behavior cloning policy The on the offline dataset. training a policy ñ by replacing all offline samples with dynamics gap exceeding K (as estimated by the composite flow) with online environment samples can reduce the performance gap to the optimal online policy T with high probubility by
(1,(1 (1 7) - 7lp) (Aw, - - -V + (2(fottPPo 8)
Here Ly and Lp are the Lipschitz constants for the reward and transition function, respectively, Tpp < 1. Aw, == sup,,a Wa(Poft (-1s, a), Pon(-Is,a) a)) is the largest dynamics gap. Co and C₁ are two constants. I == RN.. (H) + /our(1/8) where H. is the same as in Theorem 3.1 and Non. the number of samples used to train the online flow.
Non
Proof. Let's first list the assumptions:
R1: Lipschitz reward. For every E A, s + r(s, a) is L--isscchitz
R2: Lipschitz dynamics. For m E {off, on), Wi(pm( s, a))), Pm (m((1ss,a))) s Lpd(s.s)) for all ss',a
R3: Contraction. 7Lp 1.
We proceed in four steps: (A) policy-depender model-gap bound; (B) a uniform high-probability W2-gap estimation bound for the rectified flow (a:); (C) learning-gal gap bounds for Tbe and (3); (D) assembly via the standard three-term decomposition.
Policy-dependent model-gap bound. For any fixed policy we cite Theorem 3.1 of 45 Step A:
for the following bound. Under R1-R3, we have
V Vo S2L,AW
Lp)
1=0
j=0
=2L,Aw(1+7)) Σ (Lp)
1=0
70
=2L (1+m) Σ (Lp)
1-77
j=0
2I,AA (1 +7))
8
Σ (pLp)
1-y
5=0
2L,AW. Aw, 11+1)
(25)
() -Tpp)
Averaging over the initial state-distribution gives us a bound on what we call the model gap:
/ (V.T (s) VJJ (s) p(ds)
MM.. (x) - "Mao (x)) =
VT (s) V (s) u(ds). J
< V Vor
Al,,A,i (1 + 7
< (1 7) 1111-110 LLp
25

---

Moreover, the bounded state space assumption in Theorem
3.1 implies bounded second moment for
both Poff and Pon- we have W, (Pott Pon) - WPffoft, Pon PPott Pon Hence, we obtain that
21, (1 + +
MMa (TT) TIMan (T)
Aw:
(26)
(1 [1))((1 - 71pp)
2L, (1 + 7)
(1 - (1-7)(1 7 Aw2
(27)
Step B: FM-based uniform W2 estimation bound (construction of ar). We need high-probability uniform control of the enror of the rectified-flow W2 estimator ensure that thresholding by k actually caps the on-policy mismatch for R.
B.I. Flow-to-map error. Let I be the Brenier OT map from Po | a) to Pon s,a) (guaranteed by standard assumptions), and let T be the terminal map obtained by integrating the learned FM velocity field from to = 1. Under the linear path and Lipschitz regularity u* and (Theorem 3.1 assumptions), a stability argument (Gronwall) yields
Exo [T(x) T(xo)l < Kstab S./ E t(x,t) v² (X,, dt,
(28)
for some finite Kstab depending on Lipschitz constants of the flow. By the "master bound" (Eq. (7) in Theorem 3.1) uniformly over E [0, 1J. with prob. 2 1 8,
B 0² 11 z
Co I Non.5 + +1 W2 (Pott: Pon) ) I No..5.
(29)
So we have
Greawall
(Xxo) - (
S
Katab
Elli 1 dt
(30)
terminal map MSF.
FM excess risk at
< Katab (Co C1 W2(Poit Pou)) I N
(31)
B.2. From map MSE to W2 error. By the coupling construction (8, (T(X)), with
Xo ~ Pott and the triangle inequality for W2,
sup (..a)
(w, a) (s)
S sup W2 WPon (s,a)
(32)
Define the coupling ((sa)) == (T, (T,T)+POrT- s, s, i.., if S == T(Xo) and S T(Xo) then (S, "(.aa) and (ss.o) E I[(po 8, aaa).ool a)). By the definition of the 2- Wasserstein distance,
W:](Pon-- s, a)) ssa)) inf
vEIl(pon Pon) - d12
(33)
S / || vll2 ((),)) E ( -
(34)
Taking square roots gives, for each (s, a),
(Popon s, a), Poul: s, s,)) I T(xo))
Finally, taking the supremum over (s, a) preserves the inequality:
sup W2(pon(- s, a). Poul: s, s,)) < sup VE (T(x)) - T(Xo))1.. (s.a)
(35)
(s.a)
Combining 35 and
32 gives
Â Áwa (s. a) Aw, (s,a) < aNo. == V Kotab (Co + C1 Wz(pon WP(Po))Pon) I N
sup (x.a)
26

---

with probability at least - 8.
B.3. Capping the on-policy gap for R. By construction of the replacement rule, for any (s, a) encountered during fine-tuning of we have either Aw, (s, a) Σ k (kept offline sample) or we replaced it by an online sample. In either case, on those (s, a) we ensure
Aw, (8,0) a) & (8, + aNo... S K + NoNon.
Therefore the on-policy gap of obeys
R W2 I K + No.on.
(36)
Step C: Learning-gap bounds for The and मे (construction of P). Recall the three-term decompo- sition for any policy
Mon (T (mm)) -on (T) (Non))) Noal (Ton)) +
(ool (TTO) Moft (T))
(Non( Tou (T)) (37)
model(a)
learning(b)
modelte)
We now bound the learning term (b) by ERM generalization.
Policies are learned by ERM on a surrogate imitation loss ((x)) over a policy class with Rademacher complexity Ry (II). There exists calibration constant Cval (depends on concentrability/Iissshhie
ness; fixed for the class) such that, for any I,
not (m) (TT S Gvall [[(T)
s'Ell ((T'))
inf
(38)
Moreover, for datasets of sizes Nott Niudd, with probability at least 1 - 8,
[(f) inf [(T') & On (NN((I))))) + (188 N
N E (Noff, Nmodh
(39)
#'Ell
for absolute constant Cn.
Let The be the behavior cloning policy trained the offline dataset of size Notf. and let be ERM on the modified dataset (size Nmod): for any (8, a) encountered, if the FM-estimated gap satisfies Aw, (s, a) > replace the offline (s, a, s') sample by an online (s, a, san) sample (collecting No. such online transitions), leaving others unchanged. Denote Nmod the resulting (modified) dataset size.
By calibration (88 and the generalization inequality (39). with probability at least 5 we have
Nof (Tām) - Tott < Cval Cn RNN (II) +
log(1/8) Nott
and similarly
Noft ( Moft (it) I Cval Cn ANaoa (IT)
log(1/8)
Nmod
By a union bound (probability - 26) and adding these two upper bounds we obtain a common envelope
max f nont (T 7odl (Tbe), noal (TT(TT)) non s 8,
(40)
with
3 : CIa RN (II) R (II)
1on(2/6) Nott
log(2/8) Nasca
(41)
Step D: Assemble bounds and compare. with high probability,
Incorporate (27 and 40 to 37 with Tbe. we have
21, Aw,
Ton (m Ton (Tbe)
(Ton (Fan)) Nott (TTT)) +
(1 - Lpp
Incorporate
and 40 to 37 with ft we have with high probability
27

---

2I, + +
Mon (Ton) Mon (Tt
(onn( noil noul (m(n)) + 3
(nc aNo.
(1 7)(1 7bpp)
The difference between these two upper bounds is:
21, +)
(1 - 7) - ~Lpp) (Aw:- - (.).)))
Absorbing Katab into Co and C gets the result.
I Proof of Proposition 3.3
We provide rigorous justification for Proposition 3.3 The key technical point is that the augmented optimal transport (OT) problem on triples (s, a, s' with cost
On ((s, a, x), (s', (s', = || - x' 1 + m s'112 + ||a all33)
conditional quadratic OT on the next-state variable s' given each fixed (s, a). provided that the (s, -mrginnas match.
reduces,
I.1 Formal Proposition
Proposition I.1 (Shared-laten) coupling recovers conditional W2). Setup. Let Y S x A denote the conditioning space and X == denote the next-state space. Let μ be a probability measure on Y. Let a, 3 be probability measures on y X such that their Y-marginals coincide:
Taa a # p4,
(42)
where san (y) Assume and B admit disintegrations with respect to μ.:
oddy, dr) = (ldd) (y(dx)) B(dy, dr) = p(dy) By (y(dx),
(43)
where y = (8, a) E y and E X.
Augmented OT problem. For 1) > O, define the augmented quadratic cost (V X32 by Cy (13, x). (v/, (1,x'')) xx)) = -11 - 2'112 + nlyy -/112,
(44)
where |V 111 ||s s' 11 lla a' d'1/2. Define the corresponding OT value
f c.p ((v, (y', x²)) dr;
(a, inf yEll((,
(45)
where [T B) is the set of couplings on (y x X)² with marginals a and B.
Assumptions (for + reduction). Assume and 3 have finite second moments, that W2(ag, By) < 00 for p.a.e. y, and that the conditional squared Wasserstein cost is p-integrable:
/ (a By) u(dy) < oc.
(46)
Claim I (conditional decomposition). As + oc.
lim W (a, 3) (a By) p(dy). +0
(47)
Claim 2 (structure of limiting couplings). Let iin E arg min I Cn dy be any sequence of optimal couplings. Then there exists subsequence Tn. Noo weakly such that:
(i) Yoo is supported on the diagonal set (1,,2, x') y'}:
28

---

(ii) admits a disintegration
(do dx, dx') = p(dy) (y(d, da'),
where By E May By) W2-optimal for p-a.e. y-
OT-FM implication (per fixed conditioning y). Fix any (s, a) and let po = NN f). Assume the offline flow satisfies
1.8t (., y) = Ctyy
For > 0 and & E N, let vo (..2 y) denote the online flow obtained by rumning OT-FM to convergence with minibatch size k. where each iteration computes an optimal transport plan between two empirical measures of size k under the augmented cost (44).
Assume the standard OT-FM regularity conditions hold for this fixed y (e.g., those in Theorem 4.2 of (477), including:
(A1) Ogy and By have bounded support;
(A2) Og admits a density and the quadratic Of' map from Or to By exists and is continuous;
(A3) the minibatch OT plan is solved exactly at each iteration.
Then, for p.a.e. y. have the two-limit convergence
lim lim Ezo~po 10000
[1 (zo, v)
". (No, 1 v).2 m)|B (ay (48)
1.2
Proof
Proof. We prove 477 by matching an upper bound and lower bound, which also implies that any limiting optimal coupling concentrates on the diagonal set {y y/). Finally, we connect the resulting conditional OT structure to OT-FM trained with the augmented matching cost
Step (Upper bound via a diagonal coupling). Since a and 3 share the same Y-marginal μr, we may couple them by matching the conditioning variable y exactly. For μ.a.e. y, let qy [a 9y) be an optimal coupling achieving W32 (ay, By). A measurable selection y + o can chosen (see. e.g., 633 Chapter 5] [52] Section 1.4]). Define & E Il(a, B)byy by idy, dz, dy', dz') p(dy) 9t (dx, dz') (d(/))
(49)
where by is the Dirac measure at y. Because 7 is supported on {u 1/ }. for every 7) > O.
/ (e, //J || 21112 (1, u(dy) / W (ay, By) p(dy).
Therefore,
3) < / limsup W (2, +-
p(dy).
(50)
Step 2 (y-transport vanishes as - 00). Let 7/ be an optimal coupling attaining W (a. B) and define
C. C.:= = / B,) ((d)) < oc.
By Step 1, (2,, < C., hence
f cy c, (2, C.
Since e, (v), (1/, x²)) 2 nlly 1/1. it follows that
/ 1lv 1212 drm < C.
(51)
11
29

---

Step (Extract weak limit supported on the diagonal). Because a and / have finite second moments, aa, 3) is tight, hence {7, is tight. Thus there exist ?n + 30 and Yoo such that Thm Too weakly. Let f(y,y') lyy y' y' and define the bounded continuous truncation fM = min Since fM is bounded and continuous,
f far dran +
f fM droc.
Moreover, 0 S fax S f, so by (5T).
J [M d700 = lim / fM drnn _ lim n
/ ddmn =
n-000
Letting - oo and applying monotone convergence yields f / = O, which implies = y holds -as. Hence Yoo is supported on ",,,;;,,')
1/)
Step (Lower bound and convergence of values). Since Toa is supported on {v = y/} and has marginals and B, it disintegrates
(dy, dx, dx') = p(dy) Ti (dx, dz'),
where "v E 11(ay, By) for p-ace. y. Let g( = || x'113 and gM == min(g, MJ. Because 0 9M S Cya+
W2 2.mm (a, 3) = d I IM d'
= f
By weak convergence, gM dm + gMs d% Letting M oo and using monotone convergence yields
(1le-=(BBdrgtt.t)) u(dy) / Wa By) p(dy).
lim W Wn, (a, B) g gd%
n-+00
Combining this with (50) proves (47) Moreover, the last inequality can only be tight
/
| r' d W (ag. for p-a.e. y,
which implies Ty is a W2-optimal coupling between Cty and By for p-a.e. y. This proves Claim 2.
Step (OT-FM with augmented matching cost; then 7) + oc). Fix any conditioning value y (s, and recall that the offline flow satisfies
(, 10
#) = ay.
Let 21 vofff (ro. 1 11 with Po. that 01 O'y
(a) OT-FM population limit for a fixed n7. Fix 17 > O. Our OT-FM procedure builds minibatch OT matchings between joint samples (Wt.3;)
and (v') 2/)) ~ 3 using the augmented cost Gy. and then trains conditional map x + vo jo.k.. (x. 2 y) by regressing onto the matched z' values.
Let 7n E [[a, 3) be an optimal coupling attaining W (a, 3). Disintegrate Yn with respect to its first marginal ac
7o (dy, dz, dy', dx') a(dy, dr) r (dy, dz'),
where 17 y.z is probability kernel on y x X. Define the induced conditional law of x' given (y,z) by marginalizing out y/:
(dx') = / r (dy' dx').
The associated barycentric projection is
by (U. x)) = / x' An ag (dx')
30

---

Under assumptions (A1)-(A3) and OT-FM convergence guarantees (e.g., Theorem 4.2 of [47], applied with the quadratic cost the augmented space), training OT-FM to convergence with minibatch size k yields the population limit
[ Il....... (vr y) br((1x))l2] 0.
lim (gg,e) 40000
(52)
In particular, by conditioning on y. for paaae fixed y.
lim E Exsay
(z, v) (v, =0
(53)
(b) Send 1) - 0o (reduction to conditional OT). Take subsequence T/ - such that Yh. Toa as Step 3. Let by, be the barycentric projection induced by Th. as above. Since 8 has finite second moment, {bn,)w Jn uniformly bounded in L²(a). so by extracting further subsequence (not relabeled) we may assume
by Doo weakly in L (ax).
(54)
Because You is supported on {y = v/} and disintegrates as
(dy, dæ, dæ') = pa (dy) Ty (dx, dz'), Ty E 11( Wg-optimal,
the conditional law of æ' given (y. æ) under Yoo is supported on the conditional coupling "/-
Under assumption (A2), for μ-a.e. y, the optimal quadratic-cost coupling Ty is induced by Monge map Ty X - X' (i.e., x' = T,(x)) a.s.. Hence the limiting barycentric projection satisfies boo (v. = (x) for a-a.e. (v.
Therefore, for u-a.e.
E Exway ||x boo (y, x = E Exway || - (x)x (ay, py)
(55)
Finally, combining (53) with the identification dnx (U, (x) and 55 we obtain for μ-a.e. y,
lim lim Exway || v2 10000 k0000
(x, v) 2 = W3 (aky. By).
Substituting x = wi (so, 1 y) with vo po yields (48)
This completes the proof.
Intuition:
Why do we need matched (s, a) marginals?
When (s, a) is continuous, is unlikely to observe many online transitions &on with exactly the same conditioning pair (s,a), As result, the OT-FM objective must couple two of transitions whose conditioning variables differ across samples. To recover the conditional Wasserstein distance (2 (- s. Pon(- | a)) for fixed (s, we add (s, a) into the OT cost:
|| s' 112 7 soullz + ||ao aon 12)l))
As grows, transporting mass across different (s, a) becomes prohibitively expensive, so the optimal coupling is forced to match samples with similar (s, a) and transport occurs primarily in the next-state space.
This reduction requires the two empirical measures to share the same marginal distribution over (s, a); otherwise, the coupling is forced to move mass in (s, a) even when r) is large. In practice, ensure this by sampling (Soff, aoft) from the online dataset Don construct the offline-gennee -enerrttd batch (then sampling $ ~ Pon (- s, a) using the offline flow), while sampling online transitions (Son, Gon, snn) independently from Don-
J Additional Experimental Results
To further increase the task difficulty, we manually modified the benchmark configurations amplify the dynamic differences beyond the given settings. For example, in the morphology task, we drastically reduced thigh segment size-making the legs much shorter, the kinematic task, we almost immobilized the back-thigh joint by severely limiting its rotation range. As shown in Table 2 our method clearly outperforms all other baselines in out of settings, while achieving ties in the remaining one.
31

---

Datasct
Name
SAC
BC-SAC
H20
BC-VGDF
BC-PAR
Ours
MR
HallCheetah (Morphology)
1239 = 107
1896 357
1294 2: 133
1342 252
1967 4: 123
2267 :: 280
MR
HallCheetah (Kinematic)
2511 476
4275 129
3787 205
3821 187
3751 1: 113
4448 1: 227
M
HallChectah (Morphology)
1078 = 166
1199 181
1277 151
1014 1014::00 141
1263 + 163
1651 = 136
M
HallCheetah (Kinematic)
2381 = 392
4630 1 169
1372 = 81
1207 = 108
1901 =: 231
4651 1: 89
ME
HaltChectah (Morphology)
1115 207
1067 208
1034 206
1048 + 202
1196 + 203
1834 = 193
ME
HaltChectah (Kinematic)
18:13 1185
2630 = 117
3739 196
3:009 = 207
3108 + 500
1537 71
Table 2: Comparison of return under the extreme difficult settings. MR Medium Replay, M Medium, ME = Medium Expert. cell is green if the method has the highest mean and improves the second best by least 2%. Cells within 2% of the top mean marked in yellow.
K Experimental Details of Gym-MuJoCo
In this section, we describe the detailed experimental setup well the hyperparameter setup used in this work.
K.1
Environment Setting
K.1.1
Offline Dataset
We use the MuJoCo datasets from D4RL 16 as our offline data. These datasets are collected from continuous control environments in Gym 4. simulated using the MuloCo physics engine [62] We focus on three benchmark tasks: HalfCheetah, Hopper, and Walker2d, and evaluate across three dataset types: medium, medium-replay. and medium-expers
The medium datasets consist of trajectories generated by an SAC policy trained for 1M steps level of the medium agent.
and then early stopped.
The medium-replay datasets capture the replay buffer of policy trained to the performance
The cedium-expper datasets are formed by mixing equal proportions of medium and expert data (50-50).
K.1.2 Kinematic Shift Tasks
We use Kinematic Shift Tasks from the benchmark 38 We select most shift level 'hard' to make the tasks more challenging
HalfCheetah Kinematic Shift: The rotation range of the foot joint modified to be:
<joint axis="O 1 O" damping- "3" name="bfoot "bfoot" pos-"O 0" range="- "..g .157" stiffness
="120" "hiee="hinee>>
<joint axis="O 1 0" damping="1.8 name= pos="O O 0 o" range="-. .1" stiffness= "60" type="hinge"/>
Hopper Kinematic Shift: the rotation range of the foot joint is modified from [-45, 45 to (-9,9) 9]:
<joint axis-"O -1 0" name-"foot__joll pos-"O 0 0.1" range-"-g ".g 9a tyye-""inge""
Walker2D Kinematic Shift: the rotation range of the foot joint modified from [-45, 45] to [99.9]:
<joint axis="O -1 0" name="foot_joint" pos="O 0 0.1" range="-9 9" typee=hinge"/ />
<joint axis="0 -1 o" name= name="foot_lee feft__efft_jonnt" pos="O 0 0.1" range="-g 9" type= "hinge" />
K.1.3 Morphology Shift Tasks
We Morphology Shift Tasks from the benchmark 38 the tasks more challenging
We select most shift level 'hard' make
HalrCheetah Morphology Shift: the front thigh size and the back thigh size modified to be:
32

---

<geom fromto="O 0. -0.02" name="bthigh" size="0.044 type= capaule"/ppa
<body tame="fshin" pos="0.0 ~0.02">
<geon fromto="O 0 -.13 0 -.15" name="bshin' rgba="0. 0.9 0.6 0.6 1" size="0.046"
cype="capsule">>
</body>
<body name="bfoot" pos=".. -.15
<geon fromto="O 04 0 -0.05" name-"fthigh" size="0.048 046" type-"capsule/>>
</body>
<body name="fshin" pos="O -.04 -0.05">
<geom fromto="O
-.13" name="fahin" rgba="0.9 0.6 0.6 size="0.046"
capsulee'
</body>
<body name- "ffoot" pos-". .11 0 -.13//>
Hopper Morphology Shift: the foot size is revised to be 0.4 times of that within the source domain:
<geom friction-"2.0 fromto-"-O. 00.052 0 0.1 0. 104 0 0.1" name- wwwot_fooooooooeoe size- "0.024"
type="capsule"/>
Walker2D Morphology Shift: the leg size of the robot is revised to be 0.2 times of that in the source domain.
<geom friction="O.9 fromto="O 1.05 0 0 0.2" hame="thigh Stme="thigh_geom geom size="0.00 05" type=" capaule"/>
<joint axis="O -1 o" name="leg. jeiinng pos="O 0.2" range="-150 o" type="hinge"/
<geom friction="0.99 fromto="O 0.2 0.1" "leg--leeegeom size- "0.04" 04" type=" capsule"/>
<geom friction="O.9" fromto="O 1.05 0 0 0.2" name "thgh left left_geme geom" rgba="." .7 .3 1" size="0.055 05" type="capsule""/
<joint axis="O 0" lefttlleelleltoiinee pos="O 0.2" range= 150 o" type= "hinge"
/>
<geom Priction="O.99 fromto="O 0 0.2 size="0. 04" cape="capsule">>
0.1" name= left_geomm rgba=".7 .7 .3 .6 1"
K.1.4 Friction Shift Tasks
Following [38] the friction shift is implemented by altering the friction attribute in the geom elements. The frictional components adjusted to 5.0 times the frictional components in the offline environment. The following is an example for the Hopper robot.
Listing 1: Geometry Definitions for Walker2D
torso
<geom friction="4.5" fromto="O 1.45 0 0 1.05" name- ntmrs"t ttorroogeoo size="0.00 "5" type-" capaule"/>
# thigh <geom friction="4.5" fromto="O 1.005 0 0 0.6" Stme="thigh_eee geom" size="0.05* type="
capsule"/>
# leg <geom friction="4.5! fromto="O 0 0.6 0.1" name=" "oeg==eeegggoooo size="O. type=" capsule"/>
foot <geom "10..0 fromto="-O. 0. 0 0.1 0.26 0.1" name fooo__goonn size="0.06"
capsul'/"
K.2 Implementation Details
This baseline leverages both offline and online transitions for policy learning. Since learning from offline data requires conservatism while online data does not, we incorporate a behavior cloning term into the actor update of the SAC algorithm. Specifically, the critic is updated using
BC-SAC:
33

---

standard Bellman loss on the combined offline and online datasets, and the actor is optimized as:
ficctor X. (-xx(-18)
min Qs (8, ar log The (11) + (.aas,))DDan (-rp(())e) [(a
a)
(56)
where A
3 myy..ay) m (s),a;)) and w R is normalization coefficient. We train BC- SAC for 400K gradient steps, collecting online data every 10 steps. We use the hyperparameters recommended in (38)
H2O: H20 46 trains domain classifiers estimate dynamics gaps and uses them importance sampling weights during critic training. It also incorporates a CQL loss to encourage conservatism. Since the original H2O is designed for the Online-Offline setting, we adapt the objective to the Offline-Online setting, The critic loss is:
Leitie Ep.. [1Qs, (s, - y) + ED [w[s, a, s (Q), - -)
+ AcQu (Ba (11s) (w(s, (((s,,,s')Q.. (s, - ED,, a (s..aa)),, i (1,
(57)
where wss, a, 8') is the dynamics-based importance weight, and Seqr. is the penalty coefficient. We set AcQu. 10.0, which performs better than the default 0.01. We reproduce H2O using the official codebase 3 and adopt the suggested hyperparameters. H2O is trained for 40K environment steps, with 10 gradient updates per step.
BC-VGDE: BC-VGDF 655 filters offline transitions whose estimated values align closely with those from the online environment, It trains an ensemble of dynamics models to predict next states from raw state-action pairs under the online dynamics. Each predicted next state evaluated by the policy to obtain a value ensemble (&), (,) forming Gaussian distribution. A fixed percentage (5%) of offline transitions with the highest likelihood under this distribution are retained. The critic loss is:
Leiii = -E(..and) (8, -
=))²
+ +B(ssans'))Da
[ (A(s,,,
Aex) (s. - v) i {1,2),
(58)
where A(s, s') denotes the fictitious value proximity (FVP), and Ae% is the ξ-quantile threshold. VGDF also trains an exploration policy. Actor training includes a behavior cloning term, as in BC-SAC. We follow the official implementation/ use the recommended hyperparameters, and train for 40K environment steps with 10 gradient updates per step.
BC-PAR: BC-PAR 37 addresses dynamics mismatch through representation mismatch, measured as the deviation between the encoded source state-action pair and its next state. employs a state encoder fo and a state-action encoder ge, both trained the target domain. The encoder loss is:
C(v. k)
E(s,a,)) ((c(ff sG(f
(59)
where SG is the stop-gradien operator. Rewards of the offline data are adjusted as:
PPAR Far 3- llge (f a.n) (s(rr))
(60)
where 3 controls the penalty strength. The actor (TT)) and critic (Qc,) are jointly trained using both offline and online data. Actor training includes behavior cloning term, similar to BC-SAC. We implement BC-PAR using the official codebase 40K environment steps with 10 gradient updates per step.
Taaaaaaseaappaaraaaa adopt the suggested hyperparameters, and train for
COMPFLOW. When training the target flow, we use quadratic cost function and employ the Python Optimal Transport (POT) library [15) to compute the optimal transport plan for each minibatch using the exact solver. Additional hyperparameters are provided in Table 3 and Table Since the exploration bonus term is closely tied to properties of the environment- such as the state space, action space, and reward structure-it is expected that the optimal exploration strength 3 varies across tasks. We perform a sweep over 3 e {0.01, 0.1, 0.2), and select the offline data selection ratio 5% from (30%, 50%)
tttpe:://gichub.com//mmi
34

---

Hyperparameter
Value
Actor network architecture
(256, 256)
Critic network architecture
(256, 256)
Batch size
128
Learning
3 x 10-4
Optimizer
Adam
25
Discount factor (7)
0.99
Replay buffer size
106
Warmup steps
10s
Activation
ReLU
Target update rate
5 x 10-3
SAC temperature coefficient (a)
0.2
Maximum log standard deviation
2
Minimum log standard deviation
20
Normalization coefficient (w)
5
Table 3: Hyperparameters for RL training.
Hyperparameter
Offline Flow
Online Flow
Number of hidden layers
6
6
Hidden dimension
256
256
Activation
ReLU
ReLU
Batch size
1024
1024
ODE solver method
Euler
Euler
ODE solver steps
10
10
Training frequency
5000
Optimizer
Adam
Adam
Table 4: Hyperparameter setup for the offline and online flows
L Experimental Details of Wildlife Conservation
We use the green security simulator in 67 The model is Markov decision process with state, action, transitions, and terminal return. We summarize the parts needed to reproduce our experiments.
State and action. At time t, the state is
8: (a.-1, 1 1),
50 (0, wo. 0),
where w. = (wi,.. w is wildlife across cells and a; = ......... a is patrol effort. The defender chooses a, E [0, 1 under the budget Σ af < B.
Adversary behavior.
At each step, the poacher places a snare cell with probability
P = logistic 2² + Baff +
Σ df-1). JEN(()
where 2 is the baseline attractiveness of cell The parameters 3 < 0 and > 0 capture deterrence from prior patrol and displacement from neighboring patrols. The realized attack
ki Bernoull(pp)).
Wildlife transition. losses:
After attacks, wildlife in each cell evolves by natural growth and poaching
w = max 0, (wi-z) a ki_ a:)
where Φ >>>> 1 is the growth rate and a > 0 is the loss per uncovered attack. This defines the transition kernel T, over states given actions:
st+1 T((s,, oz).
35

---

Figure 7: Well-hidden snares and rangers conducting a patrol to locate them. Photos credit: Uganda Wildlife Authority
Return. The episode return is the total wildlife the horizon,
N
R(ST)
Σ
w
and the expected return of policy v under environment parameters is
r 2)
E[R(st)], $1+1
Ts(st, (ss)), so (0, wo. 0).
We assume access to an offline dataset of 100,000 transitions collected in Murchison Falls National Park using well-trained SAC policy. For the online environment in Queen Elizabeth National Park, we are allowed a budget of 40,000 interactions.
M Discussion on Computational Cost
The practical cost data filtering is relatively small. For example, with a batch size 256 for training the policy network and a Monte Carlo sample size of 30, the entire filtering process takes just 0.03 seconds on an A100 GPU. We will highlight in the paper that this efficiency is due to the simplicity of our flow training objective, which follows a linear path. As result, solving the corresponding ODE at inference time very easy-A A basic Euler method with just 10 time steps is sufficient.
36

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We propose new method for reinforcement learning with shifted-dynamics data based on flow matching.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of our method in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide all proofs for the theorems, propositions, and lemmas in the Appendix.
37

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all the experimental details in Appendix K
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We have provided the code in the paper.
Guidelines:
38

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.com cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results, See the NeurIPS code and data submission guidelines (https
/ //nips. /mips.cc/publl wwtpppplccgg.ciieooocoooccccccpe www.ddeeeooeeeeeeeeeeeeeee
ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the experimental details in Appendix K
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the standard deviation across different random seeds for all tasks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
39

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the computing infrastructure information has been provided Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform with the NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal
impacts in Appendix B
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
40

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all code, data, and models we used in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo/coaaooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
41

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information
supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We did use LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM hbtr:eee/eerrpssscc////feeeeeeee for what should should be described.
42


Reviews:
[{"review_id": "qNCZMuI7tA", "reviewer": "Reviewer_vMHa", "summary": "This paper aims to maximize the sample efficiency and performance of reinforcement learning when there are shifted dynamics between the offline data collected from a source environment and the target environment being currently learned. To this end, it presents a new framework that measures the shifted dynamics between the two environments more accurately and principledly, and utilizes it to perform data filtering and active exploration strategies.", "questions": "See the above weaknesses.", "limitations": "There are no negative societal impact.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: This paper proposes a novel framework called COMPFLOW for reinforcement learning problems that utilize offline data in dynamically heterogeneous environments. Here, it develops a robust measurement of dynamic gaps based on the Wasserstein distance by utilizing synthetic flow structures and optimal transport theory, and achieves state-of-the-art performance on 15 different RL benchmarks by performing data filtering and active exploration. It is a very interesting topic considering the theoretical approach and addressing dynamics gaps, and it can be very effective in the efficient utilization of replay buffers in RL.\n\nWeaknesses: In this paper, the proposed method is very effective in reducing the dynamic gap; however, it would be beneficial to provide more evidence for the practical costs of data filtering in the replay buffer and the performance improvement due to Large Dynamics Gap Exploration Reducing the Performance Gap (Theorem 3.3), although it is discussed in limitations of this draft. It is experimentally clear that strengthening exploration in areas where the dynamic gap is large helps improve performance, but this approach has the disadvantage of introducing an additional hyperparameter. If there is any way to decide the hyperparameter, it seems that it can help improve the quality of the paper.", "was_revised": true, "final_justification": "Like other reviewers, I feel that this paper is well structured, having good theoretical explanation. Besides, the rebuttal is good for my answer. So I change my score as accept. Thanks for this valuable paper.", "rebuttal": "We thank the reviewer for the positive feedback and for recognizing the value of our theoretical approach in addressing dynamics gaps. We appreciate the acknowledgment that this is an interesting and promising direction, particularly for improving the efficient utilization of replay buffers in reinforcement learning. We provide a detailed response for each question as below.\n\n**Q1. In this paper, the proposed method is very effective in reducing the dynamic gap; however, it would be beneficial to provide more evidence for the practical costs of data filtering in the replay buffer and the performance improvement due to Large Dynamics Gap Exploration Reducing the Performance Gap (Theorem 3.3), although it is discussed in limitations of this draft.**\n\nThe practical cost of data filtering is relatively small. For example, with a batch size of 256 for training the policy network and a Monte Carlo sample size of 30, the entire filtering process takes just 0.03 seconds on an A100 GPU. We will highlight in the paper that this efficiency is due to the simplicity of our flow training objective, which follows a linear path. As a result, solving the corresponding ODE at inference time is very easy—A basic Euler method with just 10 time steps is sufficient. The advantages of using a linear-path flow are also highlighted in (Liu et al., 2023).\n\nIn addition, recent studies—e.g., (Davtyan et al., 2025) —have proposed techniques to further accelerate inference in flow-based generative models by improving data-noise coupling. These developments offer promising directions for future research. \n\nFurthermore, Figure 6 in the appendix presents a hyper-parameter study of the exploration parameter $\\beta$. As shown, removing the exploration bonus degrades performance consistently.\n\n*Liu, Xingchao, and Chengyue Gong. \"Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.\" The Eleventh International Conference on Learning Representations. 2023*\n\n*Davtyan, Aram, et al. \"Faster inference of flow-based generative models via improved data-noise coupling.\" The Thirteenth International Conference on Learning Representations. 2025.*\n\n**Q2. It is experimentally clear that strengthening exploration in areas where the dynamic gap is large helps improve performance, but this approach has the disadvantage of introducing an additional hyperparameter. If there is any way to decide the hyperparameter, it seems that it can help improve the quality of the paper.**\n\n\nIn the future, we could explore automatically optimizing this parameter by drawing on techniques developed for entropy temperature tuning in the SAC framework—for example, the metagradient-based approach proposed in Meta-SAC (Wang and Ni, 2020). We will include this point in the discussion of limitations in our revised manuscript.\n\n*Wang, Yufei, and Tianwei Ni. \"Meta-sac: Auto-tune the entropy temperature of soft actor-critic via metagradient.\" arXiv preprint arXiv:2007.01932 (2020).*"}, {"review_id": "DA5RapHMYI", "reviewer": "Reviewer_V7jc", "summary": "The paper introduces COMPFLOW, a method for knowledge transfer in RL between a source and a target domains with different dynamics. The core contribution is a composite flow matching technique that leverages optimal transport to estimate the dynamics gap between the two domains using Wasserstein distance, where the main idea is to train the flow for the target dynamics on top of the flow for the source dynamics (ie with noise replaced by samples from the latter). COMPFLOW then uses Wasserstein distance to perform both data filtering and guide online exploration to regions with large gap. Empirically, the method is tested on 3 locomotion tasks from D4RL and compares favorably to existing baselines.", "questions": "1. Please address limitations above\n2. Following the discussion about the improvement of composite vs direct flow in the scarce-target-data setting after Theorem 3.1, could you provide intuition on why this is the case? If both source and target data are scarce, so the TV bound likely holds even if the two transition kernels may be very far, why the two FM models in composite flow would not \"bias\" each other?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles a relevant and well-motivated problem, as the setting in which we need to train policies on one domain with scarce data but have additional data under different dynamics frequently arises in practice (eg, in robotics)\n2. The proposed method is novel and sound. I particularly like the idea of \"chaining\" two flow-matching models as a tool to improve predictions and ease the estimation of Wasserstein distance\n3. The proposed method is backed by theoretical results that formally justify its main components: composite flow (Th. 3.1) and exploration bonus (Th. 3.3)\n4. The paper is very well-written and easy to read (apart from some specific details I could not grasp, see questions below)\n5. COMPFLOW performs well empirically. A good amount of ablations is provided\n\nWeaknesses:\n1. Flow-matching, while powerful, may introduce non-negligible computational/implementation overhead (eg estimating the Wasserstein distance requires multiple integrations of the ODE, which may be more inefficient than eg simpler density-based models). There is no discussion about that in the paper\n2. The experiments are carried out only on 3 locomotion domains from D4RL. Adding manipulation and/or navigation tasks (eg from the same suite) would add great value to the experiments\n3. A baseline that learns from scratch in the target domain is missing, which makes it hard to understand the benefits of transferring source samples in these domains\n4. The impact of the exploration parameter \\beta (Figure 5) seems very limited (curves mostly overlap). One wonders if that component is really important: would the performance drop significantly if exploration bonus is removed?\n5. While the paper is generally well written, there are a few theoretical details which are hard to grasp:\n   - Theorem 3.1: the MSE was not formally defined. Also, it is not clear under what conditions we should expect the TV bound assumption to hold\n   - Not clear how the optimal coupling q^* is computed in L_tar(\\phi)\n   - It seems the goal of the FM models is to learn conditional distributions s' | s,a, but many places (eg the paragraph 169 - 178) give the impression that these models should actually capture the joint distributions of triples (s,a,s'). Which one is correct?\n   - Theorem 3.3: what assumptions are made on the composite flow model to require that its estimated dynamics gap is below \\kappa? Is training assumed \"perfect\", so that this model returns the exact dynamics gap and not an estimate?\n6. A bunch of existing works for the problem considered in this paper use instance re-weighting rather than filter samples (see eg [1,2] and references therein). These are not discussed / compared empirically\n\n[1] Tirinzoni, A., Sessa, A., Pirotta, M., & Restelli, M. (2018, July). Importance weighted transfer of samples in reinforcement learning. In International Conference on Machine Learning (pp. 4936-4945). PMLR.\n[2] Xie, A., & Finn, C. (2022, November). Lifelong robotic reinforcement learning by retaining experiences. In Conference on Lifelong Learning Agents (pp. 838-855). PMLR.", "was_revised": true, "final_justification": "I have increased my score after reading the authors' response and other reviews since:\n- the authors addressed all concerned about writing clarity and promised to update the paper accordingly\n- the authors provided additional experiments which I find quite valuable for the final version", "rebuttal": "We thank the reviewer for their positive feedback, and for noting that the problem is well-motivated and the proposed method is both novel and theoretically sound. We also appreciate the recognition that our approach performs well empirically and is supported by a thorough ablation study. Below, we provide our responses to the questions.\n\n**Q1. Flow-matching, while powerful ... There is no discussion about that in the paper**\n\nFlow matching is indeed more complex than traditional density-based models. However, since we train a linear-path flow as defined in Eq. (1), the corresponding ODE can be efficiently solved using the simple Euler method with only 10 steps, resulting in minimal computational overhead. For instance, with a batch size of 256 for training the policy network and a Monte Carlo sample size of 30, the entire filtering process takes only 0.03 seconds on an A100 GPU. The advantages of using a linear-path flow are also highlighted in (Liu et al., 2023).\n\nIn addition, recent studies—e.g., (Davtyan et al., 2025) —have proposed techniques to further accelerate inference in flow-based generative models by improving data-noise coupling. These developments offer promising directions for future research. We will elaborate on this point in the limitations section.\n\n*Liu, Xingchao, and Chengyue Gong. \"Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.\" The Eleventh International Conference on Learning Representations. 2023*\n\n*Davtyan, Aram, et al. \"Faster inference of flow-based generative models via improved data-noise coupling.\" The Thirteenth International Conference on Learning Representations. 2025.*\n\n**Q2. The experiments are carried ... great value to the experiments.**\n\nWe conduct an additional manipulation experiment on the Pen task from the Adroit suite (Rajeswaran et al., 2017), focusing on two types of distribution shifts: Kinematic shift, introduced by modifying the rotation ranges of all finger joints, and Morphological shift, created by shrinking the sizes of the fingers.\n\nThe results are shown in the table below. As shown, our method achieves the highest reward on both tasks.\n\n| Method   | Kinematic | Morphology |\n|----------|------------------------|--------------------------|\n| SAC      | 43 ± 18                | 53 ± 14                  |\n| BC-SAC   | 73 ± 34                | 77 ± 25                  |\n| H2O      | 538 ± 162              | 191 ± 95                 |\n| BC_VGDF  | 115 ± 63               | 93 ± 31                  |\n| BC-PAR   | 53 ± 26                | 78 ± 19                  |\n| Ours     | **682 ± 108**          | **303 ± 67**             |\n\n*Rajeswaran, Aravind, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" arXiv preprint arXiv:1709.10087 (2017).*\n\n\n**Q3. A baseline that learns ... in these domains.**\n\nWe provide the performance of SAC trained from scratch in the target domain below. As expected, its reward is significantly lower than our method.\n\n| Dataset | Task Name                   | SAC              | Ours             | Highest                       |\n|---------|-----------------------------|------------------|------------------|-------------------------------|\n| MR  | HalfCheetah (Morphology)    | 1457 ± 89        | 3119 ± 107       | Ours                          |\n| MR  | HalfCheetah (Kinematic)     | 2255 ± 197       | 5189 ± 262       | Ours                          |\n| MR  | Hopper (Morphology)         | 364 ± 82         | 355 ± 6          | SAC                           |\n| MR  | Hopper (Kinematic)          | 737 ± 547        | 1024 ± 1         | 1025 ± 0 (H2O)                |\n| MR  | Walker2D (Morphology)       | 253 ± 60         | 1094 ± 791       | Ours                          |\n| M   | HalfCheetah (Morphology)    | 1467 ± 89        | 2282 ± 287       | Ours                          |\n| M   | HalfCheetah (Kinematic)     | 2316 ± 92        | 5593 ± 44        | Ours                          |\n| M   | Hopper (Morphology)         | 396 ± 60         | 604 ± 173        | Ours                          |\n| M   | Hopper (Kinematic)          | 724 ± 535        | 1023 ± 2         | Ours                          |\n| M   | Walker2D (Morphology)       | 301 ± 177        | 886 ± 372        | Ours                          |\n| ME  | HalfCheetah (Morphology)    | 1392 ± 238       | 1485 ± 67        | Ours                          |\n| ME  | HalfCheetah (Kinematic)     | 2323 ± 97        | 5750 ± 84        | Ours                          |\n| ME | Hopper (Morphology)         | 359 ± 75         | 462 ± 89         | Ours                          |\n| ME  | Hopper (Kinematic)          | 724 ± 535       | 1022 ± 2         | 1031 ± 3 (H2O)                |\n| ME  | Walker2D (Morphology)       | 228 ± 51         | 648 ± 180        | 1103 ± 444 (H2O)              |\n\n*MR = Medium Replay, M = Medium, ME = Medium Expert.*\n\n**Q4. The impact of the ... bonus is removed?**\n\nWe provide the rewards with and without the exploration bonus, along with 90% confidence intervals, in the table below. As shown, removing the exploration bonus leads to a clear performance drop in all 5 tasks.\n\n| Domain                             |  with best $\\beta>0$        | $\\beta=0$\n|-----------------------------------|-------------------------------|----------------------------------------------|\n| Hopper Medium (Friction)          | **295 ± 61**                  | 221 ± 3                                  |\n| Hopper Medium Replay (Friction)   | **275 ± 24**                  | 232 ± 4                                   |\n| Walker2d Medium Replay (Friction) | **579 ± 129**                 | 331 ± 5                                   |\n| HalfCheetah Medium (Morph)        | **2312 ± 326**                | 1944 ± 118                                |\n| Hopper Medium (Morph)             | **635 ± 147**                 | 423 ± 26                                 |\n\n\n**Q5.1. Theorem 3.1: the MSE ... TV bound assumption to hold.**\n\nLemma G.1 in Appendix G contains the formulation of MSE. We will include it in the main text. Thanks for the suggestion.\n\nThe TV bound assumption in Theorem 3.1 is more likely to hold when the number of target samples $N_{\\rm tar}$ is small. This aligns well with our setting, where we only have limited interactions with the target domain. We will make this point clearer in the revised manuscript.\n\n**Q5.2. Not clear how ... in L_tar(\\phi)**\n\nWe provide a detailed description of this computation process in Algorithm 3 of Appendix C. In brief, we sample a batch of source and target domain data, then solve for the optimal transport plan using the Python POT package. This transport plan is subsequently used to resample source–target data pairs. We will clarify this process more explicitly in the main text in our revised manuscript.\n\n**Q5.3. It seems the goal of the ... Which one is correct?**\n\nThe goal of the FM models is to learn the conditional distribution $p(s'|s, a)$. Lines 169–178 describe the construction of the cost matrix used for solving the mini-batch optimal coupling. As explained in lines 165–168, since flow matching is conditioned on $(s, a)$, but we do not have multiple target samples $s'$ for a fixed $(s, a)$, we include $(s, a)$ in the cost function following the approach in [17]. Nonetheless, the objective of the FM model remains learning $p(s'|s, a)$. We will clarify this distinction more clearly in the revised manuscript.\n\n**Q5.4. Theorem 3.3: what assumptions ... and not an estimate?**\n\nWe do not assume perfect training. Instead, Theorem 3.3 explicitly accounts for estimation error, as reflected by the terms following $-\\kappa$ in Equation (5). Our approach does not require the estimated dynamics gap to be smaller than $\\kappa$. Rather, both our algorithm and experiments operate directly on the estimated—potentially larger and noisy—dynamics gaps. Theorem 3.3 provides a formal analysis of how such estimation errors influence performance.\n\n**Q6.A bunch of existing works ...These are not discussed / compared empirically.**\n\n[1] is a relatively early work that was not designed for deep RL tasks. It models transition dynamics using Gaussian processes and computes importance weights based on the probability ratio between the source and target domains. A policy is then trained using decision trees. As a result, the method is limited to simple environments, making it difficult to apply to our deep RL tasks.\n\n[2] focuses on a lifelong learning setting and trains a classifier to estimate importance weights. However, as noted in page 6 of [2], the weights are ultimately used to filter out samples rather than reweight them, due to numerical instability. Training a classifier to estimate importance weights is also used in our baseline H2O, which our method outperforms.\nWe will include these references and the discussion in our revised manuscript.\n\n**Q7. Following the discussion ... \"bias\" each other?**\n\nWhen $N_{\\rm tar}$ is small, it is challenging to accurately learn the transition probabilities of the target domain for both composite and direct flow models due to limited data. With such a small sample size, the observed transitions may not adequately capture the true distribution $p(s'|s, a)$. In this case, when the transition kernels between the source and target domains differ significantly, the limited samples may fail to reveal those differences.\n\nHowever, since the source and target domains share the same state space, the source domain data still provide information of the support of that space. This allows the composite flow to leverage more information about the state space, offering an intuitive explanation for why it tends to perform better."}, {"review_id": "tv6jM7h1Dq", "reviewer": "Reviewer_fhWu", "summary": "The paper tackles the offline-to-online RL setting in which the available offline data come from environments with shifted dynamics. It introduces a novel method for quantifying the dynamics gap that combines a composite-flow model with the Wasserstein distance. The approach shows strong empirical performance on a set of simulated tasks and, in my view, offers a timely contribution to the field.", "questions": "- **Assumptions on the source data.** What coverage properties are required? How many source samples are needed for a given state–action pair?\n- **Pure online RL baseline.** To demonstrate that the source data do not hinder learning in the target environment, a comparison with a policy trained from scratch solely on target-environment data will be appreciated.", "limitations": "-na-", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The idea of using composite flow models and the Wasserstein metric to estimate the dynamics gap is novel.\n- The empirical evaluation is thorough; the proposed method consistently outperforms relevant baselines across most tasks.\n    \n    Weaknesses:\n- Wasserstein distances are approximated by solving discrete OT problems on mini-batches, which adds computational overhead and approximation error. The impact of mini-batch size is not analyzed.\n- **Theorem 3.1**\n    - The bound on the total-variation distance lacks explicit rate information: does it tighten as $N_{tar}$ increases?\n    - The bound can exceed 1, which makes it vacuous; please clarify the conditions under which it is non-trivial.\n    - Because the rest of the paper measures gaps with the Wasserstein distance, presenting this bound in the same metric would improve consistency.\n- **Theorem 3.3**\n    - Is $\\Delta_\\kappa$ always non-negative? If not, under what conditions is it positive?", "was_revised": true, "final_justification": "The authors addressed the concerns and they showcased publishable material at Neurips.", "rebuttal": "We thank the reviewer for recognizing the novelty of the proposed composite flow to estimate the dynamics gap and for acknowledging the thoroughness of our evaluation. Below, we provide detailed responses to the questions.\n\n**Q1. Wasserstein distances are approximated by solving discrete OT problems on mini-batches, which adds computational overhead and approximation error. The impact of mini-batch size is not analyzed.**\n\nThe OT problems are efficiently solved using the Python POT package, which provides a multi-threaded CPU implementation. For example, with a batch size of 1024, solving an OT problem takes us only 0.18 seconds. Importantly, OT is only required to be solved during the training phase of the target flow. Once training is complete, estimating the dynamics gap as in Eq.2 for a given $(s, a)$ pair using the trained flows no longer requires solving OT problems.\n\nAdditionally, we provide below the reward versus OT coupling batch size on the HalfCheetah-Medium replay environment across different shift tasks:\n\n| Batch Size | 8            | 256          | 512          | 1024         | 2048         |\n|------------|--------------|--------------|--------------|--------------|--------------|\n| Morphology | 2508 ± 83    | 2893 ± 152   | 3097 ± 123   | 3119 ± 107   | 3258 ± 119   |\n| Kinematic  | 4607 ± 323   | 4801 ± 265   | 4992 ± 354   | 5189 ± 262   | 5096 ± 287   |\n\nAs shown, performance stabilizes once the batch size exceeds 1024. Therefore, we use a batch size of 1024 for the OT coupling in all experiments.\n\n\n**Q2.1. The bound on the total-variation distance lacks explicit rate information: does it tighten as $N_{\\rm tar}$ increases?**\n\nYes, as $N_{\\rm tar}$​ increases, the bound becomes tighter, meaning that the requirement on the similarity between the source and target domains becomes stricter. This is intuitive: when abundant target domain data (large $N_{\\rm tar}$) is available, the benefit of incorporating source domain data diminishes, as external knowledge is no longer necessary. In our setting, however, we assume limited target domain data (small $N_{\\rm tar}$), making the bound easier to satisfy. We will clarify this point more explicitly in the revised manuscript.\n\n**Q2.2. The bound can exceed 1, which makes it vacuous; please clarify the conditions under which it is non-trivial.**\n\n\nWe have state variance in the numerator, which is bounded by the square of state norm bound ($\\zeta^2$) in the denominator. Therefore, these two terms together derive a value smaller than 1. The remaining terms left are the constant $C$ in the numerator and the square root of dataset sizes in the denominator. $C$ is a constant related to the 4-th moment of the state distribution, which depends on the target domain environment itself instead of the collected dataset. Hence, having both dataset sizes larger than $C^2$ is a sufficient condition to ensure that the bound does not exceed 1. We will include this discussion in our revised manuscript.\n\n\n**Q2.3. Because the rest of the paper measures gaps with the Wasserstein distance, presenting this bound in the same metric would improve consistency.**\n\nThanks for this nice suggestion. Indeed, Theorem 3.1 can be adapted using the Wasserstein distance. The condition for Wasserstein distance is the same as the condition for TV distance under the big O notation. The only difference lies in the constant factor, which is absorbed by the big-O. We will update the bound to use Wasserstein distance in our revised manuscript. \n\n\n**Q3. Is $\\Delta_{\\kappa}$ always non-negative? If not, under what conditions is it positive?**\n\n$\\Delta_{\\kappa}$ is generally non-negative when the estimation error of the dynamic gap or $\\kappa$ is small. Although the estimation error of the dynamics gap is inevitable—particularly when $N_{\\rm tar}$ is small—we mitigate its impact through a warm-up phase, as shown in Table 3 of the appendix, where data is collected without exploration to reduce initial large estimation error. After this warm-up phase, we can choose an appropriate value of $\\kappa$ to ensure that $\\Delta_{\\kappa}$ remains non-negative.\n\n\n**Q4. Assumptions on the source data. What coverage properties are required? How many source samples are needed for a given state–action pair?**\n\n*What coverage properties are required?*\n\nOffline RL algorithms typically require some concentration conditions for performance guarantee. A common assumption is the existence of a constant $C_\\star=\\sup_{s,a}\\frac{d_{\\pi_\\star}(s,a)}{\\mu(s,a)}<\\infty$, where $d_{\\pi_\\star}$ is the occupancy of the optimal policy $\\pi_\\star$ and $\\mu$ is the empirical state–action distribution of the dataset $D$.\n\nIn contrast, our method allows for online exploration in the target domain. Consequently, optimality no longer depends entirely on the coverage of the source data. With improved coverage, the composite flow produces more accurate estimates. Therefore, by Theorem 3.3, we obtain a tighter bound on the performance gap relative to the optimal policy. We will include this analysis in the revised manuscript.\n\n*How many source samples are needed for a given state–action pair?*\n\nTheoretically, when the number of source samples for a given state–action pair increases, the estimation of the dynamics gap will be more accurate. As a result, according to Theorem 3.3, the performance gap from the optimal policy will be reduced, indicating we get a better policy in the target domain.\n\nEmpirically, our tasks involve continuous state and action spaces, making it infeasible to obtain multiple samples for an exact state-action pair. To better understand the dataset’s coverage, we compute the number of samples falling within Euclidean balls (of varying radii) centered at the mean of (s, a) pairs.\n\n\n| Dataset                         | Distance < 2 | Distance < 2.5 | Distance < 3 | Distance < 3.5 | Distance < 4 | Distance < 4.5 | Distance < 6 | Distance < 8 |\n|---------------------------------|-------------:|---------------:|------------:|---------------:|------------:|---------------:|------------:|-------------:|\n| HalfCheetah-Medium              |            0 |              2 |          56 |            267 |         800 |           1828 |       10785 |        39325 |\n| HalfCheetah-Medium-Replay       |            0 |              2 |          22 |             99 |         457 |            805 |        3867 |        12455 |\n| HalfCheetah-Medium-Expert       |            0 |              0 |           0 |              8 |          74 |            369 |        5949 |        42164 |\n\n\n**Q5. Pure online RL baseline. To demonstrate that the source data do not hinder learning in the target environment, a comparison with a policy trained from scratch solely on target-environment data will be appreciated.**\n\nWe provide the performance of SAC trained from scratch in the target domain below. As expected, its reward is significantly lower than our method.\n\n| Dataset | Task Name                   | SAC              | Ours             | Highest                       |\n|---------|-----------------------------|------------------|------------------|-------------------------------|\n| MR  | HalfCheetah (Morphology)    | 1457 ± 89        | 3119 ± 107       | Ours                          |\n| MR  | HalfCheetah (Kinematic)     | 2255 ± 197       | 5189 ± 262       | Ours                          |\n| MR  | Hopper (Morphology)         | 364 ± 82         | 355 ± 6          | SAC                           |\n| MR  | Hopper (Kinematic)          | 737 ± 547        | 1024 ± 1         | 1025 ± 0 (H2O)                |\n| MR  | Walker2D (Morphology)       | 253 ± 60         | 1094 ± 791       | Ours                          |\n| M   | HalfCheetah (Morphology)    | 1467 ± 89        | 2282 ± 287       | Ours                          |\n| M   | HalfCheetah (Kinematic)     | 2316 ± 92        | 5593 ± 44        | Ours                          |\n| M   | Hopper (Morphology)         | 396 ± 60         | 604 ± 173        | Ours                          |\n| M   | Hopper (Kinematic)          | 724 ± 535        | 1023 ± 2         | Ours                          |\n| M   | Walker2D (Morphology)       | 301 ± 177        | 886 ± 372        | Ours                          |\n| ME  | HalfCheetah (Morphology)    | 1392 ± 238       | 1485 ± 67        | Ours                          |\n| ME  | HalfCheetah (Kinematic)     | 2323 ± 97        | 5750 ± 84        | Ours                          |\n| ME | Hopper (Morphology)         | 359 ± 75         | 462 ± 89         | Ours                          |\n| ME  | Hopper (Kinematic)          | 724 ± 535       | 1022 ± 2         | 1031 ± 3 (H2O)                |\n| ME  | Walker2D (Morphology)       | 228 ± 51         | 648 ± 180        | 1103 ± 444 (H2O)              |\n\n*MR = Medium Replay, M = Medium, ME = Medium Expert.*"}, {"review_id": "74qorjDizc", "reviewer": "Reviewer_DanK", "summary": "This paper proposes a method that leverages the source offline datasets to improve sample efficiency and generalization in the target environment. Specifically, the method uses optimal transport to calculate the dynamic gap between the source task and the target task, and then selectively expands the replay buffer of the target task for agent training.", "questions": "1. I suggest that the author specify the range of values for t in the source flow and target flow in Section 3.1.\n2. To address the issue of data scarcity (mentioned in line 136) in the target domain, the author proposes using a two-stage flow to model the target data distribution (line 148), namely, first flowing from the Gaussian prior to the source data distribution, and then from the target data distribution. However, the loss function from the source data distribution to the target data distribution is still based on a small amount of target data, and I believe the issue of data scarcity still persists.\n3. Why is the loss function of the target flow shown in line 174 inconsistent with the loss function in line 16, Algorithm 3?\n4. In Proposition 3.2, suppose both the source flow and target flow can perfectly model the flow map, according to the content introduced by the author in Section 2.3, the corresponding Wasserstein cost for this formula is $(s'_{tar} - s'_{src})^2$. However, the author has mentioned that the Wasserstein cost they intend to use is the cost shown in line 172. Therefore, the author needs to clarify this point.\n5. Essentially, Equation (2) measures how large the gap is between s' that the source dynamics and target dynamics transition to under the same conditions (s, a). Is this understanding correct?\n6. Theorem 3.3 is counterintuitive because, intuitively, selecting samples from the source data that are completely dissimilar to the target data will clearly hinder the model's generalization rather than improve it. Therefore, I am skeptical about the conclusion of Theorem 3.3.\n7. Since Theorem 3.3 mentions that 'exploration in regions with high dynamics gap can reduce the performance gap relative to the optimal policy,' why, in practice, do the authors choose the source data with the lowest gap?\n8. To fully demonstrate the generalization performance of the proposed method, I suggest that the author conduct tests on source and target tasks with larger dynamic differences, such as training on HalfCheetah and testing the generalization performance on Walker2D. Alternatively, the author could consider environments in Meta-World [1], which have the same state and action spaces.\n\n    [1] Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.\n9. I suggest that the author compare the performance differences between direct flow and composite flow in ablation study, rather than comparing the MSE of the loss, as the loss may be influenced by the values of each dimension in the state and action space.\n10. How is $v^*$ obtained in line 12 of Algorithm 1? Additionally, could the author explain the meaning of $A(i,j)$ and provide a detailed calculation of the optimal transport plan's solution process?\n11. Does the author intend to express that x represents a transition in line 10 of Algorithm 2? However, the v in the loss function provided here is an unconditional velocity field, while the source flow loss in Section 3.1 actually refers to a conditional velocity field.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Advantages: The idea presented in this paper is innovative and the theory is solid. \n\nDisadvantages: Some parts of the paper are vaguely expressed, which could cause confusion for readers. Please refer to the issues in the question section and improve the article accordingly.", "was_revised": true, "final_justification": "The authors have addressed all of my concerns", "rebuttal": "We thank the reviewer for acknowledging the novelty of our idea and the soundness of our theoretical contributions. However, we believe several confusions are caused by a misunderstanding regarding our framework, specifically Theorem 3.3: we do **not** add dissimilar source data to the replay buffer. Instead, we keep offline source data in low-gap regions and explore high-gap regions in the **target** domain, and Theorem 3.3 is consistent with implementation. We will clarify this in our revised manuscript. Below, we provide detailed responses to the questions.\n\n**Q1. I suggest that ... in Section 3.1.**\n\nThe time $t$ for the source flow is [0,1], and for the target flow is [1,2], as specified in lines 145 and 149, respectively. We will clarify this more clearly in the revised manuscript\n\n**Q2. To address the issue of data scarcity ... data scarcity still persists.**\n\nOur goal is to develop an algorithm that achieves better performance using only a small amount of data, compared to directly training a flow matching model from scratch. While the performance of composite flow also degrades when $N_{\\rm tar}$​ is small, our focus is on demonstrating **relative improvements** over the direct flow baseline, rather than claiming to fully solve the problem.\n\nBoth theoretically and empirically, we demonstrate that initializing from the source domain distribution provides a more informative prior than training the flow model from scratch. This initialization leverages prior knowledge from the source domain, resulting in improved performance when target data is limited.\n\n**Q3. Why is the loss function ... in line 16, Algorithm 3?**\n\nThank you for pointing out the typo in line 174. The right-hand side should be written as $(s_{\\rm tar}' - s_{\\rm src}')$.\n\nRegarding Algorithm 3, line 16: the variable $v^*$ is defined just before line 15, and the variables $x_1$ and $x_2$ correspond to $s_{\\rm src}'$ and $ s_{\\rm tar}'$, as defined in lines 5 and 8, respectively. Therefore, the two expressions are equivalent after fixing the typo.\n\nWe will correct this typo in our revised manuscript.\n\n**Q4. In Proposition 3.2, suppose ... Therefore, the author needs to clarify this point.**\n\nThe cost function in Section 2.3 is for flow matching without conditional variables (s, a in our case). As explained in lines 165–172, we are performing conditional variable flow matching for modeling $p(s'|s,a)$. Therefore, we use the cost function in line 172, which follows the formulation proposed in [17]. We will clarify this distinction more clearly in the revised manuscript.\n\n**Q5. Essentially, Equation (2) ... Is this understanding correct?**\n\nYes.\n\n**Q6. Theorem 3.3 is counterintuitive because, intuitively, selecting samples from the source data that are completely dissimilar to the target data will clearly hinder the model's generalization rather than improve it. Therefore, I am skeptical about the conclusion of Theorem 3.3.**\n\nWe would like to clarify that we do **not** add dissimilar source domain data to the replay buffer. As defined in Section 2.1, our setting assumes a **fixed offline source** dataset, and we do not perform any exploration in the source domain. \n\nTheorem 3.3 states that if we **retain** only **low**-dynamics-gap source data in the replay buffer and actively explore high-dynamics regions of the **target** domain to replace dissimilar source data, overall performance will improve. In this process, dissimilar source domain data is filtered out.\n\nThe benefit of exploring high-gap regions in the target domain is intuitive: since the replay buffer already includes source samples from low-gap regions, exploring high-gap areas in the target domain complements the dataset and enhances overall coverage.\n\nWe will clarify this more explicitly in our revised manuscript.\n\n\n**Q7. Since Theorem 3.3 mentions that 'exploration in regions with high dynamics gap can reduce the performance gap relative to the optimal policy,' why, in practice, do the authors choose the source data with the lowest gap?**\n\nWe believe this confusion is potentially caused by a misunderstanding of the statement of Theorem 3.3, which is clarified in Q6. \n\nTo summarize, the exploration of the high dynamics gap regions in Theorem 3.3 takes place in the **target** domain. The source domain consists of a fixed offline dataset, and **no** exploration is performed in the source domain. Our approach selectively uses offline source data in low-gap regions while actively exploring high-gap regions in the target domain. Therefore, the setup in Theorem 3.3 is fully consistent with our implementation.\n\nWe will make this clarification more explicit in our revised manuscript.\n\n**Q8. To fully demonstrate ... which have the same state and action spaces.**\n\nThank you for the suggestion. The tasks used in our paper are drawn from a public benchmark in this area [31], where each task includes three levels of difficulty. In our experiments, we already adopt the most challenging level, which corresponds to the largest dynamics gap within the benchmark.\n\nFollowing your suggestion to further increase the task difficulty, we manually modified the benchmark configurations to amplify the dynamic differences beyond the given settings. For example, in the morphology task, we drastically reduced thigh segment size—making the legs much shorter, in the kinematic task, we almost immobilized the back-thigh joint by severely limiting its rotation range. Our algorithm clearly outperforms all other baselines in 4 out of 5 settings, while achieving a tie in the remaining one. \n\n| Dataset | Task Name                | SAC           | BC_SAC       | H2O          | BC_VGDF      | BC_PAR       | Ours         |\n|---------|--------------------------|---------------|--------------|--------------|--------------|--------------|--------------|\n| MR      | HalfCheetah (Morphology) | 1239 (107)    | 1896 (357)   | 1294 (153)   | 1342 (252)   | 1967 (123)   | **2267 (280)**   |\n| MR      | HalfCheetah (Kinematic)  | 2511 (476)    | 4275 (129)   | 3787 (205)   | 3821 (187)   | 3751 (113)   | **4448 (227)**   |\n| M       | HalfCheetah (Morphology) | 1078 (165)    | 1199 (184)   | 1277 (154)   | 1014 (141)   | 1263 (163)   | **1654 (136)**   |\n| M       | HalfCheetah (Kinematic)  | 2331 (392)    | 4650 (169)   | 4572 (84)    | 4207 (108)   | 4304 (231)   | 4651 (89)    |\n| ME      | HalfCheetah (Morphology) | 1115 (207)    | 1067 (208)   | 1034 (206)   | 1048 (202)   | 1196 (203)   | **1834 (193)**   |\n| ME      | HalfCheetah (Kinematic)  | 1843 (1185)   | 2650 (447)   | 3739 (196)   | 3409 (207)   | 3108 (500)   | **4537 (74)**    |\n\n*MR = Medium-Replay, M = Medium, ME = Medium-Expert*\n\nRegarding your suggestion on HalfCheetah and Walker2D: while these tasks differ significantly in dynamics, they also differ in the physical meaning of their state and action spaces. Our method is designed for settings where the state and action spaces are consistent and only the transition dynamics vary. \n\nAs for Meta-World, although tasks have different goals and objects, they share the same underlying transition dynamics $p(s'|s,a)$, which differs from our setting where the source and target domains have distinct transition models. Our formulation (lines 74–80) specifically targets scenarios with varying dynamics while keeping the rest of the MDP structure fixed.\n\n\n**Q9. I suggest that the author ... in the state and action space.**\n\nWe report the reward values for Direct Flow and Composite Flow in the table below, using the medium offline dataset. As shown, Composite Flow consistently achieves lower reward compared to Direct Flow. Note that Direct Flow cannot compute the Wasserstein distance between the source and target transition probabilities using Proposition 1. Instead, we estimate the Wasserstein distance for a given $(s,a)$ pair by sampling 100 transitions from the source and target flow respectively.\n \n| Task             | CompFlow | DirectFlow |\n|------------------|------------------------|---------------------------|\n| HalfCheetah Morph | **2282 ± 287**             | 1538 ± 328                |\n| Hopper Morph      | **604 ± 173**              | 442 ± 65                 |\n| Walker2D Morph    | **886 ± 372**              | 532 ± 314                 |\n\n**Q10. How is $v^*$ obtained ... solution process?**\n\n$v^*$ is computed in line 10 as the difference between each pair of initial and terminal points.\n\n$A$ is the optimal transport plan, represented as a doubly stochastic matrix. Each entry $A(i,j)$ denotes the fraction of mass transported from source sample $i$ to target sample $j$. A larger value of $A(i,j)$ indicates a higher probability that the pair $(i,j)$ will be sampled together.\n\nTo compute $A$, we solve the following linear program:\n$$\\min_A \\sum_{i,j} A(i,j) \\cdot c(i,j) \\quad \\text{subject to} \\quad A \\mathbf{1} = \\mu, \\quad A^\\top \\mathbf{1} = \\nu, \\quad A(i,j) \\geq 0$$\nwhere $c(i,j)$ is the cost function, and $\\mu, \\nu$ are uniform distributions over the source and target samples, respectively. We implement this using the POT library. We will clarify this in the revised manuscript.\n\n\n**Q11. Does the author intend ... actually refers to a conditional velocity field.**\n\nYes, $x_t^{(i)}$ in line 10 is a linear interpolation between the initial point $x_0^{(i)}$ and the final point $x_1^{(i)} \\doteq s'^{(i)}$. The velocity field $v$ depends on the sampled $s'$, which itself is drawn from the conditional distribution $p(s'|s, a)$ given a specific $(s, a)$ pair. Therefore, although not explicitly written, the velocity field $v$ is indeed conditioned on $(s, a)$, making it a conditional velocity field.\n\nThe source flow training follows the standard framework of conditional variable flow matching as established in prior literatures—see, for example, Eq. (5) in (Park et al., 2025)\n\n*Seohong Park and Qiyang Li and Sergey Levine. ''Flow Q-learning.'' The Forty-second International Conference on Machine Learning, 2025.*"}]

Author rebuttal:

Meta-review: This paper represents a clear technical contribution with several distinguishing qualities:
* Methodological Innovation: The composite flow approach elegantly solves the disjoint support problem that plagues existing methods
* Theoretical Rigor: Solid theoretical foundations with practical algorithmic implications
* Empirical Strength: Consistent, significant improvements across diverse benchmarks
* Real-World Relevance: Addresses a fundamental challenge in transfer learning and domain adaptation for RL
Decision: Accept (spotlight)