Bridging Successor Measure and Online Policy Learning with Flow Matching-Based Representations
Authors: Haosen Shi
Jianda Chen
Sinno Jialin Pan
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
representation learning
flow matching
Primary area: reinforcement learning

Abstract: The Successor Measure (SM), a powerful method in reinforcement learning (RL), describes discounted future state distributions under a policy, and it has recently been studied using generative modeling techniques. Although SM is a powerful predictive object, it lacks compact representations tailored for online RL. To address this, we introduce Successor Flow Features (SF2), a representation learning framework that bridges SM estimation with policy optimization. SF2 leverages flow-matching generative models to approximate successor measures, while enforcing a structured linear decomposition into a time-invariant embedding and a time-dependent projection. This yields compact, policy-aware state-action features that integrate readily into standard off-policy algorithms like TD3 and SAC. Experiments on DeepMind Control Suite tasks show that SF2 improves sample efficiency and training stability compared to strong successor feature baselines. We attribute these gains to the compact representation induced by flow matching, which reduces compounding errors in long-horizon predictions. The code is available on https://github.com/Shiien/successor-flow-representation-implementation .

Published as a conference paper at ICLR 2026
BRIDGING SUCCESSOR MEASURE AND ONLINE POLICY LEARNING WITH FLOW MATCHING-BASED REPRESEN-
TATIONS
Haosen Shi', Jianda Chen², Sinno Jialin Pan!
'The Chinese University of Hong Kong, AAt International (haosen. shi aislink, sinnopane) cuhk, edu.nk jianda. chw..ant com
ABSTRACT
The Successor Measure (SM), powerful method in reinforcement learning (RL), describes discounted future state distributions under a policy, and has recently been studied using generative modeling techniques. Although SM is powerful predictive object, it lacks compact representations tailored for online RL. To ad- dress this, we introduce Successor Flow Features (SF*). a representation learning framework that bridges SM estimation with policy optimization. SF leverages flow-matching generative models to approximate successor measures, while en- forcing a structured linear decomposition into time-invariant embedding and a dime-dependent projection. This yields compact, policy- aware state-action features that integrate readily into standard off -oolcy algorithms like TD3 and SAC. Experi- ments on DeepMind Control Suite tasks show that SF² improves sample efficiency and training stability compared to strong successor feature baselines. We attribute these gains to the compact representation induced by flow matching, which reduces compounding errors in long-horizon predictions. The code available on
INTRODUCTION
key reason for the success of deep reinforcement learning (RL) in complex, sequential real- world tasks is its ability to learn meaningful representations automatically. This is often achieved through neural network architectures with specific inductive bias and efficient representation training algorithms. Effective representations generalize well across differrnnt observations and environments, give accurate value estimation, make efficient planning over long decision sequences, and achieve robustness when encountering new observed states (Kulkarni et al., 2016). However, how to find general and efficient representation learning method that does not rely entirely on reward functions but focuses on environment dynamics is a crucial problem, especially for complex tasks with a continuous state space, sparse reward, and long decision sequences.
Successor Representation (SR) (Dayan) 1993) offers promising approach by decoupling the reward function from environment dynamics. It captures the expected future state occupancy under given policy, offering a dual interpretation: can be viewed as a linear basis for state-action value functions, equivalently, as compact representation of aefinitehhorizon discounted visitation distributions. However, SR is inherently limited to discrete state spaces. To overcome this limitation, Successor Feature (SF) (Barreto et al. 2017) introduced, incorporating reward-relevan feature mapping along with Temporal Difference (TD)-style learning algorithm. A key challenge, however, remains the design of appropriate feature mapping, which still open problem (Ollivier 2025).
More recently, Successor Measure (SM) emerged as generalization of SR that directly models the discounted future state occupancy. Unlike SF, which relies on predefined features, SM describes distributions in principle in infinite-dimensional space and is typically estimated via generative models. Recent advances such as the geometric horizon model (GHM) (Thakoor et al. 2022). the y-model (Janner et al. 2020). and TDFlow (Farebrother et al.) 2025) utilize generative modeling and generalized TD learning to estimate SM, facilitating applications policy evaluation and transfer

---

Published as a conference paper at ICLR 2026
learning. In particular, TDFlow builds recent progress in generative modeling, specifically flow matching techniques (Lipman et al. 2023). enabling direct, simulation -reeee trajectory generation between distributions. This makes it highly efficient for continuous and high-dimensional settings. Furthermore, the mixed nature of flow matching aligns naturally with TD-style training.
The stability and efficiency of flow matching make it exceptionally well-suited for modeling SM, it mitigates compounding errors over long horizons and enhances scalability in complex environments. However, online RL imposes stricter demands, requiring representations that not only retain the predictive power of SM but also adapt quickly to newly observed experiences. These requirements motivate a new framework that unifies the robust long-term forecasting of SM. the stable and efficient generative learning of flow matching, and the rapid adaptability essential for effective online RI. In this paper, we introduce Successor Flow Features (SF²), new framework that leverages flow matching to approximate SM. SF² enforces structured decomposition into a time- invariant low-dimensional embedding of state-action pairs and a time-dependent projection operator. This design offers several key advantages: (i) the time-invariant embedding is tailored for online RL, enabling joint training with value functions and seamless integration into existing algorithms like TD3 (Fujimoto et al. 2018) and SAC (Haarnoja al. 2018); (ii) the dime-dependen projection enables generative models to reconstruct SM while decoupling policy-dependent and environmental structure. We evaluate 2F2 by integrating it into TD3 and SAC on continuous-control benchmarks from Mujoco Playground (Zakka et al. 2025). Our results demonstrate improved average performance over standard basclines. While preliminary, these findings indicate that combining flow-based generative modeling with successor representations is highly promising direction for scalable RL representation learning. Our contributions are threefold:
We propose a generative model with linear projection structure to approximate SM via flow matching.
We introduce SF², an informative representation for online RL that is trained jointly with value functions.
We provide empirical evaluation challenging continuous-control tasks, together with diagnostic studies of bootstrapped flow matching.
To the best of our knowledge, SF² is the first approach to explicitly integrate successor measures with policy optimization for online RL representation learning. We emphasize that SF² is an initial step in this direction rather than a complete solution: rigorous theoretical guarantees and broader empirical validation remain important open challenges.
2 PRELIMINARY
In this paper, we use uppercase serif fonts to denote a set S, (S) to denote the space of probability measures over a measurable set S, uppercase capital letters to denote random variables (e.g., S) and R" to denote the n-ddimensional real space.
Markoy Decision Process We consider a discounted Markov decision process M = PS, which includes the state space S, discrete or continuous action space A, transition kemnll P x A + (S),, reward function r S x A + R, and discount factor E [0, 1). Following the setting of Blier al.] (2021), the state space is measurable (either continuous or discrete). In an MDP, an agent interacts with the environment by observing the current state E S, selecting an action a; A according to policy æ, and then receiving a reward r(st, a)) while transitioning P(-(st, az). The objective of reinforcement learning algorithms is to find to a new state St+1 (A) that maximizes the expected cumulative reward, or the value function a policy π : (*((s = Ex D
- 7 7rr(s,, a,) so s) for any state s E S, where Ex denotes expectation under the distribution induced by policy स interacting with the MDP. The value function satisfies the Bellman equation: (*( EacA ([lls) [r(s, a) + Eves P(s"|s, as ())")
Flow Matching Flow Matching (FM) (Lipman et al. 2023 is a technique used in generative modeling to learn mappings between distributions. Define time- dependent diffeomorphic map Ó day, k() R x J0, 1] R" governed by an Ordinary Differential Equation (ODE): dk e( k) with time & E 1). where 2g := (zo). We use the notation rather than t for the ODE's time parameter distinguish it from the timestep / used in MDPs. A Continuous Normalizing Flow, one
2

---

Published as a conference paper at ICLR 2026
kind of Neural Ordinary Differential Equations (Chen al. 2018), employed to parameterize the vector field V/2 = (((,,)) R x [0, R as (. k) and determines the flow dynamics.
To find the training target of the parameterized dime-dependent vector field, FM introduces mix- ture representation approach to estimate the marginal vector field as a mixture of conditional vec- tor fields that condition each data point from Purget- For example, given a prior distribution Po(z) = (0, I,) and a sampled data point 21 from the target distribution p1 (z) = Purger (2), FM constructs conditional probability paths py (z( = (x; (4 (21 [n). The correspond- ing conditional üime-depender diffeommrphic map and the conditional vector field are given by (1, = (x(x))x + (x1) and déu (2,21) respectively. The training objective
dk
| (c((((E)21)) k-
do,(e, x1)
(())
dk
where ((, 1) represents the uniform distribution over the interval [0, 1]. This objective minimizes the squared difference between the parameterized vector field and the target vector field at randomly sampled time points, states, and noise values. The time parameter A uniformly sampled to ensure the model learns the entire trajectory from the prior distribution to the target distribution.
Samples generated by solving 21 In + So (1 k)dk where zo Po with standard ODE solvers (Gautschi] 2011). When additional conditions imposed as for the vector fields us (z. ks c) and the sampled data conforms to the distribution x ~ q(Icc), the FM framework is also capable of constructing flexible conditional generative models.
Successor Measure The SM (Blier et al. 2021) is probability distribution over states that captures the expected discounted future state visitations under given policy, a transition kernel, and state- action pair. Formally, for policy म, the SM μ² (X|s, a) represents the probability of visiting state E X C S when starting from state-action pair (s, a) and following policy æ, with geometric discounting:
|r = (1 )
Σ Et 80 = 8, ag
where the expectation is taken over all possible trajectories generated by starting at state s, taking action a, and then following policy ı for all subsequent steps. The indicator function 1 equals 1 when the state at time t belongs to the set and otherwise.
Similar to the Bellman equation for the value function, the SM satisfies the Bellman equation Blier et al. 2021
( = =1- + Σ P(s'ss, a)
Σ (dd1ss)) d'). d'EA
(1)
e'ES
This recursive formulation reveals that the SM can be interpreted as mixture distribution between the immediate state distribution induced by the transition kernel (with weight --) and the bootstrapped future state distribution (with weight 7). The conditional generative models can be employed to predict the SM utilizing the recursive form equation I In general, the learning objective is formulated as maximum likelihood estimation problem, which aims to find the optimal generative model by solving:
max ((Ix)) (l (MM
(1s (1w.a)) μ (X|s,
(2)
While this objective provides general framework, the specific loss function needs to be adapted according to the choice of generative model. For instance, in the Geometric Horizon Model (GHM) (Thakoor et al.] 2022) (also referred to as the 7-model (Janner et al.) (020 different implementations employ distinct training losses, such those based VAE (Kingma & Welling 2013) or GAN (Goodfellow et al. 2014). In this work, we focus on the best pesst-erforming variant, the flow matching used in TDFlow (Farebrother et al. 2025). which utilizes modified flow matching loss. We will elaborate on its details in the next section. further discussion about the benefits of using flow matching for SM through explicit mixture viewpoint can be found in Appendix A.2
3

---

Published as a conference paper at ICLR 2026
3 SUCCESSOR FLOW FEATURE
3.1 FLOW MATCHING FOR SUCCESSOR MEASURE LEARNING
The SM's mixture structure is particularly well-suited for flow matching approaches, allowing us to directly model the interpolation between immediate transitions and future state distributions. For learning 1 (s'|s, a) with its corresponding parameterized time-dependent vector field us (2, k 8, a), utilize its natural mixture representation:
n (s' |s a) (1 P(s (s((ss a) +
(s (s'18", aa",
which yields the corresponding FM training objective for the parameterized vector field (ar 8,0) on given tuple (s, a, s'):
Lrow (0) =(1 - pp) + Loooessrapping (0)
/ )") (e)
|
=(1 - -1)))
dk
( Se) k,), dønle, se) 2
+ (W
(3)
dk
where (0) is used to learn the transition distribution and Lrootstrappngg (0) uses a temporal difference form where bootstrapping distributions induced by uo(-,,, s', a') serve nº (-|s', , a'). However, this sampling procedure introduces substantial computational costs due to the need for multiple network evaluations during the generation process.
To enhance training efficiency, we leverage the fact that sampling from the SM can be achieved through an ODE solver(Euler method as an example): s, Mo(-|s,a a) is equivalent s. Fuler(c, uo( a)) where ~ (0, 7.). Motivated by the TD2-CFM loss formulation from (Fare brother et al. 2025), we directly align the vector fields conditioned on different state-action pairs at the same noise level instead of generating full successor states and then comparing them. This approximation avoids expensive ODE integration while preserving the consistency between local flow directions. Intuitively, if two vector fields agree on their evolution, their generated distributions will also agree.
Chootstrapping (0)
~
ckk''s((1s"" Jh OD (a))
[1uo(tres ki 8, a) - uo (rk, ki 8 "",'"')l' "²)""]
This approach aligns the vector fields conditioned on different state-action pairs at the same noise level, eliminating the need fully generate denoised states and then apply the Φs transformation. This form substantially reduces the need for small integration steps in the ODE solver, decreasing computational overhead while maintaining performance. We provide detailed analysis of the trade-off between computational efficiency and model performance in our ablation studies presented in Section 4.3 It is noted that the final loss used for learning is based on expectation over transition tuples from the current policy и. We model conditional vector fields on the latent space induced by the flow parameterization (Section 2 This avoids explicit density ratios on and ensures that training targets remain well-defined even when the SM is provided implicitly via pushforwards.
3.2 SUCCESSOR FLOw FEATURE FROM ESTIMATED SUCCESSOR MEASURE
Following the estimation of the SM, we employ linear projection formulation to derive a compact feature representation.
Definition 3.1 (Successor Flow Feature) We define the Successor Flow Feature (SF²) on state- action pair (s, a) as the output of the mapping i6 Rding x R + Rd, which generates the time- dependent conditional vector field u(s', k. s, a) as linear projection with a cime-conditioned matrix field Ç Rdim dims x [0, 1] R Rdddimg,
u(s', k. 8, a) C(s', (ss ((s,,))
where (8, a) is fime-invariant and captures the sufficient dimension reduction property.
4

---

Published as a conference paper at ICLR 2026
In contrast to conventional conditional generative models that combine conditions, timestamps, and noised imputs through complex non-linear transformations, our approach employs ame-invarian feature (b, a.) that interacts only the final stage with the matrix field Ç. This architectural choice promotes the encoding of temporal structures within Ç that are essential for effective downstream representation learning. This linear projection approach has been explored in prior work (Shribak) et al. 2024). which extracts spectral features from environmental transition dynamics to enhance reinforcement learning performance.
The representation function (888, a) achieves the Sufficient Dimension Reduction (SDR) (uuuuiiu al. 2009 by establishing conditional independence Se TL (s. a) (sss, a). i.. p ( (,)
4 (se ((s thereby ensuring that the extracted representations comprehensively capture all relevant information about how state-action pairs relate to successor states. Additionally, this formulation exhibits universal approximation properties (Sasaki & Hyvärinen 2018), enabling it to theoretically approximate any target function arbitrary accuracy, which makes particularly effective for modeling SM across diverse policies and environments.
(x.a)
((A,a) Rd
ODE: No(Py, A.0.) Samples from (+)a,o)
des dk
(41.))
((d,k) geboames)
Figure I: Schematic visualization of how v, S, and u interact. (s,,, a) encodes the representation of the current sate-action pair. (s())) provides time- varying projection over future states s'. Their inner product defines the vector field u(s', k. a), which describes how future-state probability mass evolves in the flow-matching objective.
flow
3.3 CONNECTION TO SUCCESSOR REPRESENTATION AND DIFFUSION SPECTRAL REPRESENTATION
Connection to Successor Representation Let's consider the one-step gradient updating the parameterized & neural network with parameters using equation with equation 4 Junder a transition tuple (s, a, s') and sampled & ((,,, 1), (.. When K let (e( kr + k)e, (e.z). x - e, we have: dk
Lnow (how(0)) (1sss (1-k),, ()) (L)), (s'-))) -) 1|<(2,, as, k) Ts ad)l?
As approaches O, we can make the approximation:
<(ks' + (1- k)), x Ç(e, 0), and <(*,, k) R (),,
The intermediate point Fę is approximately obtained through single ODE transformation step:
3 + kCte, (((ss,
Substituting these approximations into the loss function yields the semi-gradient, where stop the gradient backpropagation on the bootstrapped target. We have
VoL = 2 (11 7) (5(e,0) (s. (s' -c) + (S(e, ((ss,))ss))))))))))))))))( - ((cxa)-((e,))) ( (o)(s,
which can be rewritten more concisely as:
VeL ~ 2) - (11-)) (s'-)( + ((((), ())
This formulation reveals a temporal difference learning structure where the target combines: (1) direct supervision component (1 -s' - e) representing immediate information. (2) A discounted bootstrapped component ru(s', a') <(e, 0) that propagates future representations.
Rearranging into Bellman-lik equation, we have
(s, (1 - ()(( ()) - + (')6
5

---

Published as a conference paper at ICLR 2026
where (-)+ denotes the Moore-Penrose pseudoinverse. This formulation reveals that our ap- proach learns a Successor Representation with Dayan's definition (Dayan 1993). In our case, (1 - (L-,)) )+ (s' <) serves as the basic feature that captures immediate transitions, while the recursive structure (s),a) (1 - (immediatee feature) + "8b(8' ; a') serves as the bootstrapped part. The process incorporates novel element where the next state s' undergoes Gaussian noise perturbation before being projected onto the column space defined by (C(e, This can be interpreted as learning basis for the state space that is robust to perturbations, enabling more effective representation of the expected future state occupancy distribution. This yields an SR-like recursion on & under fixed Ç. We emphasize this is an approximation to motivate design; we do not claim exact equivalence to SR. The exploration of how the representation behaves and what properties captures when takes values significantly away from remains an open question for future investigation.
Connection to Diffusion Spectral Representation (Shribak et al., 2024) As y approaches zero, approach bears resemblance to Diffusion Spectral Representation (Shribak al. 2024), which employs diffusion models (Song et al. (02I) rather than flow matching and targets transition probabil- ities instead of SM. The incorporation of future state transitions enables features encode transition dynamics across extended time horizons. In Section 4 we conduct empirical comparisons between SF² and its variant with ? performance compared to approaches that focus solely on short-term transition prediction.
demonstrate that SF² achieves better area-uns urea-under-ous CudUr-))rrv(AUUC)
3.4 PRACTICAL POLICY OPTIMIZATION WITH FLOW SUCCESSOR REPRESENTATION
In this paper, we consider combining the proposed representation learning method with standard online reinforcement learning algorithms continuous action spaces. We choose SAC (Haarnoja ict al., 2018) and TD3 (Fujimoto et al.J 2018) as base algorithms. The learned representation is only used for building the actennctionn value function (1hass, a)). And the policy will be implicitly influenced through the VaQ(Vo(s, a)).
To enhance learning stability and performance, implement two complementary techniques:
Value Alignment: We augment the flow-matching objective with value prediction component.Cooal Lnow + A.valuu. where À controls the relative weight of value prediction. The value loss follows the standard temporal difference formulation:
Cyllue ((a,,,,,') (Q(vb(s.a)) (r max Q(vo(s' ,')))
This approach is compatible with various RL algorithms and can incorporate techniques such as double Q-learning (Van Hasselt et al. 2016) for improved target estimation.
Generative Model Smoothing: We employ exponential moving average (EMA) target networks that update pa- rameters according
Algorithm Training se within Off-Policy RL
(state, action, next state, next action) tuple (s, a, s', a'), networks (1. v/, ,'), and target for value learning y'. which depends on the base algorithm
Input:
Ogr (1
T 7) + TŌp and 0g. = (1 -) + T0; during bootstrapping phases, con- sistent with established flow match- ing training practices (Lipman al.. 2023). We perform an ablation analy- sis of the effectiveness of this moving average coefficient in Section 4.3 Fur- thermore, the EMA-updated parame- ters 8, also work in the target network for the estimation of the value func- tion, providing an additional layer of stability to learning dynamics. The overall training objective Lioual for learning SF², when embedded in off- policy RL, is shown Algorithm For the TD3-based methods,
2: Sample ~ (O, I), k 1))
3: 8t = k. s' e E k)), Starget = 5'
4: Compute features and next state loss:
5: Lnow = =|yi(s, a)"'(a))s" ((s,, k) Starget
6: Generate state using numerical integration, start with
2 =
7: kistares kiend 0,
8: kimid = - [kistart + kied)
9: dr = (' "'('''''TTS''( (') (50 ('',aa'''' C' kstart): ), kimdd)
10: x = x (Kend - ketarss
11: Compute generation loss:
12: Croossurapping = =1,((,,)) aa. (lc,kk) (/ ad'""
13: Lvalue (Q(1b(s,)) - -y//²
14: Return Liotal = 7) + 7 boootstrapping + XLraatuc

---

Published as a conference paper at ICLR 2026
mi Q2). where Qi and Q2 the target Q-networks evaluated at the next state s' and
next action a' sampled from the target policy. For the SAC-based method, y = min(Q' Q2) - log 57 where is the temperature parameter that determines the trade-off between maximiz- ing expected reward and entropy, and is updated according to the original SAC paper (Haarnoja et al.
2018).
4 EMPIRICAL EVALUATION
4.1 EXPERIMENTAL SETUP
We implemented all experiments using JAX (Bradbury et al., 2018) and Deepmind Haiku (Hennigan 2020 to leverage hardware acceleration. For the DeepMind Control Suite Tunyasuuuuakkoo
et al.
al. 2020). we utilized GPU-accelerated version, the MuJoCo Playground (Zakka 2025). All aggortthms were edited from their respective implementations in the Brax library (Freeman et al.
2021). Each experiment conducted on single NVIDIA GeForce RTX 4090 GPU. Detailed
architectural specifications, hyperparameter configurations, and environment-speccfic parameters are provided in Appendix G
Flow sampling uses Fuler integration with 2 function evaluations (NFEs) unless otherwise noted; we sample ~ 24(0, 1) and base noise E ~ ~V(O I), and condition vector fields on a) and (s', specified in Section 3 We report wall-clock time for representative settings (Section 4.3 and keep baseline parameter counts comparable by reusing encoder widths and depths across methods (Appendix G
4.2 EXPERIMENTS ON CONTINUOUS ACTION SPACES WITH OFF-POLICY LEARNING
We evaluate SF2 on seven diverse tasks from the DeepMind Control Suite (Tunyasuvunakool et al. 2020). selected to represent range of challenges in dynamics complexity, reward struc- ture, and control difficulty. We integrate our approach with two commonly used off-policy algorithms: SAC (Haarnoja al. 2018 and TD3 (Fujimoto et al., 2018). For comparison with the SF method, we also include Chua et al. (2024). J. strong SF method designed for the online RL setting. Their work reports substantial improvements over prior SF approaches. To provide fair and rigorous compari- son, we implemented four baseline methods: TD3Sim/SACSSm, which closely follows the approach described in the Chua et al.] (2024) with the TD3/SAC algorithm, and TD3SimLap/SAACSim... which removes the Q-function alignment constraint and incorporates orthogonality objective for feature learning via the graph Laplacian. All methods use identical neural network architectures, the same number of environment steps, and the same number of parallel environments as our proposed approach.
Following the suggestions described in Patterson et al. 2024), we summarize learning dynamics and aggregate performance across seven environments using 15 random seeds for each algorithm variant. For each algorithm variant, extract evaluation reward curves (mean of episodic returns over evaluation steps using 1000 trajectories) and compute the Orea-Unner--uuns AUC is normalized per environment by linearly scaling to [0, 1] using that environment's minimum and maximum in order to compare across different environments. For each algorithm family (SAC or TD3), the aggregate panels report the median, interquartile mean (IQM; the mean between the 25th and 75th percentiles), mean, and optimality gap (one minus the mean), with 95th percentile confidence intervals computed using 5000 samples over normalized scores. The results in Figure 2 reveal that incorporating SF? enhances the performance of TD3 and SAC across most environments over their standard versions. with greater improvements when using the full successor version (? 0.99). Our method consistently outperforms both baseline algorithms and the transition version (7 0.0). demonstrating the importance of incorporating a longer temporal horizon representation learning. Full per-environment learning curves for both TD3 and SAC variants are deferred to Appendix C. (Figure 7).
(AUC). Then the
Interestingly, the performance gains (compared with baseline) are more pronounced for TD3 than for SAC, suggesting that our method may be particularly beneficial for algorithms that struggle with exploration or representation learning, and deterministic policy may improve the training efficiency for the online successor measure learning. Additionally, the reduced standard deviations in many cases indicate that S not only improves performance but also enhances stability. Approaches based
7

---

Published as a conference paper at ICLR 2026
TD3 frmm Performance normallzed
Median
Mean
Optimaliyy
mm
T33 2.0
PATTAIOON SCOrES
ty Aggrepate Performance enni, nottriizeed AUC)
Optimality
566-671
$45.594
340 Lap
80
0.2
2.4
3.5
0.3
1.0
0.0
02
0.4
0.6
0.0
3.8
0.8
0.4
2.6
0.
3.5
0.
0.4
0.5
0.9
Figure 2: IQM performance
DeepMind Control tasks. Panels summarize the aggregate AUC for TD3 (upper) and SAC (lower) variants, comparing vanilla baselines, SF-based baselines, and our SF² with transition (7 0.0) and successor (7 0.99) horizons.
on SF tend to struggle on sparse reward tasks based on TD3 experiments, as the majority of transitions yield reward of 0.O, making it difficult effectively learn the task weight In contrast, method does not depend on this mechanism and thus avoids the associated performance degradation observed in such sparse-reward settings.
4.3 HYPERPARAMETER ANALYSIS
We examine the influence of three key hyperparameters in our method on the AcrobotSwingup task: exponential moving average (EMA) coefficient, number of denoising steps, and feature size. We use the mean episode return over the final 50k steps to verify.
(a) Analysis of exponential moving average coefficient with varying numbers with varying numbers on si networks. T on performance.
(b) Performance evaluation (c) Training time evalua- (d) Influence of feature size
denoising steps.
denoising steps.
Figure 3: Systematic analysis of hyperparameter effects 2 performance. (a) Exponential moving average coefficient - demonstrating stability- aubilit--erffrmancee trade-offs, (b) performance sensitivity denoising step count in the sampling procedure, (c) computational cost scaling with respect to denoising steps, and (d) feature size effects on algorithm performance.
EMA Parameter. Figure Sa demonstrates that the EMA coefficient 7 inversely correlates with performance. Peak results oceur
0.1 for TD3 and 0.01 for SAC, with performance
8

---

Published as a conference paper at ICLR 2026
declining 7 approaches 1.0. Despite considerable variance across configurations, these findings suggest that more stable target network updates (smaller т values) enhance learning dynamics in our framework. The results highlight the critical importance of proper calibration, large values can substantially impair policy effectiveness.
Denoising Steps. Our analysis reveals that varying the number of denoising steps produces compa- rable performance outcomes (Figure(3b) though computational costs increase proportionally with more steps (Figure 3c). We observe that even with minimal denoising steps (1-2), both algorithms maintain robust performance. suggesting that aligning the bootstrapping part can rely on a rough sampling process without requiring extensive iterative refinement. The computational efficiency analysis in Figure 3e further confirms that small number of denoising steps provides an optimal balance between performance and computational overhead, as the default choice in experiments.
Feature Size. As shown in Figure 3d the two algorithms respond differently to feature size changes. SAC with SF² demonstrates similar final 50k steps returns across various feature sizes with low variance, suggesting effective representation learning even in reduced dimensions. TD3 tends to benefit from larger feature sizes, though variance increases, while SAC remains stable across sizes. This indicates that the deterministic policy gradient method particularly benefits from richer feature representations.
4.4 COMPUTATION COMPLEXITY
The additional time overhead of our method due to the additional representation learning and the additional neural network. Here we use the comparison between TD3 and TD3 with SF² as an illustration, Each standard TD3 update step consists of one step of critic updating and one step of actor updating, which is consistent with TD3 with SF². The main distinction is that an additional feature representation update step: for each update step, TD3 with SF performs seven forward passes and one backward pass through the network, and two forward passes and one backward pass through the v network. Under identical experimental conditions, the original TD3 method's s running time AcrobotSwingup is 659 seconds, while, as reported in Figure 3c| SF² with denoising step takes approximately 1300 seconds, about twice as long. Our experiments demonstrate significant improvements in downstream performance, jurtormncee.ustffiinng this trade-off. We will further optimize training costs in future work.
RELATED WORK
Representation learning in RL. Reconstruction based methods have been employed for feature
extraction from observations (Hafner et al. 2019 Yarats et al. (202T). Contrastive learning techniques
have emerged as a powerful paradigm for learning discriminative state representations (Laskin et al.)
2020: Stooke et al. 2021 Zheng al. 2023). bisimulation metrics offer more formal approach
to learning state abstractions by grouping behaviorally equivalent states (Zhang et al., 2021 Castro
ct al. 202 2023). World models learn to capture the environment's 's dynamics,, allowing agents to plan or learn in a learned latent space (Gelada et al. 2019; Seo et al., 2022 Hainer et al. 2025).
Spectral decomposition methods decompose state and actions into low-rank spectral features (Wang
et al. 2021b Yang & Wang 2020; Shribak et al. 2024). SF² uniquely bridges successor measures
and online RL: (I) Unlike world models (Gelada et 2019 Seo et al. 2022 Hafner et al., (0255)
and spectral methods (e.g.. (Shribak et al. 2024)) that ignore policy-dependen horizons, SF"
explicitly encodes discounted future distributions via dow-matched successor measures; (2) While reconstruction methods focus on regenerating the observation, our method considers policy and environment dynamics; (3) Bisimulation methods (Zhang et al., 2021, Castro et al. [002 (023) emphasize state similarity with reward, but SF² also optimizes features (1s a) from environmental dynamics not only from the value alignment.
Successor Measure. SM predicts future state distributions under given policy, effectively cap- turing the expected discounted future state occupancy. This concept is closely related to Successor Representations (Dayan 1993) and Successor Features (Barreto et al. 2017 , decoupling environ- ment dynamics from reward structures, facilitating efficient policy evaluation and transfer. Blier et al. (2021) offers formal mathematical definition of SM and introduces how to estimate it for

---

Published as a conference paper at ICLR 2026
value function evaluation. Wiltzer al. (2024) further enhances these approaches by modeling the full distribution of future state occupancies, providing richer representations for downstream decision-making. GHMs and y-models extend the notion of modeling discounted state visitation distributions, creating a continuum between model-free and model- based RI. (Thakoor et al. 2022 Janner et al., 2020). Agarwal et al. (2025); Touati & Ollivier (202T) also employ SM to build representation in the zero-shot RL setting under precollected offline dataset, always need an extra exploration policy collect, which allows for optimal policy inference under other given reward functions. SF fundamentally advances this paradigm by introducing the subsequent flow characteris- ties (ss, a) (Definition 3.1) as structured linear decomposition of the flow field, enabling principled representation learning that greatly expands the scope of application beyond previous methods.
6 CONCLUSION
In this work, we proposed the Successor Flow Feature (SF2) framework, which leverages flow matching and linear-spectral decomposition address the challenges estimating and integrating successor measures in online RL. By explicitly modeling the mixture structure of successor measures, our method provides useful state-action representations that facilitate efficient online policy learning and planning. Through extensive empirical evaluations across discrete and continuous control tasks, we demonstrated that using SF consistently improves performance over standard baselines. Our results underscore the promise of flow-based generative modeling for successor features, paving the way for future research on scalable, expressive, and efficient RL representations.
REPRODUCIBILITY STATEMENT
We facilitate reproducibility by providing an anonymized source-code repository the supplementary materials. For every experiment, we specify the random seed used, and we document all imple- mentation and training details in Appendix G and Together, these references are sufficient for independent researchers replicate reported results.
ACKNOWLEDGMENTS
The research work described in this paper was conducted in the JC STEM Lab of Machine Learning and Symbolic Reasoning funded by The Hong Kong Jockey Club Charities Trust. And we would also like to thank the reviewers for their thorough reading of our manuscript and for their constructive suggestions, which have undoubtedly improved the final version.
REFERENCES
Siddhant Agarwal, Harshit Sikchi, Peter Stone, and Amy Zhang. Proto successor measure: Represent- ing the behavior space of an agent in a dynamical system. In International Conference on Learning Representations, 2025. URL htthhttp:pt http:::::////peeeviieeee net/ forum ?id=s95v100cII
André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30 (NIPS pp. 4055-4065. Curran Associates, Inc., 2017.
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint. arXiv preprint arXiv:2101.07/23, 2021.
Michat Bortkiewicz, Wladek Palucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arezewski, Lukasz Kuciński, and Benjamin Eysenbach. Accelerating Goal-Conditioned RL Algorithms and Research. In International Conference Learning Representations, 2025. URL httpps://axxi....
pdf/ Tttf/2408.1155 11052
James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and
10

---

Published as a conference paper at ICLR 2026
Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://qithui com/ jammmjaaammm/aa
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo:
Improved rep- resentations via sampling- based state similarity for markov decision processes. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL FeRl.ttpss: ptttps://openrevis
net. rum? i d= p (0t.PP6kkm@T..gg
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. A kernel perspective on behavioural metrics for markov decision processes. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
n dttp://ooromm..dmmmPxxXX11 d=
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Raymond Chua, Arna Ghosh, Christos Kaplanis, Blake A Richards, and Doina Precup. Learning successor features the simple way. Advances in Neural Information Processing Systems, 37: 49957-50030, 2024.
Peter Dayan. Improving generalization for temporal difference learning: The successor represstttin...........icccoo Neural computation, 5(4):613 624, 1993.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603-333 3-6600003.60.... 2022.
Jesse Farebrother, Mattco Pirotta, Andrea Tirinzoni, Remi Munos, Alessandro Lazaric, and Ahmed Touati. Temporal difference flows. In Forty- International Conference Machine Learning, 2025. httttp:://penneeieww..co /ttp:///eee..rrr.oooaao/c////a// 3677333yyy
C. Daniel Freeman, Erik Frey, Anton Raichuk. Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation, 2021. URL http: //github. htt.h/bb..cmmm/ooo//////////////
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Kenji Fukumizu, Francis R Bach, and Michael Jordan. Kernel dimension reduction in regression.
The Annals of Statistics, pp. 1871-1905, 2009.
Walter Gautschi. Numerical analysis. Springer Science & Business Media, 2011.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. DeepMDP: Learning continuous latent space models for representation learning. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2179-2179, PMLR, 09-15 Jun 2019, htt stllittpss:/prrceeiiiii mlr wttp:m.tw.....////e//aee.
html
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812.05905 2018. URL
05905
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha. Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555 2565. PMLR, 09-15 Jun 2019. URL https::ts//.o htttps:/pprocedi: ngs mlr wrwass/99/haafae. 9a. html

---

Published as a conference paper at ICLR 2026
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647-653, 2025, doi: 10. 00010088441588--05500844444 URL
https: pettss//viie nwtura. htt.u//ee..oooorrcceeeeeee/aa11/
027044
Tom Hennigan, Trevor Cai, Tamara Norman, Lena Martens, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. URL http://qittie...ee http:/ttthhhhto..g httt::///gtthbb..ccm//eeeemmmmm nd/ ha ku
Michael Janner, Igor Mordatch, and Sergey Levine. y-models: Generative temporal difference learning for infinite-horizon prediction. In Advances in Neural Information Processing Systems, 2020.
Diederik P Kingma and Max Welling. arXiv:1312.6114.220 2013.
Auto-encoding variational bayes.
arXiv preprint
Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, Yu Murata, Keigo Habara, Haruka Kita, and Shin Ishii. Pgx: Hardwarr--cccelerate parallel game simulators for reinforcement learning, In Advances in Neural Information Processing Systems, volume 36, pp. 09716-45743, 2023.
Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv: 2016.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised repre- sentations for reinforcement learning. In Hal Daumé III and Aarti Singh (eds.), Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 5639-5650. PMLR, 13-18 Jul 2020. URL https //proceedings.oo mlr www.rrprees/vvili wwt.l//ll/laanaaa.... laskin2Ca. html
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The Eleventh International Conference on Learning Repre- sentations, 2023. URL http:tts::/ /ttttpp:///pereviiieee .nt. ottpee/wwwwwee.tttoooooooooooeoe
Yann Ollivier. Which features best for successor features? arXiv preprint 27X999250..10790. 2025.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforce- ment learning. Journal of Machine Learning Research, 25(318)11-63 2024.
Stefano Peluchetti. Non-denoising forward-time diffusions. arXiv preprint arXiv:2312. 14589, 2023.
Hiroaki Sasaki and Aapo Hyvärinen. Neural-kernelized conditional density estimation. arXiv preprint arXiv:1806.01 005444006..7. 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL htttp://arxiv.org/ www.aaawww..r...aaaoooo abs/
1707.06347
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Dfta-efficient reinforcement learning with self- predictede representations. In International Conference Learning Representations, 2021. URL
ner/forum?
(dJuCQ@fZwRaUJ.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. 6th Annual Conference on Robot Learning, 2022. /ttl:htttp:///peeree//
Dmitry Shribak, Chen-Xiao Gao, Yitong Li, Chenjun Xiao, and Bo Dai. Diffusion spectral repre- sentation for reinforcement learning. Advances in Neural information Processing Systems, 37: 110028-110056, 2024.
Yang Song, Jascha
Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. bcsre-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https:: /ttttts://ppeeeee///ee http////pp///eee////e/p net net/forum??
id==xTTG12RRH5
12

---

Published as a conference paper at ICLR 2026
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International conference on machine learning. pp. 9870-9877 PMLR. 2021.
Shantanu Thakoor, Mark Rowland, Diana Borsa, Will Dabney, Rémi Munos, and André Barreto. Generalised policy improvement with geometric policy composition. In International Conference on Machine Learning, pp. 21272-21307. PMLR, 2022.
Ahmed Touati and Yann Ollivier. Learning one representation to optimize all rewards. Advances in Neural Information Processing Systems, 34:13-23, 2021.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020. ISSN 2665-9638. doi: https://doi.
2020. 100022. URL https:: www. wwtp://wwcceecceeeettt.tc com/ science/
art article/piii.. /ttp:iis226659822
Hado Van Hasselt, Arthur Guez, and David Silver.
Deep reinforcement learning with double learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Leonid Nisonovich Vaserstein. Markov processes denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii ((3):6472,, 1969.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep generative learning via schrödinger bridge. In International conference on machine learning, pp. 10794-10804, PMLR, 2021a.
Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Provably efficient reinforcement learn- ing with linear function approximation under adaptivity constraints. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 1322-135336 Curran Associates, Inc., 2021b. URL https:://pooceeeet/ettt
neur ips cctppper_paeeceeeeeeeee wtw.ss//appeee////// .pff]
Ei
Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Yunhao Tang, André Barreto, Will Dabney, Mare G Bellemare, and Mark Rowland. A distributional analogue the successor representation. In Proceedings of the st International Conference on Machine Learning, pp. 52994-53016 PMLR, mlx
2024. URL
html
Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In Hal Daumé and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10756-10756. PMLR, 13-18 Jul 2020. URL / wwt.s/m....p....eeeeeeemeeemm yang20h. html
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. Proceedings of the aaai conference on artificial intelligence, volume 35, pp. 10674-10681, 2021.
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire. Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carlo Sferrazza, Yuval Tassa, and Pieter Abbeel. Mujoco playground: An open-source framework for gpu- accelerred robot learning and sim-to-real transfer., 2025. URL htttttttt: thub. com/ com/ e- leeeoggleeeeemmmee nd/ muujoco_p Sarooo_ppaape round
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021. bttts:://openreiiiie net htw.:r/rwwwweeeeffoouunmmm
1d==2FCwDKRRBU
Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. S (ttextt((AAO)): Temporal latent action-driven contrastive loss for visual reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
13

---

Published as a conference paper at ICLR 2026
A
THEORETICAL FOUNDATION
A.1 CHALLENGES OF MIXED DISTRIBUTION IN GHMS
The learning procedure for GHMs using generative models, particularly those without explicit density estimation, follows a principled sampling-based strategy maximizing the objective in equation 2 Transition tuples (s, a. s') sampled from given environment interactions or a pre-collected dataset. For each tuple, an indicator variable is sampled from Bernoulli distribution with parameter 7 to determine which distribution be learned by GHMs: 1) with probability 1 7, learn from the true observed successor state s'; and 2) with probability % learn from bootstrapped samples, where a state sampled from the generative model, pu-ls', a²), conditioned on the observed next state and next action sampled according to the current policy. This procedure implements the recursive structure of the normalized successor measure (NSM) by sampling.
While theoretically sound, mixture-based learning struggles when is close to the data distribu- tion is dominated by the bootstrapped samples from the generative model, introducing significant bias early in training-wiin minimal anchoring to real data. In the following
and suncurracy-espciiaae section, we demonstrate this challenge via a Gaussian mixture example, where learning the mixed distribution grows increasingly difficult y approaches unity.
Target Distribution
VAE Samples
GAN Samples
FM Samplex
Semplen(Explich Misture)
65-
0.5
3.5
04
2.5
0.4
3.4
0.3
1.2
0.2
22
0.
60
0.3
0.0
20
3.0
Figure 4: Visualization shows flow-matching methods better capture the multimodal structure of Gaussian mixtures compared to other generative models, such as GANs and VAEs, which is crucial for SM learning with explicit mixture targets.
A.2 EMPIRICAL VALIDATION ON MIXTURE DISTRIBUTIONS (GAUSSIAN AS AN EXAMPLE)
We evaluate several generative models on one-dimensional Gaussian mixture that is simplified yet insightful setting. The target distribution is shown in the left-most part of Figure 4 We train VAEs, GANs, and FM models, and observe that learning becomes difficult as the mixing parameter y 1.
10 Sample Quality over Training (Lower is Better)
GAN
FlowMatching
How.Mtchhngttsille Mature
Wasserstein Distance
282
430
Training
Figure 5: Wasserstein distance between true distribution and generated samples.
Under = 0.99, Figure 4 provides visualizations of the final generated samples, and Figure 5 reports the Wasserstein distance (Vaserstein) 1969) between generated samples and the ground truth
2To obtain longer horizon length and align with the discount factor used in the value function definition, the discount factor y is expected he very close 1, e.g. y 0.99.
14

---

Published as a conference paper at ICLR 2026
distribution over the training process. These results show how accurately each generative model captures the underlying mixture distribution. Specifically, visualization results show that VAEs and GANs struggle capture the multimodal structure, while FM exhibits a stronger ability to model the target distribution.
In the next section, we will introduce how explicitly leverage the mixture structure and correspond- ing mixture weights to further enhance the FM approach for more efficient and accurate learning of complex multimodal distributions.
A.3 EXPLICIT MIXTURE ORJECTIVE FOR FLOW MATCHING
To enhance learning of mixture distributions p ((mix)) (x) = (x) + (1 (1-7) we introduce an explicit mixture objective within the flow matching framework. Unlike black-box density estimation, our method directly encodes the known compositional structure of the target distribution into the training objective. This improves computational efficiency and accelerates optimization convergence.
Let uo k) denote the parameterized time-dependent vector field, the marginal distribution on time = standard Gaussian distribution, and Φx(c, 2)) the conditional time-dependent diffeomorphic map. Let ~ p! mmmix) k ~ U(0, 1).c (N0,, In) and (z(e,z) dk flow matching objective for mixture distributions is,
(6x(6.2) the original
[C] (e))lllll(( vx(c, drdkk
L Ek,z,
on vc(c,)))]==E,
Since nix is a mixture, we can decompose this into:
C-Ee [C/ (117) (1- (1-7)p(0)x0)) ((oo)))))l(((((() (luo(O((e,)), (e x)1? drdk
=? E e. al (ll || x20) k)- un(e, +[[()) c.k x ((
uoloule, k) vx(e,x(2)
La
+
This explicit decomposition enables training by sampling from each mixture component and reweight- ing the loss according to its corresponding mixture weight. By leveraging the known structure of the mixture, this approach aligns closely with the recursive formulation of the NSM, which also exhibits a mixture form rooted in the Bellman equation, which is also noted in (Farebrother et al. (2025). As shown in Figure and Figure our proposed method, FlowMatching (Explicit Mixture), outperforms standard baselines by effectively capturing the multimodal structure of the target distribution. It is worth noting that not only can flow matching methods exploit this form of distribution mixing, but also diffusion models (Song et (021 and bridge tridge-based models (Wang et al. (2021a), which all rely on the diffusion mixture representation (Peluchetti) 2023).
B EXPERIMENTAL SETUP AND COMPARISONS
B.1 COMPARISON WITH OTHER REPRESENTATION LEARNING METHODS
To align with our method, we configure SPR (Schwarzer et al. 2021 with K=1, on the immediate successor transition, just like our approach. Since the SPR method was originally designed for image input, while our environment uses state input, we use Gaussian noise simulate data augmentation. In order to select a suitable augmentation setting, we perform a hyperparameter sweep over the Gaussian noise magnitude the AcrobotSwingup environment.
1, so it relies only
Following the sensitivity analysis in Figure 6 we adopt the best-performing noise standard deviation of 0.05 and reuse it across all 7 DeepMind Control Suite environments without further tuning. This configuration is used to generate the aggregate IQM statistics in Figure 2 and the full learning curves in Figure 7 Ensuring that improvements stem from representational benefits rather than environment- specific hyperparameter adjustments. In this configuration, our proposed method exhibits better performance.
15

---

Published as a conference paper at ICLR 2026
Figure SPR augmentation sweep. Performance of SPR on AcrobotSwinguy for different Gaussian noise magnitudes used in the augmentation pipeline. The noise level of 0.05 is selected for the cross- environment experiments.
C EXPERIMENTAL RESULTS
C.I MAIN RESULTS ON DEFAULT SETTINGS
Figure presents the full learning curves for 7 DeepMind Control Suite tasks using TD3 and SAC variants, complementing the aggregate IQM statistics shown in the main text.
C.2 ROBUSTNESS ANALYSIS
To rule out the possibility that our gains come only from a specific chosen training budget, we reran every experiment with IM-timestep budget and 15 random seeds. Figure reeports the IQM aggregates for TD3 and SAC under this stricter setting, showing that SF still surpasses vanilla and SF-based baselines by clear margin. Figure 9 then provides the per-environment learning dynamics, confirming that our method maintains its advantage even when learning is constrained to only 1M timesteps. We also study the effect of gradient steps and generalization to observation noise. Figure 10 (left) shows that allocating more gradient steps to the feature updater steadily improves the IQM score, highlighting that SF² benefits from thorough feature optimization. For the noise experiment, seeds, store the checkpoints, and evaluate on perturbed observations with Gaussian noise injected into the observation space. Figure IO (right) reports the AUC under different noise levels; because SF² achieves higher performance even before perturbation, it also sustains better AUC after noise injection.
train the IM-timestep TrrobotSwinguyy environment using six random
C.3 EXTENSION DISCRETE ACTION SPACES WITH IMAGE INPUT
We conducted additional experiments on the MinAtar benchmark from PGX (Koyamada et al. 2023 using PPO (Schulman al. 2017). Because our method constructs representations over state-action pairs (s,a), a). replaced the standard PPO state-value estimator V with action-value estimator Q(s,a) a) and trained with TD-À target (A = 0.95). We also adopted separate actor and critic networks for baseline and methods. MinAtar provides image- like observations and discrete action space; to make our approach applicable in this setting, we evaluated two implementations: (i) linear interpolation along image channels (shown as SF² w/o AE), and (ii) butoencoder-baseee pipeline in which the generative model is trained experiments
the latent space (shown as SF2 w AE). All with discount factor 7 0.9. Results are presented in Figure 11] where report both IQM (interquartile mean) statistics and training curves. These results provide preliminary evidence that our method can operate effectively in image-based environments with discrete actions.
16

---

Published as a conference paper at ICLR 2026
020
M pwy
TD3
y-
0.0
TD3 0.99
TDJ
TD3 Sim
Tratiig
Acrobetteedng
Carpelebsaneeioam
Cantol
Cheesahaer
570
Hoppertic
Fandukkmmiiw
070
SAC
SAC
SACSin-Lap
SACSIR
y
SACSP ym0.9
Figure 7: Learning curves across tasks. Each plot reports training curve with 95th percentile confidence intervals (bootstrap using 5000 samples) for TD3 (upper) and SAC (lower) with vanilla baselines, SF-based baselines, SPR (Schwarzer et al. 2021). and our SF² with transition 0.0) and successor (ry 0.99) horizons.
D DETAILED ALGORITHMS
We demonstrate the detailed algorithm training process of SF2 when combined with TD3 and SAC, respectively.
17

---

Published as a conference paper at ICLR 2026
Toafunily Agpiggate Parkmmance arvi, nomalied AUI
Media
Mean
Colmaliy Cap
100 yv0
24
06
SACromts Apprigats Perfommnce 17 enn. nomaledd AUG
Nollyt
10M
Moan
Ortiolly Orp
580
00
Figure 8: IQM performance 1M timesteps. Aggregate AUC statistics for TD3 (upper) and SAC (lower) at IM-step training budget using 15 seeds.
Algorithm 2 TD3 with SF²
1:
Initialize 0, v. Ç. larget networks 0' +- 0, o' +- d. v/l +- C + replay buller M do
2:
for episade
3:
Initialize environment, get initial state 40
4:
for stop=
5:
(*+) 55 is large enough then for G gradient steps Sample batch (s, a, r, from
(0, execute observe store (Re, 8841) in B8
6:
7:
8:
9:
min 1.2 2ol (s',
10:
6, a V o. (Qo, (v y)² a+ (u,
11:
12:
(#, () (v.C) - asn V. v.c C.s
13:
step mod d=0 then
14:
0 0 QaQu, (0) To (s))))
15:
8' r0 + (1 T)6'
16:
6' To + (1 r)6
17:
T)
18:
c' + 7CC + (1 Te)e'
19:
end
20:
end for
21:
end if
22:
23:
end for
18

---

Published as a conference paper at ICLR 2026
400
kdhan
TD3
628
TD3 SP y-
TD3-SFF y 0.99
Sim-Leap
200
30
13h0
Ac Aontootiim
Carbar Catbsabbaaeeeiiar
Dredaniun
630
266
SAC SAC-Sim
SAC Sim
400
SAC SP y- 0.99
Figure 9: Learning curves at 1M timesteps. Each plot shows the median and 95% CIs (bootstrap, 5000 samples) over 15 seeds for TD3 (upper) and SAC (lower) with vanilla baselines, SF-based baselines, SPR (Schwarzer et al. 2021), and our SF² with transition (7 = 0.0) and successor (y 0.99) horizons.
Algorithm 3 SAC with SF²
1:
Initialize 0, d. v:: target networks of +- 8, 6' + d. v/ + replay buffer B
2:
episode
3: Initialize environment, get initial state
4:
for step = T (ee) ir B is large enough then
5:
(/, execute observe #e41i store (fe, (641) B
6:
7:
for G gradient steps
8:
Sample batch (e, a, 5') from
9:
6e mit Por (v' Far
10:
og Vo (Q, (v) y)²
11:
au (p(s)
y)
12:
(e. c) + (v. asa V Voc
13:
step mod d=0 then
14:
8
VaQ (8) To (=))))
15:
6' rB
16:
5' - 7 + (1 r)
17:
T)p
18:
C' + TCC [C)
19:
end
20:
end for
21:
end if
22:
end for
23:
end for
19

---

Published as a conference paper at ICLR 2026
Sneriation Sraadee Pheformanne dap with Paroo Bradindd
Bride
Figure 10: Left: Effect of training budget. IQM of the AUC as we vary the number of gradient
steps dedicated to representation learning, highlighting the gains from additional feature optimization. Right: Generalization under Gaussian observation noise. Interquartile mean performance of policies trained for IM steps (six seeds) when cvaluated on Gaussian Gaussian-noise- noise- pertturre observations. sF maintains the highest returns.
07
33
205
10.0
15
120
Bcad
5f1y-000 -ot we AC
5111y wAA
00
33
200
25
10.2
AUC Aggregate Comparison
Wedion
IOM
Mean
Optimality Gop
5P WE
Sasel ne-
nommakoed
nomaaozee
Figure 11: MinAtar results. Upper: Learning curves for PPO variants on MinAtar environments with 95th percentile confidence intervals (bootstrap using 5000 samples). Lower: Aggregate IQM performance. Interquartile mean AUC statistics across MinAtar environments comparing vanilla PPO, SF² without autoencoder (w/o AE), and SF with autoencoder (w AE).
E
THE USE OF LLMS
The authors used LLMs to polish the language and improve readability. All Al-generated content was thoroughly reviewed and revised by the authors, who take full responsibility for the final content.
EXPERIMENT WITH CRL
We present an additional integration experiment integrating 2 into Contrastive RL (CRL) on the Ant locomotion task.
20

---

Published as a conference paper at ICLR 2026
steps)
200
CRL SF2
175
150
Eval spiode reward
125
100
25
10
Figure 12: Return curve Ant environment with CRL (Eysenbach al. 2022) and CRL with SF
based on JaxGCRL (Bortkiewicz et al. 2025) (5 seeds).
G NETWORK ARCHITECTURE AND HYPERPARAMETERS
G.1 TWO GAUSSIAN EXAMPLE IN SECTION A.7
The target distribution is a one-dimensional bimodal Gaussian mixture with means μ4 =-20 and 2.0, standard deviations σ 0.3 and 02 =070.7 and a mixture weight of 0.01 (i.e. (=0.99,, The generative model is two-layer MLP with hidden dimension 256 and ReLU activations. Training runs for 1500 steps with batch size 32 and learning rate 10-3; an exponential moving average of the model weights is maintained with decay rate 0.995. Performance is logged every 100 steps and evaluated every 200 steps. evaluation time, 4000 samples are generated using 100 numerical integration steps, and the Wasserstein distance to the true distribution is computed.
G.2 DEEPMIND CONTROL SUITE IN SECTION 4
21

---

Published as a conference paper at ICLR 2026
Table 1: Network and Training Parameters
Parameler
Value
Network Architecture
Hidden Layer
policy
(512, 512, 512)
Q-Network Layer Normalization
True
Policy Network Layer Normalization
True
Feature Size
512
Zeta Network Hidden Layer Sizes
(512, 512)
Embeddiig Size
Activation Function
ReLU
Kernel Initializer
LeCun Uniform
Training Parameters
Number of Timestops
5M: CartpolcallanceSpree,, CarpoolSwingupSparesc Fish-
10M: Actobot, Hopper, CheetahRun, Pendulum-
Swingup
Number of Evaluntions
100
Reward Scaling
1.0
Max Episade I Length
1000
Normalize Observations
True
Action Repeat
(4 for PendullumSwingUUp)
Learning
Number of Envireaments
128
Batch Size
Gradient Updates per Step
Max Replay Size
4,194,304 304 (1048576 4)
Min Replay Size
8192
Discounting Factor
0.99
Policy Delay
Noise Clip
0.3
Smoothing Noise
0.2
Exploration Noise
0.2
Optimizer Paramelers
Alpha Optimizer Leaming
Policy Optimizer Learning Rate
lc-4
Q- D-Netwark Optimizer I Leaming Rate
le-4
Psi-Zeta Optimizer Leaming Rate
lc-4
Spehod-Specific Parameters
TD3/SAC Gamma Successor
0.99
TD3/SAC Tau Zeta
0.005
TD3/AAC Denoising Steps
Function Evaluations
22


Reviews:
[{"review_id": "DCWI1sW69N", "reviewer": "Reviewer_77u8", "summary": "This paper proposes Successor Flow Features ($SF^2$), which uses flow-matching technique to approximate Successor Measure (SM). The authors design a linear decomposition of the vector field $u$, separating the time-conditioned and time-invariant components. The time-invariant part is viewed as the learned representation of state action pair and is used by downstream reinforcement learning algorithms. Experiments are conducted on DeepMind Control Suite, and the results demonstrate the superiority of the proposed algorithm.", "questions": "1.\tSome of the notations used in this paper are not properly introduced. \n\n    a)\tThe notation of Successor Measure introduced in lines 135-136 is defined as a probability distribution over state sets, but it is used later as a probability distribution over states (lines 170-171).\n\n    b)\tThe notation of $ODE(\\cdot, \\cdot)$ (line 187) is not clearly introduced as a function, although the ODE used by flow matching is described in line 104.\n2.\tIt would be better to provide a complete version of RL training pseudocode in Algorithm 1. The current version includes only value loss, but omits policy loss, which is confusing.\n3.\tThere are existing works that have employed successor measures for RL. The author is encouraged to emphasize the importance of integrating successor measures for online RL representation learning.\n4.\tThe learned representation is only used as input to $Q$ function. Is there a way to allow the policy $\\pi$ to also leverage the learnt representation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe linear decomposition of $u$ is the key innovation of this paper and provides a clear architectural insight. It is well-motivated, and both its efficiency and sufficiency are thoroughly discussed.\n2.\tThe proposed method can be used as a plug-in module and can be integrated in a wide range of RL algorithms.\n3.\tThe experiments are clear and comprehensive, covering effectiveness, hyperparameter sensitivity and efficiency. Experimental details are clearly organized in appendices, making them easy enough to be reproduced.\n4.\tExperimental results clearly demonstrate the superiority of the proposed algorithm over the baseline algorithms, which are strong enough.", "weaknesses": "1.\tThe presentation of this paper is not sufficiently clear (see Questions for details)\n2.\tThe literature review is somewhat limited. For instance, the paper does not cite Agarwal et al. [1], “Proto Successor Measure: Representing the Behavior Space of an RL Agent” (arXiv:2411.19418, 2024), which appears to be highly relevant to the topic and should be discussed for completeness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqJ9kS0nuN", "reviewer": "Reviewer_Cb48", "summary": "This paper proposes Successor Flow Features (SF²), a representation learning framework that integrates flow matching with Successor Measure (SM) to bridge SM estimation and online policy optimization, to enhance sample efficiency of RL training. SF² enforces a structured linear decomposition into a time-invariant embedding and a time-dependent projection, enabling seamless integration with off-policy algorithms like TD3 and SAC. Empirical results on DeepMind Control Suite tasks show improved sample efficiency and training stability over successor feature baselines. The core idea of combining flow-based generative modeling with successor representations is theoretically novel, but several critical gaps limit its comprehensiveness and practical impact.", "questions": "1. Does SF² bring better generalization performance?\n2. In what RL scenarios is SF² most effective?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The integration of flow matching with SM for online RL is a creative extension of both generative modeling and successor representation paradigms. The structured linear decomposition (time-invariant embedding + time-dependent projection) addresses SM’s lack of compact, online-friendly representations, which is a non-trivial technical contribution.\n2. Experiments on 7 DeepMind Control Suite tasks demonstrate consistent performance gains over standard TD3/SAC and SF-based baselines. The detailed hyperparameter analysis (EMA coefficient, denoising steps, feature size) and ablation studies add rigor to the results.\n3. The paper establishes meaningful links between SF² and existing methods (Successor Representation, Diffusion Spectral Representation), providing conceptual clarity and grounding the framework in prior literature.", "weaknesses": "1. Insufficient Justification for Successor Representations in Online RL: The paper frames sample efficiency as a key advantage, but successor representations are inherently designed for reward-dynamics decoupling (enabling generalization/zero-shot transfer). No evidence is provided that SF² outperforms state-of-the-art sample-efficient representation methods (e.g., SPR, CURL) that do not rely on successor paradigms. This raises questions about whether successor representations are necessary for online RL’s core goal of sample-efficient single-task learning.\n2. Missing Comparisons with Critical Baselines: The baselines are limited to standard TD3/SAC and SF variants (TD3Sim/SACSim). The absence of comparisons with representative sample-efficient RL methods—particularly SPR (Self-Predictive Representations), which directly targets data efficiency via self-supervised representations—undermines claims of advancing the state-of-the-art in online RL.\n3. Unproven Generalization to Pixel-Based Environments: All experiments use structured state features (e.g., joint angles, velocities) from DeepMind Control Suite. The paper provides no analysis or modifications for pixel-based inputs (high-dimensional, unstructured data), which are common in real-world online RL. It remains unclear if SF²’s linear decomposition and flow matching components can adapt to image data.\n4. No Evaluation on Discrete Action Spaces: The work exclusively focuses on continuous control tasks with TD3/SAC. Successor representations (e.g., SR) originated in discrete state-action spaces, but SF²’s performance and adaptability to discrete actions are unaddressed. This limits the framework’s generality, as discrete action scenarios (e.g., game playing, recommendation systems) are central to online RL.\n5. Complexity: The paper notes SF²’s ~2x longer running time than standard TD3 (e.g., 1300s vs. 659s on AcrobotSwingup).\n\nReference\n\nSPR: Schwarzer, Max, et al. \"Data-Efficient Reinforcement Learning with Self-Predictive Representations.\" ICLR 2021.\n\nCURL: Laskin, Michael, Aravind Srinivas, and Pieter Abbeel. \"Curl: Contrastive unsupervised representations for reinforcement learning.\" ICML 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5R8e0EmrSv", "reviewer": "Reviewer_kuve", "summary": "In RL, the successor measure is the probability mass of ending up in a set of states conditioned on starting in some state, taking an action, and then following some policy. It generalizes the well-known idea of the successor representation to continuous state spaces. This paper introduces a method for estimating the successor measure using the generative modelling technique of flow matching. This method is then used as an auxiliary task to shape an online RL agent’s representation. Empirical evaluation on a set of continuous control tasks show that the method increases the final expected return achieved by the agent.", "questions": "I don't fully understand how the SM learning auxilliary task differs significantly from prior work on SM learning. I understand the novelty comes from combining with online model-free RL. In addition to responding to the weaknesses raised above, I'd appreciate gaining some insight into the novelty of learning the SM.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- While prior work appears to have studied how to estimate the successor measure, this work appears to be the first to do so for model-free, online RL agents.\n- While I didn’t fully appreciate how the SM estimation part of the algorithm is distinct from prior work, I thought it was interesting how the work integrates a boostrap objective with an generative modelling technique.", "weaknesses": "- I’m not convinced that the experiments substantiate the claims made in the paper. Experiments are only repeated for 5 trials and the paper reports large standard deviations. Given the spread and limited trials, it’s plausible that the novel method does not improve upon the simpler base algorithm (SAC or TD3). Please see “Empirical Design in Reinforcement Learning” for a great reference on why these details matter.\n- The paper contains a lot of vague, overclaiming language: “achieves superior performance” (superior in what respect?) “remarkable stability” (what is remarkable about it).\n- After going through the experiments in detail and ignoring the statistical issues, I think the best claim would be “our method had the largest final expected return achieved when we stopped training.” While final performance obtained is a valid objective to study, it would be valuable to understand how the termination of training was selected. In the appendix, it is mentioned that training went for 10 million timesteps. This is substantially longer than what is usually used on thest environments (1 million seems more common). Why use the longer time? And, in any case, how can we be sure that performance wouldn’t be different for different cut-off points?\n- Abstract claims sample efficiency improves — I don’t see any result (e.g., learning curve) substantiating this claim as only final numbers are given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NmTjnqOIBo", "reviewer": "Reviewer_ydVH", "summary": "The paper proposes SF$^2$, a flow-matching based representation that approximates the successor measure (SM) while enforcing a linear decomposition: time-invariant $\\psi(s,a)$ and time-dependent $\\zeta(\\cdot|k)$. SF$^2$ is trained jointly with off-policy RL via a mixture-structured flow matching loss and a value-alignment term, and shows improved average performance and stability on seven DMC tasks, with explicit analysis of EMA, denoising steps, and runtime trade-offs. The work also offers an interpretation that, in a small-$k$ regime, $\\psi$ updates resemble succesor representation (SR)-style TD recursion.\n\n---\n**Review summary.** The proposed framework introduces a practically effective bridge between SM and flow matching. Despite some theoretical and scalability gaps, the paper offers solid empirical evidence, clear motivation, and transparent ablation studies. With stronger formal grounding and broader experiments, this work could become a notable contribution to representation learning in RL. However, the reviewer feels that the main contribution lies more in practical engineering and integrative design than in fundamental theoretical advancement. Therefore, the reviewer assigns an initial score of 6 and plans to revisit this rating after the authors address the concerns and questions raised in this review.", "questions": "- Could the authors formalize the small-$k$ connection between $\\psi$ updates and SR Bellman recursions? The reviewer wonder that what assumptions does this approximation hold, and what are the limits of this equivalence.\n- As the reviewer above-mentioned, some DMC tasks show large performance variance. Could the authors provide the number of seeds and statistical tests (e.g., t-test, bootstrap) to confirm significance?\n- The reviewer just wonder how SF2 would extend to pixel-based or image-based input environments. Would $\\psi$ and $\\zeta$ share encoders or require hierarchical separation?\n- Have the authors compared the proposed solutions to recent flow-based RL algorithms?\n    - How does the explicit mixture formulation distinguish the proposed solution theoretically or empirically from them?\n- How do the authors think that shared encoders or low-rank $\\zeta$ to reduce the cost? Is it a good direction or not?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The reviewer thinks that the integration of flow matching and SM is novel and potentially impactful. The method makes the flow-based generative paradigm compatible with online RL, which is a meaningful step beyond prior SM works that focused solely on offline or evaluation settings. Additionally, the connection to diffusion spectral representation is insightful. The discussion of mixture distributions in Appendix A is illuminating, showing that flow matching can better capture multimodal structure than VAEs or GANs (It’s a given, perhaps, anyway), lending theoretical and empirical support to the method’s design.\n\n---\n**Writing**\n- The paper is written in a clear, structured, and mathematically disciplined manner. \n- Equations are well motivated and the derivations are logically consistent. In specific, they provide a clear connection between the mixture structure of the SM and the mixture learning dynamics of Flow Matching.\n- This bridige leads to a principled training objective, which is combining direct transition and bootstrap terms weighted by ($1-\\gamma$ and $\\gamma$). \n- The related work section (section 5) is broad and situates SF² well among SR, SM, and flow-based RL.\n\n---\n**Methodology**\n- The paper presents a creative connection between SM and flow matching. By enforcing a time-invariant $\\psi(s,a)$ and time-dependent $\\zeta(\\cdot|k)$, the method introduces a new linear factorization of dynamics.\n- The semi-gradient derivation in section 3.3 gives a clear heuristic connection to SR, showing that $\\psi$ updates naturally resemble a TD-style recursion with a flow-matching twist.\n- The design promotes a low-dimensional sufficient statistic for state–action pairs, which provide a clear path to sufficient dimension reduction.\n- Algorithm 1 and the embedding into SAC/TD3 are modular, allowing the method to be plugged into standard pipelines without major architectural redesign.\n\n---\n**Experiments**\n- Across all seven DMC tasks, SF2 outperforms or matches baselines, with clear reporting of mean/std, wall-clock training time, and ablations.\n- The consideration of $\\gamma =0$ and $\\gamma = 0.99$ variants shows how long-horizon modeling contributes to performance. \n- The reviewer thinks that hyperparameter analyses are informative. For example, the study of EMA rate, denoising steps and feature size. These ablations provides practical insight into stability-efficiency trade-offs.\n- In section 4.3 and 4.4, ablations results and computational reports are transparent.", "weaknesses": "While the overall empirical presentation is strong, the paper would benefit from a more comprehensive and centralized description of computational details in the main text. Moreover, a clearer breakdown of which components dominate computational load (flow-matching updates vs value alignment vs.actor updates) would make the method’s scalability story much more convincing.\n\n---\n**Writing**\n- Despite overall clarity, Section 3 contains dense mathematical exposition with several unreferenced variables and occasional overuse of inline equations. Additionally, the reviewer thinks that the interplay between $\\psi, \\zeta$, and the vector field $u\\theta$ could benefit from schematic visualization for intuition.\n- Some claims, for example, “SF$^2$ provides compact, expressive, and robust representations,” might seem like overstatements given the limited empirical evidence and absence of stress tests on diverse modalities or noise regimes.\n\n---\n**Methodology**\n- While the combination of SM and flow matching is powerful, the reviewer thinks that theoretical novelty is somewhat incremental. Each mathmatical part, e.g., flow-mathcing, SM, SDR, has appeared in prior work, so the reviewer thinks that the contribution lies more in enginerring integration than in new theory.\n- The derivation in Section 3.3 is heuristic. The claimed connection to SR via the pseudoinverse formulation is intuitive but lacks a formal proof of convergence or representational equivalence. The reviewr thinks that the pseudoinverse handling during training is not described concretely, leaving ambiguity about numerical stability.\n- The approximation in Equation (4) that aligns vector fields instead of performing full ODE integration is efficient but somewhat ad-hoc; the implications for consistency and bias in the learned SM are not analyzed theoretically.\n- Claims of universal approximation are untested, that is, no experiments validate $\\psi$’s robustness under domain shift or adversarial perturbation, which would strengthen the theoretical claims.\n\n---\n**Experiments**\n- The experiments is limited to state-based DMC tasks. It remains unclear whether the proposed solution scales to high-dimensional observation spaces or more difficult problems, for example, issac gym-based robotics or partially observable environments.\n- Comparisons omit several modern flow/diffusion-based RL methods, such as, DSLR, TDFlow, contrastive RL. \n- No qualitative analyses, for example, UMAP, t-SNE, or value map, are presented to interpret what $\\psi$ or $\\zeta$ actually learn. \n- Some reported variances are large, especially in P-SwU and C-SwU-S, and the authors do not report statistical significance tests or confidence intervals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)