ICML 2025

Temporal Difference Flows
Jesse Farebrother 1 2 † Matteo Pirotta 3 Andrea Tirinzoni 3 R´emi Munos 3
Alessandro Lazaric 3 Ahmed Touati 3
Abstract
Predictive models of the future are fundamental
for an agent’s ability to reason and plan. A com-
mon strategy learns a world model and unrolls
it step-by-step at inference, where small errors
can rapidly compound. Geometric Horizon Mod-
els (GHMs) offer a compelling alternative by di-
rectly making predictions of future states, avoid-
ing cumulative inference errors. While GHMs
can be conveniently learned by a generative ana-
log to temporal difference (TD) learning, existing
methods are negatively affected by bootstrapping
predictions at train time and struggle to generate
high-quality predictions at long horizons. This
paper introduces Temporal Difference Flows (TD-
Flow), which leverages the structure of a novel
Bellman equation on probability paths alongside
flow-matching techniques to learn accurate GHMs
at over 5× the horizon length of prior methods.
Theoretically, we establish a new convergence re-
sult and primarily attribute TD-Flow’s efficacy to
reduced gradient variance during training. We fur-
ther show that similar arguments can be extended
to diffusion-based methods. Empirically, we vali-
date TD-Flow across a diverse set of domains on
both generative metrics and downstream tasks, in-
cluding policy evaluation. Moreover, integrating
TD-Flow with recent behavior foundation models
for planning over policies demonstrates substan-
tial performance gains, underscoring its promise
for long-horizon decision-making.
1. Introduction
Predictive modeling lies at the heart of intelligent decision-
making, enabling agents to reason and plan in complex
environments. In Reinforcement Learning (RL), this pre-
†Work done at Meta 1McGill University 2Mila - Qu´ebec AI
Institute 3FAIR at Meta. Correspondence to: Jesse Farebrother
<jfarebro@cs.mcgill.ca>, Ahmed Touati <atouati@meta.com>.
Proceedings of the 42 nd International Con...