What Happens Next? Anticipating Future Motion by Generating Point Trajectories
Authors: Gabrijel Boduljak
Laurynas Karazija
Iro Laina
Christian Rupprecht
Andrea Vedaldi
Venue: ICLR 2026 Poster (2026)
Keywords: motion generation
point trajectories
flow matching
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We develop a video-like diffusion model that generates (quasi-)dense motion trajectories rather than pixels, achieving significantly more accurate and efficient motion generation than prior motion forecasters and video generators.

Abstract: We consider the problem of forecasting motion from a single image, i.e., predicting how objects in the world are likely to move, without the ability to observe other parameters such as the object velocities or the forces applied to them. We formulate this task as conditional generation of dense trajectory grids with a model that closely follows the architecture of modern video generators but outputs motion trajectories instead of pixels. This approach captures scene-wide dynamics and uncertainty, yielding more accurate and diverse predictions than prior regressors and generators. Although recent state-of-the-art video generators are often regarded as world models, we show that they struggle with forecasting motion from a single image, even in simple physical scenarios such as falling blocks or mechanical object interactions, despite fine-tuning on such data. We show that this limitation arises from the overhead of generating pixels rather than directly modeling motion.

Published as a conference paper at ICLR 2026
WHAT HAPPENS NEXT? ANTICIPATING FUTURE Mo- TION BY GENERATING POINT TRAJECTORIES
Gabrijel Boduljak
Laurynas Karazija
Iro Laina
Christian Rupprecht
Andrea Vedaldi
Visual Geometry Group, University of Oxford
ABSTRACT
We consider the problem of forecasting motion from single image, i.e., predicting how objects in the world are likely to move, without the ability to observe other parameters such as the object velocities or the forces applied to them. We formulate this task conditional generation of dense trajectory grids with model that closely follows the architecture of modern video generators but outputs motion trajectories instead of pixels. This approach captures scene-wide dynamics and uncertainty, yielding more accurate and diverse predictions than prior regressors and generators. Although recent state-of-the-art video generators are often regarded as world models, we show that they struggle with forecasting motion from a single image, even in simple physical scenarios such as falling blocks or mechanical object interactions, despite fine-tuning on such data. We show that this limitation arises from the overhead of generating pixels rather than directly modeling motion.
INTRODUCTION
We consider the problem of forecasting motion from a single image, i.e., predicting how objects in the world are likely move. This task is representative of an agent trying to infer what may happen next given only its limited observations of the environment. Because a single image does not fully specify the observed physical system, many different futures are possible and must be predicted as potential outcomes. Yet, these predictions are not arbitrary: they must be consistent with the image, physical principles, and facts about the observed objects that are known a priori.
Modeling such prior is important in many applications of AI, such as generating realistic videos, policy learning (Wen et al., 2024a; Yang et al., 2025; Bharadhwaj al., 2024b), model- control (Ding et al., 2024; Mazzaglia et al., 2024; Yang et al., 2024c;b;a), and other problems that require an understanding of physical phenomena.
As others before us, particularly in robotics (Wen et al., 2024a; Yang et al., 2025), we formulate motion forecasting as predicting the trajectories of points in the input image. However, unlike prior work, we formulate this problem generating the trajectories conditioned on the observed image. This stochastic formulation is more appropriate as it can model the forecasting ambiguity as a distribution over possible futures, which can then be sampled to produce likely realizations.
Increasingly powerful video generators (Polyak et al., 2025; HaCohen et al., 2024b; Wan et al., 2025; Brooks et al., 2024; Parker-Holder et al., 2024; NVIDIA et al., 2025) address similar forecasting problem, predicting a video starting from a single image. We thus suggest making our formulation similar to many such video generators, and in particular, Lipman et al., 2022). However, instead of generating pixels, we generate their trajectories on a grid.
flow matching (Liu et al., 2023b;
Previous motion forecasters (Wen et al., 2024a; Yang et al., 2025; Bharadhwaj al., 2024b) generally focus on predicting the motion of selected image points, for example, those that land on the arm of a robot. In contrast, our formulation, inspired by video generation, is (quasi-)dense: we predict the motion of all points in grid. This allows the model to reason about the entire scene jointly (Karaey al., 2024b). This is beneficial because, as time passes, objects that may be too far apart interact initially may eventually collide.
"Correspondence gabri jele robots...... .o..ac.ukkkk

---

Published as a conference paper at ICLR 2026
Input
Next frame
Middle of the video
Last frame
Time
Figure 1: State-of-the-art video generators frequently produce unrealistic motion. Even state- of-the-art models, such as WAN 14B shown here, often struggle to produce accurate, realistic and coherent motion. Common failure modes include distorted object geometry, objects splitting into multiple parts, objects disappearing, and objects spontaneously appearing throughout the video.
We also ask whether video generators can more than provide a good architecture. Many have suggested that training video generators on billions of videos is an effective way of learning world models (Ha Schmidhuber, 2018). If so, we could reduce motion forecasting to video generation, for example, by applying point tracker to the generated video (Bharadhwaj et al., 2024a).
Even so, we hypothesize that predicting point trajectories directly can be significantly simpler and more efficient than predicting pixels followed by inferring motion from the generated video. Trajectories capture motion directly, whereas videos need to be further translated into an estimate of motion (Ko et al., 2023; Patel et al., 2025). Using trajectories also injects two inductive biases, object permanence and temporal coherence, that general video generators struggle with (Motamed et al., 2025; Kang al., 2024) (Figure 1).
To better contrast video and track generators, we minimize the difference between architectures and compare a trajectory forecaster trained from scratch video generators trained billions of videos. We show that even state-of-the-art video models like WAN (Wan et al., 2025) struggle with predicting the motion of objects in relatively simple simulated scenarios (Figure 1) even after fine-tuning them on these domains. Hence, it is unclear whether training video models on billions of general-purpol videos allows them to learn basic physical consistency. In contrast, learning predict tracks much more effective in this respect, even without pre- pre-training on massive datasets.
For evaluation, we primarily consider synthetic scenarios, where it is possible to simulate different events starting from the same initial image. This facilitates assessing the ability of model to capture the distribution of possible futures. To do so, we further adopt the motion distributional metrics from the video generation literature (Brooks et al., 2024). However, these population metrics are coarse and may not capture well the physical plausibility of the predicted motion. We thus consider scenarios where can also test directly for aspects of physical consistency, such maintaining the shape of rigid objects.
We experiment using simulated physical scenarios from Kubric (Greff et al., 2022), LIBERO Liu et al. (2023a). and Physion (Bear et al., 2022). We also consider the real-world setting using Physics101 (Wu et al., 2016). In all these cases, show the advantage of our formulation compared to previous point track forecasters as well as video generators.
To summarize, our contributions are as follows:
1. We formulate the problem of mmage-based motion forecasting as generating the trajectories of a grid of image points, utilizing an architecture inspired by popular video generators.
2. We show that our model outperforms previous point track forecasters because it uses generation instead of regression, which models uncertainty, and considers points across the scene instead of focusing on a small subset of active points, which captures context better.
3. We further compare learning trajectory predictors from scratch to using state-of-the-art video generators pre-trained on billions of videos (further fine-tuned on our experimental domain) and show that the former can learn motion more efficiently and accurately.
4. We evaluate models using several different synthetic and real scenarios, and include metrics that test directly for certain aspects of physical plausibility such as rigidity.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Visual motion forecasting. We consider the problem of forecasting possible motions of objects. expressed as points, in a scene, given single image. Several variants of this task have been studied in the literature. Among the most relevant are Any-point Trajectory Modeling (ATM) (Wen et al., 2024b), Yang et al. (2025) Tra-MoE (Yang et al., 2025) and Track2Act (Bharadhwaj et al., 2024b) which focus on robotic control. ATM uses CoTracker (Karacy et al., 2024b) to pseudo-label the LIBERO dataset (Liu et al., 2023a) by tracking the motion of a robotic arm performing object manipulation tasks. Tra-MoE improves ATM with a mixture of experts. Similarly, Track2Act pseudo- labels both real-world action (Goyal et al., 2017; Damen et al., 2022) and robotics datasets (Brohan et al., 2022; Walke et al., 2023). ATM and Tra-MoB deterministically regress 32 points, while Track2Act generates 400 trajectories using diffusion (Ho al., 2020). However, these methods focus only on active points. placed on the robot actuator and targets, and condition their predictions on goal (expressed as a goal image in Track2Act). In contrast, we predict trajectories for many more points, placing them uniformly on a grid, and forecast them independently of whether they should be static or dynamic. This way, we cover the whole scene, modeling motion arising from the properties of the world. Recently, Pandey et al. (2024) introduced a training-fre free method that leverages Motion-I2V (Shi et al., 2024), a pre-trained image- to-flo model, to discover potential object motion within given image. The method uses hand-crafted energy functions to guide the image-to- tos-tto-ffooo to-flow generator, aiming to separate object and camera movement. However, it can only handle single, pre-segmented object, and it is constrained by the underlying image-to-floe flolow generator. Walker et al. (2016) also consider image-to-motion generation but take a different approach: they use DCT-based linear compression to encode trajectory offsets and employ VAE to generate future point trajectories directly from static images. This approach has two key limitations. First, the linear DCT compression lacks both the expressivity and the regularized latent structure that would facilitate effective generative modeling. Second, their evaluation metrics do not assess whether the sampled future trajectories are physically plausible or whether the distribution of generated samples accurately captures the true multi- modal nature of possible outcomes. Li et al. (2018) also consider generating motion from still image, but focuses on image-to-video generation rather than motion prediction itself. They formulate image-to-video task as two-phase process: first predicting future optical flow maps from a static image, then translating these flow maps into RGB frames. This approach has several key limitations. First, optical flow motion representation cannot guarantee long-term temporal consistency due to independent frame-pair estimation, leading to error accumulation, and fails under occlusion. In contrast, CoTracker (Karaev et al., 2024a) and AllTracker Harley et al. (2025) estimate jointly to ensure better temporal consistency and tracking through occlusions. Second, while both methods employ VAΣs, their roles differ fundamentally: they use a VAE to directly generate flow maps, whereas we use a VAE to construct regularized latent space for generative modeling with rectified flow, providing better structure for sampling diverse futures. Third, similarly to Walker et al. (2016), their evaluation relies solely on RMSE between predicted and ground-truth flow maps, whereas we rigorously assess whether sampled futures preserve physical plausibility (rigidity), capture the true distribution of possible outcomes (FVMD), and generalize to dui-of-distribution object shapes.
point trajectories, which state-of-the-art trackers like
Measuring generation quality. Several metrics have been proposed to evaluate the quality of image and video generation. These include IC (Salimans et al., 2016), FVD (Unterthiner et al., 2018), VBench (Huang et al., 2024), and VideoPhy (Bansal et al., 2024). Each set of metrics captures different aspects of generation quality, such as fidelity, diversity, and physical plausibility. VBench (Huang et al., 2024). in particular, considers the quality of motion by assessing whether generated frames can be adequately interpolated with a pre-trained video interpolation model. This, however, only checks whether the motion is smooth and predictable from adjacent frames. We concentrate on motions that are both accurate and plausible.
3 METHOD
In our work, a point trajectory is a sequence of 2D pixel coordinates ((zy. Bo), (I1, y1) ........ (xT, yr))
describing a point's 's position over time, starting from its initial position (Io, 3o). We formulate image- based motion forecasting as the problem of predicting the trajectories of a quasi-dense grid of image pixels, representing the motion of the objects contained in given image.
3

---

Published as a conference paper at ICLR 2026
Imoge
Emmoddr
Flow Field
Decoder +
34p
Quay
Trajectory Laceuts
Selve (lx..))
Trajectory Lateats
Figure 2: Method overview. We generate future trajectories from a single image using a flow matching denoiser that operates in the latent space trajectory VAE.
Formally, the image is tensor I Hx where = and W are the image height and width in pixels. We predict the motion of the image points for 'T' steps. The density of the tracks is controlled by sampling a grid of tracked points with stride 1. Hence, the trajectories form a tensor x E R xTx2, Our goal
3 is the number of color channels, and H
predict from I.
Because this prediction problem is highly ambiguous, we cast it as learning a conditional distribution over possible trajectories. Thus, we take x to be a sample from a random variable X and learn the distribution p(X I).
This task is similar to video generation, except that, instead of generating RGB values, we generate point coordinates. Consequently, we construct this generator using techniques similar to those underlying modern video generators. In particular, we adopt a latent flow matching approach to trajectory prediction (Fig. 2). This involves encoding the trajectories in compact latent space (Section 3)1) and then learning a denoising neural network operating in this space (Section 3.2). Both components use similar neural network architecture (Section C.1).
3.1 TRAJECTORY LATENT SPACE
Rather than generating the trajectories directly, generate corresponding latent code obtained using a variational autoencoder (VAE) (Kingma & Welling, 2013). The VAE comprises an encoder function mapping x to (the mean and variance of) latent code and a decoder function mapping z back to x.
The code e R xTxD is a tensor with shape similar to but with an additional spatial downsampling factor r E NN and a latent dimension D. Since our generative model operates on short windows E {16, 24, 30)), we not compress time.
Because the trajectory grid is only (quasi- (quasi-)dense and may not cover all parts of an object, we provide the corresponding image as input to both the encoder and decoder. This auxiliary input improves the model's ability to reason about object boundaries, shapes, and geometry. The encoder (μz, oz) xTx2 x HxCC +
ø(x I) is thus mapping Φ IR outputting the parameters of a Gaussian distribution N No with mean uzz and variance oz. The decoder x (z I) is mapping vh # W. xTxD l8 HxWXC mean of the reconstructed trajectories.
R = xTx2
R 12 xTx2
R w xTx2 outputting the
The model is trained with a B- 3-VAE objective (Higgins al., 2017), using Huber loss Ls for reconstruction and KL divergence to regularize the Gaussian code posterior Nocxn) with respect to the normal distribution Ng. For training sample (x, I), the loss is:
Ca-VAE(0, Φb ER~Noosn [Ls (x, (iz D))] B DKL (Nacs)) |
We implement both the encoder and decoder using symmetric spatio-temporal transformer, dis- cussed in Section C.I. 1. To downsample, we fold the temporal dimension into the batch dimension, encode (spatial) trajectories, and patchify. To encode trajectories,
concatenate normalized spatial coordinates (x, y) with their learnable Fourier features (Li et al., 2021). These encoded trajectories then patchified with non-overlapping 2D convolution with kernel size and stride r, which
4

---

Published as a conference paper at ICLR 2026
outputs the number of channels matching model embedding dimension. Next, we unfold the temporal dimension to obtain tensor hene E R where D is the model dimension. A spatio-temporal transformer encoder processes hone and finally linearly projects the embeddings to the mean and variance of the latent code distribution, yielding a latent representation 2 E R x xTxD after sampling with the reparameterization trick (Kingma & Welling, 2013). To decode, we linearly project the latent code to a hidden representation haee E 7 # x " xD This representation is then processed with spatio-temporal transformer decoder, matching the encoder. Finally, the outputs of the decoder are projected to the trajectory patch dimension and assembled into the trajectory grid x E R # x xTx2
3.2
SAMPLING TRAJECTORIES USING FLOW MATCHING
Having mapped the trajectories to latent space, we can now learn generative model for the latent code z, namely the conditional distribution pZZ[I). We do with rectified flow flow matching formulation (Lipman al., 2022; Liu et al., 2023b).
Briefly, let Z Z be distributed as p(Z and let Zo ~ NO, I) be normally distributed. Define straight path z, = (1 - Zo tzj connecting the noise sample Zg to the target sample z1. The velocity of the path at any intermediate point z, is constant and given by v(z,, Zo: t) = de z = 21 Zo. We learn neural network 6(z,, t) that estimates the expected velocity with respect to all paths passing through 2, at time t. (conditioned on D), by minimizing the Rectified Flow (RF) loss:
CRV (xr (b) TT). (Z) 1 I, v(z,, Zo. +)1
ZE = (1 t)zz + tZ1.
Here, Zo (0, 7) is a normal sample, (21, I) is a training sample, and f Uniform (0, 1] is a random time step. At test time, we draw samples from the target distribution p( (I) by first sampling Zo ~ (O /) from the normal distribution and then moving it towards = ž along the path defined by the velocity field o(z,, I), which amounts to integrating an ODE.
3.3 TRAINING
During training, only one ground truth future is observed for each initial condition, which reflects the setting where real data provides only one ground truth future. At inference time, however, we aim to produce multiple plausible hypotheses. This is challenging because the model must infer the existence of multiple possible futures from nearby training examples. However, simple interpolation between training samples is not necessarily physically plausible. For example, naive interpolation between rigid motions does not, general, yield a rigid motion. This setup very different from training lext-to-image/videe models, where we have thousands of examples matching a caption.
3.4 MEASURING THE GENERATION QUALITY
Predicting possible scene motion from a single image is a highly ambiguous task. Hence, regression metries, which assume single possible ground truth output, Best-of-K Mean Square Error (MSE), which is the lowest eттor obtained between pairs of generated and simulated trajectories x for each image I. To compute this, we simulate possible trajectories for each image I (randomizing the initial velocities) and compare them to K generated trajectories x.
suitable. We instead report the
We also assess the statistical plausibility of the generated trajectories using motion distributional metrics. In particular, we use the FVMD (Liu et al., 2024) metric, which calculates the Fréchet distance between generated and simulated trajectories using histogram- based features. However, because FVMD compares the generated and simulated versions of the marginal distribution p(X) ET ([[]). does not evaluate whether the generated motions X are plausible for specific image I. To address this, also compute the FVMD image-wise (FVMD (S)) evaluate the conditional distribution p (I()), which is feasible since we generate and simulate multiple trajectories per image.
Finally, we evaluate the physical plausibility of the generated motion. Using the mask of each object (available as part of the simulated data), we identify which trajectories belong to each object. We then measure whether these 2D trajectories could have arisen from an underlying rigid 3D object. As a 'rgiidity' metric, we repurpose the method of (Karazija et al., 2024), which posits that trajectories stacked into matrix should exhibit low-rank structure. We define the LRTI. score the mean
5

---

Published as a conference paper at ICLR 2026
Frobenius norm between predicted trajectories, collected into matrix, and their truncated SVD reconstruction at rank 5. Intuitively, if objects fail to maintain shape or if not all points move cohesively, the LRTL score increases because the reconstruction cannot adequately represent such linearly independent motions.
FVMD and LRTL are complementary metrics. For instance, FVMD may fail to detect if point velocities are shuffled within the spatio-temporal window used to compute motion statistics. On the other hand, LRTL minimized if there is no motion at all, even if the generated trajectories are dissimilar to the ground truth.
4 EXPERIMENTS
In this section, we begin by comparing our method with regression-basee baselines to highlight the effectiveness combining stochastic trajectory generation with grid-hased global scene reasoning. We then present comparisons with generative methods. We show that our method surpasses generative trajectory baseline, underscoring the advantages of our proposed architecture. We also present results comparing with large-scale image-to-ving -ideo generators, which we apply for the motion prediction task. We further study why RGB video is a suboptimal proxy for modeling motion, underscoring the importance of point trajectories the appropriate modality for motion generation. Finally, we conclude with results on real-world data.
4.1 COMPARISON WITH REGRESSION METHODS
We compare our method with ATM (Wen et al., 2024a) and Tra-MoE (Yang et al., 2025), regression- based trajectory predictors, using the LIBERO robotics datasets. For these methods, official implementation and the checkpoints. ATM and Tra-MoE reported the success rate of policies trained on generated trajectories, without directly assessing the quality of the generated trajectories themselves. Here, we are interested in the motion prediction problem; we thus adopt the MSE evaluation metric, since we have only one ground truth. Since ATM and Tra-MoE regress trajectories from the initial frame and a text instruction, we extend our method with text conditioning, using the same BERT model to process text as the baselines. Details are in Section D.4.1. For evaluation, we favour the baselines by choosing trajectories according to the number of points and filtering scheme they employ during training. In contrast, our method directly predicts trajectories at every other pixel, which include evaluation points subset. To fairly compare with regression baselines, we compute results for our method in three ways:
their
McanTT Calculate the average of k samples to form the average predicted trajectory. The mean prediction used evaluate metrics, checking whether a method recovers the correct mode. Mean Compute metric for each & samples, averaging the results. Min Compute metric for each & samples, taking the minimum of the results.
Table 1: Comparison with ATM on LIBERO datasets using MSE
LIBERO-90
LIBERO-10
Model
Side
Effector
Side
Effector
ATM (A: = 1)
23.07
67.37
31.02
69.96
Ours (MeanT, k 8)16.70
52.70
23.69
58.35
Ours (Mean, l: = 8) 18.32
60.47
26.71
66.35
Ours (Min, 8) 10.99
32.01
13.86
35.93
Tables and show that our method considerably outperforms the baselines, whether we form an average trajectory (MeanT) or consider individual samples for expected performance (Mean) or the best-case scenario (Min). This suggests that modeling uncertainty in motion generation is more important than domain -emain- architectural changes, such as the mixture of experts in Tra-MoE. Results for different & are in Section A.1. We also study qualitative outputs (Fig. 3), where method can sample diverse yet consistent predictions. We attribute this to modeling full scene motion using grid. This is particularly important given the effector view, where camera motion is uncertain.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with Tra-MoE LIBERO datasets using MSE.
GOAL
OBJECT
SPATIAL
LIBERO-10
Model
Side
Effector
Side
Effector
Side
Effector
Side
Effector
Tra-MoE (k 1) 27.56
105.92
14.07
48.78
37.62
88.22
40.54
82.23
Ours (MeanT, 8)15.85
71.41
8.94
30.65
15.46
54.71
26.20
63.62
Ours (Mean, k 8) 17.46
87.38
10.26
36.91
16.91
65.48
31.73
78.50
Ours (Min, 8) 10.52
37.41
5.57
10.95
33.25
13.52
34.58
Input
GT
ATM
Ours
Side View
Elfector View
Figure 3: Qualitative comparison on LIBERO for the task "Pick up the book the right and place under the cabine! shelf". Unlike the baseline (ATM), we sample diverse predictions for entire scene, particularly beneficial for the uncertain effector view, where camera is attached to the effector.
4.2 COMPARISON WITH GENERATIVE METHODS
Table 3: Motion generation quality on Kubric (Greff et al., 2022). Our method shows better adherence the ground truth motion over multiple metrics. f. model fine-tuned Kubric dataset.
Model
FVMD
FVMD (S)
Best of K
LRTL
Diffusion-Bared Trujectory Generators
Track2Act (Bharadhwaj al., 2024b)
16735
22509
250.8 127.0
15.8
Ours (I.)
13745
17838
14.1
Diffusion- Sifueeio-BBsedd Video Generators
WAN 14B (Wan et al., 2025)
34573
42987
184.6
35.1
Stable Video Diffusion (Blattmann et al., 2023)
30173
39494
235.7
37.2
LTX- -iddeo (HaCohen al., 2024a)
24722
32019
205.1
17.0
WAN 1.3B (Wan al., 2025)
23608
30712
192.6
42.1
DynamicCrafter! (Xing et al., 2023)
413998
50123
239.9
51.8
Stable Video Diffusion! (Blatmann al., 2023)
17099
22799
152.2
30.1
WAN 1.3B* (Wan et al., 2025)
14864
20010
162.8
26.6
Table Motion generation quality on an out-of-distribution version of Kubric. 1_ model fine- tuned to Kubric dataset.
Model
FVMD
FVMD (S)
Best of K
LRTL
Diffusion- -Based Trajectory Generators Track2Act (Bharadhwaj et al., 2024b)
15751
19608
278.6 127.2
19.7
Ours (L)
12221
14949
15.9
Diffusion Beetusion-BBeeddd Video Generators
DynamicCrafter! (Xing et al., 2023) Stable Video Diffusion! (Blattmann et al., 2023)
43248
49092
230.5
58.8
16113
19780
127.7
31.7
WAN 1.3B\ (Wan et al., 2025)
13253
16547
128.2
27.3

---

Published as a conference paper at ICLR 2026
Table 5: User study. Our model ranked better than (SVD and WAN 1.3B) 52% of the time.
Model
1st (%) 2nd (%) 3rd (%)
ELO
SVD
20
42
929
WAN 1.3B
28
36
36
987
Ours
52
26
22
1084
We now conduct an in-depth evaluation of our proposal with various generative approaches that model trajectories pixels for predicting motion. We train our model on the MOVi-A variant of Kubric (Greff et al., 2022), which features geometric primitives of various colours being launched into the centre of the scene, falling, and colliding. For evaluation, containing 16 scenes. In each seene, we sample 64 different initial velocity configurations, resulting in 1, 1,024 unique evaluation settings. We condition both our method and the baselines to predict the next 23 frames based on an initial frame. CoTracker3 (Karaev et al., 2024a) is used to obtain trajectories following video generation for baselines. Tables 3 and 4 contain the results.
generate a new split of data
We compare against Track2Act, a method that also generates point trajectories. Specifically, Track2Act represents an image as a single vector encoded by a small ResNet18, flattens trajec- tories, and performs standard attention. In contrast, we perform spatio-temporal cross-attention with all patch tokens from the conditioning frame, encoded by DINOv2 (Oquab et al., 2024). Our method significantly outperforms Track2Act, despite it being more than twice large. We attribute this to the stronger inductive bias offered by cross- coossaattnntion This provides more information about object location and geometry, resulting in lower LRTL. Further, in Table 14, we show that the same conclusions hold when we train our method on pseudo ground truth trajectories from CoTracker3.
Next, we explore whether image-to-video generators can solve our task without fine-tuning. Despite significant pretraining and assumed generality, they struggle. We observe that LTX (HaCohen et al., 2024a) shows low LRTL score but high distributional metrics. Upon inspection, the model struggles to maintain the object shape and generates sudden motions, causing point tracking to fail.
We further fine-tune all three video baselines the Kubric dataset minimize domain gaps and give the generators an opportunity to learn the motion patterns and invariants present in the data. Even after this adjustment, our model continues to outperform all the baselines by a clear margin. Notably, trajectory-based methods tend to produce far more rigid motion, as reflected by their substantially lower LRTL. This result supports our hypothesis that RGB-space generation introduces excessive overhead, leading to implausible non-rigid motion, visible as object shape inconsistencies (Figure 1).
We also carry out a similar quantitative evaluation using the outtt--distributiin version of the Kubric dataset, which we generate using different set of object primitives. Table 4 shows that our method performs favourably in this setting as well, though all methods show increased metrics, indicating slightly affected performance out-of-distribun
Finally, we validate the results of our quantitative evaluation with a user study. We choose the methods according to Best-Of-K. We show 16 different scenes to 20 users and ask them to rank three models in order of preference for what they think is the most plausible, realistic depiction of future scene motion. More details are in the supplementary. We report the results in Table 5. We find that our model ranks as the best 52% of the time, with an ELO score significantly higher than other methods.
4.3 SIGNIFICANCE OF MODALITY FOR MOTION GENERATION
Motivated by the results in Tables 3 and and the qualitative evidence in Fig. 1, we hypothesize that motion implausibility arises from the overhead associated with pixel-level RGB generation. Specifically, RGB synthesis requires the model to allocate its capacity to low-level appearance factors such as lighting and texture, thereby reducing its focus on motion accuracy or physical plausibility. To evaluate this hypothesis, we fix our base architecture and ablate only the output modality. We downsample RGB such that the latent shapes of the RGB generator and the trajectory generator comparable. We first verify that CoTracker3 reliably extracts motion from video generators (Table 23) and that RGB VAEs achieve high reconstruction accuracy (Table 7). This confirms that motion errors
8

---

Published as a conference paper at ICLR 2026
Table 6: Switching modality from RGB to point trajec-
Table 7: RGB VAEs reconstruct Kubric with minimal error..
tories considerably improves motion generation quality. Joint diffusion of RGB and trajectories substantially im- proves the motion quality extracted from generated RGB.
VAE PSNR SSIM LPIPS
Kubric (I--issributton))
Latent Space
Latent Shape
FVMD FVMD (S) Best-Of-K LRTL.
SVD
36.06
0.97
0.04
Kubric (In (i-DDistribtion)
SD3.5
37.31
0.99
0.02
SVD
24x16x 16x4
20589
26789
195
48.5
WAN 37.36
0.97
0.03
SD3.5
24x 16x
16592
21869
147
33.7
Kubric
WAN
7x16x
17320
22867
160
31.1
SVD
31.76
0.94
0.05
SD3.5
Tracks 24x24x 16
15399
20414
136
28.2
SD3.5
33.86
0.97
0.03
Tracks
24x16x 16x8
12221
14950
127
15.9
WAN 32.88
0.95
0.04
Kubric (Ou- Of-Diistributinn)
SVD
24x 16x 16x4
18740
22865
155
49.3
SD3.5
24x 16x 16
13661
17004
115
28.0
WAN
7x16x16x16
15155
18761
132
30.0
SD3.5 Tracks 24x24x 16x
13386
16694
109
24.6
Tracks
24x16x 16x8
12062
14748
129
18.4
in generated videos stem from unrealistic motion, not tracking or autoencoding artifacts. We train all RGB and trajectory generators under the same setup: identical training procedure, duration, and hardware. Table 6 strongly supports our hypothesis. In particular, trajectory-basee flow matching generates motion that significantly closer to the ground truth distribution while better respecting the rigidity invariant. Since our trajectory model operates on latents of comparable dimensionality (Table 6), this is not due to the reduced dimensionality, but rather to the superior choice of modality.
Motivated by this observation, we then explore image-to-video generation a downstream application and present preliminary evidence that our trajectory generation method can be used enhance motion quality in synthesized videos. To this end, as before, we adopt fixed base denoising architecture while varying the imput modality, For video generation, we select the StableDiffusion..5 (SD3.5) RGB latent space, which demonstrates superior motion quality compared SVD and WAN (Table 6). Then, we train a specialized VAE that encodes 32x32 trajectory grids conditioned on 128x128 images. This trajectory VAE is the same our 256x256 VAE but operates lower resolution to align the latent tensor dimensions with the RGB VAE.
We then train two generative models: (1) an RGB-only model that generates video from the SD3.5 latent space (SD3.5 in Table 6), and (2) a joint model that simultaneously generates video and trajectories from both the SD3.5 RGB latent space and the trajectory VAE latent space (SD3.5 + Tracks in Table 6). The RGB-only model generates 16-channel SD3.5 latents, while the joint model generates 24-channel inputs latents by concatenating RGB latents ( (16 channels) with trajectory latents (8 channels). This concatenation is sensible because the trajectory VAE grid structure matches the RGB VAE grid structure. the joint model, we discard the directly generated trajectories and extract trajectories from generated RGB using CoTracker3, This evaluation measures the motion quality of the generated RGB frames themselves, rather than the explicitly generated trajectories.
As shown in Table 6, jointly diffusing RGB frames and trajectories significantly improves the quality of motion extracted from the generated RGB. This improvement is consistent across all metrics. The substantial gain in the rigidity proxy metric (LRTL) indicates that the generated motion is more rigid and thus more physically plausible. This result further confirms the advantage of directly generating motion represented point trajectories.
4.4 RESULTS ON REAL-WORLD DATA
We further evaluate our method on two real-world datasets: Physics101 and Cityscapes. Physics enables study of object physical properties from unlabeled videos, while Cityscapes provides urban driving scenarios captured across 50 cities with diverse egocentric motion and viewpoints. To- gether, these datasets allow us to examine broader range of physical phenomena and interactions while assessing our method's performance on motion forecasting under unconstrained, real-world conditions.

---

Published as a conference paper at ICLR 2026
Physics101: Physics101 consists of roughly 10000 video clips containing 101 objects of various materials and appearances (shapes, colors, and sizes). We evaluate five different physical scenarios, namely fall, liquid. multi, ramp, and spring. Our evaluation set contains 1450 different initial conditions, with a single ground truth per initial condition. Due to the high cost of sampling from video generators (Table 22), we sample ground truth from CoTracker3. As only a single ground truth is available, we report MSE.
from each method and compare with the single pseudo
Results in Table 8 show that overall our method shows comparable or better performance than large-scale fine-tuned WAN, with better performance overall. Analysis in Figure 6 shows that our method produces fewer outliers. In many cases, achieves 10x lower MSE.
Table 8: Comparison with WAN on Physics101 using MSE. Our method outperforms WAN overall and in 3/5 evaluated scenarios, including the most complex Multi scenario.
Physical Scenario
Model
Fall
Liquid
Multi
Ramp
Spring
Overall
WAN (1.3B)
16.05
4.48
21.88
37.53
70.48
30.08
Ours (B)
19.78
6.00
15.65
36.35
65.31
28.62
Cityscapes: Our Cityscapes training set consists of approximately 3000 video clips of 30 frames of resolution 224 x 448, captured across 19 different cities, towns, and open roads. For evaluation, use 500 validation clips from three unseen locations. We employ CoTracker3 to generate pseudo ground truths for both training and evaluation. We evaluate on 28 x 56 trajectory grids, sampling from each method times, every time with a different seed. Given that only single pseudo ground truth future is available per clip, we adopt the evaluation protocol from Section 4.1.
As shown Table 9, our method substantially outperforms both Track2Act and fine-tuned WAN across all metrics, producing more accurate motion overall. Interestingly, WAN frequently hallu- einates in scenes where the car makes turn (Figure 11), generating RGB outputs that confuse the point tracker, while Track2Act often fails to account for (distant) objects in the scene, such as pedestrians or cars. These limitations may be consequence of their limited input image conditioning: Track2Act uses small pre-trained ResNet to represent the entire scene as a single vector for adaptive normalization. In contrast, we condition on patch-level tokens from DINO through cross-attention in every block, providing richer information about the input image. Figure 12 demonstrates that our method also works on soft bodies, such as pedestrians.
Table 9: Comparison Cityscapes using MSE. Our method significantly outperforms both Track2Act and the fine-tuned WAN 1.3B in motion forecasting.
Model
MeanT
Mean
Min
Track2Act
7037.88
9305.63
4393.58
WAN (1.3B)
2650.59
3495.43
1799.04
Ours (L) 1475.68
1565.06
1240.6
Finally, in Section A.4, we present a qualitative study to showcase our method's ability to generalize from synthetic training data from Physion (Bear al., 2022) to real-world scenes that we recorded. Reproducibility details, evaluation procedures, and design choice studies are in the Appendix.
5 CONCLUSION
In this work, we address motion anticipation from single image by formulating as the conditional generation of dense trajectory grids. Our results highlight the benefits of modeling uncertainty in the motion of the entire scene over prior trajectory regressors and generators. We extensively evaluate our approach in simulated settings, assessing diversity, physical consistency, and user preference. We also show that large-scale pretrained video generators underperform in motion prediction, even in simple physical scenarios such falling blocks or mechanical interactions, simulated or real data. By switching the output modality of our method, we experimentally show that this limitation arises from the overhead of generating RGB pixels rather than directly modeling motion trajectories.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03522 2024. 3,
Fan Bao, Chongxuan Li, Jiacheng Sun, and Jun Zhu, Why are conditional generative models better than unconditional ones?, 2022. URL https: //axxii//////vv wwwg/oorg/oooo abs a2t// 62. 34
Daniel M. Bear, Elias Wang, Damian Mrowca, Felix Binder, Hsiao- Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating physical prediction from vision in humans and machines, 2022. URL https: ///a/aaavvvv/////////////aaa//aa org/ abs /2006 06.06...00 08261. 10
Homanga Bharadhwaj. Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arX1v:2409.16283, 2024a. 2
Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2Act: Predicting point tracks from internet videos enables generalizable robot manipulation. In Proc. ECCV, 2024b. 1, 7,33
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. URL
https: / /ttp://aaiivvvv/ org/ abss /2ts//211111 2311.15127.7
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Hawley, Jasmine Hsieh, Jost Tobias Hsu, Julian Ibarz, Kanishka Jain, Ryan Julian, Kenz Konolige, Sergey Levine, Yao Lu, Lluis Castrejon Luu, Henry Luo. Michael Memory, Sumeet Nakaema. Janavi Patravali, Ingrid Peng, Sudeep Peri, Rawiparrot Quilbe, Abrin Rajeswaran, Nikhil Rao, Khem Retana, Daniel Riser, Pierre Sermanet, Balakumar Singh, Anikait Singhal, Zhuo Tan, Alex Tchuiev, Jose Toloba, Vincent Vanhoucke, Filipe Veiga, Ted Wu, Fei Xu, Yan Xu, Zheyuan Xu, Jiaming Yan, Andy Yau, Helen Ye, Peter Yu, Tianhe Yue, Andy Zeng, Shuang Zhang, Aleksandra Antonova, Misha Bajracharya, Steven Bohez, Betsy Boling, Konstantinos Bousmalis, Shixiang Chowdhury, Daniel Collins. Todor Davehev, Yotam Derudder, S. M. Ali Eslami, Andrew Garcia, G. Garcia, Diego de Las Gasso, Kamyar Ghugre, Ofir Gottesman, Fangchen Gu, Ted Hand, Jonathan Harris, Linda Hee, Daniel Hennes, Kuhan Hertkorn, Nick Ho, Alex Huang, Brian Irpan, Itamar Ito, Shruthi Jariwala, Takayuki Jeong. Kyle Johnson, Smit Joshi, Leslie Pack Kaelbling, Dmitry Kalashnikov, Igor Kamenev, Masashiuristic Kaneeda, Jiri Kloss, Allen Ko, Robert Ku, Andy Kudo, Peter Le. Tsang-We Wei Edward Lec, Chen Li, Yunfei Li. Zhen Lin, Edward Liu, Po-Wei Liu, Yang Liu, Yu- Yei Liu, Kyle Luhman, Stefan Lundberg, Yutaka Ma, Ryan Mahdavi, Viktor Makoviychuk, Vaibhav Malik, Coline Marcelo, Yevgen Markov, James Martin, Roberto Martin- Corey McHugh, Staton McMahon, Clayton Merrill, Jonathan Michelman, Toki Migimatsu, Alborz Milstein, Peter Mineault, Igor C. Mordatch, Erfan Morena, Sripriya Naga, Vidhya Nadan, Sriram Narasimhan, Kyle Oslund, Alexander Pachev, Krishnan R. Parbigata, Peter Pastor, Dmitry Pavlichenko, H. Charles Pham, Michael Piekutowski, David Pinkas, Ivan Popov, Anish Purohit, Ilija Radosavovic, Kanishka Rao. Robert Reid, Jessica Reyes, Michael Ritter, Christopher Rivera, Ricardo Rodriguez, Fredrik Rooms, Krishan Roy, Michael Ryoo, Cameron Salter, Suvranu Sankar, Stefan Schaal, Eric Schrader, Pannag Shah, Gregory Shakhnarovich, Junlin Shen, Ethan Sherman, Enna Shteto, Albert Song, Cameron Sowell, Hubert Stokking, Daniel Su, Ansh Sud, Alex Taksin, Arthur Tan, Garrett Thomas, Altay Topcubasi, Eric Tung, Ekin Tzeng, Patrick Van Der Smagt, Mel Vecerik, Paulo Veiga, Bo Wang, Eric Wang, Karl Welker, Cameron White, Paul Wojcik, Andy Wong, Chien-Yar Xiao, Peng Xu Xia, Jing Yang, Tianli Yang. Michihiro Yasunaga, Amir M Yazdani, Fei Yi, Sherry Young, Mengyuan Zhang, Tianhao Zhang, Yifeng Zhu, Yixin Zhu, and Joseph Zito. RT-1: Robotics transformer for real- world control at scale. arXiv preprint 2rXiv:22 2022.

---

Published as a conference paper at ICLR 2026
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. Technical report, OpenAI, 2024. 1,2
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kaza- kos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-ION 1000 International Journal of Computer Vision (JCV), 130(2):331-355 2022. doi: 10.1007/s| 1263-021-015333
Jingtao Ding, Yunke Zhang, Yu Shang. Yuheng Zhang, Zefang Zong, Jie Feng. Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, Fengli Xu, and Yong Li. Understanding world or predicting future? comprehensive survey of world models. arXiv, 2411,14499, 2024.
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: tracking any point with per-frame initialization and temporal refinement. In Proc. CVPR, 2023. 32
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow trans- formers for high-resolution image synthesis, 2024. URL pettps://arxte wwrgivvorrgooooooo wwtp/rgg///aaeeeee//.//eeeeeaeee 03206. 31
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller- Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In Proc. ICCV, 2017. 3
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gmanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. In Proc. CVPR, 2022. 2, 7,8
David Ha and Jürgen Schmidhuber. World models. arXiv, 1803. 10122, 2018. 2
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.000 100003,,0 2024a. 7,8
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Bitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion, 2024b.
URL attps://azzes
org/ abs/ /25//25000 200.. 00103. 3,
Adam W. Harley, Yang You, Xinglong Sun, Yang Zheng, Nikhil Raghuraman, Yunqi Gu, Sheldon Liang, Wen-Hsuan Chu, Achal Dave, Pavel Tokmakov, Suya You, Rares Ambrus, Katerina Fragkiadaki, and Leonidas J. Guibas. Alltracker: Efficient dense point tracking at high resolution, 2025. URL https: /ttpvvaaivvv////////////.././.aa org/ abs /2006. 07310.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-VAB: Learning basic visual concepts with a constrained variational framework. In Proc. ICLR, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan Hsuan-Tien Lin (eds.), Proc. NewrIPS, 2020. 35
12

---

Published as a conference paper at ICLR 2026
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench; Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21807-21818, 2024. 3
Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective, 2024. URL https: /taps:://axiii: otgg
02385. 2, 36
Nikita Karaev, Jurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos, 2024a.
URL htttt:h//aaxiiii org/ abs abss/4100 11831. 3, 8, 30
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together, 2024b. URL https:t/axxiv...omm org/abs/
07635. 3,330
Laurynas Karazija, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Learning segmentation from point trajectories. Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2024.
D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 4,5
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences, 2023. URL httpss://arxiivoorrg abs/2310 08576.
Ruining Li, Gabrijel Boduljak, Jensen, and Zhou, On vanishing variance in transformer length generalization, 2025. URL
wrwg abaaab/s/2200 02827. 35
Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self-supervised 0370000000
representation generation method, 2024. URL
34
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable Fourier features for multi- dimensional spatial positional encoding, In Proc. NeurlPS, 2021. 4
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Flow-grounded spatial-temporal video prediction from still images, 2018. URL peetps:/aarxive org//bs/ 1807.00 095.5.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv.cs, abs/2210.02747, 2022. 1,5
Bo Liu, Yifeng Zhu, Chongkai Gao. Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint 2rX:v:2306.003310 2023a. 2,
Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjie Liao. Fry 'echet video motion
distance: A metric for evaluating motion consistency in videos. arXiv preprint 24Xiv:2407.16124 2024. 5
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In Proc. ICLR, 2023b. 1,5
Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation, 2025, URL https: //arxiv. orroor///aoo abs /20022 03048. 29, 30, 31
Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, and Sai Rajeswar. GenRL: cewimmodalfounndatti world models for generalization in embodied agents. In Proc. NeurIPS, 2024. 1
13

---

Published as a conference paper at ICLR 2026
Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models learn physical principles from watching videos? arXiv, 2501.09038, 2025. 2, 36
NVIDIA, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai. Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge. Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Y Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Alice Luo, Qianli Ma. Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty. David Page. Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren. Vasanth Rao Naik Sabavat, Ed Schmerling. Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang. Heng Wang. Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang. Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang. Yuxuan Zhang, Qingqing Zhao, and Artur Zolkowski. Cosmos world foundation model platform for physical ai. arXiv, 2501.03575, 2025. 1
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang. Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2024. 8
Karran Pandey, Matheus Gadelha, Yannick Hold-Geoffin Karan Singh, Niloy J. Mitra, and Paul Guerrero. Motion modes: What could happen next?, 2024. URL https: :/app:////axiii/////a org/ /bs/ 2412.00148. 3,
Jack Parker-Holder, Philip Ball, Jake Bruce, Vibhavari Dasagi, Kristian Holsheimer, Chris- tos Kaplanis, Alexandre Moufarek, Guy Scully, Jeremy Shar, Jimmy Shi, Stephen Spencer, Jessica Yung, Michael Dennis, Sultan Kenjeyev, Shangbang Long, Vlad Mnih, Harris Chan, Maxime Gazeau, Bonnie Li, Fabio Pardo, Luyu Wang, Lei Zhang, Frederic Besse, Tim Harley, Anna Mitenkova, Jane Wang, Jeff Clune, Demis Hassabis, Raia Hadsell, Adrian Bolton, Satinder Singh, and Tim Rocktäschel. Genie 2: A large-scale foun- dation world model, 2024. URL https: http://eeepedddd/////// google/ / /ddiiisoooo// scover/bloog verie-2-a--1 wae---rree-seee scale e- foundati www.iiiioooooodddddeeee
Shivansh Patel, Shraddhaa Mohan, Hanlin Mai, Unnat Jain, Svetlana Lazebnik, and Yunzhu Li. Robotic manipulation by imitating generated videos without physical demonstrations, 2025. URL https: //arsi///aivv./// org/ abs 2507.5040 00990. 2
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https:
//taxivv.rg/abbbaeeeee/ 221...22...2 09748. 31, 34,
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang. Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Ramb- hatla, Sam Tsai, Samanch Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy. Shelly Sheynin. Siddharth Bhattacharya. Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei- Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng. Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang. John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson. Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: A cast of media foundation models, 2025. URL https: //tp///aavvv////////////////aa org/ abs /41 1..0000...... 13720. 1,
14

---

Published as a conference paper at ICLR 2026
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016. 3
Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang. Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang. Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Motion-i2v: Consistent and controllable image-to-video generation with explicit motion modeling, 2024. URL https:
org/ alss 2101.11977..
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:18/2.01777, 2018. 3
Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. Softmax is not enough (for sharp size generalisation), 2025. URL attps:://arxe org/ /2410.
01104. 35
Rahul Venkatesh, Honglin Chen, Kevin Feigelis, Daniel M. Bear, Khaled Jedoui, Klemen Kotar, Felix Binder, Wanhee Lee, Sherry Liu, Kevin A. Smith, Judith E. Fan, and Daniel L. K. Yamins. Understanding physical dynamics with counterfactual world modeling, 2024. URL https:
//arxiv. org/ abs/ 2312. 0 06721. 20
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng. Tony Z. Zhao, Philippe Hansen-Fstrudd Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. BridgeData V2: dataset for robot learning at scale. In Conference on Robot Learning (CoRL), volume 229 of Proceedings of Machine Learning Research, pp. 1723-1736. PMLR, 2023. 3
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders, 2016. URL 1606. 07873.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang. Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang. Mengyang Feng. Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng. Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang. Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu. Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, 2025. URL hwwps:tpp........... / aaxiii wtwaaaiwiv.g.aaaaaaeeeeaeaaeaaae wrg/abs///aooo....oooo 2503. 20314 4. 1, 2, 7,30
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning, 2024a. URL https: http/axxvvvvi....o.aaa org/ooggg//oo.. 2401. 00025. 1, 6, 34
Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv, 2401 00015,0 2024b. 3
Jiajun Wu, Joseph Lim, Hongyi Zhang, Joshua Tenenbaum, and William Freeman. Physics 101: Learning physical object properties from unlabeled videos. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 39.1-39.12 12. BMVA Press, September 2016. ISBN 1-901725-59-66 doi: 10.5244/C.30.39, URL https: //xx . doi org 5244 C. 30.33.3....
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors, 2023. URL httph:tt//// htapps://axxi....// wrg. 2310 12190. 2190.7
Jiange Yang, Haoyi Zhu, Yating Wang, Gangshan Wu, Tong He, and Limin Wang. Tra-moe: Learning trajectory prediction model from multiple domains for adaptive policy conditioning, 2025. URL
https://arxivv wowlvvorrg/aeee hltg//bb//445 2.11.115111111111111111111111111 14519. 3, 6,
15

---

Published as a conference paper at ICLR 2026
Sherry Yang, Yilun Du Bo Dai, Dale Schuurmans, Joshua B. Tenenbaum, and Pieter Abbeel. Probabilistic adaptation of black- box text-to-video models. In Proc. ICLR, 2024a.
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kael- bling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In Proc. ICLR, 2024b.
Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. In Proc. ICML, 2024c.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
In this supplementary material, provide the following:
1. Ethics Statement
2. Additional results and comparisons
3. Interactive animations (provided in the vi visualizat folder):
(a) Qualitative comparison versus the video-generation baselines on Kubric.
(b) Qualitative comparison versus the robotics baselines.
(C) More real-world examples.
4. Detailed implementation information, including model architecture, training and sampling nyperparameters, and reproducibility settings such as required hardware and estimated reproduction time.
5. Details about the user study.
6. User study scenes (provided in the user Study_scennt folder):
7. The Use of Large Language Models (LLMs) Statement
8. Limitations
A ADDITIONAL RESULTS AND COMPARISONS
A.1 MORE QUANTITATIVE RESULTS ON LIBERO
In Tables and 11, we report the performance of our method with different numbers of samples per initial condition (k). The results indicate that our approach is fairly robust to the choice of k. though using more samples generally leads to better performance across evaluation metrics, highlighting the advantages of diversity in generation.
Table 10: Comparison with ATM LIBERO datasets using MSE.
LIBERO-90
LIBERO- 10
Model
Side
Effector
Side
Effector
ATM (k: = 1) 23.07
67.37
31.02
69.96
h:
Ours (Mean) 17.89
57.64
26.18
63.47
Ours (Min)
17.89
57.64
26.18
63.47
k:
Ours (Mean) 18.31
62.56
27.89
71.98
Ours (Min)
14.90
46.86
21.88
53.41
Ours (Mean)
18.32
60.13
26.22
66.
Ours (Min)
12.77
38.68
16.93
44.33
1: =
Ours (M.anT))6.70
52.70
23.69
58.35
Ours (Mean) 18.32
60.47
26.71
66.35
Ours (Min) 10.99
32.01
13.86
35,93
17

---

Published as a conference paper at ICLR 2026
Table 11: Comparison with Tra-MoE on LIBERO datasets using MSE.
GOAL
OBJECT
SPATIAL
LIBERO-10 10
Model
Side
Effector
Side
Effector
Side
Effector
Side
Effector
Tra-MoB (k 1)27.56
105.92
14.07
48.78
37.62
88.22
40.54
82.23
k ==1
Ours (Mean)
16,69
91.89
10.52
34,00 34.00
17.39
61.36
34.78 34.78
72.00
Ours (Min)
16.69
91.89
10.52
17.39
61.36
72.00
A 2
Ours (Mean)
16.77
92.18
10.10
37.40
17.08
62.92
35.86
84.51
Ours (Min)
14.03
61.68
7.88
27.56
14.07
45.11
27.06
56.88
k
Ours (Mean)
17.04
88.14
10.26
37.19
17.11
62.71
33,44
77.95
Ours (Min)
11.63
46.69
6.76
22.85
12.39
38.69
18.38
43.54
k ==8
Ours (MeanT)
15.85
71.41
8.94
30.65
15.46
54.71
26.20
63.62
Ours (Mean)
17.46
87.38
10.26
36.91
16.91
65.48
31.73
78.50
Ours (Min) 10.52
37.41
5.57
18.08
10.95
33.25
13.52
34.58
A.2 EXTENSION TO MULTIPLE FRAMES
Input Frames
First sample
Second sample
Third sample
Fourth sample
t == 0
= 1
conditioned on input
conditioned on 2 jmput frames
Ground truth
Figure 4: Trajectory generation diversity with varying number of input frames. Input frames and the corresponding ground truth future rollout are shown alongside generated samples with different random seeds. Single-frame input produces diverse trajectories reflecting higher uncertainty without velocity information. Two-frame input yields more accurate trajectories with reduced diversity as velocity observed. Colors distinguish different object instances.
Our model can easily extended support video or multi-frame inputs. Currently, we condition on the initial frame using cross-altention over tokens extracted from the initial frame. To incorporate multiple frames, the model would simply cross-attend to tokens from all conditioning frames, requiring major architectural changes or training setup changes. Please see Section C.1 for details. To demonstrate this, here we train and evaluate a multi-frame version of our method on Kubric.
Theoretical formulation. We train single model that generate future trajectories from variable number of input images. Formally, the multi-frame model learns conditional distributions of future trajectories given a variable number of input images, {p(X I, I2,.. ..... L)) where E T x x2 are future trajectories on a spatial grid and E N denotes the maximum context length.
18

---

Published as a conference paper at ICLR 2026
Implementation. We cross-attend with tokens from all conditioning frames. To encode temporal ordering of the input frames, we add learnable frame index positional encodings to the frame patch tokens before cross-attention. During training, randomly vary the context length. The trajectory VAE remains unchanged, conditioned only on the initial frame I
Evaluation. We evaluate the same model with either 1 or 2 input frames identical test data. The test set the same as in Section 4.1, including 16 scenes each for in-distribution and out-of- distribution object shapes, with 64 rollouts per scene. Each rollout shares the same initial object placement but varies in initial velocity. Single-frame prediction is highly ambiguous due to unknown initial velocity. With two frames, velocity is observable and the Kubric data generation process is actually deterministie. However, occlusions still introduce some ambiguity in predictions. Therefore, we sample predictions per input configuration (i.e., number of input frames) and report the mean squared error (MSH) between generated and ground truth trajectories. Here, we follow the evaluation protocol from Section A.I since the Kubric data generation process becomes deterministic when velocity is observed.
Results. Table 12 shows that multi-frame samples outperform single-frame samples across met- rics and both dataser settings (i.e., in-distribution out-of-distribution object shapes). Qualitatively (Figure 4), multi-frame predictions are more accurate and exhibit lower diversity, reflecting reduced uncertainty when velocity is observed. Notably, Figure 5 shows that the model adapts its prediction diversity based on the amount of conditioning information provided. This also demonstrates that generation from single frame is indeed more challenging and that the model knows how to incorporate more information.
Input Frames
First sample
Second sample
Third sample
t 0
conditioned 1 input frame
conditioned on 2 input frames
First two GT steps
First two sampled steps
Figure 5: Initial steps of samples with varying number of input frames. Input frames and the corresponding ground truth future rollout are shown alongside generated samples with different random seeds. Single-frame input produces diverse trajectories reflecting higher uncertainty without velocity information. Two-frame input enables better estimation of the initial velocity, consequently reducing trajectory diversity and demonstrating that the model correctly incorporates more context.
Table 12: Trajectory generation performance with varying input frames. Mean Squared Error (MSE) decreases substantially when using input frames compared 1 frame across all metrics and datasets. MeanT, Mean, and Min refer to different aggregation methods over predicted trajectories.
MSE (MeanT)
MSE (Mean)
MSE (Min)
Kubric (I--Disribbution) input frame
403.287
478.369
368.784
2 input frames
253.263
264.963
226.989
Kubric
input frame
275.284
305.264
255.165
input frames
219.097
233.268
209.901
19

---

Published as a conference paper at ICLR 2026
A.3 MORE QUANTITATIVE RESULTS ON PHYSICS101
Figure 6: MSE Error Analysis Physics 101. Compared to WAN, method does have extremely wrong predictions (top right corner). Moreover, there are many examples where our method achieves 10x lower MSE (upper left part).
Ourr betser: 56.11
104
WAN (log masi))
Ours WAN
10
WAN better: 4..0%
Ours MSE (Jog scals)
A.4 QUALITATIVE RESULTS ON PHYSION
Input
10
20
30
Figure 7: Real-worid Generalization. Despite training only synthetic data, without motion blur, our model generalizes to real, unseen objects and different viewpoints in the wild.
Due to limited computational resources, we cannot train on large-scale real-world motion datasets. Instead, we investigate whether our method can be trained on more diverse synthetic data and still be effective in generalizing to real-world scenarios. We train our model on Physion Venkatesh et al. (2024), synthetic dataset, and reserve set of real-world scenes solely for evaluation. Because our real-ve real-world dataset is too small for robust quantitative analysis, we focus on qualitative results, shown in Fig. 7. More animations in the supplementary material. In particular, method
20

---

Published as a conference paper at ICLR 2026
transfer knowledge of the physical laws leamndd in the synthetic environment real scenes with unfamiliar objects, backgrounds, camera viewpoints, and textures. It effectively captures and combines multiple physical phenomena, including gravity, collisions, and force propagation. Although the generalization is not perfect, we believe that this provides strong evidence and solid foundation for future work in scaling our method.
A.S COMPARISON WITH MOTIONMODES
Although ModionModes (Pandey et al., 2024) adresses different problem (e.g. it requires segmen- tation mask for the moving part of the scene), we include as baseline because, like our method, generates predictions conditioned on single input image.
use the ground-truth segmentation mask of the entire scene to construct foreground mask, where pixels corresponding to any object are set to and background pixels to 0. This mask is provided to MotionModes. For fairness, we resize the input image to match their resolution and aspect ration. Then, using their official implementation, our model, using identical random seeds. Tracks are extracted following the procedure described in their paper and linearly interpolated from 16 frames (their prediction horizon) to 24 frames (ours). Evaluations are performed on both in-distribution and out-of-distribution samples. Results are Table 13. Our method clearly outperforms MotionModes.
generate the same number of samples as
Table 13: Comparison with ModionModes Kubric. Our method clearly outperforms Motion- Modes.
Model
FVMD
FVMD (S)
Best of K
LRTL
Kubric (In- (n-Diisribution)
MotionModes(Panddg et al., 2024)24698
30125.0
252.7
43.7
Ours (B)
5713.7
8309.7
146.7
8.3
Ours (L)
6470.3
9293.0
131.4
6.4
Kubric )
MotionModessPandeee et al., 2024)21538
25115.0
178.6
43.4
Ours (B)
5118.1
6989.3
160.4
8.5
Ours (L)
5916.9
7926.4
142.4
7.4
21

---

Published as a conference paper at ICLR 2026
A.6
MORE QUALITATIVE RESULTS ON PHYSICS101
Input
t t= 10
== 20
= 30
Figure 8: Qualitative results of our method on Physics 101. Our method can generate the motion of both rigid and non-rigid objects (first row), model force propagation (first and second row), and integrate multiple physical phenomena with different material properties (third row).
Initial Condition
20
Figure 9: Qualitative results of our method on Physics101. The visualization overlays predicted motion trajectories on the initial frame. The color gradient represents temporal progression: blue indicates early timesteps, purple shows intermediate stages, and pink marks later timesteps. Our method can generate the motion of both rigid and non-rigid objects (last three rows), model force propagation (first and second row), and integrate multiple physical phenomena with different material properties (third row and forth row). More examples on the next page.
22

---

Published as a conference paper at ICLR 2026
Initial Condition
10
Figure 10: More qualitative results of our method on Physics101. The visualization overlays predicted motion trajectories on the initial frame. The color gradient represents temporal progression: blue indicates early timesteps, purple shows intermediate stages, and pink marks later timesteps. Our method can generate motion of both rigid and non-rigid objects and understand multiple physical phenomena with different material properties (third row and forth row).
For a clearer understanding of dynamics illustrated in Figures to 10, please refer to the animated visualizations available in our supplementary website materials.
23

---

Published as a conference paper at ICLR 2026
A.7 QUALITATIVE RESULTS ON CITYSCAPES
Input
==
t= 15
Ours
WAN
Figure 11: Qualitative comparison with WAN on Cityscapes. Our method produces more accurate motion, especially in turning scenarios. WAN frequently hallucinates during turns, generating RGB outputs that confuse the point tracker. In contrast, our method maintains coherent scene-wide motion throughout these challenging conditions, involving large camera and object motion.
24

---

Published as a conference paper at ICLR 2026
Input
t= 15
Figure 12: Predictions of our methods on Cityscapes. Our method produces accurate motion of non-rigid bodics, such as humans.
25

---

Published as a conference paper at ICLR 2026
B
IMPACT OF DESIGN CHOICES
Table 14: Ablations of our method on Kubric: estimated tracks (2), no VAE (4), model sizes (1,3,5).
Model
FVMD
FVMD (scene)
Best of K
LRTL
Ours (L)
13745
17838
127.0
14.6
(2) Ours (L) + CT
14766
19123
140.3
16.4
(3) Ours (B)
12221
14950
127.2
15.9
Ours w/o VAE
14519
18551
137.8
23.7
(5) Ours (S)
13065
17119
177.6
23.6
Table 15: Ablation of VAE on LIBERO using MSE (I: = 8). Our method performs better with VAE.
LIBERO-90
LIBERO-10 10
Model
Side
Effector
Side
Effector
Ours (B, VAE) (MeanT)18.66
57.08
23.67
80.33
Ours (B, w/o VAE) (Mean)
20.8
66.69
27.81
92.98
Ours (B, w/o VAE) (Min)
12.55
33.17
14.99
42.47
Ours (B, VAE) (MeanT)
16.70
52.70
23.69
58.35
Ours (B, VAE) (Mean)
18.32
60.47
26.71
66.35
Ours (B, VAE) (Min)
10.99
32.01
13.86
35.93
Table 16: Ablation of VAE Physics101 using MSE. Our method overall performs worse with VAE.
Physical Scenario
Model
Fall
Liquid
Multi
Ramp
Spring
Overall
Ours (w/o VAE)
19.78
6.00
15.65
36.35
65.31
28.62
Ours (VAE)
21.30
4.39
14.51
42.15
75.98
31.67
Source of trajectories. We experiment with changing the target distribution for our model on Kubric from ground truth trajectories those estimated using CoTracker, to support larger-scale, real-world applications, which may rely on estimates of motion. We train both the VAE and the denoiser entirely from scratch using CoTracker trajectories. Comparing (1) (2) in Table 14, training on CoTTracke output causes a modest performance drop relative to using the actual ground truth. Nevertheless, our method performs comparably or better than the strongest alternative method.
Model scale. In Table 14, we also experiment with varying the size of our denoiser between large (L., base (B), and small (S) for (1), (3), (5), respectively. Larger models exhibit less jitter and better geometry preservation, as evidenced by lower LRTI and qualitative results across the evaluation dataset.
Latent space. We investigate whether it is necessary to predict trajectories in a latent space with additional downsampling using the VAE. We adjust the patch size such that both the latent flow matching and raw trajectory models process inputs of identical dimensionality. In Table 14, compare the two settings ((3) and (4)). showing that latent flow matching consistently outperforms flow matching on point coordinates. We further investigate this gap in Figure 13, where evaluate the sample variance' of trajectories as the training progresses. Without a VAE, the denoiser collapses single mode. Since the ground truth is multi-modal, with infinitely many plausible future
Let X E K XXXxxxxXX> he a tensor of K samples for given scene (initial image). The scene sample variance, denoted x( x(X), X),ii is defined
x
(K (e,h.w) ) where ft.h.w
X.t.
KTHW
26

---

Published as a conference paper at ICLR 2026
Scene Sample Variance (x)
300
200
Ours (Latent)
Ours (Raw)
100
100
150
200
250
300
350
400
Epoch
Figure 13: Scene sample variance (k) on Kubric, shown with one standard deviation around its mean over the dataset. Unlike denoising latent codes, using raw coordinates leads single mode collapse.
LRTL.
30
Ours (Latent)
Ours (Raw)
20
100
150
200
250
300
350
400
Epoch
Figure 14: LRTL through training on Kubric. Our method produces more plausible motion with VAE.
trajectories, this collapse leads to poor coverage of the target distribution, adversely affecting the distributional metrics and best-of-K. K. We hypothesize that latent modeling is superior because the VAE latent space is smooth and thus easier model than the raw coordinate space. To qualitatively verify this smoothness, in Figure 15 we show that it is possible to interpolate between distinct sets of plausible ground truth trajectories in the latent space.
Table 15 shows that our method performs better with VAE on robotics data. Table 16 shows that our VAE ablation is less conclusive on real data. Overall, our method performs slightly worse with VAE, but there are 2/5 scenarios where it performs better. A qualitative inspection suggests that the effect arises from increased diversity, consistent with our observations on Kubric (Figure 13). Since our evaluation on real data is less extensive (only one sample initial condition), we place greater emphasis on the results obtained from Kubric and LIBERO in the context of our method performance with VAB.
Input
GT (A = 0)
À = 0.25
A = 0.5
X = 0.75
GT (A = 1.0)
Figure 15: Decoded latent space interpolations, >(A) = (1- Azr, where z, and z, are latent codes of two different sets of ground truth future trajectories for the same initial condition (Input).
27

---

Published as a conference paper at ICLR 2026
Table 17: Effect of sampling density on motion quality. Denser models consistently outperform
sparser ones across all metrics. As grid density increases, the Best-Of-K MSE decreases substantially, demonstrating that modeling more points in the scene produces more accurate motion generation. The best result is shown in bold, and the second best is underlined.
Grid Size
FVMD
FVMD (S)
Best-Or-K
LRTI.
Base
Ours (8x8)
163.43
219.81
147.37
1.77
Ours (16x 16)
162.29
214.00
123.08
1.54
Ours (32x32)
155.94
187.56
78.29
1.50
Large
Ours (32x32)
146.28
186.92
72.75
Sampling density. We evaluate three grid configurations: 8x8, 16x16, and 32x32. For each configuration, train separate VAE using shared architecture and hyperparameters, varying only the input size (sampling density). All VAEs are trained for 400 epochs, with model selection based validation set reconstruction error (MSE). We then train separate denoising models using the base network architecture, following the input modality ablation (Section 4.3). Each denoiser is trained for 400 epochs with checkpoints saved every 25 epochs. Following the protocol in Section 4.3, we evaluate models using three categories of metrics: distributional (dataset-level and scene-level FVMD), pointwise (Best-Of-K MSE), and data-invariant (LRTL). For each denoiser, we generate samples from the final three checkpoints (epochs 350, 375, and 400) and report the best score among them. To ensure fair comparison, all models are evaluated on common 8x8 subgrid of points.
Table 17 contains the results. Denser models consistently outperform sparser ones. As grid density increases, Best-Of-K MSE decreases substantially, indicating that modeling more points on the scene leads to more accurate generation. Simultaneously. LRTI. decreases with higher density, showing that the generated motion becomes more rigid, better matching the data invariant. This improvement is also evident in the qualitative results. Together, these findings highlight contribution: modeling the motion of the entire scene rather than sparse (active) points.
Since the 32x32 model generates at least 4x more points than the bascline, also report results for 32x32 model with a large denoiser (roughly 5x more parameters than the base version, Table 18a) while maintaining the same VAE. These results demonstrate that scaling the architecture yields further improvements and that generating more points benefits from increased model capacity.
28

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
C.I SHARED ARCHITECTURE
The encoder decoder vb, and denoiser are all based on variants of the same architecture, derived from Latte Ma et al. (2025). Originally introduced as a text-to-vided denoiser, we adapt it into an image-conditioned spatio-temporal trajectory transformer that functions as either a VAE or a denoiser. x7x and image tokens
The model operates on two types of tokens: trajectory tokens x f E R P
extracted from the image I.
For the VAE, is produced by encoding and patchifying the trajectory, as described Section 3.1. For the denoiser, corresponds to rescaled latent codes, 1,z where 7 Rº is the per-channel standard deviation computed on the training set. In both cases, image tokens are DINOv2 patch features projected to the model dimension vía a linear layer. The model alternates between spatial and temporal transformer blocks, folding the corresponding dimension into the batch dimension. To incorporate image context, we extend each Latte block with learnable, gated cross-attention mechanism over the image tokens f. Each block thus consists of selffattention (applied only to trajectory tokens), cross-attention (where trajectory tokens serve as queries and image tokens as keys/values), and a pointwise MLP. After all spatio-temporal blocks, the output projected and reshaped either to the latent code shape (if denoising) to the full trajectory grid (for the VAE).
We consider three different model configurations: small (S), base (B), and large (L).
i
Σg
X
Reshape
Reslupe
Reshage
Linear
Linear
Linear
(b n (b ad
(b t) td
(b n d (b td
Denoising Block
Autoencoding Block
Autoencoding Blockk
T
T (b n (l td
(be) (d
(b (d
f
Demoising Block
Autoencoding Block
Autoencoding Block
Patchity
DINO
Embed
Patchily
DINO
Linex
DINO
(a) Denoiser
(b) Encoder
(c) Decoder
Figure 16: Shared Architecture. The Denoiser é, Encoder d, and Decoder vé all use the same architectural building blocks. The primary difference lies in the type of blocks they alternate: the Denoiser è uses Denoising blocks, while the Encoder ó and Decoder &p use Autoencoding blocks, illustrated in Fig. 17.
The overall architecture of each component in method is illustrated in Fig. 16. As previously discussed, our method comprises three neural networks: the Denoiser (velocity prediction model, Fig. 16a), the Encoder (Fig. 16b), and the Decoder (Fig. 16c). All three networks share the same architectural building blocks, alternating between spatial attention and temporal attention blocks. The inputs the model are linearly projected model dimension. Then, they are processed by
29

---

Published as a conference paper at ICLR 2026
Denoising Block
Autoencoding
Block
Scale
Scale
MLP
MLP
The
Th.
Adaptive Norm
Adaptive Norm
Scale
Scale
Attention
Cross
Adaptive Norm
Adaptive Norm
Scale
Scale
Self Attention
Self Attention
71.00
Adaptive Norm
MLP
Adaptive Norm
Trajectory Tokens
Trajectory Tokens
(a) Denoising Block
(b) Autoencoding Block
Figure 17: Detailed overview of our attention blocks. Both the Denoising Block (Fig. 17a) and the Autoencoding Block (Fig. 17b) are adapted from the DiT blocks used in Latte Ma et al. (2025). We extend these blocks by introducing image conditioning through gated cross-attention with image features f. In the Denoising Block, a temporal embedding t is used to predict shift and scale parameters a, B3, and yy for gating and adaptive normalization. These parameters are predicted by block-specific MLP. In contrast, the Autoencoding Block uses learnable constants for these parameters.
stack of apatio-tempont blocks. After all spatio-tempora blocks, the output is projected and reshaped, either to the latent code shape (if denoising) the full trajectory grid (for the VAE).
In spatial attention, the attention mechanism operates across trajectory tokens at a fixed timestep. Conversely, temporal attention attends tokens along the temporal axis within the same trajectory. This is implemented by reshaping the input tensor to fold either the temporal or spatial dimension into the batch dimension, followed by an application of either the Denoising block (in the velocity prediction model, Fig. 17a) or the Autoencoding block (in the autoencoder, Fig. 17b). Although the recent works in video generation Wan al. (2025) use full attention (i.e., attention along both spatial and temporal axes), we apply the above-described factorized attention due to the quadratic computational cost of full attention. addition, existing point trackers Karaev al. (2024b;a) demonstrated exceptional accuracy and efficacy of factorized atlention in point tracking.
Both the Denoising block and the Autoencoding block are illustrated in Fig. 17, and they follow shared architectural design. Each block thus consists of self-attention (applied only to trajectory tokens), cross-attention (where trajectory tokens serve as queries and image tokens as keys/values), and pointwise MLP. Since our task involves predicting trajectories from images, condition the network on image features f, which are extracted using DINO. In each block, we apply cross-attention between the trajectory tokens and the image features f. The resulting cross-attention output is then combined with the previously computed features through a leamaale additive gating. The learnable gating is the addition of previously computed features and scaled gated features, while gating is
30

---

Published as a conference paper at ICLR 2026
pointwise multiplication with the given scale parameter. In the denoising block, the gating parameters are predicted by an MLP, whereas in the autoencoding block, they are learnable constants.
The main distinction between the Denoising block and the Autoencoding block is due to the additional input used during denoising: a time embedding. This embedding conditions the Denoiser on the flow matching timestep, effectively informing the model of the expected noise level in the input. Following the design Latte Ma et al. (2025) and DiT Peebles & Xie (2023), we encode the timestep using sinusoidal positional encoding, followed by multilayer perceptron (MLP). We condition the denoising model on timestep encoding with Adaptive Normalization. We implement Adaptive Normalization by first applying RMSNorm without elementwise affine parameters and then shifting and scaling the result by the parameters regressed by an MLP. In Autoencoding block, we do not have the timestep encoding we still apply the same normalization and gating. In this case, shift and scale are simply learnable constants. Unlike the original Latte Ma et al. (2025), which does not apply QK Normalization, we apply QK Normalization to every attention block in the network. We found this crucial for training stability, consistent with findings from existing works Polyak et al. (2025); Esser et al. (2024); HaCohen et al. (2024b) that apply transformers flow matching denoising diffusion. Since the most recent methods Polyak et al. (2025) implement QK Normalization by applying RMSNorm to queries and keys, we follow the same method. Thus, we replaced all LayerNon layers with RMSNo ensure consistent normalization throughout the network.
To summarize, the difference between ours and the Latte blocks is that we 1) apply gated cross- attention with image features, 2) apply QK Normalization, 3) use RMSNo rm instead of LayerNorm.
C.2 MODEL CONFIGURATIONS
We experiment with three different denoising model configurations, shown in Table 18.
Table 18: Model Configurations.
(a) Denoiser Configuration
Model
Blocks (N)
Hidden size
Heads
Parameter Count (M)
Small (S)
192
7.2
(B)
384
41.7
Large (L)
768
220.0
(b) VAE Configuration
Model
Encoder Blocks
Decoder Blocks
Hidden size
Heads
Parameter Count (M)
Base (B)
12
12
384
6
57.8
For Kubric, we extract image features using DINOv2 (Small), for others using DINOv2 (Large).
C.3 MODEL SELECTION
For every dataset, we have reserve a validation set for model selection. For VAEs, we select the best model based on the smallest validation reconstruction loss (LI). For denoisers, model selection criterion is the best validation Best-of-K K metric.
C.4 TRAINING
We use the identical initialization method Latte. We implement the models PyTorch and train using Distributed Data Parallel (DDP) with automatic mixed precision in bfloat 1. For flow matching, we sample timesteps using logit-normal method, following MovieGen Polyak et al. (2025). Depending on model size and the dataset, we use different hardware and train for different duration. All the experiments are executed on the internal SLURM compute cluster. Table 19 and Table 20 contain model and training hyperparameters, for each experiment. To reproduce all the experiments in this paper, we estimate total compute time of 32 GPU days. Training resources for each experiment in Table 21.
31

---

Published as a conference paper at ICLR 2026
Table 19: Denoiser Hyperparameters.
(a) Kubric (S)
(b) Kubric (B)
(c) Kubric (L)
Hyperparameter
Value Setting
Hyperparameter
Value Setting
Hyperparameter
Value Setting
patch size
patch size
patch size
training epochs
training epochs
training epochs
effective batch size
64
effective batch size
64
effective batch size
32
batch size per GPU
16
batch size GPU
16
batch size per GPU
gradient accumulation
gradicnt accumulation
gradient accumulation
optimizer
AdamW
optimizer
AdamW
optimizer
Adam
learning
hase learning rate Ir sehoduler
10-5
learning
10-5
schoduler
lincar warm-op
lincar warm-up
schedaler
lincar warm-ap
wamn-up steps
864
warm-up steps clip grad norm latent shape longth
wamm-up steps
3238
clip grad torm
1.0
clip grad norm
1.0
Intent shape
24 16 16
24 16 16
lntent shape
24 16 16 x
length
length
sampling method
Ealer
sampling method
Euler
sampling method
Eiller
sampling
10
sampling steps
sampling
10
sampling atol
x 10-7
sampling atol
x 10-
sampling atol
x
sampling ctol
1 10
sampling rtol
10
sampling rtol
1 10
(d) LIBERO
(c) Physion
(n) Physics101
Hyperparameter
Value Setting
Hyperparameter
Value / Setting
Hyperparameter
Value Setting
patch size
patch size
patch size
training epochs
1500
epachs
40
training epochs
750
effective batch
effective batch size
32
effective batch size
32
hatch size per GPU
16
hatch size per GPU
4
hatch size GPU
gradicut accumulation
gradicut accumolation
gradicut accuniulation optimizer
optimizer
A
optimizer
Adam
Adam
base learning
6 10-5
hase learning Ir scheduler warm-up steps clip grad lntent shape
10-5
base learning
10-5
scheduler
lincar warm-up
warm-op
scheduler
lincar warm-up
warn steps
1448
2048
warm-up steps
2048
clip
grad
lntent shape
x 16 x
30 x 16
latent shape
30 16 x 29
track leugth
16
track length
30
track length
30
sampling method
Faler
sampling method
sampling method
Baler
sampling steps
10
sampling steps
10
sampling steps
10
sampling
sampling
sampling atol
sampling stol
1 10
sampling stol
x 10-5
sampling stol
x 10
C.5 EVALUATION CoST
C.5.1 KUBRIC
We evaluate all models 2,048 videos (2 benchmarks (in-distribution, OOD), rollouts, 16 scenes). Evaluation in this field is generally difficult due to the cost of running video generators (sampling for the single initial condition and seed takes more than 2 minutes for most of the baselines; Table 22). In total, our evaluation on Kubric used roughly 500 GPU hours. Note that training our method (L) takes 8.5 days = 204 GPU hours, implying that evaluation 2x more expensive than training the largest configuration method.
D EXPERIMENTAL SETUP
D.1 COTRACKER'S ACCURACY ON OUR BENCHMARK
We evaluate CoTracker using standard point tracking evaluation (Doersch al., 2023) metrics and protocol. Table 23 contains the results.
D.2 ASSESSING METRICS
We investigate the validity of our distributional metrics in Table 24, Here, make use of our Kubric evaluation set, but we partition it into two sets, such that each scene has 32 possible futures. Intuitively, ground truth data should be very good predictor of itself. We compare the values against simply running CoTracker to predict the motion of points on the ground truth video. All metrics are lower when ground truth is used to predict ground truth.
32

---

Published as a conference paper at ICLR 2026
Table 20: VAE Hyperparameters.
(a) Kubric
(b) LIBERO
Hyperparameter
Value / Setting
Hyperparameter
Value / Setting
patch size
patch size
training epochs
400
training epochs
400
effective batch size
effective batch size batch size per GPU gradient accumulation optimizer
batch size per GPU gradient accumulation optimizer
4
AdamW
Adam
base learning rate
6 x 10
base learning rate
x 10
clip grad norm latent shape
1.0
clip grad norm latent shape
1.0
24 16 x
16
16 x 16 x
track length
track length
beta
1 x 10-6
beta
x 10-6
(c) Physion
(d) Physics101
Hyperparameter
Value / Setting
Hyperparameter
Value Setting
patch size
patch size
2
training epochs
24
training epochs
400
effective batch size
16
effective batch size
16
batch size per GPU 4
batch size per GPU 4
gradient accumulation optimizer
gradient accumulation optimizer
I
AdamW
AdamW
base learning rate
6 10
base learning rate
6 x 10
clip grad norm latent shape
1.0
clip grad norm latent shape track length
1.0
x 16 16
30
29
track length
30
30
beta
1 x 10-6
beta
1 10-6
Table 21: Required resources.
(a) Denoiser
(b) VAE
Model
GPUs
Training Time (Days)
Model
GPUs
Training Time (Days)
Kubric
Kubric
Small (8)
4 A6000
Base (B)
x A6000
4
Base (B)
2 x H100 NVL
LIBERO
Large (L)
x H100 NVL
4.5
Base (B)
x A6000
LIBERO
Base (B)
x H100 NVL
Physion
Physion
Base (B)
x A6000
Base (B)
4 L40s
Physics 101
Physics107
Base (B)
x A6000
Base (B)
2 x H100 NVL
D.3 PROTOCOL FOR VIDEO GENERATORS
Data Preprocessing: The Kubric training dataset is originally rendered at 256 256 resolution. However, most video generation baselines assume 16:9 aspect ratio with resolutions such as 320x 512 (DynamicCrafter). 3 (Stable VideoDiffussii with pre-training resolution and aspect ratio, we bilinearly upsample Kubric videos such that the shorter side matches the target resolution, and pad the longer side with black bars to preserve the aspect ratio and center the content. For example, for DynamicCrafter, we upsample Kubric to 320x320 and add symmetric vertical padding produce x10 videos.
640x480 (WAN). To ensure compatibility
Temporal Horizon Adjustment: As the baselines vary in their output temporal horizon, we modify each implementation to produce 24 frames at 12 fps, ensuring consistent evaluation across models.
Fine-tuning Protocol: For DynamicCrafter and WAN2.1, we use the official implementations. For SVD, due to the absence of an official training script, adopt the Hugging Face version and
33

---

Published as a conference paper at ICLR 2026
Table 22: Model size and cost. We compare inference time, memory usage, and parameter count.
Model
Time (8)
Peak GPU Memory (GB)
Denoiser Size (M)
WAN
248.2
32.3
14000
DynamicCrafter!
134.7
12.3
1487
SVD
41.8
12.7
1525
Ours (L.
2.9
1.9
220
Ours (B)
1.0
0.9
41.7
Ours (S)
0.5
7.2
Table 23: Point-tracking accuracy. CoTracker3 performs exceptionally benchmark data.
Benchmark
Occlusion Accuracy
Average Jaccard
Avg. Points Within Threshold
Kubric (Out-Diee hitt-Distribution)
91.9
84.9
91.5
Kubric (In-Distribution)
91.3
82.2
90.2
implement a custom training pipeline based on publicly available details. We fine-tune each video generator for the same amount of time as it takes to train our method.
Trajectory Extraction: We employ the official CoTracker3 implementation to extract trajectories from generated videos. For evaluation, all trajectories are resized 256 x 256 to match the resolution of the ground truth.
D.4 PROTOCOL FOR TRAJECTORY METHOOSS
D.4.1 REGRESSION METHODS
We compare method with ATM (Wen al., 2024a) and Tra-MoE (Yang et al., 2025). To ensure a fair comparison, we carefully design the following training and evaluation protocol.
Using the same dataset and the identical train/validation split each baseline, slice videos into 16-frame windows, matching baseline prediction length. For each window, we extract point trajectories every other pixel (e.g., 64x64 grid) using Co'Tracker. Since both baselines condition trajectory generation on both the initial frame and text instruction, we extend our method with text conditioning. Specifically, we extract pooled BERT embedding for each text instruction using the same version of BERT as both baselines. Following baselines, we use the same model dimension and first project the instruction embedding with an MLP. Then, we concatenate the projected text embedding with the timestep embedding for flow matching. This is used for conditioning through adaptive normalization, as the original Latte and DiT (Peebles & Xie, 2023). To ensure a fair comparison, we do not apply advanced conditional sampling techniques, such as classifier-free guidance, to our method. We simply train and sample, providing just the input image and the instruction. It is worth noting that this also gives the basclines an advantage due to the experimental observation that unconditional diffusion models are generally worse than the conditional ones terms of sampling quality (Bao al., 2022; Li et al., 2024). We compare with each baseline independently, because they are trained on different subsets of the training data. We train our model using raw trajectory grids without any additional preprocessing. For evaluation, we give baselines advantage by selecting evaluation trajectories using the filtering method they use during training. Specifically, we consider only those trajectories whose temporal variance exceeds a fixed threshold, taken directly from bascline codebase. If there are no such trajectories for a given window, we simply discard the window. Since both baselines predict only 32 trajectories, we first sample uniformly at random 32 trajectories from the set of filtered trajectories. To obtain baseline predictions, we use their official implementation and the checkpoint. For our method, we simply predict trajectories for every other point. Next, from these densely sampled trajectories, points (i.e. initial positions) of the 32 sampled evaluation trajectories. Because only a single (pseudo) ground-truth trajectory set is available per initial frame, distributional metrics like FVMD proposed variants are unsuitable. Instead, we adopt a simple regression metric mean square error, computing the average Fuclidean distance between the i our samples and (pseudo) ground-truth
select trajectories corresponding to the query
34

---

Published as a conference paper at ICLR 2026
Verification of distributional metries. We check whether data is good predictor of itself by partitioning the dataset. We compare this to simply predicting the motion on true videos. The metrics are lowered, showing sensitivity. Note, distributional metrics are sensitive to the number of samples, which here is set to 32.
Table 24:
Metric
GT vs GT
CoTracker vs GT
FVMD (scene)
8979.95 114.241
21331.69
Best of K
191.304
trajectories. We report results for our method for l E {1,2,4,8), For ATM and Tra-MoE, = 1 always because they deterministic.
D.4.2 DIFFUSION-RANE BASED METHODS
Firstly, we briefly introduce the Track2Act (Bharadhwaj et al., 2024b). The method builds on the original diffusion transformer (DiT) (Peebles & Xie, 2023) model trajectory generation with two-frame conditioning, namely the start image and the goal image. Both start and goal frame are encoded using a pre-trained ResNet18, producing one embedding vector per frame. These embeddings concatenated, flattened, and linearly projected to the model dimension, then injected into the network via adaptive normalization layers to guide generation. The model is trained on variable-length tracks (100-400 frames). The prediction horizon is 8 future frames. Each track is flattened along the channel dimension. The method uses original DDPM (Ho et al., 2020) formulation with 1000 timesteps. Here are the modifications applied their method:
1. Dropping the goal frame. We simply drop the goal frame and use only start frame embedding vector for conditioning.
2. Extending prediction horizon. Track2Act originally predicts the next eight frames. We extend the prediction horizon to 24 frames to align with our Kubric training and evaluation setup.
3. Fixing the number of generated points. Because Track2Act is built on a DiT (Peebles & Xie, 2023), and transformers are known generalize poorly to -iitt--distritution sequence lengths (Li et al., 2025; Veličković et al., 2025), we standardize the number of generated points. Recall that our method predicts 1,024 points arranged on a 32 x 32 grid, whereas the original Track2Act is trained output at most 400 uniformly sampled points. For fair comparison, we therefore train Track2Act to produce exactly 1,024 points with the same 32 x 32 grid arrangement used by our method and in evaluation.
We train the method on our dataset using the official publicly available codebase, following the default hyperparameter settings provided by the authors.
D.4.3 PHYSICS101
The dataset consists of roughly 10000 video clips containing 101 objects of various materials and appearances (shapes, colors, and sizes). Since this dataset was collected using high-resolution camera (1080p), resulting in videos where majority of the objects we preprocessed the dataset such that the objects and their interactions are centered, preserving the aspect ratio but reducing the resolution 256x464, for computational reasons. There are five different physical scenarios, namely fall, liquid, multi, ramp and spring. Please see the dataset for more information about these. For computational reasons, extracted 32x58 trajectory grids using CoTracker3. All the clips consist of 30 frames, 1856 trajectories. Our training set contains 9252 different initial conditions with the single ground truth. Our test set contains 1450 different initial conditions and single ground truth per initial condition.
small compared the background,
E
USER STUDY
We carried out a user study comparing our model, fine-tuned SVD and fine-tuned WANL.3 model. The study consists of 16 questions, asking the respondents rank the three models from best to
35

---

Published as a conference paper at ICLR 2026
worst in each question. We did not identify the models in the study, but randomly gave them label as "Option 1", "Option 2", and "Option 3" for each question. We show example of the question in Fig. 18. All questions identical except the scene animation changes. We include all scenes alongside this supplemental document.
Inital Frame
Option
Option
Option
Example
theacoee abows, how roald rank aptana??
Option?
Option
(Bea)
Rane
O
(Winss)
O
Figure 18: Example of user study question. The study contained 16 questions showing the animation of methods, which were assigned names random for each question.
F MORE RELATED WORK
Studies on the implausibility motion generated videos. Several studies have highlighted physical implausibility in video generation. To quantify this, Motamed et al. (2025) introduced Physics-IQ. novel benchmark dataset evaluating the physical understanding video generation models. Their findings reveal that, while current models exhibit impressive visual realism, their understanding of fundamental physical principles remains limited. Errors include the spontaneous (dis)appearanne of objects and physically implausible object interactions (e.g., objects passing through each other). In a related study, Kang et al. (2024) investigated whether vidco generation using latent diffusion learn solid mechanics from simple 2D dataset governed primarily by rigid body mechanics. Their results suggest that scaling up model size improves performance within the training distribution and aids combinatorial generalization but does not lead to accurate motion synthesis out-of-distribution. VideoPhy Bansal et al. (2024) conducted a large-scale user study assessing whether generated videos followed physical common sense, observing limited performance even for the latest video generators.
G ETHICS STATEMENT AND BROADER IMPACT
Our work offers computationally efficient alternative for inferring motion. This approach has the potential to significantly reduce the resource demands of motion forecasting, making it more accessible and deployable in real-world scenarios, especially on edge devices or in bandwidth- constrained environments. By eliminating the need for video input and the additional processing required for vidco-bascd tracking, model can democratize access to future motion understanding, benefiting fields such as robotics, autonomous navigation, assistive technology, and video editing.
However, the ability to infer motion dynamics from static image raises ethical considerations. In particular, if used in surveillance or behavioral prediction, such technology could be misapplied to infer intentions or future movements of individuals without their knowledge consent. These concerns underline the importance of deploying such technologies transparently, with safeguards to protect privacy and civil liberties.
36

---

Published as a conference paper at ICLR 2026
Moreover, the model's reliance learned priors from training datasets may introduce biases. Future work should explore robustness and domain adaptation to ensure that the benefits of this research extend across diverse contexts.
H USE LLMs IN OUR WORK
We use LLMs refine and rephrase our text, well to assist in generating visualizations, which are included both in the main body the paper and in the supplementary material.
I LIMITATIONS
Due to computational resource constraints, our work is limited to synthetic data and small-scale real- world experiments. Moreover, lower conditioning image resolution and further downsampling applied by patchification in the encoder reduce the accuracy of predicted tracks around object boundaries for model. Additionally, while method demonstrates strong performance datasets like Kubric, LIBERO, Physics 101, and also some generalisation to real-world scenarios from Physion, is currently limited from achieving broader in-the-wild generalisation. Future efforts will focus on addressing these limitations by scaling datasets and improving the resolution of the image features.
37


Reviews:
[{"review_id": "6fFTppJH2T", "reviewer": "Reviewer_h2xU", "summary": "This paper addresses the problem of forecasting object motion from a single image—predicting potential object movements without observing velocities or applied forces. It formulates the task as conditional generation of dense point trajectory grids, drawing inspiration from modern video generator architectures but outputting trajectories instead of pixels. The model uses a trajectory VAE for latent space mapping and flow matching for sampling, capturing scene-wide dynamics and uncertainty.", "questions": "In the absence of a task description (e.g., for the action \"Pick up the book on the right and place it under the cabinet shelf\"), how does the paper address the diversity of action prediction, and are there any controllable settings integrated into the pipeline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It innovatively redefines motion forecasting as dense trajectory grid generation, moving beyond prior work focused on sparse active points (e.g., ATM, Tra-MoE) or pixel-based video generation, enabling better global scene dynamics modeling.\n2. The study uses diverse datasets and comprehensive metrics to assess accuracy, diversity, and physical plausibility, with ablations strengthening conclusions", "weaknesses": "- **Lack of Downstream Task Evaluation for Video Generation**: While the paper argues that trajectory generation outperforms video generation for motion forecasting, it does not test how the two approaches perform on downstream tasks that rely on motion, e.g., robot manipulation, video interpolation, or action prediction.   \n- **No Analysis of Query Point Sampling Density**: The paper only discusses ablations of latent code dimensions but does not explore the impact of query point sampling density.   \n- **Insufficient Real-World Data Results and Visualizations**: Real-world validation is limited to the Physics101 dataset without testing on other real datasets, e.g., egocentric motion datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V7seuVrVJJ", "reviewer": "Reviewer_VFf8", "summary": "This paper addresses the problem of predicting object movement from a single static image, where future motion is inherently uncertain. The authors propose a method that directly generates a dense grid of point trajectories (coordinates) conditioned on the input image, rather than generating future video frames (pixels). They utilize a generative architecture (similar to modern video generators, specifically using flow matching) to model the distribution of possible future movements.", "questions": "1. The point I understand the least is why, for a scenario where object motion needs to be predicted, it is limited to using only a single static image. Alternatively, is there a way to adapt this model to also support video or multi-frame input? And how would its performance be?\n2. It seems to lack some relevant paper citations, such as Flow-grounded spatial-temporal video prediction from still images, papers about future frame synthesis/prediction", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Generates point trajectories instead of pixels, bypassing the complexity of video generation. This allows the model to efficiently learn physical dynamics and object consistency.\n2. Uses a generative model to predict the distribution of trajectories, producing multiple physically plausible futures, which better reflects real-world information gaps.\n3. Achieves superior performance in physical reasoning tasks compared to prior trajectory regressors and state-of-the-art video generators pre-trained on massive datasets.", "weaknesses": "1. Predictions are based on a single image, lacking precise initial velocity or force data. Accuracy heavily relies on the model's learned \"physical common sense\" rather than precise observation. I think this task setting quite strange.\n2. Evaluation primarily uses synthetic datasets (e.g., Kubric, Physion). Its generalization capability to complex, \"in-the-wild\" real-world physics (e.g., fluid, soft bodies) remains questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZfSNrjcSQY", "reviewer": "Reviewer_GBJN", "summary": "In this paper, the authors focus on efficient approaches to forecasting dynamics in an image. To this end, they propose, instead of modeling pixels directly, to forecast point trajectories. They utilize a beta-VAE combined with Flow Matching to generate future point trajectories conditioned on an image. The authors then comprehensively evaluate the performance of their model, looking at both statistical and physical plausibility. They evaluate on robotic datasets such as Libero as well as synthetic datasets (Kubric). The also include a user study of performance.", "questions": "1. Please cite the paper above. It is similar to approach mentioned in this paper.\n\n2. Please elaborate more on the user study, and would it be possible to include more results from Physics 101?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper explores a less studied avenue of visual dynamics: predicting pixel motion versus pixels themselves. This approach makes intuitive sense.\n\n2. Quantitative evaluation is comprehensive and convincing. \n\n3. The paper is well-written and easy to understand.", "weaknesses": "1. There is a prior work, \"An Uncertain Future: Forecasting from Static Images using Variational Autoencoders\", Walker et al, ECCV 2016, which is similar and also uses a VAE to forecast pixel trajectories from a single image. This work should be cited.\n\n2. Results of the user study are somewhat weak (preferred over other approaches only 52% of the time).\n\n3. It would be helpful to see a few more qualitative results from the Physics 101 dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)