Direct Motion Models for Assessing Generated Videos
Authors: Kelsey R Allen
Carl Doersch
Guangyao Zhou
Mohammed Suhail
Danny Driess
Ignacio Rocco
Yulia Rubanova
Thomas Kipf
Mehdi S. M. Sajjadi
Kevin Patrick Murphy
Joao Carreira
Sjoerd van Steenkiste
Venue: ICML 2025 poster (2025)
Keywords: metrics
videos
motion
point tracking
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a novel approach based on auto-encoding point tracks that can be used to evaluate motion in generated videos.

Abstract: A current limitation of video generative video models is that they generate plausible looking frames, but poor motion --- an issue that is not well captured by FVD and other popular methods for evaluating generated videos. Here we go beyond FVD by developing a metric which better measures plausible object interactions and motion. Our novel approach is based on auto-encoding point tracks and yields motion features that can be used to not only compare distributions of videos (as few as one generated and one ground truth, or as many as two datasets), but also for evaluating motion of single videos. We show that using point tracks instead of pixel reconstruction or action recognition features results in a metric which is markedly more sensitive to temporal distortions in synthetic data, and can predict human evaluations of temporal consistency and realism in generated videos obtained from open-source models better than a wide range of alternatives. We also show that by using a point track representation, we can spatiotemporally localize generative video inconsistencies, providing extra interpretability of generated video errors relative to prior work. An overview of the results and link to the code can be found on the project page: trajan-paper.github.io.

Direct Motion Models for Assessing Generated Videos
Kelsey Allen
Carl Doersch'
Guangyao Zhou' Mohammed Suhail Danny Driess'
Ignacio Rocco
Yulia Rubanova'
Thomas Kipf'
Mehdi Sajjadi
Kevin Murphy'
Joao Carreira' Sjoerd van Steenkiste 22
Abstract
1. Introduction
As generative video models become increasingly capable, the community needs more powerful methods for automat- ically evaluating the quality of generated videos. State-of- the-art models are getting better at generating what appear to be plausible looking frames, yet they still struggle to put together coherent motion (Brooks et al., 2024). The current gold standard for assessing video quality is collecting hu- man judgments, but these are expensive to obtain and not scalable as metric for regularly measuring improvements in modeling capabilities, e.g. throughout training.
A current limitation of video generative video models is that they generate plausible looking frames, but poor motion an issue that is not well captured by FVD and other popular methods for evaluating generated videos. Here we go beyond FVD by developing a metric which better measures plausible object interactions and motion. Our novel approach is based on auto-encoding point tracks and yields motion features that can be used to not only compare distributions of videos (as few as generated and one ground truth, or as many as two datasets), but also for evaluating motion of single videos. We show that using point tracks instead of pixel reconstruction or action recognition features results in a metric which is markedly more sensitive to temporal distortions in synthetic data, and can predict human evaluations of temporal consistency and realism in generated videos obtained from open-source models better than a wide range of alternatives. We also show that by using a point track representation, we can spatiotemporally lo- calize generative video inconsistencies, providing extra interpretability of generated video errors relative to prior work. An overview of the results and link to the code can be found on the project page: traaann-aaa
Existing metrics, such as those based on Fréchet Video Distance (FVD) (Unterthiner et al., 2018; Ge et al., 2024; Luo et al., 2025), can capture certain elements of plausibility, but are more sensitive to frame-level content effects (Ge et al., 2024), and depend on access to the underlying training distribution which is not always available. Further, these approaches cover only one way of evaluating generated videos, i.e. by comparing entire distributions, and it remains unclear how to evaluate pairs of videos, or individual videos, with this approach.
Here we propose new method that we show addresses many of these issues, by directly modelling 2D video mo- tion. Given single generated video, estimate lowelevel, temporally-extendee motion features as point tracks using the publicly available BootsTAPIR model (Doersch et al., 2024). Next, we (auto)encode these features to obtain dense high-level motion features using a novel TRAJectory Au- toeNcoder (TRAJAN) architecture. We can then use the TRAJAN latent space to compare distributions of videos (as few one generated and one real, or many as two datasets), or the reconstruction error from TRAJAN esti- mate per-video motion inconsistencies.
github. io.
'qqual contribution 'Google DeepMind Goooge Research. Correspondence to: Kelsey Allen <krallen& google.com>>, Sjoerd van Steenkiste <svansteenkiste@ google.com>.. Author contri- butions: KA conceptualized the idea of using track-based latent motion features to measure video quality, moving from distribu- tional metrics to per-video or video-video metrics. KA, SVS co-led the project and ran all the experiments. GZ invented an initial pro- totype of the TRAJAN architecture, which was further developed by CD. MS supplied the WALT checkpoints. All authors advised on the project direction and contributed to writing the paper.
Point tracking, by design, separates the semantics from the motion content by focusing on features necessary for predicting motion without reconstructing the whole scene (Figure 1). This means that will likely focus on plausible motion irrespective of semantic information. We show that this is indeed the case across the three major ways in which the community generally uses metrics to evaluate videos:
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
For individual videos, we measure how well TRA-

---

Direct Motion Models for Assessing Generated Videos
methods at predicting human ratings, although with room for improvement, suggesting an area for further research.
2. Related Work
Metrics for assessing various aspects of video generation quality can broadly be categorized into a) operating on indi- vidual videos without a reference, b) comparing a generated video to real "ground truth" reference, and c) comparing a distribution of generated videos with a reference distribution. Though these areas are generally considered separately, here we propose TRAJAN as a unifying model for measuring motion consistency in all of these three settings.
(a) Unrealistic object appearance, but plausible motion.
Metries for individual videos. Several recent metries have been proposed for evaluating text-to-video models on per- video basis, since in this case a reference video is typically unavailable. However, major focus has been on the seman- tic alignment between the prompt and its generated video. CLIPScore (Hessel et al., 2021) utilizes CLIP (Radford et al., 2021) to compare individual video frames with the text in the embedding space (Gu et al., 2024; Liu et al., 2024b; Wu al., 2023). The main issue with CLIPScore that it mainly captures semantics of individual frames, instead of dynamic properties of videos such as consistency over time, physical plausibility, or motion quality (Bansal et al., 2024b). Even for image generation tasks, it has been shown that such metrics are not always aligned with human judge- ment (Otani et al., 2023). Beyond CLIP, other approaches propose to use vision-language models (VLMs) to judge the semantic adherence between the prompt and the video (Wu al., 2024; Bansal et al., 2024b), either zero-shot or by finetuning a VLM on human annotations of video quality (Huang et al., 2024). DOVER (Wu et al., 2022b) is a model trained to predict the average human subjective perception quality of video. In addition to semantic adherence, Video- Phy by Bansal et al. (2024b) evaluate videos by whether they follow physical commonsense by finetuning VLM (Bansal al., 2024a) this task.
(b) Realistic looking hands, but which change appearance from frame-to-frame, with fingers morphing in and out of existence.
Figure 1. Predicted point trajectories from TRAJAN, colored by their reconstruction error with respect to BooIsTAPIR (well recon- structed track, poorly reconstructed track) (Doersch et al., 2024). See t stajan-paper: . gi thub. 1o for videos. The video in (a) has plausible motion but implausible frame-level appearance, while the video in (b) has plausible frame-level appearance but implausible motion (the fingers morph and disappear between frames). TRAJAN correctly predicts this discrepancy by focusing on motion irrespective of appearance.
JAN is able to reconstruct the original input tracks ($5.1). We find that this reconstruction score is more or equally predictive of human ratings for realism, appear- ance, and motion consistency of generated videos from 10 different open-source models than a wide range of alternatives (both appearance- and motion-based).
2. For pairs of videos, we compute a distance between their TRAJAN embeddings ($5.2), which better cap- tures similarities in motion even when appearance- based lixel-level metrics suggest that they are different.
Compared to these metrics, our approach is not focused on semantic adherence between a prompt and a video, but rather on the motion and appearance consistency of a video independent of the prompt. Our approach based on point tracks inherently takes the temporal evolution of the frames into account, which we show crucial for matching human judgements of video quality per-video basis.
3. At the distribution level, we show that among several other choices- - including VideoMAE v2 (Wang et al., 2023b), I3D (Carreira& & Zisserman, 2017), and motion histograms (Liu et al., 2024a)- - the features learned by TRAJAN are markedly more sensitive for detecting synthethic temporal distortions ($5.3).
Metrics for paired videos. Evaluating the quality of gener- ated video against ground truth frames often involves pixel- wise and perceptual metrics, such as Peak Signal-to-Nois Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). PSNR measures pixel-level discrepancies using Mean Squared Er- ror, but its pixel dependency makes it overly sensitive to
As an additional contribution, we conduct a detailed human study generated videos. We collect new dataset of ratings that focuses on four aspects of motion: quality of interactions, the realism of the objects and behavior, consis- tency of appearance and motion, and the speed of objects and the camera. While acknowledging these are somewhat subjective, we find that TRAJAN typically outperforms prior
2

---

Direct Motion Models for Assessing Generated Videos
small, often imperceptible variations. SSIM (Wang et al., 2004) improves on this by comparing local patterss in bright- ness, contrast, and structure to better align with human perception. LPIPS (Zhang et al., 2018) uses a pre-trained deep networks (e.g.. VGG (Simonyan & Zisserman, 2014), AlexNet (Krizhevsky et al., 2012)) to compute similarity based on feature maps, capturing higher-level perceptual cues. While these metries provide valuable insights into frame-level similarity, they evaluate each frame indepen- dently and thus fail to capture temporal coherence- an es- sential component of video realism.
For distribution-level comparisons, we compute the Fréchet distance between the latents in two datasets PR and Pg as in prior work (Unterthiner et al., 2018). It is defined as dR Pa) = minx,y E|X - Y which simplifies to d( (nn, Pc) = | HG 2 +rr(E EG Le) when PR and Pc are multivariate Gaussians. For video pair comparisons, we simply take the I.2 distance between the latents. For single video evaluations, we use the ordinal number (such as reconstruction error) to determine quality.
3.1. TRAJAN
Metrics for video distributions. FVD (Unterthiner et al., 2018) is a standard metric to compare a distribution of real world videos with generated ones (Bugliarello et al., 2024). However, Ge et al. (2024) show that FVD is biased to- wards the content of individual frames, possibly because of how the underlying 13D (Carreira & Zisserman, 2017) feature extractor was trained. Ge et al. (2024) propose to use VideoMAE features to address this, while concurrent work by Luo et al. (2025) considers a number of alternative appearance- based feature representations. Liu et al. (2024a) also propose a new feature representation, but they compute histogram-based motion features derived from estimated point tracks. Our approach is closely related in that we also use point tracks, however, key difference is that we use learned TRAJAN features, which we find to perform markedly better. Additionally, we demonstrate how TRA- JAN can be used to assess the quality of individual videos.
Point tracking models like PIPs+ (Zheng et al., 2023) or BootsTAPIR (Doersch et al., 2024) can track points in ar- bitrary videos surprisingly well. Point tracks are inherently linked to motion rather than appearance, making them a strong candidate for assessing motion quality in videos. However, set of point tracks is inconvenient to work with because the tracks are orderless, making it difficult to com- pare two sets without a ground-truth reference. Further, we must also contend with missing data due to occlusions.
To address this problem, we introduce the TRAJectory Au- toeNcoder (TRAJAN, Figure 2). At high level, TRAJAN is trained to reconstruct point tracks starting from random queries across space and time, and can provide both la- tent representation and a reconstruction score. TRAJAN operates on a set of point trajectories S = { coming from, e.g., BootsTAPIR, where St.j = (Zz.j Vt.j 9t.5) cor- responds to x and y positions and occlusion flag o at time for the jth trajectory, and trained to reconstruct a sepa- rate set of query trajectories Q = {gt.}) } similarly randomly sampled from the video.
Benchmarks & generative video models. The field of generative video models is rapidly evolving. Most current methods for text-to-video models are either based on diffu- sion (He et al., 2022; Ho et al., 2022b;a; Singer et al., 2022: Luo et al., 2023; Wang et al., 2023a; Zhou et al., 2022; Khachatryan et al., 2023; Blattmann et al., 2023b) or trans- formers (Villegas et al., 2022; Wu et al., 2021; Hong et al., 2022; Wu et al., 2022a). Hence, there have been several efforts evaluate such generative models (Liu et al., 2024d; Huang et al., 2024). VideoPhy (Bansal et al., 2024b) is benchmark that focuses on prompts where a model has to obey the laws of physics, such as marbles rolling down a slanted surface. We evaluate TRAJAN both on VideoPhy (Bansal et al., 2024b) and EvalCrafter (Liu et al., 2024c).
Architecture. For a given track j. we first embed all loca- tions (Ze.j. Vt.j f) with sinusoidal embedding across space and time and project to C channels. Then we add "readout" token of length C, and perform self-attention to all tokens for track j. using (1 or.)) as an attention mask, which helps the representation be invariant to occluded points. After self attention, we discard all tokens except the "readout" token to obtain fixed-lengti C-channel representation of each track. Next, we use Perceiver (Jaegle et al., 2021 2-2te- ap- proach to encode all track tokens, namely by cross- -ooss-attedding from a learned set of 128 latent tokens to all track readout tokens, followed by self-attention. Finally, we project the latent tokens to a lower dimension resulting a fixed-size 128 64 dimensional representation Φs of the tracks.
3. Methods
To obtain metrics for evaluating motion in videos, we fo- cus on two aspects: how to extract latent representations from videos, and how to compute ordinal-valued metrics for videos that can directly be understood as ranking individ- ual videos as being better or worse. We propose TRAJAN, which gives us latent representations and can be used di- rectly as metric on a per-video basis.
To ensure this representation encodes the dense motion, we train it to reconstruct tracks from the video using a decoder. However, since we want to represent the motion as a dense field, we want it to be invariant to the specific query points used to obtain the support tracks. To achieve this, we let the decoder reconstruct held-out tracks not included in the
3

---

Direct Motion Models for Assessing Generated Videos
Encoder
Decoder
Sel-attentinn U A/
Transforme w/ CVd geiatin
Modon 33011
Comp maad mothonsieeess
Corroeooe
uttentte U et
bet attentnn Am
Menb
Compaime letrta
(4bb qeery paina aaniraatin
Priskt
Vouec for icoo
tolen, part
Vodion ideett
number Trmmes
parper
Line salorconn
Quarry points
(a) Point trajectory encoder
(b) Query point decoder
Figure 2. TRAJAN- - The trajectory encoder encodes (variable-sized) set of point trajectories (Fr.)) Mh.j. Or.j. () into compressed motion latent dy of fixed size using a Perceiver (Jaegle et al., attention mask, making the representation invariant occluded points. The decoder takes this latent és and predicts for query point (Ig, Va. fq) fg) different input and query points, the model learns to represent dense motion field.
transformer architecture. The occlusion flag o is used in the
the point track that goes through this point at all other times, as well as their occlusion flag. By training the autoencoder on
input. That is, the decoder takes Φs and a query point (Zg, Mgs tg), and outputs the track that goes through this point. We up-project the tokens in Φs to higher dimension with an operator U, and compute "readout" token via sinusoidal position encoding of (Ig, Ma. (g)) Next, we apply self-attention to the full token set, and discard all but the readout token. We finally apply a linear projection to the readout token to obtain (27.3%- of) for every frame, where of an occlusion logit trained by sigmoid cross-entropy, and x/ 3t are trained with an L1 loss. In Appendix B.2.1, we provide more details about training and architecture.
Motion histograms Liu et al. (2024a) propose to evaluate motion consistency in generated videos by estimating ve- locity and acceleration from point tracks. Inspired by HOG features (Dalal & Triggs, 2005), they partition the resulting volumes into 4 x 5 x 5-sized tubelets and accumulate the magnitude of the values at each angle (using 8 bins) within a tubelet. Motion features are obtained by concatenating the resulting 1D histograms obtained for each tubelet using both velocity and acceleration. We apply this approach to 16-frame chunks of 64 64 densely sampled point tracks obtained from BootsTAPIR (Doersch et al., 2024) to yield 9216-dimensional vector describing the motion within the corresponding 16-frame video.
Average Jaccard. An advantage of TRAJAN is that it not only computes representations that describe the motion con- tained in a video, but that it can also be used to evaluate the motion directly: the model was trained on real motions, so expect unrealistic motions to be more difficult to reconstruct. Thus, propose Average Jaccard (Doersch et al., 2022) (AJ) to measure the accuracy of the autoencoder reconstructions relative to the tracks it receives as input. Av- erage Jaccard combines occlusion and position accuracy. For given threshold 5 Jaceard, considers "true positives* (TP) to be predictions which are within 8 of the ground truth. "False positives" (FP) are predictions that are farther than 8 from the ground truth (or the ground truth is oc- cluded), and "false negatives" (FN) are ground truth points where the prediction is farther than 8 (or occluded). Jaccard, is TP/(TP + FP+ + FN), and Average Jaccard averages Jaccardy over several thresholds (see also Appendix B.2.1).
Optical flow Another natural candidate for evaluating motion in videos is to make use of Optical Flow. Unlike point tracks, flow is usually estimated between consecutive frames, which might affect its ability to consider long-term motion patterns. Prior work has examined the warping er- ror as a loss to make videos more temporally consistent (Lai et al., 2018), and was used more recently as a metric of temporal consistency in generated videos (Liu et al., 2024c).
For given pair of frames, the warping error is obtained by computing the pixel-wise difference between the second frame and the "warped prediction" computed using the op- tical flow prediction applied to the first frame, while also accounting for predicted points of occlusion. Following Liu et al. (2024c), we calculate the warp differences on every two frames, and average across pairs obtain the final score. To estimate optical flow we use a version of RAFT (Teed & Deng, 2020) with added improvements (Sun et al., 2022; Saxena et al., 2024) (see Appendix B for details). RAFT iter- atively updates a flow field based on multi-scale 4D correla- tion volumes computed from learned features for all pairs of pixels. Due to the size of these volumes, we only consider the negative warping error (such that higher error corre-
3.2. Alternative models
3.2.1. MOTION-BASHI
For comparison to TRAJAN, we consider two alternative cottion-based metrics: one based on histograms over point track trajectories, and one based on optical flow.
4

---

Direct Motion Models for Assessing Generated Videos
sponds to a lower ordinal value) as per-video metric as cal- culating covariance matrix would otherwise be intractable.
decoder converts the latent state back pixels by querying the latents through cross-attention with fixed grid-based fea- tures (Sajjadi et al., 2022; Jaegle et al., 2022). We slightly modify the original model by decoding the next-frame from the "corrected" state to simplify the implementation. van Steenkiste et al. (2024) showed how MooG learns represen- tations that encode both appearance and motion, as they can be decoded into both reconstructed video and point-tracks using shallow decoder heads. However, due to the size of the latent space, we will only consider its prediction error in the form of PSNR as single-video metric.
3.2.2. APPEARANCE-BASED
Inspired by FVD, we also consider a number of methods that operate directly on RGB. Although such methods are freely able to focus on content or motion related information, prior work encountered content-bian (Ge et al., 2024), especially for classification models.
I3D The I3D network an inflated 3D convolutional neu- ral network based on the Inception architecture for image classification that can be applied to videos (Carreira & Zis- serman, 2017). Unterthiner et al. (2018) proposed to use the logits of an 13D network that was trained to perform action recognition on the Kinetics-400 dataset (YouTube videos of humans performing various actions (Carreira & Zisserman, 2017)) as a feature space for comparing videos. For a single-video metric from I3D, we use the negative entropy over the action class predictions (meaning lower scores indicate worse quality).
4. Human evaluation
To evaluate whether the proposed metrics are useful, we compare them to human judgements. We propose to source fine- grained human annotations for generated videos, focus- ing on the following dimensions. Consistency: Whether the appearance and motion of objects and background appear consistent over time; Interactions: Whether the interactions between the objects are realistic (if any take place); Realism: Whether the objects and background are depicted realisti- cally; and Speed: Whether the motion of the objects and camera is perceived being slow, normal, or fast.
VideoMAE VideoMAE (Tong et al., 2022) is Masked Autoencoder (MAE) trained on videos using a self- supervised reconstruction objective, which was previously found to reduce frame-level bias relative to I3D when used for evaluating videos (Ge et al., 2024). Here we use VideoMAF-v2 (Wang et al., 2023b), which is trained on a mixed set of raw video datasets. Importantly, the reconstruc- tion objective for VideoMAE-v2 not autoregressive it is not trained to predict the future, rather it is trained to "fill in" missing patch tubes from a video. In practice, VideoMAF. v2 is usually further fine-tuned on specific downstream tasks to deal with domain shift.
Together, these are intended to capture common failure modes of generative video models, such as objects changing in shape unnaturally, discrete jumps or jitter, poor interac- tions, such as objects that blend when they come together, or highly implausible scenes. We allow humans to provide real-valued judgements where possible reflect that generated video might typically meet the criteria above only to a variable degree. We refer to Appendix B.3 for details about questions, including an overview of the evaluation UI.
5. Results
We consider two variants of VideoMAE-v2, the pre-trained model (VideoMAE and a model that has been further fine-tuned for action recognition the SSv2 dataset (Goyal et al., 2017) (VideoMAE-SSV2). For VideoMAE-PT. we pool over patches to calculate the embeddings, and use the negative L2 loss on the decoded patches as the single-video metric. For VideoMAF-SSV? we follow Ge et al. (2024) and choose the penultimate layer before classification for the embeddings, and the negative entropy over action class predictions as the single-video metric.
We evaluate TRAJAN and alternative approaches in three different scenarios for how one might make use of them metrics in practice: conducting per-video ($5.1) qual- ity assessments, comparing generated to real videos ($5.2),
0.9
model
NolrValleyy
VideoMAE
-0.40
TRAJAN
0.0
gen2
f3
-0.75
-0.500
MooG MooG (van Steenkiste et al.. 2024) is a recurrent model trained for next-frame prediction. operates by maintaining and updating internal state constituting of off-the-grid latent tokens, which can be decoded to predict the next frame. These latents are first randomly intialized, and then updated on each iteration by a transformer model which cross attends to the image features of the correspond- ing frame, followed by a set of self-attention layers. The
0.6
december
videcratteev1..
Human rated realism
Human rated realism
Figure 3. TRAJAN (right) captures the rank ordering of human preferences for different generative video models very well rel- ative to VideoMAE (left). When correlated with human realism ratings, TRAJAN achieves a spearman rank coefficient of 0.9, while VideoMAE only achieves 0.07.
5

---

Direct Motion Models for Assessing Generated Videos
EvalCrafter
VidcoPhy
Motion Consist.
App. Consist.
Realism
Interacts
Motion Consist.
App. Consist.
Realism
Interacts
Method
V-MAEpT L2
-0.00
0.00
0.04
-0.05
-0.03
-0.06
0.06
0.02
I3D Entropy
-0.01
-0.03
-0.02
0.03
0.09
0.08
0.14
0.09
RAFT Warp
0.28
0.27
0.25
0.13
0.20
0.26
0.18
0,03
MooG PSNR
0.21
0.19
0.17
0.05
0.11
0.16
0.07
0.01
TRAJAN
0.29
0.29
0.27
0.19
0.25
0.32
0.29
0.09
Inter-raters
0.49
0.46
0.47
0.53
0.48
0.46
0.48
0.53
Table Spearman's 's rank coefficients between human ratings and automated metries for subset of videos from EvalCrafter (Liu al.,
2024c) and VideoPhy (Bansal al., 2024b) (higher is better). Inter-rater is the standard deviation of human responses (lower is better).
ERE
Figure Humans rate this video to be poor in terms consistency realism questions. automated metries score this positively. this case, none of the metrics capturing the unexpected disappearance and morphing of the glass.
and comparing sets of videos distributional- ($5.3). Across all settings, find that TRAJAN consistently per- forms better than the alternatives in its sensitivity to motion artefacts. We report additional results in Appendix A and complete experimental details in Appendix B.
EvalCrafter
VideoPhy
Camera speed
Object speed
Camera speed
Object speed
Method
RAFT Mag.
0.32
0.57
0.12
0.63
TRAIAN- Len.
0.26
0.58
0.16
0.47
TRAJAN- Radii
0.42
0.01
0.39
-0.28
5.1. Assessing generated videos individually
Inter-rater
0.06
0.08
0.07
0.08
In many cases, we want metric that can be used to de- termine the quality the motion for a single video, i.e. without reference.
Table 2. Spearman's 's rank coefficients between human ratings and automated metrics for the amount of motion generated vidcos from the EvalCrafter and VideoPhy datasets.
increasingly common to encounter generative video models for which we do not have access training data, that too computationally expensive to sample many videos from. Here investigate metrics to evaluate different aspects of motion on per-video ba- sis. We source generated videos from the "solid-solid" split of VideoPhy (Bansal et al., 2024b), which contains videos generated by different models, and EvalCrafter (Liu et al., 2024c), using 11 different models. Although these datasets include human evaluations, they focus aggregate metrics such "motion quality" and do not contain enough infor- mation to evaluate consistency between raters. Therefore, in addition to the original ratings, we conduct our own hu- man evaluation focusing on individual motion dimensions as outlined in $4. Each video is rated by 3 raters sampled randomly from total pool of 10. We then z-score each rater's responses to account for different raters using the sliding scale differently (Appendix B.3).
lated dimensions. We also show overall correlations for TRAJAN and VideoMAE against human-rated realism for the 11 different generative video models from EvalCrafter in Figure 3. All numbers are reported as Spearman's 's rank coefficients. We find interesting and surprising results from the human study. First, the humans themselves are consistent with each other. We do not have sufficient num- bers of overlapping examples for rater pairs, compute the inter-rater standard deviation or for each video for each question type and average across all videos in the dataset. Larger or therefore indicates more discrepancy be- tween raters a given question type. For example, given that ratings were z-scored by participant, the standard devi- ation for single rater across the dataset would be 1.0. A of 0.5 therefore indicates that there is approximately half as much variation in rater responses to a single question as there is variation in a single rater across the dataset.
instead
A summary of our main findings are shown in Table for consistency related dimensions, and Table 2 for speed re-
Across all question types, in both datasets, TRAJAN best correlates with human judgements, and performs similar
6

---

Direct Motion Models for Assessing Generated Videos
Autrage bocard over time
0.8
Average jaccard Jaccad
0.6
-04
0.2
Figure 5. The Average Jaceard (reconstruction quality higher AJ means higher reconstruction quality) from TRAJAN can be used detect generated video inconsistencies at specific points in space and at specific moments in time. Here we show two examples (top and bottom). the left the AJ averaged over all points for each frame of the video. This highlights moments in time where there are temporal inconsistencies overall. On the right we visualize the AJ for each visible point track for 4 consecutive frames centered on the frame having the lowest AJ score (indicated with the dashed lines on the left). Red indicates poor reconstruction (low AJ) while blue indicates good reconstruction (high AJ). In the top case, the motion is consistent over most of space, there is overall higher AJ across the points. the bottom case, the glove's appearance morphs over the frames, leading to worse reconstruction for points located on and immediately around the glove.
to RAFT for consistency- relattd categories. While this may be expected for motion and appearance consistency, TRAJAN also correlates best for realism both for the video overall, and for the object interactions. This holds not just on a per-video basis, but also tracks which generative video model humans prefer overall for realism (Figure 3). If condition on only high motion videos (those rated by humans as having camera speeds greater than or equal to "medium"), where we expect motion- metrics to be most informative, we see even stronger correlations (and lower inter-rater os), especially for the EvalCrafter dataset (Table and Appendix A). The point tracks can also be used measure motion amount (Table 2), with both point track lengths and optical flow magnitude being predictive of object speeds, and point track radii (see Appendix B.2. for details on how these are calculated) being predictive of camera speed.
Method
Consist.
Quality
Visual
T2V
Subj.
V-MAEpT
-0.19
-0.16
-0.06
-0.07
-0.02
I3D
0.09
0.24
0.22
0.25
0.14
RAFT
0.08
-0.01
0.09
0.13
0.15
MooG
0.13
0.08
0.01
0.11
0.03
TRAIAN 0.33
0.24
0.28
0.25
0.23
Table 3. Spearman rank coefficients the original EvalCrafter dataset for their categories of temporal consistency, motion quality, visual quality, text-to-video similarity, and subjective likeness, when holding the overall amount of motion fixed.
ever, we found that human judgements in this dataset (across temporal consistency, visual quality, text-to-video similar- ity, and subjective likeness), were mostly driven by sim- ple measures of overall motion (i.e. videos with significant motion are worse, regardless of how realistic that motion appears). We therefore re-analyze the data controlling for overall motion by binning videos based on their optical flow magnitude (RAFT- Mag), and then evaluate Spearman rank coefficients within bins falling into the top half of the dataset (those where there was significant motion). We av- erage the coefficients across bins. Here, TRAJAN strongly outperforms other metrics in explaining temporal and vi- sual consistency, well subjective likeness, suggesting that it captures more than just overall motion magnitudes in explaining motion quality.
However, even TRAJAN not a perfect predictor of hu- man judgements. For example, in Figure 4, a beer glass disappears by collapsing into the table. All metrics rate this video highly, as the motion is smooth and easily tracked. However, this is a very unrealistic motion, since there is nothing that would cause the glass to collapse. Future work would be needed to develop metrics with expectations about what should happen rather than simply tracking what does happen.
Original VideoPhy dataset As further external valida- tion, we report the area under the ROC curve using the original human labels for physical consistency contained in the VideoPhy dataset (Figure 6). RAFT, MooG and TRA- JAN are the top-performing models in this case. We also
Original EvalCrafter dataset To ensure that our metrics are not biased to the human evaluations we collected, we also compare to the human ratings released on the Eval- Crafter dataset by the original authors in Appendix A. How-
7

---

Direct Motion Models for Assessing Generated Videos
1.0
ROC-AUC Physical Cammonsense on VideaPhy
Method
PSNR
SSIM
V-MAEPT
0.35
0.53
0.8
I3D
0.24
0.26
Tue Pocttvv Rate
TRAJAN
0.23
0.13
0.6
VideoCan 0.71
Table 4. Spearman's p between latent space distances and PSNR/ /
0.4
TRAJAN AUC 0.55
RAFT 0.64
SSIM scores for high motion videos generated by WALT.
0.2
MooG AUC = 0.52
VideoMAE
I3D AUC 0.47
0.0
sideo-to-vides similarity better than pixel-based scores.
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
To do so, we take the WALT video diffusion model (Gupta et al., 2024) and train it on the Kinetics-600 dataset (Carreira et al., 2018) to predict future video frames conditioned 2 real latent frames. We sample checkpoints from throughout training, and calculate video embeddings using TRAJAN, VideoMAE and I3D on 2364 generated and corresponding real samples with high motion as estimated by the length of the point trajectories.
Figure 6. ROC-AUC Scores for physical consistency on Video- Phy dataset. Note that VideoCon is fine-tuned human labels, whereas all other methods are not.
compare to VideoCon as proposed in Bansal et al. (2024b) as method for evaluating physical consistency in generated videos, which fine-tunes a vision-language model on sub- set of the human labels. Without any fine-tuning on human data, TRAJAN performs strikingly well, which indicates its generality for evaluating motion in videos originating from different models and datasets (i.e. prompts).
In Table 4, we show that the distances in model latent space are not well correlated with either PSNR SSIM for any of the models, and the correlation is lowest for TRAJAN. This suggests that the TRAJAN latent space is capturing something fundamentally different from pixel error, which we highlight with examples in Figure 7. In particular, TRA- JAN is sensitive to differences in motion but not appearance, while distances in the VideoMAF and I3D latent spaces are more sensitive overall recognizability. This suggests that the TRAJAN latent space is useful for measuring similar- ities in the motion between videos, even when the exact pixels or overall appearance are not a perfect match.
Spatiotemporal error localization with TRAJAN Fi- nally, we demonstrate how TRAJAN can be used to localize generative video errors in both space and time for an indi- vidual video. In particular, TRAJAN allows us to calculate reconstruction quality (through the Average Jaceard metric applied to the reconstructed points) for each point in space over short windows in time to determine where and when motion and appearance inconsistencies occur.
Figure 5 shows a generated video which TRAJAN scores as highly consistent (top) and highly inconsistent (bottom). Looking at the progression over time on the left, the con- sistent video maintains reasonably high Average Jaccard. However, for the video at the bottom, where the glove's appearance morphs throughout the beginning of the video before stabilizing, we see a dramatic change in Average Jaccard. Visualizing the spatial errors around these poorly reconstructed frames (on the right) shows that the points on the glove are contributing the most to the low score.
5.3. Comparing video distributions
Finally. we demonstrate how to compare empirical distributions of videos, which is the usual paradigm for metrics based on FVD (Unterthiner et al., 2018). We obtain real videos from the UCF101 dataset, which consists of 13,320 videos recorded in the wild that show humans performing different types of actions (Soomro et al., 2012), and synthetically corrupt them using the 5 levels of "elastic transformations" from Ge et al. (2024). Corruption levels 1.1 and 1.2 apply low frequency deformations, while 2.1, 2.2, and 2.3 apply high frequency deformations (Figures 13 & 14 Appendix B) There are two corruption modes: a spatiotemporal (ST) mode, where video frames are distorted independently using different parameters, and a spatial (S) mode where the same set of parameters is used to distort all the frames. Ge et al. (2024) propose to use the ratio of these modes to measure how sensitive metrics are to temporal corruptions.
5.2. Comparing real and generated video pairs
Many video models are evaluated by conditioning on a few frames of real video, and then predicting how that video should unfold in the future (Whitney al.. 2023; Wang et al., 2017; Lin al., 2020). However, comparing future generated and real frames is not straightforward. Compar- isons in pixel space can suffer from inherently uncertain futures such as objects that might enter the frame, or pan- ning a camera such that new parts of the scene are visible (Figure 7a). In this section, we investigate whether com- parisons in the latent space of various models can capture
We first apply the standard technique of computing the Fréchet distance between the latent representations com-
Parameters were obtained via personal correspondence.
8

---

Direct Motion Models for Assessing Generated Videos
Gemerated
Cerre
Real
(a) A generated video from WALT where the camera pans new parts the scene that were not initially visible (e.g. the wheelchair). The objects on the right are inhereatly impossible to predict, which leads to large pixel error between the generated and the real vidco. However, since the camera motion correct, the distance between their TRAJAN embeddings small.
(h) A generated viden from WALT with incorrect predicted motion. Despite the incorrect motion, the I3D and VideoMAE latent space distance between the generated and real videos is very low (rank 38, 74 2364). while the distance in the TRAJAN embeddings is high (rank 1655 2364).
Figure 7. WALT extrapolations for two vidcos (top: generated, bottom: real).
'emporal Sensitivity Fröchet
Temporel Sensilivity Reconstructon
8-
VideoMAE-LZ L2
20-
VideoMAE PT
VideoMAE 55V2
ST 5 ranto
13D Entropy
VideoMAE-SSVZ Entropy
1
TRAJAN AA]
10
TRAJAN
RAFT Warp
Motion Hist
MooG PSNR
Figure& 8. Comparing different methods for detecting spatiotemporal distortions (ST) relative to spatial-only distortions (S) on the UCF-101 dataset (Ge et al., 2024) in terms (left) Fréchet distances latent space and (right) per-video ordinal scores. TRAJAN and RAFT warp error are particularly sensitive to spatiotemporal distortions (high ST / S ratios), while motion histograms (Liu et al., 2024a) and appearance- based methods are equally sensitive to spatiotemporal vs. spatial-only distortions. See Figure 13 and Figure 14 for examples.
puted for videos within each set, using TRAJAN and four of the models presented in $3.2. Figure 8 (left) reports the average temporal sensitivity across different levels of cor- ruption strength in Ge et al. (2024). TRAJAN is the most sensitive to temporal distortions.
and real video pairs even when the appearance is not main- tained. Finally, TRAJAN presents an alternative FVD for measuring generated video quality, as it is significantly more sensitive to motion artefacts at the distribution level. Together, these results suggest an alternative approach to evaluating generated videos: not just by using a different backbone for FVD, but by moving away from costly distribu- tional comparisons all together, and focusing on comparing generated real video pairs in latent space or evaluating individual videos (all three are supported by TRAJAN).
To bridge distribution level and single-video metrics, we compute the latter for each video and take the average across the set (Figure right). TRAJAN and the RAFT warp error particularly sensitive temporal distortions, while motion histograms (Liu et al., 2024a) and appearance-based methods are not. Although distributional comparisons yield the highest overall sensitivity, the strong performance of TRAJAN suggests that per-video metrics based on motion can be used as replacement for distribution-bart based metrics such as FVD if a reference dataset is not available or if sample sizes are limited.
Despite the strengths of TRAJAN, we also found that there are shortcomings. By conducting detailed human study, we showed that while TRAJAN predicted human judgements better than range of alternative motion- and appearance- based metrics. there are videos for which no automated metric captures human ratings. To capture hu- man judgements in these scenarios, future work will need to develop more advanced models that can make more accurate judgements not just about how something does move, but how it should move if it was obeying physical principles.
6. Conclusion
We investigated different methods for evaluating motion in generated videos, and proposed TRAJAN, a novel architec- ture for auto-encoding dense point tracks. We showed that TRAJAN's reconstruction error correlates remarkably well with human judgements of motion consistency, appearance consistency, realism, and even visual quality, and can be used to localize spatiotemporal errors in generated video. TRAJAN can also be used to compare motions of generated
Furthermore, individual people seem to care about different aspects of videos for evaluating basic properties like tempo- ral and appearance consistency, leading to low consistency in evaluations across human participants. This suggests that future work is needed to determine how to elicit better judge- ments from human raters, and suggests caution for using humans as gold standard for adjudicating between models.

---

Direct Motion Models for Assessing Generated Videos
Acknowledgements
Bugliarello, E., Moraldo, H. H., Villegas, R., Babacizadeh, M., Saffar, M. T., Zhang. H., Erhan, D., Ferrari, V., Kin- dermans, P.-.,, and Voigtlaender, P. Storybench: mul- tifaceted benchmark for continuous story visualization. Adv. Neural Inform. Process. Syst. 36, 2024.
We would like to thank Mike Mozer for helpful comments and suggestions to improve our paper, and Viorica Pa- traucean, Yiwen Luo, Lily Pagan, and Nishita Shetty for help preparing the human study. We would like thank Songwei Ge and Ge Ya Luo for sharing details about the video corruptions used in their work.
Butler, D. J., Wulff, J., Stanley, G. B., and Black, M. A naturalistic open source movie for optical flow evaluation. A. Fitzgibbon et al. (Eds.) (cd.), Eur. Conf. Comput. Vis., Part IV, LNCS 7577, pp. 61 1-15.. Springer- Verlag, October 2012.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. By presenting novel evaluation framework for generated videos, this work has the potential to aid in the detection of generated video content. It also has the potential to allow generative video model designers to better understand whether their models will appear realistic to human observers. However, we do not feel that these risks outweigh the potential benefits. There are many other potential societal consequences of our work, but do not feel that they must be specifically highlighted here.
Carreira, and Zisserman, A. Quo vadis, action recogni- tion? new model and the kinetics dataset. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 6299 6308, 2017.
Carreira, J., Noland, E.,, Banki-Horvath, A., Hillier, C., and Zisserman, A. A short note about kinetics-600. arXiv preprint arXiv: 1800.00084..444 2018.
Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang. S., Xing, J., Liu, Y., Chen, Q., Wang, X., et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19511 19512, 2023.
References
J. Kiros, J. R., and Hinton, G. E. Layer normalization. NeurlPS Deep Learning Symposium, 2016.
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang. X., Weng, C., and Shan, Y. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. JEEE Conf. Comput. Vis. Pattern Recog., pp. 7310-7320, 2024.
Bansal, H., Bitton, Y., Szpektor, I., Chang, K.-W., and Grover, A. Videocon: Robust video-language alignment via contrast captions. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 13927-13937, 2024a.
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. Scannet: Richly-annotated 3d recon- structions of indoor scenes. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 5828-5839 2017.
Bansal, H., Lin, Z., Xie, T., Zong, 7., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W., and Grover, A. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.033500 2024b.
Dalal, N. and Triggs, B. Histograms of oriented gradients for human detection. In Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recog,, volume 1, pp. 886-893. leee, 2005.
Binkowski, M., Sutherland, D. J., Arbel, M., and Gretton, A. Demystifying mmd gans. In Int. Conf. Learn. Represent., 2018.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers 22 billion parameters. In Int. Conf. Mach. Learn., pp.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.151277 2023a.
PMLR, 2023.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 22563- 22575, 2023b.
Doersch, C., Gupta, A., Markeeva, L., Recasens, A., Smaira, L.., Aytar, Y., Carreira, J., Zisserman, A., and Yang, Y. TAP-vid: A benchmark for tracking any point in a video. Adv. Neural Inform. Process. Syst., 35:13610-
13626, 2022.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y.,
Doersch, C., Luc, P., Yang, Y., Gokay, D., Koppula, S., Gupta, A., Heyward, J., Rocco, I., Goroshin, R., Carreira, J., et al. BootsTAP: Bootstrapped training for tracking- any-point. In Asian Conf. Comput. Vis., 2024.
Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. Video generation models as world simulators. 2024.
10

---

Direct Motion Models for Assessing Generated Videos
Esser, P., Chiu, J., Atighehchian, P., Granskog, J., and Ger- manidis, A. Structure and content-guided video synthesis with diffusion models. In Int. Conf. Comput. Vis., pp. 7346-7356, 2023.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv22210002303, 2022a.
Ge, S., Mahapatra, A., Parmar, G., Zhu, J.-Y., and Huang, J.-B. On the content bias in fréchet video distance. In IEFE Conf. Comput. Vis. Pattern Recog., pp. 7277-7288, 2024.
Ho, J.. Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. Video diffusion models. Adv. Neural Inform. Process. Syst. 35:8633-88 2022b.
Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. Vision meets robotics: The kitti dataset. Int. J. Robotics Res., 32 (11):1231-1237) 2013.
Hong, W. Ding, M., Zheng, W. Liu, X., and Tang, J. Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. arXiv preprint arXiv:2205.1586. 2022.
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Fueller-Freitag, M., et al. The" something something" video database for learning and evaluating visual common sense. In Int. Conf. Comput. Vis., pp. 58455544555 58522, 2017.
Hotshot-XL. Hotshot-xl.
https://hugginggaat soottshhoooooothttt.....
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q. Chanpaisit, N., et al. Vbench: Com- prehensive benchmark suite for video generative models. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 21807- 21818, 2024.
Grauman, K., Westbury, A., Bymne E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang. H., Liu, M., Liu, X., et al. Ego4d: Around the world in 3,000 hours of egocentric video. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 18995-19012, 2022.
Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J. Perceiver: General perception with iterative attention. In Iut. Conf. Mach. Learn., pp. 4651- 4664. PMLR, 2021.
Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duck- worth, D., Fleet, D. J., Gnanapragasam, D., Golemo, F., Herrmann, C., et al. Kubric: A scalable dataset gener- ator. In IEFE Conf. Comput. Vis. Pattern Recog., pp- 3749-3761, 2022.
Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al. Perceiver IO: A general architec- ture for structured inputs & outputs. In Int. Conf. Learn. Represemt, 2022.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf. B., and Smola, A. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
Gu, Y., Zhou, Y., Wu, B., Yu, L., Liu, J., Zhao, R., Wu, J. Z., Zhang, D. J., Shou, M. Z., and Tang, K. Videoswap: Customized video subject swapping with interactive se- mantic point correspondence. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 7621-7630 2024.
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang. Z., Navasardyan, S., and Shi, H. Text2video- zero: Text-to-image diffusion models are zero-shot video generators. In Int. Conf. Comput. Vis., pp. 15964- 2023.
Gupta, A., Yu, I.,, Sohn, K., Gu, X., Hahn, M., F.FF,, Essa, I., Jiang, L., and Lezama, J. Photorealistic video generation with diffusion models. In Eur. Conf. Comput. Vis., pp. 393-411. Springer, 2024.
Kim, J., Kim, S., and Yoo, J. Stream: Spatio-temporal evaluation and analysis metric for video generative mod- els. In Int. Conf. Learn. Represent. 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. Int. Conf. Learn. Represent., 2015.
He, Y., Yang. T., Zhang, Y., Shan, Y., and Chen, Q. La- tent video diffusion models for high- long video generation. arXiv preprint arXiv:2211. 1.1221,, 2022.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Adv. Neural Inform. Process. Syst., 25, 2012.
Hendryeks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. In Int. Conf. Learn. Represent., 2018
Lai, W.-S., Huang, J.-B., Wang, O., Shechtman, E., Yumer, E., and Yang, M.-H. Learning blind video temporal con- sistency. In Eur. Conf. Compul. Vis., pp. 170-185, 2018.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L.. and Choi, Y. Clipscore: A reference- frreencee-feeeeeee evaluation metric for im- age captioning. Conf. on Emp. Meth. in Nat. Lang. Proc., 2021.
Lin, Z., Wu, Y.-.,, Peri, S., Fu, B., Jiang. J., and Ahn, S. Improving generative imagination in object-centric

---

Direct Motion Models for Assessing Generated Videos
world models. In Int. Conf. Mach. Learn., pp. 6140-6149. PMLR, 2020.
language supervision. In Int. Conf. Mach. Learn., pp. 8748-8763. PMIR, 2021.
Liu, J., Qu, Y. Yan, Q. Zeng, X., Wang, L., and Liao, R. Fréchet video motion distance: A metric for eval- uating motion consistency in videos. arXiv preprint 2XXv:2407.166244 2024a.
Sajjadi, M. S., Meyer, H., Pot, E.,, Bergmann, U., Greff, K., Radwan, N., Vora, S., Lučić, M., Duckworth, D., Dosovitskiy, A., et Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 6229-6238, 2022.
Liu, J.-W., Cao, Y.-P., Wu, J. Z., Mao, W., Gu, Y., Zhao, R., Keppo, J., Shan, Y., and Shou, M. Z. Dynvideo- e: Harnessing dynamic nerf for large-scale motion-and view-change human-centric video editing. In IEEE Conf. Comput. Vis. Pattern Recog. pp. 7664-7674, 2024b.
Saxena, S., Herrmann, C., Hur, J., Kar, A., Norouzi, M., Sun, D., and Fleet, D. J. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. Adv. Neural Inform. Process. Syst., 36, 2024.
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang. Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalcrafter: Benchmarking and evaluating large video generation models. In IEEE Conf. Comput. Vis. Puttern Recog., pp. 22139-22149, 2024c.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large- scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Singer, U., Polyak, A., Hayes, T., Yin, An, J., Zhang, S., Hu, Q. Yang, H., Ashual, O., Gafni, O., et al. Make-a- video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Liu, Y., Li, L., Ren, S., Gao, R., Li, S., Chen, S., Sun, X., and Hou, L. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. Adv. Neural Inform. Process. Syst. 36, 2024d.
Soomro, K., Zamir, A. R., and Shah, M. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
Luo, G. Y., Favero, G. M., Hao I Luo, Z.. Martinea,,, A., and Pal, C. Beyond fvd: Enhanced evaluation met- rics for video generation quality. In Int. Conf. Learn. Represent., 2025.
Sterling, S. Zeroscope. https: https/pppppttpee////oggggnn..eoo perspensee/eeosccpeee_2
Luo, Z., Chen, D., Zhang. Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., and Tan, T. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint ar 23v3: 08320, 2023.
Sun, D., Vlasic, D., Hermann, C., Jampani, V., Krainin, M., Chang, H., Zabih, R., Freeman, W. T., and Liu, C. Autoflow: Learning a better training set for optical flow. In IEEE Conf. Comput. Vis. Pattern Recog,, pp. 10093- 10102, 2021.
Mayer, N., 1lg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., and Brox, T. A large dataset train con- volutional networks for disparity, optical flow, and scene flow estimation. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 4040-4048, 2016.
Sun, D., Herrmann, C., Reda, F., Rubinstein, M., Fleet, D. J., and Freeman, W. Disentangling architecture and training for optical flow. In Eur. Conf. Comput. Vis., pp. 165-182. Springer, 2022.
Moon Valley. moonval ley com/.
Moonvalley.
https: https://www.
Teed, Z. and Deng, J. Raft: Recurrent all-pairs field trans- forms for optical flow. In Eur. Conf. Comput. Vis., pp. 402-419. Springer, 2020.
Otani, M., Togashi, R., Sawai, Y., Ishigami, R., Nakashima, Y., Rahtu, E., Heikkil"a, J., and Satoh, S. Toward verifi- able and reproducible human evaluation for text-to-image generation. In JEEE Conf. Comput. Vis. Pattern Recog., pp. 14277-14286, 2023.
Tong. Z., Song. Y., Wang, J., and Wang, L. Vidcomae: Masked autoencoders are data-efficient learners for self- supervised video pre-training. Adv. Neural Inform. Pro- cess. Syst. 35:10078-10092 2022.
Pictures, F.
Floor33 pictures. morphstudio. con/.
https:/ https://www.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 11 2018.
Pika. Pika pika.art.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural
van Steenkiste, S., Zoran, D., Yang, Y., Rubanova, Y., Kabra, R., Doersch, C.,, Gokay, D., Heyward, J., Pot, E., Greff,
12

---

Direct Motion Models for Assessing Generated Videos
K., Hudson, D. A., Keck, T. A., Carreira, J., Dosovitskiy, A., Saijadi, M. S. M., and Kipf, T. Moving off-the-grid: Scene-grounded video representations. In Adv. Neural Inform. Process. Syst. 2024.
Wu, C., Liang, Ji, L., Yang, F., Fang, Y., Jiang, D., and Duan, N. N"uwa: Visual synthesis pre-training for neural visual world creation. In Eur. Conf. Comput. Vis., pp. 720-736. Springer, 2022a.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. Adv. Neural Inform. Process. Syst, 30, 2017.
Wu, H., Liao, L., Chen, C., Hou, J., Wang. A., Sun, W. Yan, Q., and Lin, W. Disentangling aesthetic and technical effects for video quality assessment of user generated content. arXiv preprint arXiv:221 1.04944, 2(5):6, 2022b.
Venkataramanan, S., Rizve, M. N., Carreira, J., Asano, Y., and Avrithis, Y. Is imagenet worth I video? learning strong image encoders from long unlabelled video. In Int. Conf. Learn. Represent., pp. 1-21, 2024.
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W. Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a-video: One-shot tuning of image diffusion models for text-to- video generation. In Inf. Conf. Comput. Vis., 2023.
Villegas, R., Babaeizadeh, M., Kindermans, P.-J., Moraldo, H., Zhang, H., Saffar, M. T., Castro, S., Kunze, J., and Erhan, D. Phenaki: Variable length video generation from open domain textual descriptions. In Int. Conf. Learn. Represent., 2022.
Wu, J. Z., Fang, G., Wu, H., Wang. X., Ge, Y., Cun, X., Zhang, D. Liu, J.-W., Gu, Y., Zhao, R., et al. Towards better metric for text-to- video generation. arXiv preprint M:Xiv:2401.7775 2024.
Xiong, R., Yang. Y., He, D., Zheng. K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In Int. Conf. Mach. Learn., pp. 10524-10533. PMLR, 2020.
Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., and Zhang, S. Modelscope text-to-video technical report. arXiv preprint 11t.:23308006571,, 2023a.
Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang. Y., and Qiao, Y. Vidcomae v2: Scaling video masked autoencoders with dual masking. IEEE Conf. Comput. Vis. Pattern Recog,, pp. 14549-14560, 2023b.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with ex- pert transformer. arXiv preprint arXiv:2408.0600 2024.
Wang, W., Zhu, D., Wang, X., Hu, Y., Qiu, Y., Wang, C., Hu, Y., Kapoor, A., and Scherer, S. Tartanair: A dataset to push the limits of visual slam. In IEEE/RSJ Int. Conf. Intell. Robots Syst. pp. 9909-4916 IEEE, 2020.
Zhang, D. J., Wu, Z., Liu, J.-W., Zhao, R., Ran, L., Gu, Y., Gao, D.,, and Shou, M. Z. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. In- ternational Journal of Computer Vision, pp. 1-15, 2024.
Wang, Y., Long, M., Wang. J., Gao, Z., and Yu, P. S. Predmnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. Adv Neural Inform. Process. Syst., 30, 2017.
Zhang, R., Isola, P., Efros, A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog. 2018
Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al. Lavie: High- quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.151 1503, 2023c.
Zheng, Y., Harley, A. W., Shen, B., Wetzstein, G., and Guibas, L. J. Pointodyssey: large-scale synthetic dataset for long-term point tracking. In Ini. Conf. Comput. Vis., pp. 19855- 19865, 2023.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process., 13(4): 600-612, 2004.
Zheng, Z.., Peng, X., Yang. T., Shen, Ci, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, march 2024. URL https://github com/hpcattech/Open-
1(3):4.
Whitney, W. F., Lopeza T., Pfaff, T., Rubanova, Y., Kipf, T., Stachenfeld, K., and Allen, K. R. Learning 3d particle-based simulators from rgb-d videos. Int. Conf. Learn. Represent., 2023.
Zhou, D., Wang, W., Yan, H., Lv, W., Zhu, Y., and Feng, J. Magicvideo: Efficient video generation with latent diffu- sion models, arXiv preprint arXiv:2211.1101 2022.
Wu, C., Huang, L., Zhang, Q. B., Ji. L., Yang, F., Sapiro, G., and Duan, N. Godiva: Generating open- domain videos from natural descriptions. arXiv preprint arXiv:2104 /4806, 2021.
13

---

Direct Motion Models for Assessing Generated Videos
A. Additional Results
A.1. MMD with different backbones
An alternative to using Fréchet distance for comparing embeddings is to make us of the Maximum Mean Discrepancy (Gretton et al., 2012) as a distance function. This idea was first proposed for comparing distributions of images in Binkowski et al. (2018), and briefly explored for video in Unterthiner al. (2018). MMD is kernel-based approach to comparing distributions without assuming particular form (such as Gaussianity when using Fréchet distance). An unbiased estimator of the squared MMD is:
m k(zz Fzgg
m 12
kx Mr)
1s
7 k(ye 93 n(n 1)
- 2
Σ
Σ
(1)
m(m - 1)
me
where {x: ) and {uy } are samples drawn from the respective distributions we wish to compare, and k(:,)) is kernel function. Here we use the polynomial kernel function k(a, b) == (a" b + 1)3 as in Unterthiner et al. (2018).
In Figure 9 we report the results of using MMD instead of the Fréchet distance for calculating the sensitivity of different models to temporal corruptions on the UCF-101 dataset. Defined as in (1), the MMD can occasionally be negative. If the MMD is negative (which was the case for the motion histograms model for several corruption levels), we replace it with le 6. With MMD, all models are significantly more sensitive to temporal corruptions, with motion histograms showing the largest sensitivity. Upon further investigation we noticed that this is primarily due to 3 cases where the spatial-only corruptions gave negative MMD scores.
Temporal Sensitivity MMD
Log (ST/ 5 ratio)
8-
VideoMAE PTT
6-
VideoMAE SSV2
I3D
4-
TRAJAN
Motion Hist
2-
Figure 9, MMD for UCF-101 corruptions
A.2. Metric changes during training
We measure how each of the different metries tracks WALT's (Gupta et al., 2024) training progress at different checkpoints in training (Figure 10). For the distribution-level metrics, we see similar performance using TRAJAN, VideoMAE or I3D. For the ainleevideoo metrics, only TRAJAN tracks WALT's training reasonably well. However, most of the change in all metrics is driven by the first 100k steps, after which all metrics become more-or-less constant.
WALT training
distributional metrics
WALT training single video metrics
1.00-
1.0
Normalized score
VideoMAE-PT
Normalized score
VideoMAE-PT
0.75
I3D
0.8
13D
TRAJAN
TRAJAN
0.50
MooG
0.25
0.4
0.00
Logitraining steps) (1e6)
Logitraining steps) (1e6)
(a) Distribution-level metrics.
(b) Single-video metrics.
Figure 10. Metrics over training iterations for WALT.
14

---

Direct Motion Models for Assessing Generated Videos
(a) Generated (top) with real (bottom) from WALT. In this example, the camera pans to the left. The pixel error between these two videos is very high, since the camera moves the left, does not fill in the left side of the image successfully. nor does it predict exact spacing of the grating. However, since motion all the objects is correct, distance between these videos in the TRAJAN feature space is low.
(b) Generated (top) with real (hottom) from WALT. In this example, the golfer's arm moves into the frame from the top. The pixel error between these two vidcos is high, because the color of the man's 's jacket is incorrect, and there is generated background object the top of the video that does actually exist. However, the distance the TRAJAN feature space is low, because motion of the golfer's arm is correct.
Figure II. Additional paired WALT generations.
A.3. Additional WALT examples for generated vs. real videos
In Figure 1l show additional examples of comparing real vs. generated videos from the WALT model (Gupta et al., 2024) which have different amounts of average pixel error. In Figure 1la, there is very high pixel error because of objects that could not be seen as the camera pans to the left, and in Figure 11b, the pixel error is reasonably high because the predicted color of the golfer's jacket is incorrect, and an extra background object is generated which does not exist. In both cases, the distance between the real and generated video in TRAJAN feature space is low, because the motion of the camera (in the first case), or the man (in the second case) is correct.
A.4. Additional results on VideoPhy (Bansal et al., 2024b) and EvalCrafter (Liu et al., 2024c)
Table 5 shows the results of different automated metrics on the full EvalCrafter (Liu et al., 2024c) dataset using their labels. We found that across the dataset, RAFT Mag, the average optical flow magnitude across the video, a very good predictor of human judgements in all conditions. This suggests that, on average, videos that are faster are more poorly rated by humans generally.
15

---

Direct Motion Models for Assessing Generated Videos
Method
Consistency
Visual
T2V
Subj.
VideoMABPr
-0.19
-0.12
-0.05
-0.09
I3D
-0.00
0.01
0.00
-0.01
RAFT - Warp
0.61
0.42
0.18
0.34
RAFT Mag
-0,58
-0.37
-0.17
-0.32
MooG 0.42
0.23
0.10
0.22
TRAJAN
0.52
0.36
0.20
0.32
Table 5. Results on full EvalCrafter (Liu et al., 2024c) dataset.
EvalCrafter
VideoPhy
Motion Consistency
Appearance Consistency
Realism
Motion Consistency
Appearance Consistency
Realism
VidCoMAEPT
0.06
-0.05
0.01
-0.01
0.04
0.01
I3D
-0.05
0.08
-0.04
0.14
0.09
0.19
RAFT
0.21
0.19
0.11
0.30
0.40
0.09
MooG
0.10
0,08
-0.01
0.19
0.31
0.13
TRAJAN
0.44
0.51
0.34
0.33
0.41
0.14
Rater
0.43
0.41
0.41
0.47
0.46
0.44
Table Spearman's 's rank coefficients between human ratings and automated metrics for medium high motion subset the data. this subset, people are more consistent in their scores (lower rater σ), and in most cases, TRAJAN also better predicts human ratings. In the case VidcoPhy realism, there are specific failure modes in the dataset where some models produce vidcos with inconsistent motion but realistic individual frames, see Figure 12a, but humans still rate these as realistic.
Table 6 gives results for both EvalCrafter and VideoPhy when looking only at medium and high motion samples (those with human ratings of "medium" or "high" on the camera motion question). In these cases, TRAJAN provides better correlations to human ratings, and humans are more consistent with each other (lower inter-rater σs). This suggests, similar to the results on the external human datasets for EvalCrafter, that TRAJAN capturing human ratings consistency beyond just overall amount of motion.
We also provide further qualitative examples of how people rate videos from the VideoPhy (Bansal al., 2024b) and EvalCrafter (Liu et al., 2024c) datasets in human study in Figure 12. In Figure 12a, people rate the video realistic, despite the fact that the video is temporally inconsistent. In Figure 12b, people rate the video unrealistic and inconsistent, even though the motion of the dogs playing poker looks reasonable. In Figure 12c, people also rate the video as both unrealistic and inconsistent in appearance, even though the appearance does not change. People therefore seem strongly affected by the semantics of the objects even when prompted to ignore this information.
Validation Study. We validate the UI for our human study by conducting a second human study on the same subset of videos obtained from the EvalCrafter dataset. For this study, we replaced the slider scales with preset answer options corresponding 5-point Likert scale. Here option implies a video that is very unrealistic (for questions about realism) or inconsistent (for questions about consistency) along this dimension, and option 5 implies video that is very realistic (for questions about realism) or very consistent (for questions about consistency). For this study we used 5 raters per question.
Results are reported in Table 7, where we post-processes in the same way as was done for Table 1. To make the results more comparable, we mapped the 1-5 Likert scale onto the slider scale by interpreting each seale as increments of 20. can be seen how TRAJAN and RAFT best correlate human judgement similar to in Table (left) for consistency-relater categories and realism, while TRAJAN performs clearly better for object interactions.
16

---

Direct Motion Models for Assessing Generated Videos
(a) Example from the VideoPhy dataset sampled every 5 frames. This sample is scored in the upper half of the dataset for realism and appearance consistency by humans, and receives physical consistency score of from the original VideoPhy dataset.
(b) Humans rate badly on consistency realism metrics. automated metrics score highly. this case, motion consistency and appearance consistency qualitatively appear good, but people seem unable to separate realism from consistency.
(c) Humans rate badly on realism metrics. automated metrics score this highly. In this case, motion consistency appearance consistency qualitatively appear good, but people seem unable to separate the realism of the soccer football from consistency.
Figure 12, Examples from VidcoPhy and Craferafter with human ratings.
17

---

Direct Motion Models for Assessing Generated Videos
Method
Motion Consistency
Appearance Consistency
Realism
Interacts
V-MAEPT L2
0.04
0.07
0.10
-0.07
I3D Entropy
-0.05
-0.06
-0.02
0.02
RAFT Warp
0.39
0.43
0.29
0.18
MooG- PSNR
0.28
0.32
0.26
0.09
TRAJAN
0.38
0.40
0.26
0.26
Inter-rater
0.09
0.07
0.09
0.12
Table Validation Study. Spearman's rank coefficients between human ratings and automated metrics for a subset of videos from EvalCrafter (Liu et al., 2024c) (higher is better). Inter-rater is the standard deviation of human responses (lower is better).
B. Experiment Details
B.1. Datasets
B.I.1. UCF101
For our experiments with synthethic distortions, we make use of the UCF101 dataset (Soomro et al., 2012), which consists of 13,320 videos recorded in the wild that show humans performing different types of actions. We make use of the version available at https: / /www. tensorflow
alog/ saoog//ucfo/ /uff001 having 256 x 256 resolution. Similar to prior work (Ge et al., 2024), obtain the 'ground-truth' reference distribution by combining the first 32 frames of videos in the train and test split.
To obtain distorted videos, we consider the elastic transformation from Ge et al. (2024). Details were obtained via personal correspondence with the authors, which we reproduce in the following.
Elastic Transformation (Ge et al., 2024) The reference implementation for this transformation is https: htt__///cmmaeeee__.m__ corruptions py, which works by first performing an affine- and then an elastic transformation. The reference implementation suggests five levels of degradation (Hendrycks Dietterich, 2018). The first two levels (referenced as 1.1 and 1,2 in ??) primarily trade-off the strength of the affine transformation with that of the elastic transform, such that 1.2 creates more global distortions to the shape of objects. Levels 2.1, 2.2, and 2.3 fix the affine transformation small intensity, and only increase the strength of the elastic part. This results in much more local distortions associated with high-freyuency noise. Ge et al. (2024) adopt these sample levels, except that they adjust the resolution parameter from 244 to 128. Here we adopt the same distortion levels as in Ge et al. (2024).
//github. com/ seendryckk/robustee
To compute the temporal sensitivity of metric, Ge et al. (2024) proposes to evaluate them on both spatially distorted videos and on spatiotemporally distorted videos, and report the ratio of the two. In practice, the elastic transformation is applied at per-frame level, where the distortion levels parametrize distribution of corruptions that sampled from. When using the same seed between frames in a video, the same corruption is applied to each frame, creating only a spatial effect. To obtain a spatiotemporal effect, a different corruption is sampled for each frame in the video.
In Figures 13 & 14 we show an example of a UCF-101 video with spatial- or spatio-temporal corruptions applied using the five levels of elastic transformation from Ge et al. (2024). Note that in the spatial setting, the same corruption is applied to each frame, while in the spatiotemporal setting the corruption parameters are resampled for each frame.
B. 1.2. VIDEOPHY
We make use of the "solid-solid" split of VideoPhy (Bansal et al., 2024b) available for download at https: / huggingface. cocaace.oo//ae co/datasets potisets/videe wideophysiccss/ hwtpppyyssss/videpppppssss videophy..y... test test__ublie____eeepppeeeptttpptt public. These include generated videos from several competitive models, including Cog VideoX (2b and 5b) (Yang et al., 2024), Gen-2 (Esser et al., 2023), LaVIE (Wang et al., 2023c), OpenSora (Zheng et al.), Pika (Pika), SVD (Blattmann et al., 2023a), VideoCrafter2 (Chen et al., 2024), and ZeroScope (Sterling). Note that at the time of preparing this submission, we were unable to download videos contained in the dataset belonging to Luma Dream Machine, which were made available at a later date, We pre- pre-processed all videos to 256 x 256 resolution.
18

---

Direct Motion Models for Assessing Generated Videos
(a) Spatial-only corruptions level 1.1
(b) Spatial-only corruptions level 1.2
Spatial-only corruptions level 2.1
(d) Spatial-only corruptions level 2.2
(c) Spatial-only corruptions level 2.3
Figure 13. UCF-101 vidco with spatial corruptions using the five levels clastic transformation from Ge et al. (2024).
19

---

Direct Motion Models for Assessing Generated Videos
(a) Spatiotemporal corruptions - level 1.1
(b) Spatiotemporal corruptions level 1.2
(c) Spatiotemporal corruptions level 2.1
(d) Spatiotemporal corruptions level 2.2
(c) Spatiotemporal corruptions level 2.3
Figure 14, A UCF-101 vidco with spariotemporal corruptions comptioon using the five levels clastic transformation from Ge et al. (2024).
20

---

Direct Motion Models for Assessing Generated Videos
B. 1.3. EVALCRAFTER
The EvalCrafter dataset (Liu et al., 2024c) contains generated videos for diverse of prompts and is available for download at ht tps: /hwgggngfacee.eeecmcc///aaa/aaae co/ ///aaaaseesss comparable size to VideoPhy, while preserving diversity as much possible, randomly select 104 generated vidcos from 11 of the text-to-video models (the original 5 from the EvalCrafter human evaluation dataset, and 6 additional models). These models are:
wwl.i//aaaaaaaee:..aaaaaaaaaaaae _ere___v_____ See____V________a__ Dataset. To obtain
1. Gen2 (December) (Esser et al., 2023)
2. MoonValley (MoonValley)
3. PikaLab (December) (Pika)
4. Show-1 (Zhang et al., 2024)
5. VideoCrafter 1.0 (Chen et al., 2023)
6. Hotshot-XL, (Hotshot-Xo)
7. PikaLab (Pika)
8. Gen2 (Esser et al., 2023)
9. Floor33 Pictures (Pictures)
10. ZeroScope (Sterling)
11. ModelScope (Wang et al., 2023a)
We obtain 1 1144 videos in this way, which preprocess to 256 x 256 resolution.
1.4. WALT GENERATIONS
We train a WALT (Gupta et al., 2024) diffusion model (214M params) for frame-conditions video generation on the Kinetics-6o 600 dataset (Carreira et al., 2018). The model architecture and hyperparameters are consistent with those used in the original paper. Training is conducted for 495,000 iterations with batch size of 256, using videos at a resolution of 128 x 128 with 17 temporal frames. Following WALT, the model is designed to predict three future latent frames conditioned on the initial two latent frames. To evaluate performance, we compute metrics at key checkpoints, using model weights saved at steps 1, 100, 0000, 200, 000, 300, 000, 400, 000, and 495, 000. We upsample generated videos to 256 x 256 resolution and make use of the first 16 frames. We show the distribution level metrics and single video metrics evaluated on samples generated at these intermediate steps in Figure 10.
B.2. Models
B.2.1. TRAJAN
We train TRAJAN on video dataset of publicly accessible videos following Doersch et al. (2024), aiming for high-quality and realistic motion. We use videos tagged as lifestyle and one-shot videos, but omit those videos from categories with low visual complexity or unrealistic motions, e.g. tutorial videos, lyrics videos, and animations. We select only 60fps videos with over 200 views, without cuts or overlays. However, a key difference from prior work (Doersch et al., 2024) is that we use 150-frame clips, whereas prior work focused on 24-frame clips for bootstrapping point tracking. We sample 15 million such videos, and for each video, we choose 4096 points uniformly at random across space and time for every clip and track them using the public BootsTAPIR (Doersch et al., 2024) model.
There are three key challenges for representing set of point tracks as a dense vector where distances are meaningful. I) point tracks are orderless, in the sense that they cannot be arranged into a sequence of tokens: unlike image pixels, tracks do not occur on a grid, and there may be no single video frame where all tracks are visible. Therefore, we wish to have a permutation-inn invariian representation. 2) occlusions should be treated missing data rather than fundamental part of
21

---

Direct Motion Models for Assessing Generated Videos
the motion, since two motions may be very similar in the real world even if one is partially occluded. Finally, 3) although are representing dense signal (every point in the image corresponds to single track), for computational reasons we only receive finite number of samples from the underlying function. The exact points that are chosen for tracking are not important for representing motion, so we aim for our representation to be invariant to the specific chosen points.
Let S = {st.y) } be point trajectories and occlusions, where $1.5 E IR33 = (#r.5 Ut.j 0t.5) corresponds to x and y positions and occlusion flag o at time t for the jth track. As mentioned, we embed all (#t.j Mt.j. t), add a readout token, and perform self-attention using (1 - ot.)) as an attention mask, which helps us achieve point 1) above, as the transformer sees these points as 'masked' rather than some dummy value. After self attention, we discard all tokens except the "readout" token, which now provides fixed-length C-channel representation of each track. We then apply Perceiver (Jaegle et al., 2021) to encode all track tokens; we apply no additional position encoding, meaning that the entire representation is permutation invariant, following principle 1) above. Finally, we project the latent tokens to a lower dimension resulting in fixed-size 128x x 64 dimensional representation Φs of the tracks.
As mentioned, we achieve point 3) above by decoding points that are not included the autoencoder input. The autoencoder knows which track to decode because we give query point Zq. Mq. &g on any frame, and we output the track that goes through this point. Note that this means that the autoencoder can actually output truly dense motion information even though it receives only finite sample as input. As stated, we up-project the tokens in Øs to higher dimension with operator U, apply transformer with readout token, and apply a linear mapping to occlusion logit of and = v/- We train them with softmax cross entropy and Huber loss, respectively, with a weight of 5000 on the Huber loss and le - 8 on the cross entropy; the disparity between loss weights is due to the fact that we mostly want the representation to be mostly invariant to occlusion and focus on motion. In initial experiments, we found that setting these weights equally led to worse performance in correlating with human judgements of realism in generated videos. We find that a naive linear up-projection operator U tends result in poor temporal localization of the query point, as the model struggles to cross-attend the correct latent tokens. We find that we can improve performance by using an upsampling operator that first linearly up-projects, and then extracts window of each token, concatenating along the channel axis. That is, the up-projected representation for the f(d')) Jpt pt + 128)), where f is a linear projection, [-] represents indexing, and
1th token is U = comcat( [f ( f((s)) is a stride. This can be seen as specializing the motion tokens for time I, so the model can more easily identify what motion information is relevant to this query. We train with Adam (Kingma & Ba, 2015) with a warmup cosine learning rate schedule with 1000 warmup steps and a peak learning rate of 2e-4 for 1M steps with a batch size of 64.
Note that not all of the videos would like to evaluate have 150 frames of motion. Therefore, we train the model to also encode shorter clips. Given motion from a 150-frame clip, for half of the examples, we (uniformly) randomly sample an 'end' frame, and mark all points after certain length as 'occluded', For these examples, we only apply the loss to the frames before the 'end' frame.
We evaluate reconstruction accuracy with Average Jaccard, following TAP- Vid (Doersch et al., 2022). Given a threshold 8 "true positives" (T are predictions which are within 8 of the ground truth. "False positives" (FP) P) are predictions that are farther than 8 from the ground truth (or the ground truth is occluded), and "false negatives" (FNN are ground truth points where the prediction is farther than 8 (or occluded). Jaccards is (TP + FP + FN), and Average Jaccard averages Jaccardy over several pixel thresholds. We use the same thresholds proposed in TAP-Vid (Doersch et al., 2022); namely, we resize all trajectories as if they had come from a 256 x 256 video, and use thresholds of 1, 2, 4, 8, and 16 pixels. We find that this model is quite accurate for real-world training data, with average points within threshold of 85.3 when evaluated on held-out data from the same distribution, and an average jaccard of 55.8, roughly meeting the performance of the underlying BootsTAPIR tracker.
Architecture details For our transformer implementation, we use the standard design from Vaswani et al. (2017) with the pre-layer norm configuration from Xiong et al. (2020), and some of the additional improvements introduced in Dehghani et al. (2023). Specifically, we use the RMS norm applied to the keys and queries before computing attention weights, and execute self- and cross-attention paths (not the MLP path) in parallel. Another Layer Normalization layer (Ba et al., 2016) is applied to the output. Transformer hyperparameters are given in Table 9 with names given by their corresponding descriptions in the main text.
Additional hyperparameters for the sinusoidal positional embeddings, and the dimensionalities of projection operators, are given in Table 8.
22

---

Direct Motion Models for Assessing Generated Videos
Sinusoidal emhedding (number of frequencies)
32
Track token projection dimensionality (C) Compression dimensionality
256
64
Up- pp-projection dimensionality
1024 128
Query point encoder dimensionality
1024
Table 8, Positional encoding and projection operator hyperparameters for TRAJAN.
Transformer name
Attention type
QKV size
Layers
Heads
MLP size
Input track transformer Perceiver-style tracks latents Up-projection latent transformer in decoder
SA
64
2
8
1024
CA
64x8
3
2048
CA
64
3
2048
Track readout transformer
CA
64x8
4
1024
Table 9. Transformer architecture hyperparameters for TRAJAN. SA self-atiention, CA = cross-attention.
Track motion radii calculation From the point tracks output by BootsTAPIR (Doersch et al., 2024), we can calculate two metrics of their overall motion: (1) their track lengths E, (z - xz)² + (y++1 - )t)3), masking out non-visible tracks, and (2) their track radii (Figure 15), calculated as the radius of the smallest enclosing circle from the start to the end of a visible track. The track lengths will cover jittery movements or objects changing direction, while the motion radii will capture the maximum distance, consistent direction, for each point.
B.2.2. MOTION HISTOGRAMS
Our implementation of motion histograms (Liu et al., 2024a) is based pettss:/gittie com/DSL-Lab/ fwwehht-reiieeeee viteeet-viieooooiioro
motionn-oe mnotion-distanne distance. They propose to evaluate motion consistency in generated videos by estimating velocity and acceleration from point tracks. Inspired by HOG features (Dalal & Triggs, 2005), they partition the resulting volumes into 4 x 5 5-sized tubelets and accumulate the magnitude of the values at each angle (using 8 bins) within a tubelet. Motion features are obtained by concatenating the resulting ID histograms obtained for each tubelct using both velocity and acceleration. We apply this approach 16-frame chunks of x 64 densely sampled point tracks obtained from BootsTAPIR (Doersch et al., 2024) to yield 9216-dimensional vector describing the motion within the corresponding 16-frame video.
B.2.3. RAFT
To estimate optical flow we use the improved version (Sun et al., 2022) of RAFT (Teed & Deng, 2020) used as baseline in Saxena et al. (2024). Since we are only concerned with obtaining the best possible flow model, we use a RAFT model which was trained on a large mixture of standard optical flow datasets. These include Sintel (Butler et al., 2012), KITTI (Geiger
Figure 15. Examples of point trajectory radii. Circles are shown around maximum amount of motion. Radii are calculated from these circles.
23

---

Direct Motion Models for Assessing Generated Videos
alll,,013 2013), Kubric (Greff et al., 2022), TartanAir (Wang et al., 2020), FlyingThings (Mayer et al., 2016), and AutoFlow (Sun al., 2021).
B.2.4. MOOG
MooG (van Steenkiste et al., 2024) is recurrent model trained for next-frame prediction. It operates by maintaining and updating an internal state constituting of off-the- grid latent tokens, which can be decoded to predict the next frame. These latents are first randomly intialized, and then updated on each iteration by a transformer model which cross attends to the image features of the corresponding frame, followed by a set of self-attention layers. The decoder converts the latent state back to pixels by querying the latents through cross-attention with fixed grid-based features (Sajjadi et al., 2022; Jaegle et al., 2022).
The original MooG implementation distinguishes between a "predicted" state and "correctedd" state, where the former only depends on the previous state, while the latter additionally integrates the current observation. To avoid the model only reconstructing the current observation, the loss is placed on the predicted state in this case. Here we slightly simplify the implementation by using placing the loss on the corrected state, but using the next observation when computing the loss. We train MooG for 600K steps on a mixture of datasets, including Kinetics-700 (Carreira et al., 2018), SSv2 (Goyal et al., 2017), ScanNet (Dai et al., 2017), Ego4D (Grauman et al., 2022), and Walking Tours (Venkataramanan et al., 2024).
B.3. Human Study
To evaluate whether the proposed metrics are useful, we compare them to human judgements. Prior works mainly source human labels at a coarse-grained level using pair-wise comparisons between videos. For example, Unterthiner et al. (2018); Luo et al. (2025) ask humans to compare videos from two different sources (either different models, or with different levels of noise applied to them) to determine which of the two looked better, or alternatively report that their quality was indistinguishable. Liu et al. (2024a) take similar approach using three pairwise comparisons: if a human expresses the same preference for at least two of the pairs then they are determined to prefer that model.
An alternative is to ask humans to evaluate individual videos as in Kim et al. (2024). There they use a 6-point scale to evaluate realism of videos, and a 3-point scale for motion or "temporal naturalness". Similarly, Bansal et al. (2024b) asks humans indicate with yes/no whether videos "follow Physies Laws Physical Commonsense". Liu et al. (2024c) asks humans to score individual videos based on their "motion quality" and "temporal consistency" using 5-point scale.
A possible concern when sourcing human labels in this way is that the questions might be too open epeedended, which limits their usefulness for evaluating motion in generated videos and for developing corresponding metrics. More generally, it isn't clear what motion-related dimensions are measured when asking about "physical commonsense" or when having humans express a preference for one generated video over another. To improve upon this, we propose to source fine- -raiggained human annotations for generated videos using the questions detailed in Figure 16. To avoid conflating perfect consistency in videos that have motion with videos that contain no motion, we first ask humans about whether video contains any motion at all, and only follow-up with the next two consistency related questions if they answer "Yes". Similarly, we first ask humans about whether interactions take place in video. If they answer "Yes", then the slider for rating how realistic these interactions are appears. The other questions are available at all times and not conditioned on any of the previous answers.
We made use of a rater pool of 10 raters, and used raters per question (randomly assigned). Before conducting the full human study, we asked raters to rate 10 questions, and gave feedback on their responses. For example, after an initial pilot, we noticed that many of the ratings were either 0 or 100 (both extreme endpoints of the slider), with few responses in between, and encouraged raters to make use of the full range of the slider to express degrees of agreement. We also noticed that camera motion was initially incorrectly evaluated, which we clarified. Feedback was also provided once we had obtained the full results for VideoPhy, before starting EvalCrafter. In particular, we noted that many videos with inconsistent motion (such as the objects or camera jittering/ / jumping around) received high scores for motion consistency. This is not desirable as for this data it can be assumed that the objects/ / camera are expected to move in a smooth and continuous manner, which we clarified. We encountered several videos with plausible looking individual frames, but that are stitched together in unnatural ways (no continuity between the frames in terms of content, see Figure 12a), which received high realism scores. In this case we provided feedback that this question concerns the entire video as whole and not the sum of individual frames.
24

---

Direct Motion Models for Assessing Generated Videos
SUBMIT
Does the video contain motion?
Yes No
the of objects background consistent time?
the motion of the objects and background consistent over time?
Do any interactions take place between the objects in this video? ?
How realistic are the interactions between the objects in this video?
Are the objects and background depicted in this video realistic? O
How fast the motion of the objects this video?
None
Slow
Medium
Fast
Slow
Fast
00:00/00:06
Figure A screenshot of rater UI. The video plays automatically on repeat as soon opened, Videos were rendered 8ips for ease of rating.
25


Reviews:
[{"review_id": "zWYLM9g351", "reviewer": "Reviewer_zaWZ", "summary": "The authors propose a new video evaluation metric using point tracks instead of pixel reconstruction or recognition features, which can evaluate temporal consistency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and feedback. Please see [https://sites.google.com/view/trajan-videos-anonymous/home](https://sites.google.com/corp/view/trajan-videos-anonymous/home) for additional experiments on score interpretation.\n\n> “The corrupted images as demonstrated in figure.4 do not seem to be low-level distorted (like blurred or locally rotated and shifted). That casts doubt on whether this proposed TRAJAN metric can indeed capture the distortions of real motions”\n\nThe distortions in Figure 4 are taken from prior work by [Ge et al. (2024)](https://arxiv.org/abs/2404.12391) which were chosen to highlight challenges faced by existing metrics (like FVD) in addressing *temporal* above and beyond *appearance-based* distortions. These distortions include local noise (Corruption 2.3, Figure 4e) and local warping (Corruption 1.2, Figure 4b) and were designed to capture realistic distortions from cameras or other sensors. __This experiment primarily highlights the advantage of TRAJAN over prior work (including Ge et al, 2024) for being sufficiently sensitive to temporal rather than appearance-based distortions (i.e. motion biased), and we therefore matched the experimental conditions from prior work__.\n\nHowever, we agree with the reviewer that synthetic distortions are not necessarily representative of real motion distortions. This informed our choice to investigate different model’s sensitivities to the kinds of distortions introduced by state of the art video generation methods.\n\nOur evaluations on VideoPhy and EvalCrafter (as referenced in your review) are doing precisely this. For example, in Appendix B.1.3 we describe how “[...] we randomly select 104 generated videos from 11 of the text-to-video models (the original 5 from the EvalCrafter human evaluation dataset, and 6 additional models).” to conduct a human study. __In Table 1 and 2, we report how TRAJAN is the top-performing model that best correlates with human judgements of motion/appearance consistency and realism.__\n\n>”Some other methods also include motion distortion such as EvalCrafter [1] , VBench [2], or SIFT-sora [3]. This paper should compare against those methods.”\n\nThe __RAFT-Warp error__, which we report in our paper, __is the one used in EvalCrafter, with a related RAFT-based metric in VBench for assessing motion quality and distortions (“Temporal Quality - Dynamic Degree”)__. Across all of our experiments, __TRAJAN performs better than or equal to RAFT-Warp for assessing motion quality and realism__ including on the EvalCrafter dataset.\n\nSIFT-sora is a compelling method for interpretability, but can *only* be applied to videos where the motion solely comes from the camera. __If any of the objects move independently of the camera (which is the case in many generated videos), SIFT-sora cannot be applied__.\n\n> “Some work like [3] works on pure geometry for estimating motion score, but this work still uses representation learning, which lacks explainability a little. Including more analysis on score interpretation will be helpful.”\n\nThank you for this suggestion – we have added a detailed investigation of whether TRAJAN scores can provide more interpretability. Since TRAJAN measures the reconstruction of individual point tracks (Figure 1), it is possible to localize errors in both space and time (with badly reconstructed tracks, shown in red, indicating where the largest errors are). __We provide 3 further examples of this kind of score interpretation [here](https://sites.google.com/view/trajan-videos-anonymous/home)__.\n\nIn each example, we show on the left the full video, in the center the Average Jaccard across all points for each frame of the video, and on the right a clip from the video centered on the frame with the worst overall Average Jaccard. __In the first two cases, there is a major change in object appearance partway through the video, which is picked up in the time course plots. Looking at individual points, the greatest errors are centered on the part of the video that unnaturally changes in appearance, as indicated by the red and white colored point tracks.__ In the last case, this video maintains consistent motion throughout, as evidenced by the consistently high Average Jaccard. Most points are well reconstructed, with the only exceptions being points on the border of the moving object, which are generally ambiguous (their motions could go either way).\n\nWe will update the paper to discuss this aspect of our approach in greater detail and include more examples of localizing motion errors in this way.", "claims_and_evidence": "The claims seem to be supported well in this paper.", "methods_and_evaluation_criteria": "The evaluation makes sense that demonstrates that this proposed metric is better at capturing motion distortions.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is valid but the corrupted images as demonstrated in figure.4 do not seem to be low-level distorted (like blurred or locally rotated and shifted). That casts doubt on whether this proposed TRAJAN metric can indeed capture the distortions of real motions. \nSome other methods also include motion distortion such as EvalCrafter [1] , VBench [2], or SIFT-sora [3]. This paper should compare against those methods. \n\n[1] Liu, Yaofang, et al. \"Evalcrafter: Benchmarking and evaluating large video generation models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Huang, Ziqi, et al. \"Vbench: Comprehensive benchmark suite for video generative models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Sora Generates Videos with Stunning Geometrical Consistency", "relation_to_broader_scientific_literature": "This work can be impactful if they have a good motion quality assessment method as validated thoroughly.", "essential_references_not_discussed": "N/A/", "other_strengths_and_weaknesses": "See above. \nSome work like [3] works on pure geometry for estimating motion score, but this work still uses representation learning, which lacks explainability a little. Including more analysis on score interpretation will be helpful.", "other_comments_or_suggestions": "N/A/", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HHxgKviuMR", "reviewer": "Reviewer_P3xr", "summary": "This paper proposes TRAJAN, a novel motion-focused evaluation framework for assessing the quality of generated videos. Unlike traditional metrics like FVD that emphasize appearance, TRAJAN uses auto-encoded point tracks to assess temporally extended motion features. It supports distribution-level, video-pair, and per-video evaluation. Experiments show TRAJAN achieves higher sensitivity to temporal distortions and better alignment with human judgments across benchmarks (EvalCrafter, VideoPhy), outperforming existing motion and appearance-based metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and feedback. We were pleased to see that you found our approach “methodologically sound and well-motivated” and that “All claims are thoroughly validated by extensive empirical evaluation”.\n\nYou also mentioned how “The paper [...] provides strong conceptual justification for modeling motion independently from semantics via point track encoding.” and how  “This work fills a critical gap in the video generation literature by emphasizing motion quality assessment over frame-based metrics.” \n\n>“TRAJAN performance still misses edge cases where semantic context is crucial”, “Reliance on point tracks may limit applicability in occlusion-heavy … scenes”\n\nWe highlight and discuss this first point about semantic context in Figure 6, where __none__ of the metrics can capture the unexpected disappearance of the beer glass. Fully solving this challenge would require learning a fully accurate world model. This is an exciting future direction, but it is outside the scope of our work.\n\nOcclusion-heavy scenes are similarly challenging for all existing automated metrics which we compare to here (optical flow, action recognition, and masked auto-encoding will similarly suffer in occlusion-heavy scenes, although possibly less so than point tracking).\n\n>“Reliance on point tracks may limit applicability in … textureless scenes”\n\nWorking with textureless scenes is a general challenge for point tracking. In practice, many of the videos we evaluate are textureless. In such cases, the base point tracker does produce relatively poor point tracks which can drift and be inappropriately marked as occluded. However, because TRAJAN is trained to do point track *reconstruction*, it can *learn* this from data, and the reconstruction errors on textureless scenes are still, on average, reasonably small. __Some examples of this can be seen  [here](https://sites.google.com/view/trajan-videos-anonymous/home) (second section) where point tracks for textureless parts of the background are still well reconstructed (shown in blue)__. This is an advantage of using a trained autoencoder over the point tracks instead of calculating a metric directly on the point tracks themselves. Thank you for raising this point, we will add a discussion of this to the paper.", "claims_and_evidence": "The paper makes several key claims:\n\n1. TRAJAN provides better sensitivity to temporal distortions than prior metrics like FVD, VideoMAE, or I3D.\n2. TRAJAN embeddings correlate better with human ratings of motion consistency, realism, and interaction quality.\n\nAll claims are thoroughly validated by extensive empirical evaluation.", "methods_and_evaluation_criteria": "The TRAJAN model architecture is methodologically sound and well-motivated. It uses a Perceiver-style transformer to encode dense point tracks and reconstruct them from query points. \nComparisons with other motion-based (e.g., RAFT warp error, motion histograms) and appearance-based (e.g., VideoMAE, I3D) metrics are comprehensive and insightful.", "theoretical_claims": "The paper does not introduce formal theoretical analysis but provides strong conceptual justification for modeling motion independently from semantics via point track encoding.", "experimental_designs_or_analyses": "The experiments are rigorous and cover multiple evaluation modalities:\n\n1. Controlled distortions on UCF101 to measure temporal sensitivity.\n2. Video-to-video comparisons using WALT.\n3. Per-video quality assessments using human ratings from EvalCrafter and VideoPhy.", "relation_to_broader_scientific_literature": "This work fills a critical gap in the video generation literature by emphasizing motion quality assessment over frame-based metrics.", "essential_references_not_discussed": "The paper comprehensively reviews prior work. No critical omissions were identified.", "other_strengths_and_weaknesses": "Weaknesses:\n1. TRAJAN performance still misses edge cases where semantic context is crucial (e.g., object disappears unrealistically).\n2. Reliance on point tracks may limit applicability in occlusion-heavy or textureless scenes.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NuiOYr41l0", "reviewer": "Reviewer_ANes", "summary": "This paper proposes TRAJAN, an architecture designed to obtain dense high-level motion features using tracks predicted by the BootsTAPIR model. The authors demonstrate that these extracted features can effectively measure pairwise distances between videos in terms of motion, as well as evaluate the temporal distortions within individual videos. While the authors conduct extensive experiments to showcase the effectiveness of the learned TRAJAN features, several claims remain inadequately supported. Additionally, the paper's presentation requires substantial improvement.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and feedback. \n\n> there appears to be a contradiction between interpretations of Figure 5.\n\nTRAJAN is sensitive to differences in motion between videos (whether they be from a camera, or from an object). In Figure 5(a), the generated and reference video motions are the same for both the camera and the objects – the camera pans to the left in both cases, and none of the objects move independently of the camera, even though they have different appearances between the two videos. In Figure 5(b), the camera does not move, but the man moves in a distinctly different way. As a result, TRAJAN predicts a small distance in the first case (all motions are the same), and a large distance in the second case (the motion of the object is different). *There is therefore no contradiction in Figure 5.*\n\n> is the TRAJAN embedding more sensitive to camera motion or subject motion?\n\nWe separately analyze how well TRAJAN captures human evaluations of generated videos when they have either high camera motion (but low subject motion) or high subject motion (but low camera motion) for the EvalCrafter dataset. The results are provided [here](https://sites.google.com/view/trajan-videos-anonymous/home) in the last section.\n\n__In both settings, TRAJAN outperforms all baselines__. TRAJAN also captures human judgments better in the high camera motion vs. high object motion setting (motion consistency 0.53 vs. 0.30; appearance consistency 0.56 vs. 0.29; realism 0.45 vs. 0.24). However, we also found that human raters were less consistent within the high object motion setting, suggesting that this may be a generically harder task (for both humans and models). \n\n> Table 3 shows that the RAFT score outperforms TRAJAN-Len. in measuring object speed (...). This finding potentially diminishes the value of the proposed metric.\n\n> The authors should provide more insights on their motivation. Is the proposed TRAJAN feature intended to serve as a standard metric to evaluate motion magnitude or plausibility for video generation models? The paper would benefit from clearer articulation of the broader purpose and application of this work.\n\nThe proposed metric was not primarily intended for measuring object and camera speed. We included this result as an interesting addition to the __main arguments of the paper__: __TRAJAN is useful as a metric for measuring motion quality and motion differences in videos by simultaneously providing a compact latent space representation of motion and reconstruction errors for evaluating individual videos__. \n\nImportantly, in this work, and for the first time, we study metrics across distribution-level comparisons, video-video comparisons and evaluations of individual videos in terms of their motions. This involved adapting several prior methods, such as I3D, to the per-video setting as well as exploring new approaches, such as MooG, for evaluating videos. We also conducted a human study that includes fine-grained motion categories for evaluating metrics. All in all we identify numerous shortcomings of existing approaches (including RAFT) and propose a new metric based on point tracks (TRAJAN) that works well across all of the settings we tested.\n\n__To our knowledge, TRAJAN is the first metric that can operate across the distribution-, video-video comparison-, and single-video evaluation- levels, providing a compelling metric for motion in all cases.__\n\n> The authors conducted comprehensive experiments [...]. However, they did not provide results on using the TRAJAN metric to compare existing video generation models.\n\nOur human evaluations on VideoPhy and EvalCrafter are using video samples from existing video generation models. Based on your feedback we have recomputed the results from Table 1 comparing human judgements and metric scores separately for each model and then aggregating. __We found that TRAJAN correlates significantly better with human evaluations of which video model creates the most realistic videos: 0.76 (TRAJAN) 0.53 (RAFT) 0.43 (VideoMAE) 0.51 (I3D) 0.58 (MooG)__.\n\n> The paper is poorly written. Figure Quality. Citation Inconsistency. Capitalization. References. Formatting Issues. Metric Description.\n\nWe were surprised that you found this to be the case and listed “poor presentation” as one of the reasons for recommending weak reject. The examples you listed, such as citation forms, subsection capitalization, and spacing, can be easily addressed for a camera-ready paper. We will ensure the camera-ready version of our manuscript will have each of these addressed. Thank you for pointing these out.  \n\nWe provide videos for Figure 1, with additional experiments, [here](https://sites.google.com/view/trajan-videos-anonymous/home). We hope that by ensuring that the formatting concerns are addressed in a camera-ready version of the manuscript, the “comprehensive evaluations” and “insights to the community” that you highlight might lead you to reconsider your score.", "claims_and_evidence": "The claims in Section 5.2 are not all well-supported:\n- The authors claim that \"TRAJAN latent space is capturing something fundamentally different from pixel error\" (line 352). This claim is supported by the evidence presented in Table 2.\n- However, there appears to be a contradiction between interpretations of Figure 5. In Figure 5 (left), the authors claim \"since the camera motion is correct, the distance between their TRAJAN embeddings is small.\" Yet in Figure 5 (right), even when camera motion is identical in two videos (both static), the TRAJAN difference is large. This raises an important question: is the TRAJAN embedding more sensitive to camera motion or subject motion?\n\nFurthermore, Table 3 shows that the RAFT score outperforms TRAJAN-Len. in measuring object speed and performs comparably in evaluating camera speed. This finding potentially diminishes the value of the proposed metric.", "methods_and_evaluation_criteria": "The authors conducted comprehensive experiments demonstrating correlations between the TRAJAN metric and human evaluation. However, they did not provide results on using the TRAJAN metric to compare existing video generation models. Such analysis would provide valuable insights into the performance of current methods and help researchers better understand the proposed metric's effectiveness.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Please refer to the sections above.", "relation_to_broader_scientific_literature": "The paper builds upon prior work in evaluating video generation models, focusing on measuring temporal distortions in generated videos.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Please see comments above and below.", "other_comments_or_suggestions": "The paper is poorly written. Below is a non-exhaustive list of issues identified during review:\n\n- Figure Quality: Figure 1 is blurry and difficult to interpret, even after zooming in and carefully reading the captions. I suggest embedding videos into the PDF or including them in supplementary material for better clarity.\n- Citation Inconsistency: The citation format varies throughout the paper. For example, one venue is listed as \"ICLR,\" while another appears as \"ICLR 2024-Twelfth International Conference on Learning Representations.\" A consistent citation format should be maintained.\nInconsistent Capitalization: Some subsection titles have the first letter capitalized (e.g., Sec. 3.2), while others do not (e.g., Sec. 5.1-5.3). Consistency in formatting is necessary.\n- References: The SSv2 dataset is mentioned in the main paper but is only cited in the supplementary material. All references should be properly included in the main text.\n- Formatting Issues: The spacing between Section 3.2 and Section 3.2.1 titles is irregular.\n- Metric Descriptions: For TRAJAN-Len. and TRAJAN-Radii metrics, clear descriptions or direct links to their corresponding explanations should be provided rather than placing this information randomly in the Appendix.", "questions_for_authors": "The authors should provide more insights on their motivation. Is the proposed TRAJAN feature intended to serve as a standard metric to evaluate motion magnitude or plausibility for video generation models? The paper would benefit from clearer articulation of the broader purpose and application of this work.\n\nThis paper potentially offers insights to the community. However, based on the poor presentation and unsupported claims noted above, I recommend a weak reject.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents work on assessing video generation methods.  The paper had divergent reviews after the rebuttal phase.  On the positive side, the proposed TRAJAN model architecture is methodologically sound and well-motivated.  Comprehensive comparisons with other motion-based and appearance-based metrics are made, from a perspective of alignment to human judgements.

The main negatives raised by all the reviewers included details over the method, with its potential for sensitivity to camera motion or subject motion.  Importantly, there was a concern over lack of comparison of existing video generation models: does the proposed metric provide insights on video generation.

On balance, the paper does produce a state of the art method for evaluating video generation, which is demonstrated via alignment to human evaluations.  While there are concerns over the generality and the aspects of the videos the method is measuring, it does represent a step forward in machine learning for video generation.
Decision: Accept (poster)