VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Authors: Hila Chefer
Uriel Singer
Amit Zohar
Yuval Kirstain
Adam Polyak
Yaniv Taigman
Lior Wolf
Shelly Sheynin
Venue: ICML 2025 oral (2025)
Keywords: Video generation
Motion understanding
Diffusion models
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: Despite tremendous recent progress, generative video models still struggle to capture real-world motion, dynamics, and physics. We show that this limitation arises from the conventional pixel reconstruction objective, which biases models toward appearance fidelity at the expense of motion coherence.
To address this, we introduce **VideoJAM**, a novel framework that instills an effective motion prior to video generators, by encouraging the model to learn *a joint appearance-motion representation*. VideoJAM is composed of two complementary units. During training, we extend the objective to predict both the generated pixels and their corresponding motion from a single learned representation. 
During inference, we introduce **Inner-Guidance**, a mechanism that steers the generation toward coherent motion by leveraging the model's own evolving motion prediction as a dynamic guidance signal.
Notably, our framework can be applied to any video model with minimal adaptations, requiring no modifications to the training data or scaling of the model.
VideoJAM achieves state-of-the-art performance in motion coherence, surpassing highly competitive proprietary models while also enhancing the perceived visual quality of the generations.
These findings emphasize that appearance and motion can be complementary and, when effectively integrated, enhance both the visual quality and the coherence of video generation.

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Hila Chefer
Uriel Singer'
Amit Zohar!
Yuval Kirstain
Adam Polyak Yaniv Taigman
Lior Wolf
Shelly Sheynin
https: github. io io/videjjan pappee
paper, gi github.
/o/
"A ballet dancer twirls on the surface of a at sunset"
"A skatehoarder performs jumps"
"An acrobat executing handstand on narrow beam above water"
"Fingers press into shinmering slime ball"
Figure 1. Text-to-video samples generated by VideoJAM. We present VideoJAM, a framework that explicitly instills a strong motion prior to any video generation model. Our framework significantly enhances motion coherence
wide variety of motion types.
Abstract
aging the model's own evolving motion predic- tion as dynamic guidance signal. Notably, our framework can be applied to any video model with minimal adaptations, requiring no modifica- tions the training data or scaling of the model. VideoJAM achieves state-of-the-art performance in motion coherence, surpassing highly competi- tive proprietary models while also enhancing the perceived visual quality of the generations. These findings emphasize that appearance and motion can be complementary and, when effectively in- tegrated, enhance both the visual quality and the coherence of video generation.
Despite tremendous recent progress, generative video models still struggle to capture real-world motion, dynamics, and physics. We show that this limitation arises from the conventional pixel reconstruction objective, which biases models to- ward appearance fidelity at the expense motion coherence. To address this, we introduce Video- JAM, novel framework that instills effective motion prior to video generators, by encourag- ing the model to learn joint appearance-motion representation. VideoJAM is composed of two complementary units. During training, we extend the objective to predict both the generated pix- els and their corresponding motion from single learned representation. During inference, we in- troduce Inner-Guidance, mechanism that steers the generation toward coherent motion by lever-
1. Introduction
Recent advances video generation showcased remark- able progress in producing high-quality clips (Brooks et al., 2024; KlingAI, 2024; Polyak et al., 2024). Yet, despite con- tinuous improvements in the visual quality of the generated videos, these models often fail to accurately portray mo- tion, physics, and dynamic interactions (Kang et al., 2024; Brooks et al., 2024) (Fig. 2). When tasked with generat- ing challenging motions like gymnastic elements (c.g., a cartwheel in Fig. 2(b)). the generations often display severe
Work done while the first author intern GenAI, Meta. 'GenAl, Meta "Tel Aviv University. Correspondence to: Hila Chefer <hilach70@@ gomi..com>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
(a) Basic motion
(b) Complex Motion
Physics
(d) Rotational motion
"Jogger's Jest olong meck coast" "A mon performing cortwheel"
"A enjoying hula looping"
"A hand hoolding yellow spinner"
Figure 2. Motion incoherence in video generation. Examples of incoherent generations by DiT-30B (Peebles & Xie, 2023). The model struggles with (a) basic motion, e.g., jogging (stepping on the same leg repeatedly); (b) complex motion e-g., gymnasties; (c) physics, egg. object dynamics (the hoop passes through the woman); and (d) rotational motion, failing to replicate simple repetitive patterns.
deformations, such as the appearance of additional limbs. In other cases, the generations exhibit behavior that contradicts fundamental physics, such objects passing through other solid objects (e.g.. hula hoop passing through a woman in Fig. 2(c)). Another example is rotational motion, where models struggle to replicate a simple repetitive pattern of movement (c.g., a spinner in Fig. 2(d)). Interestingly, these issues are prominent even for basic motion types that are well-represented in the model's training data (e.g.. jogging in Fig. 2(a)), suggesting that data and scale may not be the sole factors responsible for temporal issues in video models.
prior, we introduce Inner-Guidance, a novel inference-time guidance mechanism. Unlike existing approaches (Ho & Salimans, 2022; Brooks et al., 2023), which depend on fixed external signals, Inner-Guidance leverages the model's own evolving motion prediction as a dynamic guidance signal. This setting requires addressing unique challenges: the motion signal is inherently dependent on the other con- ditions and the model weights, making the assumptions of prior works invalid and requiring a new formulation (Sec. 2, App. A). Our mechanism directly modifies the model's sampling distribution steer the generation toward the joint appearance appearance-motion distribution and away from the appearance-only prediction, allowing the model to refine its own outputs throughout the generation process.
In this work, we aim to provide insights into why video models struggle with temporal coherence and introduce a generic solution that achieves state-of-the-art motion gen- eration results. First, we find that the gap between pixel quality and motion modeling can be largely attributed to the common training objective. Through qualitative and quanti- tative experiments (see Sec. 3), we show that the pixel- pixel-based objective is nearty invariant to temporal perturbations in generation steps that are critical to determining motion.
Through extensive experiments, we demonstrate that ap- plying VideoJAM to pre-trained video models significantly enhances motion coherence across various model sizes and diverse motion types. Furthermore, VideoJAM establishes new statc-of-the-an in motion modeling, surpassing even highly competitive proprietary models. These advances are achieved without the need for any modifications to the data or model scaling. With an intuitive design requiring only the addition of two linear layers, VideoJAM both generic and easily adaptable to any video model. Interestingly, Video- JAM also improves the perceived quality of the generations, even though we do not explicitly target pixel quality. These findings underscore that appearance and motion are not mutually exclusive but rather inherently complementary.
Motivated by these insights, we propose VideoJAM, a novel framework that equips video models with an explicit motion prior by teaching them a Joint Oppearance-Motionn repre- sentation. This is achieved through two complementary modifications: during training, we amend the objective to predict motion in addition to appearance, and during infer- ence, we propose a guidance mechanism to leverage the learned motion prior for temporally coherent generations.
Specifically, during the VideoJAM training, we pair the videos with their corresponding motion representations and modify the network to predict both signals (appearance and motion). To accommodate this dual format, we only add two linear layers to the architecture (see Fig. 4). The first, located at the input to the model, combines the two signals into single representation. The second, at the model's 's output, extracts a motion prediction from the learned joint representation. The objective function is then modified to predict the joint appearance-moticn distribution, encourag- ing the model to rely on the added motion signal.
2. Related Work
Diffusion models (Ho et al., 2020) revolutionized visual con- tent generation. Beginning with image generation (Dhariwal & Nichol, 2021; Rombach et al., 2022; Ho et al., 2022a; Black Forest Labs, 2024; Dai et al., 2023; OpenAT, 2024), editing and personalization (Gal et al., 2022: Ruiz et al., 2023; Chefer et al., 2024b:a), and more recently video gen- eration. The first efforts to employ diffusion models for videos relied on model cascades (Ho et al., 2022b; Singer et al., 2023) or direct "inflation" of image models using tem- poral layers (Guo et al., 2023; BarTal et al., 2024; Wu et al., 2023). Other works focused on adding an auto-encoder for efficiency (Blattmann et al., 2023b; An et al., 2023;
At inference, our primary objective is video generation, with the predicted motion serving as an auxiliary signal. To guide the generation effectively incorporate the learned motion
2

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Wang et al., 2023), or conditioning the generation on im- ages (Blattmann et al., 2023a; Girdhar al., 2024; Hong et al., 2022). Recently, the UNet backbone was replaced by Transformer (Polyak et al., 2024; Brooks et al., 2024; Genmo, 2024; HaCohen et al., 2024), mostly following Diffusion Transformers (DiTs) (Peebles & Xie, 2023).
DiT
DIT permutee video
DNT +iiceeee
DiT+VideeJAM permuted video
8.3-
To control the generated content, Dhariwal Nichol (2021) introduced Classifier Guidance, where classifier gradients guide the generation toward a specific class. Ho & Salimans (2022) proposed Clussifier-Free Guidance (CFG), replac- ing classifiers with text. Similar to Inner-Guidance, CFG modifies the sampling distribution. However, CFG does not address noisy conditions or multiple conditions. Closest to our work, Liu et al. (2022), handle multiple conditions, Ciy.r., Cn. using a compositional score estimate,
6G2-
23
46
20 03 15 600 100
Denciiing step
Figure 3. Motivation Experiment. We compare the model's loss before and after randomly permuting the video frames, using a "vanilla" DiT (orange) and our fine-tuned model (blue). The origi- nal model is nearly invariant to temporal perturbations for f I 60.
prediction with the original video, the noise, or a combi- nation of the two (Ho et al., 2020; Lipman et al., 2023) (Sec. 4.1). We hypothesize that this formulation biases the model towards appearance-based features, such color and texture, as these dominate pixel-wise differences. Conse- quently, the model is less inclined to attend to temporal information, such as dynamics or physics, which contribute less to the objective. To demonstrate this claim, we per- form experiments to evaluate the sensitivity of the model to temporal incoherence. The following experiments are conducted on DiT-4B (Peebles & Xie, 2023) for efficiency.
po(x|cr...... ............. Gn) po(ar, c1,..,,))) po pelc........ Cn)
x po( = (x) Il (o(cs/z).
where 9 denotes the model weights and p is the sampling distribution. The above assumes that ci,..., On are inde- pendent of each other and 0, which does not hold in our case, since the motion is directly predicted by the model and thus inherently depends on 8 and the conditions. Similarly, Brooks et al. (2023) assume independence between the con- ditions and model weights 6, which is, again, incorrect in our setting. See App. A for further discussion.
We conduct an experiment where two variants of videos are noised and fed to the model first, the plain video without intervention, and second, the video after applying a random permutation to its frames. Assuming the model captures temporal information, we anticipate that the temporally in- coherent (perturbed) input will result in higher measured loss compared the temporally coherent input.
The gap between pixel quality and temporal coherence is a prominent issue (Ruan et al., 2024; Brooks et al., 2024; Liu et al., 2024b; Kang al., 2024). Previous works explored motion or physies-based based signals to improve video gener- ation. Some methods use them as input for guidance or editing (Geng et al., 2024; Ma et al., 2023; Liu et al., 2024a; Cong et al., 2023; Montanaro et al., 2024). Note that their objective differs from ours since we aim to teach models a temporal prior rather than taking it as input. Other methods suggest separating content and motion generation (Tulyakov al., 20;8; Ruan et al., 2024; Qing et al., 2023; Shen al., 2024; Jin et al., 2024b). In the context of image generation, Liu et al. (2023) improves the realism of human images by incorporating spatial priors such as surface-normal and depth maps in addition to a conditioning skeleton given as input. Finally, most similar to our approach, recent works use motion representations to improve coherence in image- vi-video generation (Shi et al., 2024; Wang et al., 2024), but these are limited to models conditioned on images.
Given random set of 35, 000 training videos, noise each video to random denoising step t E [0, 99). We then examine the difference in the loss measured before and after the permutation and aggregate the results per timestep. We consider two models- the "vanilla" DiT, which employs a pixel-based objective, and our fine-tuned VideoJAM model, which adds explicit motion objective (Sec. 4).
The results of this experiment are reported in Fig. 3. As can be observed, the original model appears to be nearly invari- ant to frame shuffling until step 60 of the generation. This implies that the model fails to distinguish between a valid video and a temporally incoherent one. In stark contrast, our model is extremely sensitive to these perturbations, as is indicated by the significant gap in the calculated loss.
In App. B we include qualitative experiment demonstrating that the steps t < 60 determine the coarse motion in the video. Both results suggest that the training objective is less sensitive to temporal incoherence, leading models to favor appearance over motion.
3. Motivation
During training, generative video models take a noised train- ing video and compute a loss by comparing the model's
3

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
(a) Training
(b) Inference (step
"An urban bullet doncer performing actobatic
W
W
Video prediction
graseese
Sess GRA
Iaaned VidealA
Pi Mialin Mock
Tosned ViteolAMe
Be (ahrnss
Metion prediction
d, (
dess Ry Nsseens
Tranel VidealAAe
Exinting
Added biyer
Exising knteet
Inner-Guidance
Figure 4. VideoJAM Framework. VideoJAM is constructed of two units; (a) Training. Given an input video 21 and its motion representation di, both signals are noised and embedded single, joint latent representation using a linear layer, W The diffusion model processes the input, and two linear projection layers predict both appearance and motion from the joint representation. (b) Inference. We propose Uoner-Guidane where the model's own noisy motion prediction used guide the video prediction each step.
4. VideoJAM
The prediction, u, is obtained using the DIT. First, the model "patchifies" xy into a sequence of X p video patches. This sequence is projected into the DiT's embedding space via a linear projection, Wsn E CR x Caa, where CTAE and ODT are the embedding dimensions of the TAE and DiT, respectively. The DiT then applies stacked attention layers to produce latent representation for the video, which is pro- jected back to the TAE's space to yield the final prediction using W out IR RCrxxCra iie..
Motivated by the insights from the previous section, pro- pose to teach the model joint representation encapsulating both appearance and motion. Our method consists of two complementary phases (see Fig. 4): (i) During training, we modify the objective predict the joint appearance motion distribution: This achieved by altering the architecture to support a dual input-output format, where the model pre- diets both the appearance and the motion of the video. (ii) At inference, we add Guidanee,, a novel formulation that employs the predicted motion to guide the generated video toward coherent motion.
u(Ie, y, y, = MMx Wm. V. 0) Wouts
(4)
where . denotes the attention blocks. For efficiency, we employ models that are pre-trained as described above and fine-tune them using VideoJAM as explained next.
4.1. Preliminaries
We conduct our experiments on the Diffusion Transformer (DiT) architecture, which has become the standard backbone for video generation (Brooks et al., 2024; Genmo, 2024). The model operates in the latent space of Temporal Auto- Encoder (TAE), which downsamples videos spatially and temporally for efficiency. We use Flow Matching (Lipman et al., 2023) to define the objective. During training, given video æ1, random noise E [0, 1]. æ is noised using to obtain an intermediate latent as follows,
4.2. Joint Appearance-Motion Representations
We begin by describing the motion representation employed by VideolAM. We opt to use optical flow since it is flex- ible, generic, and easily represented as an RGB video; thus, does not require training an additional TAE. Optical flow computes dense displacement field between pairs of frames. Given two frames I, 12 E IR H x3, the optical flow, d xx22 holds that d(u, v) is the displacement of the pixel (u, v) from I in Iz. To convert d into RGB image, we compute the angle and norm of each pixel,
(00, I), and a timestep
37 = tx + (1 t) zo.
(1)
The model is then optimized to predict the velocity, namely,
Vu² + 2²
m min
av H2 W2 aa arctan 2(v, u), (5)
dr: 5 To-
0t dt
(2)
Thus, the objective function employed for training becomes, C = Ex, (Fo. [l]u(re.3. t; 0) vx 11 (3)
where m is the normalized motion magnitude, = 0.15. and a is the motion direction (angle). Each angle is assigned a color and the pixel opacity is determined by m. Our nor- malization enables the model to capture motion magnitude, with larger movements corresponding to higher m values
where is an (optional) input condition, denotes the weights, and u( y, 1; 0) is the prediction by the model.

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
and reduced opacity. By using a coefficient = 0.15 in- stead of the full resolution (y (v=² + W2), we prevent subtler movements from becoming too opaque, ensuring they re- main distinguishable. The RGB optical flow is processed by the TAE to produce a noised representation, de (see Eq. 1).
Therefore, we propose modify the diffusion score func- tion to steer the prediction toward plausible motion.
In our setting, there are two conditioning signals: the prompt, and the noisy intermediate motion prediction, d. Notably, de inherently depends on the prompt and model weights, as it is generated by the model itself. Consequently, existing approaches that assume independence between con- ditions and model weights (c.g.. Brooks et al. (2023)), are not applicable in this setting (Sec. 2, App. A). To address this, we propose to directly modify the sampling distribution,
Next, we modify the model to predict the joint distribution of appearance and motion. We achieve this by altering the architecture dual input-output format, where the model takes both noised video, 2t, and noised flow, de, and predicts both signals. This requires modifying two linear projection matrices, Win and Wout (see Fig. 4(a)).
Po ([ dilly) x
First, we extend the input projection Win to take two inputs- the video and motion latents, Z2, dz. This done by adding CT zero-rows to obtain a duoo-ppojection matrix W E R22 XCan such that at initialization, the network is equivalent to the pre-trained DiT, and ignores the added motion signal. Second, we extend W oue with an additional output matrix to obtain What out E PRCorx2-Creee layer extracts the motion prediction from the joint latent representation. Together, W dual input-output format that processes and predicts both appearance and motion.
per ([ delly)po (ul [x,, del) t2 pa (de|ze, [zt, (2),))))) y)"²
where per (Jes, dilly) is the original sampling distribution, per (o [z,, d:]) estimates the likelihood of the prompt given the joint prediction, and per (de (er, y) estimates the likeli- hood of the noisy motion prediction. The latter is aimed at improving the model's ss motion coherence, as it maximizes the likelihood of the motion representation of the generated video. Using Bayes' Theorem, Eq. 7 is equivalent to,
The added
in and W alter the model to
(p [po[( del.,)
(Por ((Fr, del.
por (I delle)
pa ( de])
Po (Te, y) (por ([2,, dile,,
As shown in Fig. 4(a), our modifications maintain the origi- nal latent dimensions of the DiT. Essentially, this requires the model to learn a single unified latent representation, from which both signals are predicted using a linear projec- tion. Plugging the above into Eq. we get,
po (Po (Tare delly)
x po (L dellv)
por ([ del)
where we omit all occurrences of (v) since y is an ex- ternal constant input. Next, we can translate this to the corresponding score function by taking the log derivative,
u+ ([r,, y.),
M[[ w Wh y: v, 0) W outs
(1 + w1 + w2) log por ([ delly) -1 Vor log po ([z,, del) w2 V log po px:(2x (v).
where [+] denotes concatenation in the channel dimension, 0' denotes the extended model weights as specified above, and u = [u", -²]] denotes the dual output, where the first channels represent the appearance (video) prediction, while the last ones represent the motion (optical flow) prediction.
(8)
Following Ho & Salimans (2022), we jointly train the model to be conditional and unconditional on both auxiliary signals, y, d by randomly dropping out the text in 30% of the training steps, and the optical flow in 20% of the steps (setting d = 0). In particular, when the optical flow signal is dropped, the loss is derived with respect to the appearance only, resulting in the training scheme described in Sect. 4.1. Thus, the overall guidance formulation during inference becomes,
Finally, we extend the training objective to include an ex- plicit motion term, thus the objective from Eq. 3 becomes,
== (e di de [||u=* + ((za, de). y, 1; 8¹) - v= [[]]
(6) where v.- = [oy, of]] is calculated using Eq. 2. Note that while we only modify two linear layers, we jointly fine-tune all the weights in the network, to allow the model to learn the new target distribution.
u + ((rr, del, y, 1; (1 + wi + w2) u (( de)), v, 1; 0') -wi - -ut ((r, 1,) - -w2 u ([r,, 6,, 1: 0').
Unless stated otherwise, all experiments use W1 = 5, W22 3, where w = 5 is the base model's text guidance scale.
At inference, the model generates both the video and its mo- tion representation from noise. Note that we are mostly inter- ested in the video prediction, whereas the motion prediction guides the model toward temporally plausible outputs.
5. Experiments
We conduct qualitative and quantitative experiments to demonstrate the effectiveness of VidcoJAM. We benchmark our models against their base (pre-trained) versions, as well as leading proprictary and open-source video models, to highlight VideoJAM's enhanced motion coherence.
4.3. Inner-Guidance
As previously observed (Ho& & Salimans, 2022), condition- ing a diffusion model on an auxiliary signal does not guar- antee that the model will faithfully consider the condition.
5

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
"A close-up of runner's 's legs as they dash through rainstorm"
"A dog jumping over a wooden hurdle in slow motion
"A gymnast performs an aerial somersault at sunrise."
"A figure skater gliding gracefully across the ice"
"Á woman doing handstand on man' shoulders" Figure 5. Text-to-vine results by VideoJAM-30B, VideoIAM enables the generation of a wide variety of motion types, from basic motion (c.g., running) complex motion (c.g., acrobatics), and improved physics (e.g., jumping over hurdle).
"A panda breakdancing a ncon-lit urban alley, graffid backdrop"
Sora
Kling 1.5
DřT-30B
DIT-30B AM
"A woman doing pull-ups on green bars"
"Giraffe running through open field"
"A woman doing headstand on the beach"
"A roulette wheel a casino floor There's small white ball spinning rapidly"
Figure 6. Qualitative comparisons between VideolAM-30B and the leading baselines- Sora. Kling, and DIT-30B representative prompts from VideolAM-benchh The baselines struggle with basic motion, displaying "backward motion" (Sora, 2nd row) or unnatural motion (Kling, 2nd row). The generated content defies the hasic laws of physics e.g., people passing through objects (DiT, 1st row), objects that appear or evaporate (Sora, DIT, 4th row). For complex motion, the baselines display static motion or deformations (Sora, Kling, 1st, 3rd row). Conversely, cases, VidcoJAM produces temporally coherent videos that better adhere to the laws physics.
6

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Implementation Details We consider two variants of the DiT text-to-video model, DiT-4B and DiT-30B, demon- strate that motion coherence is common issue for both small and large models. All of our models are trained with a spatial resolution of 256 x 256 for efficiency. The models are trained to generate 128 frame videos at 24 frames per second, resulting in 5-second video generations. Both DiT models were pre-trained using the framework in Sec. 4.1 on an internal dataset of (1000 M) videos. We then fine-tune the models with VideoJAM using 3 million random samples from the model's original training set, which constitute less than 3% of the training videos. This allows our fine-tuning phase to be light and efficient. During this fine-tuning, we employ RAFT (Teed Deng, 2020) to obtain optical flow. For more implementation details, see App. C.
Table I. Comparison of VideoJAM-44 with prior work on VideoJAM--ennhh Human evaluation shows percentage votes favoring VideoJAM; automatic metrics use VBench.
Human Eval
Auto. Metrics
Method
Text Faith.
Quality
Motion
Appearance
Motion
CogVideo2B
84.3
94.5
96.1
68.3
90.0
CogVideoSB
62.5
74.7
68.8
71.9
90.1
PyramidFlow
76.6
83.6
82.8
73.1
89.6
DIT-4B + +VideJJAM
71.1
77.3
82.0
75.2
78.3
75.1
93.7
Table 2. Comparison of Video.JAM-30R with prior work on VideaJAM--enchh Human evaluation shows percentage of votes favoring VideoJAM; automatic metrics use VBench.
Benchmarks We two benchmarks for evaluation. First, we introduce VideoJAM-henchh, constructed specif- ically to test motion coherence. Second, we consider the Movie Gen (MGen) benchmark (Polyak et al., 2024) to show the robustness of our results.
Human Eval
Auto. Metrics
Method
Text Faith.
Quality
Motion
Appearance
Motion
Cog
73.4
71.9
85.9
71.9
90.1
RunWay Gen3
72.2
76.6
77.3
73,2
92.0
Mochi
56.1
65.6
74.2
69.9
89.7
VideoJAA--encce addresses limitations in existing bench- marks, including MGen, which do not fully evaluate real- world scenarios with challenging motion. For example, MGen's second-largest category, "unusual activity" (23.4% of MGen), contrasts with our objective of evaluating real- world ("usual") dynamics. The third largest category, "scenes" (19.9% MGen), focuses on nearly static scenes in nature, thus inherently prioritizes appearance over meaning- ful motion. Even for categories that overlap with ours such as "animals", the representative example given by MGen is "a curious cat peering out from cozy hiding spot".
Sora Kling 1.5
56.3
51.7
68.5
75.4
91.7
51.8
45.9
63,8
76,8
87.1
DiT-30B + +VideoJAM
71.9
74.2
72.7
72.4
88.1
73.4
92.4
5.1. Qualitative Experiments
Figures 5, 9 present results obtained using VideoJAM- 30B. The results demonstrate a wide variety of motion types that challenge existing models such gymnastics (e.g.. air splits), prompts that require physics understanding (c.g.. fingers pressed into slime, basketball landing in a net), etc.
To construct AiccoJAM-bench. we consider prompts from four categories of natural motion that challenge video gener- ators (see Fig. 2): basic motion, complex motion, rotational motion, and physics. We use a holdout set from our train- ing data-on which no model was trained-an and employ an LLM select the top 128 prompts that best fit at least one of the four categories and describe a single, specific, and clear motion. To avoid biasing the evaluation toward a specific prompt style, we task the LLM with modifying the prompts to be of varying lengths and detail levels. A full list of our prompts can be found in App. D.
Figure compares VideoJAM with the leading baselines, Sora and Kling, and the base model, DiT-30B, on prompts from VideoJAM-bench. The comparison highlights mo- tion issues in state-of-the-art models. Even simple motions, such as running giraffe (second row), show problems like "backward motion" (Sora) or unnatural movements (Kling, DiT-30B) Complex motions, like pull-ups or headstands, result in static videos (Sora, first and third rows; Kling, first row) or body deformations (Kling, third row). The baselines also exhibit physics violations, such as objects disappear ing appearing (Sora, DIT-30B, fourth row). In contrast, VideoJAM consistently produces coherent motion.
Baselines We consider a wide variety of state-of-the-art models, both proprictary and ooen-source. In the smaller cat- egory, we include Cog Video2B, CogVide (Hong et al., 2022), PyramidFlow (Jin et al., 2024a), and the base model DIT-4B. In the larger category, we evaluate leading open- source models (Mochi (Genmo, 2024), VogVido) and proprietary models with external APIs (Sora (Brooks et al., 2024), Kling 1.5 (KlingAI, 2024), RunWay Gen3 (Run- wayML, 2024)), along with the base model DiT-30B. The leading baselines were selected using the video leaderboard.
5.2. Quantitative Experiments
We evaluate appearance and motion quality, as well as prompt fidelity using both automatic metrics and human evaluations. In all our comparisons, each model runs once with the same random seed for all the benchmark prompts. For the automatic metrics, we use VBench (Huang et al., 2024), which assesses video generators across disentangled
7

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Table Ablation study. Ablations of the primary components of our framework on VideoJAM--4B using VideolAM-bench Human
(a) Motion observed in "zoom-out"
(b) Physies abject interactions
evaluation shows percentage votes favoring VideolAM.
Human Eval
Auto. Metrics
Ablation type
Text Faith. Quality Motion
Appearance Motion
"A skydiver deploying their parachute" "A soccer player lacking a ball"
w/o text guidance
68.0
62.5
63.3
74.5
93.3
w/o Inner- Inner-Guidane Guidance
68.9
64.4
66.2
75.3
93.1
Figure 7. Limitations. Our method is less effective for: (a) motion observed in "zoom-out" (the moving object covers a small part of the frame). (b) Complex physics of object interactions.
w/ optical flow
79.0
70.4
80.2
74.7
90.1
IP2P guidance
73.7
85.2
78.1
72.0
90.4
+VideoJAM-4B
75.1
93.7
also enhances other aspects of the generation.
axes. We aggregate the scores into two categories- appear- ance and motion, following the paper. The metrics evaluate the per-frame quality, aesthetics, subject consistency, the amount of generated motion, and motion coherence. More details on the metrics and their aggregation can be found in App. C.I.
5.3. Ablation Study
We ablate the primary design choices of our framework. First, ablate the use of text guidance and motion guid- ance in our inner guidance formulation (by setting w 0, w = 0 in Eq. 8, respectively). Next, we ablate the use of motion prediction during inference altogether, by dropping the optical flow at each inference step (d = 0). Finally, we ablate our guidance formulation by replacing it with the InstructPix2Pix (IP2P) guidance (Brooks et al., 2023) (see Sec. 2, App. A). Note that the results of the DiT models in Tabs. 1, 2 also function as ablations, as they ablate the use of VideoJAM during training and inference.
For the human evaluations, we follow the Two-alternative Forced Choice (2AFC) protocol, similar to Rombach et al. (2022); Blattmann et al. (2023a), where raters compare two videos (one from VideoJAM, one from a baseline) and select the best one based on quality, motion, and text alignment. Each comparison is rated by 5 unique users, providing at least 640 responses per baseline for each benchmark.
The results are reported in Tab. 3. All ablations cause signif- icant degradation in motion coherence, where the removal of motion guidance is more harmful than the removal of the text guidance, indicating that the motion guidance compo- nent indeed steers the model toward temporally coherent generations. Furthermore, dropping the optical flow pre- diction at inference is the most harmful, substantiating the benefits of the joint output structure to enforce plausible motion. The InstruetPix2Pix guidance comparison is fur- ther indication that our muidaned formulation is most suited our framework, as it gives the second lowest result in terms of motion.
The results of the comparison VideoJAM- -bncchhehchhh for the 4B, 30B models are presented in Tabs. 1, 2, respectively. Additionally, a full breakdown of the automatic metrics is presented in App. D. The results of the comparison on the Movie Gen benchmark are presented in App. E. In all cases, VideoJAM outperforms all baselines in all model sizes in terms of motion coherence, across both the automatic and human evaluations by a sizable margin (Tabs. 7).
Notably, VideoJAMM4B outperforms the CogVideoSB baseline, even though the latter is 25% larger. For the 30B variant, VideoJAM surpasses even proprietary state-of-the-art models such as Kling, Sora and Gen3 (63.8%, 68.5%, 77.3% preference in motion, respectively). These results are particularly impressive given that Video- JAM was trained at a significantly lower resolution (256) compared to the baselines (768 and higher) and fine-tuned on only 3 million samples. While this resolution disparity explains why proprietary models like Kling and Sora sur- pass ours in visual quality (Tab. 2), VideoJAM consistently demonstrates substantially better motion coherence.
Finally, note that human evaluators consistently prefer VideoJAM in terms of visual quality and text alignment over all the ablations, further establishing that VideoJAM benefits all aspects of video generation.
5.4. Limitations
While VideoJAM significantly improves temporal coher- ence, challenges remain (see Fig. 7). First, due to computa- tional constraints, we rely on both limited training resolution and RGB motion representation, which hinder the model's ability to capture motion in "zoomed-out" scenarios where moving objects occupy a small portion of the frame. In these cases, the relative motion magnitude is reduced, mak- ing the representation less informative (Eq. 5). For example, in Fig. 7(a), no parachute is deployed, and the motion ap- pears incoherent. Second, while motion and physics are
Most critically, VideoJAM significantly improves motion coherence in its base models, DiT-4B and DIT-30B, in direct apples-to-apples comparison. Human raters pre- ferred VideoJAM's 's motion in 82.0% of cases for DiT-4B and 72.7% for DiT-30B. Raters also favored VideoJAM in quality (77.3%, 74.2% in 4B. 30B) and text faithfulness (71.1%, 71. in 4B, 30B), indicating that our approach
8

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
intertwined, leading to improved physics, our motion rep- resentation lacks explicit physics encoding. This limits the model's s ability to handle complex physics of object interac- tions. For example, in Fig. 7(b), the player's foot does not touch the ball before it changes trajectory.
Black Forest Labs. FLUX, 2024. blackforestlabb.cc
URL https:/ /
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D.,
Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V. Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.1112) 2023a.
6. Conclusions
Video generation poses a unique challenge, requiring the modeling of both spatial interactions and temporal dynamics. Despite impressive advancements, video models continue to struggle with temporal coherence, even for basic mo- tions well-represented in training datasets (Fig. 2). In this work, we identify the training objective as a key factor that prioritizes appearance fidelity over motion coherence.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In CVPR, 2023b.
Brooks, T., Holynski, A., and Efros, A. A. InstructPix2Pix: Learning to follow image editing instructions. In CVPR, 2023.
To address this, we propose VideoJAM, framework that equips video models with an explicit motion prior. The core idea is intuitive and natural: a single latent representation captures both appearance and motion jointly. Using only two additional linear layers and no additional training data, VidcoJAM significantly improves motion coherence, achiev- ing state-of-the -h--hhe-rtttt art results even against powerful proprietary models. Our approach is generie, offering numerous op- portunities for future enhancement of video models with real-world priors such as complex physics, paving the way for holistic modeling of real-world interactions.
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y.,
Jing. L.,, Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A. Video generation models world simulators. 2024.
Chefer, H., Lang, O., Geva, M., Polosukhin, V., Shocher, A., michal Irani, Mosseri, I., and Wolf, L. The hid- den language of diffusion models. In The Twelfth In- ternational Conference on Learning Representations, 2024a. http://t:://// /tpp//eveeeeeeeeeeeeeeeeee net/ forum? i id=aWWpHnEJDw.
Impact Statement
Chefer, H., Zada, S., Paiss, R., Ephrat, A., Tov, O., Rubin- stein, M., Wolf, L., Dekel, T., Michaeli, T., and Mosseri, I. Still-moving: Customized video generation without customized video data, arXiv preprint arXiv:2407.08674, 2024b.
The primary goal of this work is to advance motion model- ing in video generation, empowering models to understand and represent the world more faithfully. As with any tech- nology in the content generation field, video generation carries the potential for misuse, a concern that is widely dis- cussed within the research community. However, our work does not introduce any specific risks that were not already present in previous advancements. We strongly believe in the importance of developing and applying tools to detect biases and mitigate malicious use cases, ensuring the safe and fair use of generative tools, including ours.
Cong. Y., Xu, M., Simon, C., Chen, S., Ren, J., Xie, Y., Perez-Rua, J.-M., Rosenhahn, B., Xiang, T., and He, S. Flatten: optical flow-guided attention for consistent text- to-video editing, arXiv preprint arXiv:2310.00
2023,
Dai, X., Hou, J., Ma, C.-Y,, Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang. X., Dubey, A., al. Emu Enhancing image generation models us- ing photogenic needles in a haystack. arXiv preprint arXiv: 2023.
References
An, J., Zhang, S., Yang, H., Gupta, S., Huang, J.-B., Luo, J., and Yin, X. Latent-Shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv:2304.08477, 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs on image synthesis. arXiv preprint arXiv:2105.0525 2021.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0166 2022.
BarTal, O., Chefer, H., Tov, O.,, Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Li, Y., Michaeli, T., Wang, O., Sun, D., Dekel, T., and Mosseri, I. Lumiere: A space- time diffusion model for video generation. arXiv preprint PrXiv22401112445 2024.
Geng, D.,, Herrmann, C., Hur, J., Cole, F., Zhang, S., Pfaff, T., Doersch, C., Aytar, Y., Rubinstein,
T.,

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
M., Sun, C., Wang, O., Owens, A., and Sun, D. Motion prompting: Controlling video generation with motion trajectories, 2024.
Jin, Y., Sun, Z., Xu, K., Chen, L.., Jiang, H., Huang, Q.. Song, C., Liu, Y., Zhang, D., Song, Y., Gai, K., and Mu, Y. Video-lavit: Unified video-language pre-training with decoupled visual- tokenization. In Inter- national Conference on Machine Learning, pp. 22185- 22209, 2024b.
Genmo. Mochi 1.
com
models, s, 2024.
Girdhar, R., Singh, M., Brown, A., Duval, Q., Azadi, S., Rambhatla, S. S., Shah, A., Yin. X., Parikh, D., and Misra, I. Emu video: Factorizing text-to- text-to-video generation by explicit image conditioning. In ECCV, 2024.
Kang. B., Yue, Y., Lu, R., Lin, Z., Zhao, Y., Wang, K., Huang, G., and Feng, J. How far is video generation from world model: A physical law perspective, 2024.
KlingAI. Kling AI, 2024. URL https: /ttp://kinniiiii com/.
Guo, Y., Yang, C.. Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. AnimateDiff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint MXXi::307.047255 2023.
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In ICLR, 2023.
HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., Levin, E., Shiran, G., Zabari, N., Gordon, O., Panet, P., Weissbuch, S., Kulikov, V., Bitterman, Y.. Melumian, Z., and Bibi, O. Ltx- video: Realtime video latent diffusion. arKiv preprint arXiv:2501.00103 2024.
Liu, N., Li, S.. Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with com- posable diffusion models. ArXiv, abs/2206.01714, 2022. URL htwptttt///// /appi/app.. semanticscholaa org/ Sts//CooppIDD:243772222
Liu, S., Ren, Z., Gupta, S., and Wang, S. Physgen: Rigid- body physics-grounded image-to-video generation. In European Conference on Computer Vision ECCV, 2024a.
Ho, J. and Salimans, Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12520 2022.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, 2020.
Liu, X., Ren, J., Siarohin, A., Skorokhodov, I., Li, Y., Lin, D., Liu, X., Liu, Z., and Tulyakov, S. Hyperhuman: Hyper-realistic human generation with latent structural diffusion. arXiv preprint arXiv:2310.08577 2023.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Grit- senko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., and Salimans, Imagen video: High definition video generation with diffusion models. arXiv preprint 2rXiv:2210.02303, 2022a.
Liu, Y., Zhang, K., Li. Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., He, L., and Sun, L. Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024b.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Grit- senko, A., Kingma, D. P., Poole, B., Norouzi, M.. Fleet, D. J., and Salimans, T. Imagen video: High definition video generation with diffusion models. arXiv preprint arX1v:2210.023033, 2022b.
Ma, W.-D. K., Lewis, J. P., and Kleijn, W. B. Trailblazer: Trajectory control for diffusion-based video generation, 2023.
Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.-Y. I., and Ermon, S. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
Hong, W., Ding, M., Zheng, W., Liu, X., and Tang, Cogvideo: Large-scale pretraining for text-to-video gener- ation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
Montanaro, A., Savant Aira, L., Aiello, E., Valsesia, D., and Magli, E. Motioncraft: Physics based zero-shot video generation. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Systems, vol- ume 37, pp. 131555-123181 2024.
Huang, Z., Hc, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y., Chen, X., Wang, L., Lin, D., Qiao, Y., and Liu, Z. VBench: Com- prehensive benchmark suite for video generative models. In CVPR, 2024.
Curran Associates, Inc.,
OpenAl. Dall-E 3, 2024. URL URLhttts://openae com/ index/ dall-e-3/.
Jin, Y., Sun, Z., Li, N., Xu, K., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y., Mu, Y., and Lin, Z. Pyramidal flow matching for efficient video generative modeling, 2024a.
Peebles, W. and Xie. S. Scalable diffusion models with transformers. In ICCV, 2023.
10

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Polyak, A., Zohar, A., Brown, A., Tjandra, A.,Sia A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., Yan, D., Choudhary, D., Wang. D., Sethi, G., Pang, G., Ma, H., Misra, I., Hou, J., Wang, J., Jagadeesh, K., Li, K., Zhang. I.., Singh, M., Williamson, M., Le, M., Yu, M., Singh, M. K., Zhang, P., Vajda, P., Duval, Q. Girdhar, R. Sumbaly, R., Rambhatla, S. S., Tsai, S., Azadi, S., Datta, S., Chen. S., Bell, S., Ramaswamy, S., Sheynin, S., Bhattacharya, S., Motwani, S., Xu, T., Li, T., Hou, T., Hsu, W.N., Yin, X., Dai, X., Taigman, Y., Luo, Y., Liu, Y-C.., Wu, Y.-C., Zhao, Y. Kirstain, Y., He, 7., He, Z., Pumarola, A., Thabet, A., Sanakoyeu, A., Mallya, A., Guo, B., Araya, B., Kerr, B., Wood, C., Liu, C., Peng. C.,, Vengertsev, D., Schonfeld, E.,, Blanchard, E., Jue F., Nord, F., Liang, J., Hoffmman, J., Kohler, J., Fire, K., Sivakumar, K., Chen, L.. Yu, L., Gao, L., Georgopoulos, M.. Moritz, R., Sampson, S. K., Li, S., Parmeggiani, S., Fine, S., Fowler, T., Petrovic, V., and Du, Y. Movie gen: A cast of media foundation models, 2024.
image-to-video generation with explicit motion modeling, 2024.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y. Make-A-Video, Text-to-vidoo generation without text-video data. In ICLR, 2023.
Tay, Y., Dehghani, M., Tran, V. Q.. Garcia, X., Wei, J., Wang, X., Chung. H. W., Shakeri, S., Bahri, D., Schuster, T., al. UL2: Unifying language learning paradigms. arXiv preprint S.K.v::2255.011555 2022.
Teed, Z. and Deng, J. Raft: Recurrent all-pairs field transforms for optical flow. In European Confer- ence on Computer Vision, 2020. URL https:
wen.sssmaanicccoole.... //app.ssmatllcccool.....oooooppe 214667893.
Tulyakov, S., Liu, M.-Y., Yang. X., and Kautz, J. MoCo- GAN: Decomposing motion and content for video gen- eration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1526-1535, 2018.
Qing, Z., Zhang, S., Wang, J., Wang. X., Wei, Y., Zhang, Y., Gao, C., and Sang, Hierarchical spatio-temporal decoupling for text-to-video generation, 2023.
Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., and Zhang, S. ModelScope text-to-vin video technical report. arXiv preprint arXiv:2308. 06571, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Wang, S., Azadi, S., Girdhar, R., Rambhatla, S., Sun, C., and Yin, X. Motif: Making text count in image animation with motion focal loss, 2024.
Ruan, P., Wang, P., Saxena, D., Cao, J., and Shi, Y. Enhanc- ing motion in text-to-video generation with decomposed encoding and conditioning, 2024.
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-A- One-shot tuming of image diffusion models for rexi-to-video generation. In ICCV, 2023.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
Video:
Xu, H., Xie, S., Tan, X. E., Huang, P.-Y,, Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feicht- enhofer, C. Demystifying CLIP data. arXiv preprint arXiv:2309. 16671, 2023.
RunwayML. Gen-3 Alpha, 2024.
URL
https://runwayntt com/ wtt.:///e///eereeeeeeeeeeeee research/ -eneoduuiinngge----aaa
Xue, L.,, Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C. ByT5: Towards a token-free free future with pre-trained byte-to-byte models. TACL, 2022.
Shen, C., Gan, Y., Chen, C., Zhu, X., Cheng, L., Gao, T., and Wang. J. Decouple content and motion for conditional image-to-vides generation. In Proceedings of the Eighth- AAAI Conference on Artificial In- telligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial In- telligence, AAAT244AAATT2/TAATT"" 2024. ISBN 978-1-57735--88 28277. URL https: 28277.
AAAI Press, doi: 20.1669/aai.vv888 1609 609/6099//0 aaai..
Shi, X., Huang, Z., Wang, F.-Y, Bian, W., Li, D., Zhang, Y., Zhang, M., Cheung, K. C., See, S., Qin, H., Dai, J., and Li, Motion-i2y: Consistent and controllable

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
A. Compositional Guidance vs. Inner-Guidance
Liu et al. (2022) proposed Composable Diffusion Models where a diffusion model can be conditioned on several signals C Cn. The model's conditional sampling distribution is, therefore,
Po(I, Cn) & (a, s cn) (a II po(e)) po (c(.:....,Ca)
po (a( pa( . Cn)
(9)
where 0 represents the model weights, and is the sampling distribution. Importantly, this formulation assumes that Ciy...,Cy Cn are independent of each other and the weights of the model 6, allowing to drop the denominator pa pa(ci..... C.). Notice that this assumption does not hold in our setting, where the motion condition de is noisy and strictly dependent on the neural network, as one of its outputs, as well as the text conditioning, as serves as another input to the model.
Inspired by Liu et al. (2022), InstructPix2Pix (IP2P) (Brooks et al., 2023) used similar compositional formulation to extend Classifier-Free Guidance (Ho& & Salimans, 2022) to two conditioning signals. Formally, given two conditions C1 C2.
po(x)c1,c2) po(a, (2) po(c1 c2, (c2)))(0p)))) C2)
(10)
po(c1,02) C2)
taking the log derivative this gives us,
V log (o) v (o
+ V (og2)( logpe(<)) - log 02):
(11)
next, the IP2P formulation assumes (similar to Liu al. (2022)) that we can omit the term pe(ci, 02) since is independent of 8, which is again incorrect in case.
For completeness, our ablations in Sec. 5 compare our Inner- Cine--Gidannce formulation with that of IP2P, and find that this theoretical gap causes significant degradation in the performance. The direct interpretation of Eq. 11 to VideoJAM employed in our experiments is as follows,
ū (1T,, dol. v,t;0' = ut ([ 0)),0,;;'))
(;), 0') u+ 0)), 0*) +
W2 (u ([x,,
(.),
where the notations follow Sec. 4.3, and we employ the same guidance scales as do for Inner-Guidance, i.e. wi 3, w2 5. Note that the notations for W1, W2 reversed with respect to Eq. since IP2P condition the visual signal first and the textual signal second and order matters for IP2P, while our Inner-Guidance formulation is order invariant.
B. Motivation Experiments
Appearance
Motion
Appearane
Motion
To exemplify that steps S 60 of the generation are in- deed meaningful to determine the motion, we conduct an SDEdit (Meng al., 2022) experiment, in which we noise videos different timesteps (20, 60, 80), and continue the generation given the noised videos. In Fig. 8, we show representative appearance frame and two motion frames for each video, using RAFT (Teed & Deng, 2020) to es- timate optical flow. We observe that the coarse motion and structure of the generated videos are determined be- tween steps 20 and 60, since the generation from step 20 changes the entire video while starting from step 60 main- tains the coarse motion and structure of the input video, suggesting that they are already determined by the input noisy video. Note that the appearance may still change between steps 60 and 80 (right), whereas from step 80, both appearance and motion seem to be determined.
Imnut video
Step 80
Step Step
Figure 8. Qualitative motivation. We noise input videos to different timesteps (20, 60, 80) and continue the generation. By step 60, the vidco's coarse motion and structure are mostly determined.
12

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
"A woman hwirks hulu hoop in park during sunset"
"A gūroffe stepping gingerly along tightrope above a city plaza
"A balancing spinning ball on mountaintop"
"A dancer leaps across stage, golden confetti fluttering around them
"A professional chef chops vegetables wooden cutting board."
"A woman transitions grucefully an aerial hoop golden hour"
basketball spins, bouncing the backboard and falling into the hoop"
bear pedaling unicycle through circus tent
"A acrobat holding handstand
"On rainy rooftop, pair of hip-hop duncers lock and pop"
"An otter riding rollerblades two legy bustling city park
woman doing push-up exercise"
"A dolphin leaps out of the sea, flock of birds flying in the background"
"A skier performing spin midair"
Figure 9, Additional text-to-video results using VideoJAM-30B.
13

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
C. Implementation Details
VideoJAM-4B was fine-tuned using 32 A GPUs with a batch size of 32 for 50, 000 iterations on a spatial resolution of 256 x 256. It has a latent dimension of 3072 and 32 attention blocks (same as the base model). VideoJAM-30B was fine-tuned using 256 A100 GPUs with a batch size of 256 for 35, 000 iterations on a spatial resolution of 256 256. It has latent dimension of 6144 and 48 attention blocks (same as the base model). Each attention block is constructed of self-attention layer that performs spatiotemporal attention between all the video tokens, and a cross-altention layer that integrates the text. Both models were trained with fixed learning rate of 5e 35-66 6, using the Flow Matching paradigm (Lipman et al., 2023) (see Sec. 4.1).
During inference, we perform 100 denoising steps with a linear quadratic t-schedule using a text guidance scale of w1 5 and a motion guidance scale of W22 (see Eq. 8), other than the ablations that test these components. Additionally, we only employ the motion guidance for the first half of the generation steps (50 steps) following the conclusions from our motivational experiments (Sec. 3), as these the steps that determine the coarse motion in the video, and display less sensitivity to temporal incoherence before applying VideoJAM. In practice, InnernGu is performed similarly to Classífier-Free Guidance (Ho & Salimans, 2022), where all results are generated in batch ut (102, di)), y, 1; 0"), u+ ([vr, di), 0, 1; 0'), u- (1x;, 0), y, 4;) and the final prediction is calculated following Eq. 8. The models are trained to generate 128 frame videos at 24 frames per second, resulting in 5-second video generations.
The models operate in the latent space of a TAE, as specified in Sec. 4.1. The TAE structure follows that of Polyak et al. (2024), with temporal compression rate of x8 and spatial compression rate of 8 8. The Transformer patch size is 1 x 2 2. The text prompt conditioning is processed by three different text encoders: UL2 (Tay et al., 2022), ByT5 (Xue et al., 2022), and MetaCLIP (Xu et al., 2023).
Both DiT models were pre-trained using the framework in Sec. 4.1 on a dataset of O(100 M) videos. We then fine-tune the models using VideoJAM on under million random samples from the model's original training set, which constitute less than 3% of the training videos. This allows our fine-tuning phase to be light and efficient. During this fine-tuning, we employ RAFT (Teed & Deng, 2020) to obtain optical flow per training video.
Since each of the baselines generates videos in different resolutions, we resize the baseline results to a 256 resolution to facilitate fair and unbiased comparison. No therry-pickingg is involved in the evaluation of any of the models, and the first result obtained by each model is taken. All baselines produce the same length of videos (5 seconds), therefore we only resize the videos spatially. For the qualitative results in the website, we train an additional super-resolution model to spatially upsample the 256x x 256 videos to 512 x 512 videos. The training regime follows that of VideoJAM-30B. Note that all our experiments (besides the visualizations on the website) are in the lower 256 resolution due to resource limitations.
C.1. VBench Metrics
We employ all metrics supported by VBench on both VideoJAM- and the Movie Gen benchmark. Inspired by the protocol in the VBench paper, we split the metrics into a motion category and an appearance category. For the appearance category, we include the aesthetic quality and image quality metries, which assess the per-frame quality of the generated videos, well as subject consistency and background consistency, which assess the model's ability to maintain consistent appearance. For motion comprehension, we include the motion smoothness score, which aims to assess the realism of the motion, and the dynamic degree score which estimates the amount of motion in the generated vidcos. In other words, the motion score measures the model's ability to generate meaningful motion (i.e., non-static videos) that is also coherent and plausible.
All scores are normalized and a weighted score is calculated according to the weights suggested in the VBench paper. The full results of all VBench metrics for each benchmark are reported in App. D, E.
D. VideoJAM-benchh: Automatic Metrics Breakdown and Prompts
In the following, we provide a breakdown of the automatic metrics calculated on our motion benchmark using VBench (Huang et al., 2024) for the 4B model (Tab. 4) and the 30B model (Tab. 5). As mentioned in App. C.I, the motion metrics measure the amount of motion in the video and the coherence of the motion. the smaller model category, Cog Video2B scores the highest dynamic degree and the lowest motion smoothness. This indicates that while there is abundant motion in the generated videos, it is incoherent. The DYT-4B base model obtains the best smoothness score, and
14

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
the worst dynamic degree, indicating that it produces videos with very subtle movements. As be observed, VideoJAM strikes the best balance, where plenty of motion is generated while maintaining strong coherence.
For the larger DiT-30B model, we observe, again, that there is trade-off between the dynamic degree and the motion smoothness, where Cog CogVideo5B produces the most motion, yet it is incoherent. Among the competitive proprietary baselines, notice that Runway Gen 3 obtains very high dynamic degree, yet it has the lowest motion smoothness among all the proprietary baselines (Runway Gen 3, Sora, Kling 1.5). In Fig. 5, we show comparisons to Sora and Kling since these are the most competitive with VideoJAM according the human evaluation, which is generally considered to be a more reliable evaluation form (BarTal al., 2024; Polyak et al., 2024; Wang al., 2024). However, in the website, we include comparison to Runway Gen 3 in addition to Sora and Kling for completeness. Furthermore, Kling shows the best motion smoothness, with the lowest dynamic degree. Observe that VideoJAM, again, strikes the best balance between motion coherence and the amount of generated motion. Additionally, it outperforms the base model (DiT-301) across all motion metrics, and nearly all appearance metrics, indicating that our method improves all aspects of the generation.
For completeness, Tab. 6 presents the VBench metrics not supported for external benchmarks, calculated on the VBench prompts. The reported results demonstrate that VideoJAM improves almost all aspects of generation, indicating once again that our framework does not sacrifice video quality for improved temporal coherence.
A full list of the prompts considered in our motion benchmark is provided in App. F.
Table Breakdown of the automatic metries from VBench comparing our model and previous work on VidealAM-bench. Our method strikes the best balance between the dynamic degree (higher implies more motion) and the motion smoothness (higher implies smooth motion).
Appearance Metrics
Motion Metrics
Aesthetic Quality
Image Quality
Subject Consistency
Background Consistency
Motion Smoothness
Dynamic Degree
Method
Video2
46.9
48.9
87.8
93.9
97.1
88.6
VodVid
51.1
52.9
91.3
95.3
97.3
87.5
DiT-4B
51.8
61.4
93.0
96.7
99.3
38.3
+ +VideoJAM-4B
51.6
61.1
93.5
96.7
98.8
87.5
Table Breakdown of the automatic metrics from VBench comparing our 30B model and previous work on VideolAM--enchh. Our method strikes the best balance between the dynamic degree (higher implies motion) and the motion smoothness (higher implies smooth motion).
Appearance Metrics
Motion Metrics
Aesthetic Quality
Image Quality
Subject Consistency
Background Consistency
Motion Smoothness
Dynamic Degree
Method
VideeS
51.1
52.9
91.3
95.3
97.3
87.5
RunWay Gen3
55.1
55.1
90.7
95.2
98.4
84.4
Mochi
49.5
48.8
89.7
95.2
98.4
78.1
Sora
56.8
57.7
93.0
96.4
98.7
82.0
Kling 1.5
58.5
60.4
93,9
96.5
99.2
64.8
DiT-30B
49.2
56.8
91.3
95.5
98.8
71.1
+VideaJAM-30B
51.2
55.9
93.0
96.1
99.0
82.3
E. Movie Gen Benchmark
We employ the prompts from the official benchmark labeled as containing "high" motion since our primary objective is to estimate motion coherence. Additionally, since the Movie Gen benchmark is significantly larger than VideoJAM-benchh and mostly contains less relevant prompts (Sec. 5), we consider the baselines that provide open-source code and can run automatically. Importantly, note that the benchmark as well, allowing us to assess the direct impact of VideoJAM on large video generation model.
comparison to the pre-trained model. DiT-30B is presented for this
15

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
Table Breakdown of remaining automatic metrics from VBench comparing our model with its model, DiT-30B, on VBench prompts. Our method improves almost all metrics, indicating that VideolAM does not compromise appearance for motion enhancement.
Temporal Flickering
Object Class
Multiple Objects
Human Motion
Spatial Relationship
Appearance Style
Temporal Style
Overall Consistency
Method
Color
Scene
DiT-30B
99.7
89.1
63.0
97.0
81.2
72.0
52.1
22.1
24.2
27.5
+VideoJAM-30B
99.7
90.7
70.0
99.0
91.1
73.3
49.7
23.3
24.4
27.6
The results are reported in Tab. 7, with breakdown of the automatic metrics in Tab. 8. Similarly to the results on our motion benchmark, VideoJAM strikes the best balance between the amount of motion and the coherence of the generated motion. While VogVid consistently produces the most motion, it is also consistently the least coherent baseline. Mochi, on the other hand, suffers from the complementary problem where less motion is generated. Notably, VideoJAM outperforms all baselines, by significant margin across all metrics, both human-basee and automatic (other than the dynamic degree, where Cog Video5B scores the highest, as mentioned). Importantly, we observe a consistent improvement over the base model used by VideoJAM, DiT-30B in both the appearance and motion metrics across all evaluations, which further substantiates our method's ability to improve all aspects of video generation.
Table Comparison of V-330JAM-30B with prior work on the Movie Gen benchmark. Human evaluation shows percentage of favoring VideoJAM; automatic metries use VBench.
Human Eval
Auto. Metries
Method
Text Faith.
Quality
Motion
Appearance
Motion
CogVideo5B
61.4
77.0
78.7
70.8
88.8
Mochi
53.5
59.4
69.1
70.4
85.1
DiT-30B
60.3
64.6
66.1
70.5
87.3
+VideoJAMM300
73.7
90.8
Table 8. Breakdown of the automatic metries from VBench comparing our 30B model and previous work on the Movie Gen benchmark. Our method strikes the best balance between the dynamic degree (higher implies more motion) and the motion smoothness (higher implics smooth motion).
Appearance Metrics
Motion Metrics
Aesthetic Quality
Image Quality
Subject Consistency
Background Consistency
Motion
Dynamic
Method
Smoothness
Degree
Vid
50.9
51.9
89.5
94.7
97.5
81.6
Mochi
50.4
50.1
89.0
95.4
98.9
60.7
DIT-30B
48.7
50.6
90.8
95.3
98.9
67.8
+VideoJAMM30B
51.5
56.4
93.3
96.2
99.1
76.9
F. VideoJJAM-eench Prompts
Below, we present the full set of 128 prompts used our motion benchmark, VideolAM-benchh. The benchmark designed to diverse, encompassing simple motions (e.g., walking), complex human movements (e.g., gymnastics), rotational motions (e.g.. spinning balls), and physics-based actions (e.g.. a woman hula hooping). To ensure elarity, the prompts were refined using I.LM focus on specific motion types, enabling a precise evaluation of the model's 's ability generate coherent movement. Additionally, the prompts vary in detail and include camera instructions to test the model's performance across a wide range of scenarios.
1. AA woman performing an intricate dance on stage, illuminated by a single spotlight in the first frame. She is dressed in a long black dress and a wide-brinmed hat, with her arms raised above her head. The woman dance Argentine
16

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
flamenco dance."
2. woman doing a headstand on a beach."
3. woman engaging in a challenging workout routine, performing pull-ups on green bars."
4. "Two ibexes navigating a rocky hillside. They are walking down a steep slope covered in small rocks and dirt. In the background, there are more rocks and some greenery visible through an opening the rocks."
5. "A close-up of a runner's legs as they sprint through a crowded city street, dodging pedestrians and street vendors, with the sounds of the city all around."
6. Athletic man doing gymmastics elements on horizontal bar in city park. Male sportsmen perform strength exercises outdoors."
7. "A small dog playing with a red ball on floor..
8. A woman engaging in lively trampoline workout. The woman jumps and exercises the trampoline. The background is room with white walls and white ceiling, and there are two large windows on the left side of the wall, and mirror on the right side reflecting the womans image."
9. "A man performing handstand on wooden deck overlooking green lake surrounded by trees."
10. " Young adult female performs air gymmastic show circus arena, holding ring in hand, making twine exercise, spin around"
11. "A woman enjoying the fun of hula hooping."
12. "A man juggling with three red balls in a city street."
13. "A white kitten playing with ball."
14. "A slow-motion shot captures a runner's legs as they dash through a busy intersection, dodging cars and pedestrians, the city life bustling around them."
15. "A young girl playing basketball in a red brick wall background. The girl, fair skin and long blonde hair, is wearing green jacket and has her left arm up to throw the ball. In the mid-frame, the girl is still playing basketball, with her right hand holding the ball in front of her face. The ground is dark gray cement with some patches of grass growing through it. As the video progresses, the girl is seen playing near some grassy areas on the ground."
16. "A basketball game in progress, with two players reaching up grab the ball as spills out of the net. The player on the left has his hand outstretched, while the player the right has both hands raised high. The ball is just above their fingertips, indicating that they are both trying to grab it simultaneously. The background of the image is blurred, but it appears to be gymnasium or sports arena, with fluorescent lights illuminating the scene. As the video progresses, the players continue to jump and stretch to gain possession of the ball, their movements becoming more urgent and intense. The ball flies back and forth between them, with neither player able to secure it. In the final frame, the ball still in mid-air, the players hands reaching up to grab as the video ends."
17. "A group of basketballs floating in mid-air in slow motion, with a larger ball on the left and two smaller balls on either side in the initial frame. Overall, the video captures the dynamic and energetic movement of basketballs as they float and bounce through space."
18. "A dog playing with an orange ball with blue stripes. The dog picks up the ball and holds its mouth, conveying a sense of playfulness and energy. Throughout the video, the dog is seen playing with the ball, capturing the joy and excitement of the moment."
19. woman doing acrobatic exercises on a pole in the gym."
17

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
20. "A young man performing cartwheel a gray surface. He is dressed in orange pants, a black I-shiri, and while sneakers. As he executes the cartwheel, his right arm extended upward, and his left arm is bent at the elbow, reaching down to the ground. His right leg is extended behind him, while his left leg is bent at the knee, pointing towards the camera. The background is a featureless gray wall. The mans energy and focus are evident as he completes the cartwheel, showcasing his athleticism and coordination."
21. "A golden retriever playing fetch on a grassy field. The dog is running with a frishee its mouth, its fur waving the wind."
22. "A brightly colored ball spins rapidly a flat surface, its patterns blurring it twirls in place."
23. "A basketball spins on player's fingertip, maintaining balance while gradually slowing down."
24. "A person jogs along a forest trail dawn, their feet kicking up dirt with every stride, the filtering through the Irees casting long shadows on the path."
25. "A child jumps up and down in place, their feet leaving the ground briefty before landing again."
26. "A person lifts one knee high in a marching motion, then places their foot back down and repeats with the other leg."
27. " Professional cyclist training indoors on a stationary bike trainer."
28. "Young Adult Male Doing Handsland on the beach."
29. young woman practicing boxing in a gym."
30. "A man jumping in a pool."
31. "A man doing push-ups ledge overlooking body of water. The man appears to be doing a push-up, with his head down."
32. "A man enjoying a leisurely bike ride along a road next to a body of water during a sunset. As he pedals, he looks down at his front wheel, seemingly focused on his ride. The background features a large body of water, with a gray wall along the left side of the road in the mid-frame caption."
33. "close up shot of the feet of a woman exercising cardio fitness machine in a fitness club. As the video progresses, the legs continue to pedal the bike in smooth, consistent motion."
34. "A woman engaging in an intense workout on stationary bike while monitoring her progress on a screen." "
35. "A woman running along river with a city skyline in the background."
36. "A skier walking up snowy hill with their skis on their buck and ski poles in hand."
37. "A woman rumning through a grassy area, wearing a black tank top, gray and white leggings. and white sneakers. She is initially running on a dirt path, surrounded by trees with green leaves. As she continues to run, the scenery changes to a park, and her leggings change to a blue and white pattern. She is still running on a dirt path, surrounded by trees and green grass. The video captures her journey as she runs through the grassy area, enjoying the outdoors and the beauty of nature."
38. "A young girl coloring at her desk."
39. "A close-up of runner's legs as they dash through a rainstorm, their shoes splashing through puddles as they push forward with determination."
40. "Tracking camera shot. A kangaroo hops swiftly across open grassy plain."
41."A "A close-up view of a spiral object with a glowing center. The object appears to be made of metal and has a shiny, reflective surface. This light creates a series of concentric circles around the objects circumference, which are visible due the reflection of the light off the metal surface."
18

---

VidcoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
42. "A roulette wheel in a dimly lit room or casino floor. In the center of the wheel, there's 's small white ball that appears be spinning rapidly as it moves around the track. The ball spins around the wheel, and the wheel rotates counterclockwisee"
43. "A close-up of jogger's feet as they run along a rocky coastal path, their shoes gripping the uneven surface, with the ocean waves crashing below."
44. "A person's s hands they shape and mold clay a pottery wheel. The hands are covered in brown clay and are visible from the elbows down, with the forearms resting on top of a large yellow pottery wheel."
45. "A conveyor belt pouring out a large amount of small, brown objects into pile on the ground. The objects being poured are falling from the conveyor belt a steady stream, forming large pile on the ground below. In the background, the sky is bright blue and cloudless, providing stark contrast to the darker colors of the conveyor belt and the pile of objects."
46. "A 3d rendering of coins and small objects floating against a black background. The coins are gold, silver, bronze, and copper, with various denominations and sizes. Some have a shiny finish, while others are malte or tarnished. The scene is chaotic and dynamic, with the objects seemingly flying around all directions. As the video progresses, the coins and objects tumble and spin, creating sense of movement and energy. By the end, the screen is filled with white objects of various shapes and sizes, suggesting that something exciting is happening."
47. "A puppy runs through a grassy field."
48. "A cinematic shot of person walking along a quiet country road, their feet crunching on the gravel with every step, fields of wheat swaying in the breeze either side."
49. "A washing machine undergoing a full cycle. It begins with a top-down view of the machine filled with water and white soap suds, with two black rubber seals on either side of the stainless steel drum. The video progresses to show the drum spinning, with the suds becoming more agitated and the seals moving along with the drums motion."
50. "Sweet Cherries Stems Colliding and Splashing Water Droplets"
51. "A series of colorful balloons floating in mid-air, creating a festive and celebratory atmosphere."
52. "A cinematic shot of person jogging along riverside path, their feet rhythmically tapping against the ground, the river flowing gently beside them."
53. "A green helicopter taking off from an airport runway."
54. "A hand holding a yellow fidget spinner. The hand is fair-skinned and holds the bright yellow fidget spinner with silver bearings. The background is blurred and appears be trees against blue sky. The video captures the subtle movements of the hand as it spins the fidget spinner, creating soothing and mesmerizing visual effect. As the video progresses, the hand continues to hold the fidget spinner, showcasing its smooth and satisfying motion. The background remains blurred, adding a sense of tranquility to the scene. Overall, the video is a calming and enjoyable display of the simple pleasure of fidget spinning."
55. "A windmill spinning in green field."
56. "A bicycle wheel spins forward, moving in a circular motion while keeping balance."
57. "A waterwheel turns water flows over it, the paddles rotating consistently."
58. "A close-up of a person's 's legs as they walk through a sun-dappled forest, the light playing off their shoes as they navigate the uneven terrain."
59. "A man riding a mountain bike on dirt trail."
60. A child's 's toy top spins on smooth surface, rotating without stopping."
61. "A basketball spins on player's fingertip, showcasing balance and skill."
19

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
62. "A jellyfish swimming in shallow water. The jellyfish has translucent body with a distinctive pattern of white circles and lines. It appears be swimming just below the surface of the water, which is dark and murky due the presence of algae other aquatic plants."
63. "A cinematic shot of a person walking along a cobblestone street in a historic town, their feet making a rhythmic tap on the stones as they move."
64. "A group of horses grazing in grassy field behind black wooden fence"
65. "A fish swims forward a steady line, its tail swaying side to side as propels itself."
66. "A penguin waddles in straight line, shifting from one foot the other."
67. "A man is jumping rope on the sandy beach, with waves crashing in the background."
68. AA man enjoying water skiing on brown river with green shore and lily pads in the background. Water sprays up from underneath him as he skis across the surface of the lake."
69. "A man is swimming in clear blue pool, enjoying the cool water and the freedom of movement in the pool. As he continues to swim, he glides gracefully through the water, his arms and legs moving a smooth and coordinated rhythm..
70. "A kid rumning in the mountains of Campo Imperatore, Italy, at the sunset. He is wearing red polo shirt, blue jeans, and brown shoes. As he runs, he passes by some white rocks on the ground."
71. "A woman doing push-up exercise on beach at sunset."
72. "A woman is shown running through a field, with tall grass and wildflowers all around her. She is fair-skinned woman with long, red hair, wearing a black t-shirt and leggings, and listening to music on her phone, In the background, there are trees and more fields of greenery."
73. "A man exercising with battle ropes at gym."
74." AA person engaging in boxing workout at gym."
75. "A dark gray horse running in an enclosed corral. running towards the camera."
76. "A close-up of a runner's 's legs as they dash up a flfgight of stairs in city park, their feet hitting each step with precision and power."
77. "A man is swimming in the ocean. In the background, the sky is hazy and overexposed, with the sun shining brightly above the horizon. As the video progresses, the man continues swim, his arms moving rhythmically through the water."
78. "A herd of white cows walking down a dirt path. The cows are all facing forward and walking towards the right side of the image. The background is blurry but appears to a field or pasture."
79. "A person jogs along trail in dense forest, their legs pumping they navigate the roots and rocks that dot the puth."
80. "A young woman dances in the night bustle against the backdrop of a glowing fanfare."
81. "A man is walking down the street while pushing trash can. The man, wearing red t-shirt, blue jeans, and brown sandals, pushes the black trash can wheels."
82. "A man enjoying a mountain biking adventure through a forest. He is seen riding a black and white mountain bike down a dirt path, with his back to the camera."
83. Women's 's legs walk into the sea with waves."
84. "A young man walking on a treadmill. He is wearing a white tank top and red shorts, and has his hands on the sides of the machine as he runs."
20

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
85. " Closeup of feet of professional soccer player training with ball on stadium field with artificial turf."
86. "A helicopter flying over a forest. The helicopter is black and has two large rotor blades on top. It is flying low to the ground, with nose pointing slightly upwards."
87. "A close-up up of a person's 's feet as they walk through field of wildflowers, their shoes brushing against the blooms with each step."
88. "A man is playing basketball, dribbling the ball and making shots."
89. "A giraffe running through an open field. The background a bright blue sky with fluff white clouds."
90. "A person jogs along a city waterfront, their legs moving steadily as the sun sets, casting a warm glow over the water and the buildings behind them."
91. "A woman is doing push-ups mat in the studio."
92. " "Two dancers perform on a stage. The man stands behind the woman with his left arm is lifted over his head and the other stretched to the right. The woman lets go of the man't right hand, swinging her leg to the left and performing a pirouette. She spins four times and ends up facing the man."
93." "A woman drinks from a water bottle in forest. The woman has fair skin and brown hair. She is wearing a black jacket and black and white gloves." "
94. "Tracking camera shot. A polar bear walks across snowy landscape. It looks curiously around it plods through the snow. The background is a snowy landscape with footprints visible in the snow. Sunlight shines from overhead and casts the bear's 's shadow on the snow."
95. "A cinematic shot of person walking through desert at midday, their legs moving slowly but steadily across the sand dunes, with heat waves distorting the distant horizon."
96. "A man jumping rope on dark stage. His movements are fluid and energetic. Two spotlights shine down from above him."
97. "A woman twirls hula hoop around her waist in a park during sunset. The woman, with predium-length curly bluck hair and yellow tank top, stands on a grassy field surrounded by trees. As the hoop revolves around her waist, she shifts her hips rhythmically to keep it moving, The golden sunlight casts a long shadow behind her."
98. "A man exercises on a leg press machine at a gym."
99. "A young woman enjoys cup of coffee balcony."
100. "A man energetically bangs on drum kit. He holds drumsticks both hands and bashes on the drum kit with the drumsticks."
101. AA woman performs high knees on a beach."
102. "A "Aerial tracking camera shot. A white semi-truck drives highway."
103. "A woman is holding clear wine glass partly filled with a burgundy-colored wine. Facing forward, the woman smiles, she raises the glass with her left hand and takes a small sip."
104. "A man works on a piece of wood in a workroom. He holds a shiny silver chisel with a wooden handle in his right hand."
105. "lliced green apples are tossed in brown liquid. The apples are cut into thick slices and have shiny green skins with some light-colored speckling. They begin to rotate clockwise, flying out in every direction as the light amber liquid splashes and swirls behind them." "
106. AA baboon eats mango."
107. "A young woman vapes the living room. The woman exhales the thick, billowing smoke."
21

---

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models
108. "A woman performing an aerial hoop trick. The woman hangs from black aerial hoop attached the ceiling by rope. In the initial frame, she has her legs wrapped around the hoop and her extended outward, holding onto the hoop with both hands. Her body is twisted, looking up towards the ceiling, with her shadow cast the white wall behind her As the video progresses, she continues to hang from the hoop, her body twisted in various positions, her arms and legs wrapped around the hoop as she performs the aerial trick. The background remains the same, with shadows from the aerial hoop and the woman's 's body on the white wall."
109." "09.."MMde "Modern urban street ballet dancer performing acrobatics and jumps."
110. 'A woman doing pirouette in an empty dance studio."
" 112. "A brunette woman doing some acrobatic elements on aerial hoop outdoors."
111. "A woman dancing hip hop, street dancing in the studio. Slow motion."
113. "A woman, with long brown hair and wearing black top and gray bottoms, climbs on a pole with her right leg wrapped around it and her left arm extended upward. The background is white wall with mirror reflecting the woman's images."
114." "A man performing backflip. Slow motion."
115. "A woman dancing in gym. The woman is spinning around repeatedly."
116. "A group of duck are walking in a row, one after the other. The background a Japanese temple."
117. "Arc camera shot. A young woman doing stretches on a beach."
118. "A woman walking through a field of beautiful sunflowers. She spins coumterclockwise and laughs. A field of shoulder- length sunflowers grow in the background, with trees on the horizon stretching up towards a cloudy sky."
119. "Arc camera shot. A man playing the guitar."
120. "A boy blowing out candles on a birthday cake."
121. "A cheetah running in the Savannah."
122. "Tracking shot. A golden retriever runs through a grassy park. The dog ears flop up and down with each bounding step, and its tongue hangs out to one side. A frishee flies into view from the left, and the dog leaps into the air to catch
it. A group of people in the background claps and cheers."
123. "A young girl skips down quiet suburban street lined with trees. She has light brown skin and long, wavy black hair tied back with red ribbon. The girl wears white t-shirt, a denim skirt, and bright yellow sneakers. Her arms swing loosely as she skips"
124. "A woman doing sit-ups at a gym."
125. "A child riding his bicycle on dirt path. The background is dirt path lined with trees on either side."
126. "A runner moves at full speed along a suburban sidewalk. The background is rows of houses and trees passing by in blur."
127. "A young woman engaging in boxing workout. She is wearing red boxing gloves and white t-shirt, and has long blonde hair. In the first frame, she is slanding in front of a black punching bug, with her right arm extended and her left arm bent, ready to punch the bag. She appears focused and determined. In the second frame, she has moved to the left of the bag and is looking towards the right side of the image. She continues to punch the bag with her right arm extended and her left amm bent. In the final frane, she is still standing to the left of the bag and looking towards the right side of the image. She is still wearing her red boxing gloves and white 1-shirt, and her long blonde hair visible. The background of a blue wall with a window on the left and doorway on the right, as well as two black objects hanging from the ceiling. Throughout the video, the woman is intensely focused her workout, punching the bag with precision and skill."
128. "A brown bear walks in a grassy fiedd."
22


Reviews:
[{"review_id": "mXM06pMNfz", "reviewer": "Reviewer_RHDu", "summary": "The authors present Video-JAM (Joint Appearance-Motion representation), at an aim to capture real-world motion, dynamics, and physics, which existing video generative models struggle with handling. \nIn particular, the authors discover that the current video model training objective biases models towards fidelity, at the expense of motion coherence or temporal consistency.\nTo this end, the authors propose to train video generative models to predict the motion of the video - represented by optical flow masks - as well, thereby training the models on joint appearance (pixel) - motion (optical flow) representations.\nThis addition requires addition of a few layers (to handle the extra input and output), and training together **with** the optical flow supervision can only happen at the final finetuning stage.\nNow that the video generative model is equipped with the ability to recover the motion, the authors propose the Inner Guidance to steer the model towards coherent motion.\nVideoJAM achieves state-of-the-art performance in motion coherence, while also improving the visual quality of the generated videos.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comprehensive review of our work and the insightful suggestions. Please find below our response to the points raised in the review.\n\n__Related work on text-to-3D:__ Thank you for bringing this work to our attention. We will revise the related works section to reference this paper in the final version of the manuscript.\n\n__VBench metrics:__ Thank you for this point for discussion. VBench has two different contributions: a benchmark and a set of metrics. The benchmark is not used in our work since a significant part of the prompts there _intentionally describe static scenes_. Most categories of the benchmark (e.g., architecture, food, scenery, plants) are dominated by prompts that do not elicit any motion (e.g., “an apartment building with balcony”, “video of a indoor green plant” (grammar mistake in the original prompt), “landscape agriculture farm tractor”). Even the categories that entail motion (e.g., humans and animals) tend to focus on prompts that do not require meaningful temporal understanding (e.g., “people sitting on a sofa”, “a man with a muertos face painting”, \"a black dog wearing halloween costume”).  \nAdditionally, out of the 11 dimensions for evaluation, _6 purposefully contain prompts with no motion at all_ (e.g., the class “object class” simply contains a list of objects such as “a person”, “a bicycle”, the class “color” contains prompts that assign colors to objects such as “a red bicycle”, “a green bicycle”). Therefore, similar to concurrent leading video models (Step-Video-T2V, Veo2) we opt to employ the Movie Gen benchmark as a general quality evaluator since it is carefully curated to evaluate all aspects of the generation (similar to VBench) while also explicitly incorporating motion instructions in all the prompts (see Section 3.5 of the [Movie Gen paper](https://arxiv.org/pdf/2410.13720) for details on the Movie Gen benchmark).\n\nThe set of metrics proposed by VBench is divided into two: those that are applied by VBench to external benchmarks (see [official implementation](https://tinyurl.com/yc3xhyk5)) and a set that includes additional metrics that apply only to the VBench prompts. An example of such a metric is the VBench metric to estimate temporal flickering which requires _frozen videos_, thus the [prompts explicitly encourage no motion at all](https://tinyurl.com/yyd2z9b9). Naturally, we used _all metrics that are supported for external benchmarks_ to evaluate the performance of VideoJAM in the setting that it is intended for. This allows us to benefit from the generic and meaningful prompts from Movie Gen with the disentangled axes of evaluation from VBench.\n\nNotably, both benchmarks are not suitable for measuring motion coherence, as both tend to focus more on appearance and less on temporal understanding (L. 330-357). Thus, we construct an additional benchmark (VideoJAM-bench) to estimate temporal coherence and employ Movie Gen for a general assessment of the video quality.\n\nWe are happy to address any remaining questions by the reviewer.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.\n\nClaim 1: The current objective used for training video generative models is biased towards pixel fidelity, and neglects motion coherence.\n- Substantiated by Figure 3 and Figure 8.\n\nClaim 2. VIdeoJAM also improves visual quality of generated videos, not only the motion coherence.\n- Validated in the quantitative/qualitative results in the experiment section.", "methods_and_evaluation_criteria": "Yes. \nThe authors use both human evaluation and the automatic metrics from VBench, the currently standard suite for evaluating video generative capabilities.", "theoretical_claims": "Yes - section 4.3 sufficiently shows that since $d_t$ depends on the prompt and model weights (i.e., not independent with the model weights), the conventional CFG formulation cannot be used to guide motion coherence. The proposed inner guidance in Eq. 8 is derived with substantial motivation and correctness.", "experimental_designs_or_analyses": "The experimental designs and analyses seem sound and valid. In particular, it was nice to see that human evaluation together with the automatic metrics from VBench, since the automatic metrics may not directly reflect the actual human preferences.", "relation_to_broader_scientific_literature": "Similar endeavors were carried out in the text-to-3D / multi-view domain, where Wonder3D (Long et al., 2024) attempts to predict the normal map together with the pixel values during the generation process. Such work, including VideoJAM, hints that appropriate usage of additional modalities may help improve the generative results while not harming the generative quality.", "essential_references_not_discussed": "While not an essential reference, adding Wonder3D as a reference may help the readers to understand that the idea of engaging additional modalities is a sound way of improving generative quality. Otherwise, I could not find any essential references which was not discussed.", "other_strengths_and_weaknesses": "Strengths:\n\nS1. Strong motivation and effective results.\n\nS2. Simple and straightforward methodology without complications, yielding effective results.\n\nWeaknesses:\n\nW1. Not a strong weakness, but the paper does not include ALL the evaluation metrics provided in VBench. More results are included in Tables 4 and 5, but it would help to include the full results from VBench for the authors to fully understand the capabilities and shortcomings of VideoJAM.", "other_comments_or_suggestions": "Please refer to W1 mentioned above.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXtA0haOzM", "reviewer": "Reviewer_kPFe", "summary": "This paper presents VideoJAM, a framework that improves motion coherence in generative video models by learning a joint appearance-motion representation. It introduces two key components: predicting both pixels and motion during training, and Inner-Guidance for coherent motion during inference. VideoJAM outperforms competitive models in motion coherence and visual quality, highlighting the importance of integrating appearance and motion for better video generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and points for discussion. Please find below our response.\n\n__Human evaluation:__\nThank you for your feedback. There appears to be a misunderstanding in the review. Importantly, _VideoJAM appears in all human evaluations_. As highlighted in all table captions, _the human evaluation scores indicate the percentage of votes that favor VideoJAM_, where each result uses the Two-Alternative Forced Choice (2AFC) protocol (Tabs. 1-3, L. 371-377). For example, a motion score of 85.9% for CogVid-5B indicates that evaluators prefer the motion by VideoJAM over CogVid-5B in 85.9% of votes. Since all rows are inherently comparative to VideoJAM, the final row (\"VideoJAM vs. VideoJAM\") contains minus signs (-). To further improve clarity, we will add visual graphs of the comparisons in the Appendix, and replace the minus signs with upper arrows.  \nAs noted in the paper (L. 400-403, L. 413-419), human raters consistently and significantly prefer VideoJAM in terms of motion over all baselines.\n\n__Impact of Inner-Guidance:__\nInner-Guidance is a highly beneficial component of our method. As shown in Tab. 3, human evaluators significantly prefer the results with Inner-Guidance across all categories (68.9% vs. 31.1% for text faithfulness, 64.4% vs. 35.6% for quality, and 66.2% vs. 33.8% for motion). Also, note that _removing the optical flow prediction inherently removes Inner-Guidance_, as Inner-Guidance depends on the motion prediction. Thus, eliminating optical flow is an ablation that includes _both Inner-Guidance, and the flow prediction_ and, as expected, results in a greater performance drop.  \nTo further demonstrate the benefits of Inner-Guidance, we enclose a [qualitative ablation test](https://tinyurl.com/5c3jdab7) (best viewed in full screen). The results, generated in the same setting as in the paper, illustrate that removing Inner-Guidance often degrades motion coherence and introduces artifacts.\n\n__Inner-Guidance latency:__\nAs noted in the implementation details (L.726-731), Inner-Guidance is applied to only 50% of the steps (following Sec. 3) and performed in parallel using a batch size of 3. This results in a 1.15x slowdown compared to the standard CFG, which is a significantly lower slowdown than other multi-guidance methods (e.g., Instruct Pix2Pix incurs a 1.25x slowdown since it operates on all steps).\n\n__Additional related works:__\nWe appreciate the suggestion. The related works section will be revised to include all references proposed by the reviewers, ensuring a more comprehensive review of existing literature.\n\n__Automatic metrics vs. human evaluation:__\nThank you for raising this point. We appreciate the opportunity to clarify.\n1. It is well-documented in literature that automatic metrics often fail to align with human preferences [1,2,3]. For instance, the Movie Gen paper dedicates an entire section (3.5.3) to metrics, and highlights that \"across all tasks, we find that _existing automated metrics struggle to provide reliable results_”.\nSpecifically, temporal metrics tend to prefer either static motion since it is more “smooth” (as noted in the VBench paper \"a completely static video can score well in the aforementioned temporal quality dimensions.\"), or completely incoherent videos that contain a lot of movement (Appendix, L. 767-768). Appearance metrics inherently prefer static videos, since they are typically based on _image models_, for which video frames that contain motion are out of distribution. All of the above underscores the inherent limitations of automated metrics in capturing the dynamic and nuanced nature of video content, and the necessity for a human-based evaluation.  \n[1] Polyak et al., Movie Gen.  \n[2] Bartal et al., Lumiere.  \n[3] Girdhar et al., Emu Video.\n\n2. The [official VBench leaderboard](https://tinyurl.com/33tu3we5) further highlights these inconsistencies. Models ranked highly by the [HuggingFace (HF) human leaderboard](https://tinyurl.com/28ea397z) are ranked significantly lower by VBench. For example, Kling 1.5 is the leading baseline by both our human evaluation (Tab. 2) and HF. However, it is ranked 21st in the VBench leaderboard, below even much smaller models like CogVid-5B and Wan2.1-1.3B, both ranked much lower in the HF human leaderboard.\n\nWe enclose both automated metrics and human evaluations for completeness and believe continued research on reliable automatic metrics is essential for video generation assessment.\n\n__Supplementary materials (SM):__\nWe noticed that the reviewer indicated “no” to viewing the SM. While we understand that reviewing SM is optional, we believe that the actual video results are very important to fully appreciate the improvement enabled by VideoJAM. We kindly ask the reviewer to consider viewing the results on our SM website.\n\nWe are happy to address any remaining questions, and respectfully ask the reviewer to reconsider the score if our response has addressed all the questions.", "claims_and_evidence": "Claims: VideoJAM improves motion coherence in video generation models by learning joint appearance-motion representations, using dynamic guidance to steer generation toward coherent motion.\n\nEvidence: The paper provides comparisons showing VideoJAM outperforms competitors in motion coherence and visual quality. However, the ablation study raises doubts about the necessity of the Inner-Guidance component, as it has little effect on results.", "methods_and_evaluation_criteria": "Methods: VideoJAM introduces two key components: predicting both pixels and motion during training, and using Inner-Guidance during inference to enhance motion coherence.\n\nEvaluation Criteria: The model’s performance is evaluated based on motion coherence, visual quality, and quantitative metrics (Human Eval, Auto Metrics), but the Human Eval results for the final method are not provided.", "theoretical_claims": "The paper suggests that integrating motion and appearance representations during training leads to more coherent motion in generated videos, which is a significant improvement over previous methods.", "experimental_designs_or_analyses": "The paper includes ablation studies to assess the impact of different components like Inner-Guidance and optical flow. However, the removal of Inner-Guidance shows minimal effect, and the correlation between Human Eval and Auto Metrics is unclear.", "relation_to_broader_scientific_literature": "The paper builds on existing video generation models but emphasizes the novel integration of motion and appearance. However, it could engage more with related work on motion coherence and the role of dynamic guidance in generative models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n1.\tThe idea of using the model’s own evolving motion prediction as a dynamic guidance signal to steer generation toward coherent motion is interesting and highlights the difference from previous work.\n2.\tThe presentation is clear and easy to understand.\n\nWeakness:\n1.\tThe introduction of motion representation can significantly improve the effectiveness of motion, and this conclusion is predictable. However, I noticed in the ablation study in Table 3 that removing Inner-Guidance has little effect on the results, while removing optical flow has the largest impact. This raises concerns about the necessity of Inner-Guidance. Additionally, I would like the authors to provide information on the impact of Inner-Guidance on inference time.\n2.\tI found that there is no correlation between the Human Eval and Auto. Metrics in Tables 1, 2, and 3, and the authors did not provide the Human Eval results for the final method. Therefore, I remain skeptical about the true effectiveness of the work.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7UQvZeJVR", "reviewer": "Reviewer_QjbW", "summary": "Despite recent advancements, generative video models still exhibit significant limitations in temporal coherence, especially when modeling real-world dynamic interactions and physics. The authors identify that this issue arises fundamentally from the traditional pixel-based reconstruction objectives, which prioritize appearance quality at the expense of coherent motion. To address this critical limitation, this paper proposes VideoJAM, a framework designed to explicitly integrate motion priors into video generation. Specifically, VideoJAM introduces a unified latent representation to jointly capture appearance and motion information by predicting both signals simultaneously through the addition of two linear layers during training. Additionally, during inference, the authors propose Inner-Guidance, a novel mechanism that dynamically adjusts the sampling process based on the model's own motion predictions. Extensive experiments demonstrate that VideoJAM substantially improves motion coherence, consistently outperforming several state-of-the-art open-source and proprietary models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comprehensive feedback and the interesting points for discussion.\nPlease find below our response to the points raised in the review.\n\n__VideoJAM adaptability:__ We appreciate the feedback. While all concepts of our work can be easily generalized to any backbone, we acknowledge that this work focuses on the de-facto leading architecture, DiT. Following the review, the writing will be revised and refined to reflect this and avoid confusion.\n\n__Higher resolution and longer videos:__ The duration of our generated videos is limited by that of the base model we employ, which is 5 seconds (128 frames at 24 fps). However, to showcase the scalability of VideoJAM, we repeat the experiment from Tab. 2 on VideoJAM-30B with a higher resolution of 512x512. The results of the human study are enclosed below. As can be observed, VideoJAM is, once again, superior to the base model in all aspects, and very significantly improves motion coherence, further establishing the scalability of our method.\n| **Model** | **Text Faithfulness** | **Visual Quality** | **Motion** |\n|:------------:|:----------:|:-------:|:------:|\n| DiT-30B (512) |  30.5%    |  26.6%   |  28.9%   |\n| **+ VideoJAM** |  **69.5%** |   **73.4%** |   **71.1%** |\n\n__Inner-Guidance sensitivity test:__ Thank you for this suggestion. In accordance with common practice in diffusion papers, we selected the best scale empirically. Following the review, we enclose a qualitative sensitivity test with different Inner-Guidance scales in the [following link](https://tinyurl.com/3fx38map) (best viewed in full screen). All results are extracted in the same setting described in the paper.  \nAs can be observed, removing Inner-Guidance causes a noticeable degradation to the motion coherence (e.g., the legs of the man, the helicopter is flying backward). Importantly, Inner-Guidance demonstrates robust results across all reasonable scales (3,7) and does not display unusual sensitivities.  \nWhen significantly increasing the scale (50), the results are out of distribution and cause a degradation in the video quality and motion, which is to be expected with any other guidance signal as well (e.g., text with CFG).\n\n__Human evaluation protocol:__ Thank you for raising this point, we appreciate the opportunity to clarify this aspect of our work. The human evaluators selected to participate in our study have extensive experience with evaluating generative visual models. As a filtering criterion, each evaluator has performed _at least 1000 evaluations_ before, where _at least 90% of those evaluations have been approved by third-party examiners_. The two videos in each comparison are randomly shuffled to ensure an unbiased comparison, and all videos are generated without watermarks to avoid identification.  \nAdditionally, note that the rating of the baselines by our evaluators (Tab. 2) is very much in correlation with the [public HuggingFace video leaderboard based on human evaluations](https://tinyurl.com/28ea397z), where Kling 1.5 is the leading baseline, with Sora being the second strongest baseline.\n\nWe enclose below our instructions for the evaluators for transparency.\n\n> Hello! We need your help to read a caption, and then watch two generated videos. After watching the videos, we want you to answer a few questions about them:  \n>Text alignment: Which video better matches the caption?  \n>Quality: Aesthetically, which video is better?  \n>Motion: Which video has more coherent and physically plausible motion? Do note, it is OK if the quality is less impressive as long as the motion looks better.\n\n\nWe are happy to address any remaining questions.", "claims_and_evidence": "The authors' core method (VideoJAM) is evaluated comprehensively on the proposed VideoJAM-bench which is based on Movie Gen benchmark, utilizing clearly defined evaluation standards such as VBench metrics and the two-alternative forced choice (2AFC) human evaluation protocol. Experimental results convincingly demonstrate VideoJAM’s substantial improvement in motion coherence. However, the authors' claim regarding the adaptability and generality of VideoJAM to arbitrary video generation models currently lacks sufficient experimental validation. Specifically, all provided experiments exclusively employ DiT-based architectures (DiT-4B and DiT-30B), and no additional evidence is presented to demonstrate the generalization capability of VideoJAM to other prevalent architectures. Further validation using diverse, non-DiT video generation models is necessary to robustly substantiate this claim.", "methods_and_evaluation_criteria": "The proposed VideoJAM framework is clearly motivated, conceptually straightforward, and innovative, effectively addressing the key issue of temporal coherence in generative video models. Specifically, the introduction of a joint appearance-motion representation and the Inner-Guidance mechanism provides explicit and effective guidance toward coherent motion predictions, which is theoretically sound and well justified. Additionally, the authors propose the VideoJAM-bench, specifically constructed based on the Movie Gen benchmark, focusing explicitly on challenging motion categories, including basic motion, complex motion, rotational motion, and physics-based interactions. The benchmark is well-designed, targeted, and capable of effectively evaluating motion coherence. Furthermore, the evaluation criteria, which include both automatic metrics (VBench) and human assessments (2AFC), are thorough and rigorous, comprehensively capturing the performance of the proposed approach.\nHowever, despite the authors emphasizing the widespread use of the datasets without any modification, the relatively low resolution and short duration of the videos utilized may limit the ability to demonstrate the framework's practical effectiveness under high-resolution, longer-duration scenarios. Therefore, it would be beneficial to validate the method further on higher-resolution and longer-duration videos to fully establish its practical applicability and generalizability in real-world settings.", "theoretical_claims": "The theoretical analysis presented in VideoJAM primarily revolves around two aspects: (1) the theoretical justification and motivation behind the proposed Joint Appearance-Motion Representation, and (2) the theoretical derivation and rationale of the Inner-Guidance mechanism. Specifically, the authors clearly articulate the theoretical reasoning behind jointly modeling appearance and motion representations, addressing the fundamental limitations of traditional pixel-based objectives. Furthermore, they rigorously discuss the theoretical distinctions between their proposed Inner-Guidance and existing methods such as Classifier-Free Guidance and InstructPix2Pix guidance, emphasizing the critical dependence between motion prediction signals and model weights, a scenario not supported by existing assumptions of independence between conditions.\nHowever, the selection of specific Inner-Guidance weights appears empirical, lacking theoretical justification or sensitivity analysis. A theoretical explanation or sensitivity analysis supporting the effectiveness of these particular weight values would further strengthen the theoretical rigor and clarity of the presented claims.", "experimental_designs_or_analyses": "The authors utilize not only the established Movie Gen benchmark but also introduce the VideoJAM-bench specifically designed to evaluate motion coherence, enhancing the relevance of their experimental setup. The selection of advanced baselines, including open-source models such as CogVideo5B and Mochi, and proprietary models such as Sora and Kling, ensures fairness and reliability of comparisons. The ablation studies are thorough and clearly demonstrate the contribution of text guidance, motion guidance, and the Inner-Guidance mechanism individually.\nHowever, the experiments are limited by relatively low video resolutions and short durations (5 seconds), which may not fully reflect the model’s effectiveness in realistic, high-resolution, and longer-duration scenarios. Additionally, although a clear comparison standard (2AFC) was provided, the paper lacks detailed information on the selection criteria of human raters and the explicit evaluation protocols, potentially introducing subjectivity and bias into the experimental outcomes.", "relation_to_broader_scientific_literature": "The authors clearly identify the prevalent issue of motion incoherence in existing diffusion-based video generation models and thoroughly analyze its root cause—namely, traditional pixel-based reconstruction objectives being insensitive to temporal dynamics. Inspired by Composable Diffusion Models (Liu et al., 2022) and Classifier-Free Guidance (Ho & Salimans, 2022), they propose a novel conditional sampling mechanism termed Inner-Guidance. Unlike previous methods, VideoJAM explicitly emphasizes dependencies among conditioning signals and between conditioning signals and model weights, rather than assuming their independence. By dynamically leveraging motion predictions generated by the model itself during sampling, VideoJAM substantially enhances the coherence of generated motion. Furthermore, compared to recent motion representation approaches (e.g., Shi et al., 2024; Wang et al., 2024), which typically treat motion as an external conditioning input, VideoJAM explicitly integrates motion prediction into the training objective, systematically improving both motion generation quality and video coherence.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Although VideoJAM demonstrates superior motion coherence compared to existing methods, it still struggles with fine-grained motion details, such as hand movements, in complex scenarios (e.g., \"On a rainy rooftop, a pair of hip-hop dancers lock and pop in perfect sync, bringing energy and rhythm to the stage,\" and \"A panda breakdancing in a neon-lit urban alley, graffiti art providing a colorful backdrop\"). This limitation suggests room for future improvement in handling intricate motion coherence.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fagTKG67DC", "reviewer": "Reviewer_VpZA", "summary": "The paper proposes a joint appearance-motion learning framework for video generation. The authors are motivated by the key observation that the common pixel-based training objective is invariant to temporal perturbations. Therefore, they propose to equip the model with an explicit motion learning objective via a joint representation learning framework. In addition, the authors propose an inner-guidance mechanism that steers the prediction towards more coherent motion. The proposed method is extensively evaluated on VideoJAM-Bench and shows advantageous performance especially in motion coherence and physics following.\n\n## update after rebuttal\nThe authors have addressed all of my concerns with extensive experimental results. Therefore, I am keeping my score of Accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comprehensive feedback and the useful points for discussion. Please find below our response.\n\n__VBench prompts:__ Both VBench and Movie Gen are common benchmarks for general video evaluation. The VBench prompts are not used in our work since out of the 11 dimensions for evaluation, _6 purposefully contain prompts with no motion at all_ (e.g., “object class” contains objects: “a person”, “a bicycle”, “temporal flickering” explicitly demands no motion, “a laptop, frozen in time”). Therefore, similar to recent leading models (Step-Video-T2V, Veo2), we employ Movie Gen, which evaluates all aspects of the generation (similar to VBench) while also incorporating motion (for more details, see our response to reviewer RHDu).\n\n__VideoJAM scalability:__ We appreciate the opportunity to clarify. While it is true that (only) the automatic metrics favor the 4B model, please note:\n1. It is well-documented that automatic metrics often fail to align with human preferences [1,2,3]. For instance, Movie Gen dedicates a section (3.5.3) to metrics, highlighting that \"across all tasks, we find that _existing automated metrics struggle to provide reliable results_”.  \nAutomatic metrics tend to prefer static motion since it is more “smooth” (as noted in the VBench paper, \"a completely static video can score well in the aforementioned temporal quality dimensions\"). Indeed, the 4B model scores better than the 30B model in the automated metrics, despite the 30B model being overwhelmingly superior (see human evaluation below). This anomaly can be attributed to the 4B model's tendency to produce videos with reduced motion (DiT-4B scores only 38.3 in its dynamic degree), which aligns with the biases of automated metrics discussed above.\n|**Model**|**Text Faith.**|**Quality**|**Motion**|\n|:------------:|:----------:|:-------:|:------:|\n|VideoJAM-4B|21.9%|21.1%|18.7%|\n|**VideoJAM-30B**|**78.1%**|**78.9%**|**81.3%**|\n\n[1] Polyak et al., Movie Gen.  \n[2] Bartal et al., Lumiere.  \n[3] Girdhar et al., Emu Video.\n\n2. The [official VBench leaderboard](https://tinyurl.com/33tu3we5) shows similar inconsistencies. Models ranked highly by the [HuggingFace (HF) human leaderboard](https://tinyurl.com/28ea397z) are ranked significantly lower by VBench. For example, Kling is the leading baseline by both our human evaluation (Tab. 2) and HF. However, it is ranked 21st in the VBench leaderboard, below even much smaller models like CogVid-5B and Wan2.1-1.3B, both ranked much lower in the human leaderboard.\n\n3. To further show robustness and scalability, we enclose a study in the same setting as Tab. 2, conducted on DiT-30B at a 512x512 resolution. VideoJAM maintains its significant improvement with a _71.1% lead in motion_, which is less than 2% below the improvement for the lower resolution.\n|**Model**|**Text Faith.**|**Quality**|**Motion**|\n|:------------:|:----------:|:-------:|:------:|\n|DiT-30B (512)|30.5%|26.6%|28.9%|\n| **+VideoJAM** |**69.5%**|**73.4%**|**71.1%**|\n\n__Additional references:__ Thank you for the useful suggestions, the related works section will be revised to include them.\n\n__Scene cuts:__ We hypothesize that it is still possible to use VideoJAM by applying RAFT on each scene separately and stitching the scenes with a “white” (no motion) frame.\n\n__Inner-Guidance latency:__ Inner-Guidance is applied to only 50% of the steps and performed in parallel using a batch size of 3 (L.726-731). This results in a 1.15x slowdown compared to standard CFG, which is a significantly lower slowdown than other multi-guidance methods (e.g., Instruct Pix2Pix incurs a 1.25x slowdown as it operates on all steps).\n\n__Physics benchmark:__ Thank you for this suggestion. We arbitrarily used the first benchmark suggested, and conducted an evaluation in the same setting as Tab. 2. The results below indicate a clear advantage to VideoJAM (62.7% preference in motion).  \nNote that it is expected for the improvement in this benchmark to be somewhat decreased since optical flow is not an explicit physics representation, therefore it does not solve all physics incoherences (L. 412-418). As we mention in the conclusions, incorporating physics priors in future works is a fascinating direction. Samples from the comparison are provided in [this link](https://tinyurl.com/yhtmvx8p). We observe that these prompts are quite different from the ones employed by other benchmarks, and do not describe typical natural scenes.\n|**Model**|**Text Faith.**|**Quality**|**Motion**|\n|:------------:|:----------:|:-------:|:------:|\n|DiT-30B|48.2%|49.2%|37.3%|\n|+VideoJAM|**51.8%**|**50.8%**|**62.7%**|\n\n__Optical flow calculation on data:__ The calculation takes 13 hours in total (~1-2 seconds per sample with a batch size of 128).\nVideoJAM is a generic approach that can theoretically work with any representation, as long as there is a reliable way to encode it (e.g., RGB representation or a specialized VAE).\n\nWe are happy to address any remaining questions.", "claims_and_evidence": "The paper's claims are well supported by extensive empirical evidence. In particular, the experiment in Section 3 is insightful and effectively justifies the motivation of the paper.", "methods_and_evaluation_criteria": "The proposed method is suitable for video generation because of its ability to enhance motion coherence. The evaluation criteria are extensive, including human evaluation and automatic metrics on VideoJAM-Bench and Movie Gen benchmark, but there is room for improvement:\n\n* **W1**: It would be better to evaluate on the original VBench dataset, as this would allow comparison with more open-source baselines on more general prompts.", "theoretical_claims": "The paper makes no theoretical claims.", "experimental_designs_or_analyses": "The experimental designs are quite comprehensive, but could be improved by considering scalability:\n\n* **W2**: The most counterintuitive result is that VideoJAM-30B is inferior to VideoJAM-4B in terms of quality in human evaluation and all automatic metrics, according to Tables 1 and 2. This casts doubt on the scalability of the proposed method, since a well-known bitter lesson is that the introduction of human prior knowledge can reduce scalability. It would be better for the authors to explain the performance drop and provide additional results to prove scalability.", "relation_to_broader_scientific_literature": "The paper advances video generation [Hong'22, Kondratyuk'23, Bar-Tal'24, Brooks'24] by learning an appearance-motion joint representation. Its improved physics capability contributes to world models [Ha'18, Brooks'24].", "essential_references_not_discussed": "Several related works on using motion representation for video generation are missing [Tulyakov'18, Shen'24, Jin'24].\n\n---\n\n[1] Tulyakov, et al. MoCoGAN: Decomposing Motion and Content for Video Generation. CVPR 2018.\n\n[2] Shen, et al. Decouple Content and Motion for Conditional Image-to-Video Generation. AAAI 2024.\n\n[3] Jin, et al. Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization. ICML 2024.", "other_strengths_and_weaknesses": "Strengths:\n\n- The idea of appearance-motion joint learning in video generation is very intuitive, as the model would be more effective at ensuring motion consistency.\n- The comprehensive experimental analyses provide valuable insights into the original DiT and the new proposed model's behavior.\n- The paper is well formatted and clearly written, making it easy to follow.\n\nWeaknesses:\n\n* **W3**: Reduced flexibility. Since the optical flow can only be computed within a single scene, the proposed framework cannot effectively handle scene cuts. In this sense, the appearance-motion joint learning framework sacrifices flexibility on multi-scene video clips for better single-scene consistency.\n* **W4**: Inference overhead due to inner-guidance. According to Equation (8), each sampling step requires three evaluations, which imposes 1.5x inference cost compared to standard CFG. It would be better to discuss this and make comparisons under the same NFE budget rather than the same number of sampling steps.", "other_comments_or_suggestions": "Given the improved physics generation capability, it would be interesting to quantitatively evaluate it on recent physics-focused benchmarks such as PhyGenBench [Meng'24], Physics-IQ Benchmark [Motamed'25], WorldModelBench [Li'25].\n\n---\n\n[1] Meng, et al. Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation. 2024.\n\n[2] Motamed, et al. Do Generative Video Models Understand Physical Principles? 2025.\n\n[3] Li, et al. WorldModelBench: Judging Video Generation Models As World Models. 2025.", "questions_for_authors": "* **Q1**: What is the computational cost of extracting optical flow for 3M fine-tuning samples? Is it possible to use coarser but more efficient motion representations such as motion vector?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces VideoJAM, a straightforward approach for improving motion coherence and temporal consistency in video generation models. It works by denoising optical flow maps alongside the pixels, resulting in a joint appearance-motion representation. It is also shown how, during inference, the resulting approach can be steered towards generating videos with better motion using guidance.

3/4 of the reviewers acknowledged the author response and engaged with the authors during the rebuttal stage. Among those reviewers there is broad agreement that the contributions in this work are significant, timely, and that the claims are well supported by evidence. A few concerns, such has how VideoJAM affects performance on non-motion related categories, or whether this approach is scalable have been convincingly addressed by the authors as evidenced by all reviewers recommending accept. Remaining limitations, such as the method struggles more often with fine-grained motion categories are suitable for follow-up work. The AC agrees with the recommendation to accept and concurs that the work has the potential to be impactful and of broader interest to the video generative modeling community.
Decision: Accept (oral)