MATRIX: Mask Track Alignment for Interaction-aware Video Generation
Authors: Siyoon Jin
Seongchan Kim
Jae Ho Lee
Dahyun Chung
Hyunwook Choi
Jisu Nam
Jiyoung Kim
Seungryong Kim
Venue: ICLR 2026 Poster (2026)
Keywords: video generative model
Primary area: generative models

Abstract: Video DiTs have advanced video generation, yet they still struggle to model multi-instance or subject-object interactions. This raises a key question: How do these models internally represent interactions? To answer this, we curate MATRIX-11K, 
a video dataset with interaction-aware captions and multi-instance mask tracks. Using this dataset, we conduct a systematic analysis that formalizes two perspectives of video DiTs: semantic grounding, via video-to-text attention, which evaluates whether noun and verb tokens capture instances and their relations; and semantic propagation, via video-to-video attention, which assesses whether instance bindings persist across frames. We find both effects concentrate in a small subset of interaction-dominant layers. Motivated by this, we introduce MATRIX, a simple and effective regularization that aligns attention in specific layers of video DiTs with multi-instance mask tracks from the MATRIX-11K dataset, enhancing both grounding and propagation. We further propose InterGenEval, an evaluation protocol for interaction-aware video generation. In experiments, MATRIX improves both interaction fidelity and semantic alignment while reducing drift and hallucination. Extensive ablations validate our design choices. Codes and weights will be released.

Published as a conference paper at ICLR 2026
MATRIX: MASK TRACK ALIGNMENT FOR INTERACTION-AWAR VIDEO GENERATION
Siyoon Jin
Seongchan Kim
Dahyun Chung Seungryong Kim
Jaeho Lee
Hyunwook Choi
Jisu Nam
Jiyoung Kim
KAIST AI https://cvllb-kkeltt git.tt..gtttt thu... 10/MARRIX
"A boy noadies fir the withe areen li,, prepering to bake a
boy
"green lid bottle"
Results
MM-DiT Block
Frame
MM MMM-DIT DIT Block
MMMDDYY slock
30 RufE
Fome 2
Baseline (Failed Results)
Input
Generated
Frame
EScAl ISPA
Frome 2
Instance
Original Attention
Aligned
Attention
MATRIX (Successful Results)
(a) MATRIX Framework
(b) Attention Visualization and Generation Results
We reveal how video diffusion transformers (DiTs) represent multi-instance or subject-object interactions during video generation by focusing on their internal attention mecha- nisms. Building on these, our MATRIX framework further enhances the interaction-awarentt of video DiTs via the proposed Semantic Grounding Alignment (SGA, LSGA) and Semantic Propaga- tion Alignment (SPA, LSPA) losses.
Figure 1: Teaser:
ABSTRACT
Video DiTs have advanced video generation, yet they still struggle to model multi-instance or subject-object interactions. This raises key question: How do these models intermally represent interactions? To answer this, we curate MATRIX-11K, video dataset with interaction-awant captions and multi-instance mask tracks. Using this dataset, we conduct systematic analysis that formalizes two perspectives of video DiTs: semantic grounding. via video-to-text attention, which evaluates whether noun and verb tokens capture instances and their re- lations; and semantic propagation, via video-to-video attention, which assesses whether instance bindings persist across frames. We find both effects concentrate in small subset of interaction-dominant layers. Motivated by this, we introduce MATRIX, simple and effective regularization that aligns attention in specific layers of video DiTs with multi-instance mask tracks from the MATRIX-II1 IKK dataset, enhancing both grounding and propagation. We further propose Inter- GenEval, an evaluation protocol for interaction-awwe aware video generation. In ex- periments, MATRIX improves both interaction fidelity and semantic alignment while reducing drift and hallucination. Extensive ablations validate our design choices. Codes and weights will be released.

---

Published as a conference paper at ICLR 2026
Noun-level grounding mismatch
Hollucination
"In a chernistry lab with benches test lubes, a iormami white shakes small glass Bask."
"Ina library filled with tall wodden shelves, tosminn in black bends slightly as thick Book
X Verb-level grounding mismmatch
X Duplication
"The womenn in red blouse with long brown hair Gloses the black laptop on the desk."
"One playet in a red, jersey raises his palm, and the other playse in blue jersey slups in highh-iss""
(a) Semantic Grounding Failure
(b) Semantic Propagation Failure
Figure 2: Failure cases of existing video DiTs: (a) semantic grounding failures, where subjects, ob- jects, or their verb relations are mismatched, and (b) semantic propagation failures, where bindings break over time, leading to hallucinations or duplications. Overlays indicate the intended instances.
INTRODUCTION
Recent video diffusion transformers (DiT) have advanced text-to-video generation and manipulation of single object human, enabling applications in simulation (Soni al. 2025), AR/VR (Zhou jct al. 2025), robotics (Kim & Joo 2025) and embodied reasoning (Feng et al. 2025b). Despite these advances, DiT-based models (Yng et al. 2024 Zheng al. 2024) still struggle to generate multi-instance or subject-object interactions from text prompts (e.g.. who does what to whom).
As illustrated in Fig. D and 2 two main failures emerge: (1) semantic grounding failure, where they fail to localize subject or object specified by prompt nouns or to bind verb-specified subject- object interaction, resulting in text-video mismatch; and (2) semantic propagation failure, where this noun/verb grounding does not persist over time, causing drift, duplication, or hallucination. These observations raise key questions, How do video DiTs semantically bind and video, and how is this binding propagated to support interactions?, which motivates us to analyze and strengthen this improve interaction-awant video generation.
Fig. and Fig. 4 motivate our analysis. In 3D full attention of video DiTs Yang et al. 20244, video-to-text attention aligns noun tokens with subject and object regions and verb tokens with their interaction region, which is the union of subject and object. Video-to-video attention propagates this binding across frames, by linking interaction regions in one frame to the corresponding regions in other frames. Especially. in successful generations, these alignments concentrate in few layers and persists across frames. We regard these alignments as the binding to analyze, assessing where emerges and whether it persists across frames. To quantify this binding, the reference must provide spatial precision to verify grounding and temporal continuity to test persistence, and instance sepa- rability to disambiguate same-class instances. We therefore adopt multi-instance mask tracks the reference, since for each instance, a per-frame mask is linked by a persistent ID across the video, and the union of the subject and object masks defines the interaction region.
Since no existing dataset pairs such tracks with interaction-aware captions, we curate MATRIX- 11K, 11K videos with interaction-rich captions and instance masks tracks. With MATRIX- IK, we conduct the first systematic study of how subject-object interactions are internally represented
in video DiTs (Yang et al. 2024; Peebles & Xie, 2023 Esser et al. 2024). We analyze 3D full attention where text and video tokens interact, and study two core perspectives: semantic ground- ing, via video-to-text attention, measuring whether noun tokens localize to subject or object regions and verb tokens attend to their union; and semantic propugation, via video-to- iileo-to-vian videe attention, mea- suring whether these noun/verb groundings are preserved so identities and their interaction persists across frames. We observe both effects emerge strongly in small subset of layers, which term interaction-dominan layers, and the alignment in these layers is consistently stronger success- ful generations and weaker in failures, yielding clear process-failure contrast.
Based on these insights, we propose MATRIX (Mask Track Alignment for Interaction-Aware Video Generation), simple yet effective regularization that aligns attention in interaction- dominant layers with multi-instance mask tracks. We finetune the image-to-vided model (Yang al. 2024) with LoRA (Hu et al. 2021), condition multi-instance mask and supervise only
2

---

Published as a conference paper at ICLR 2026
Generated Vidco
subject
kerb
object
Generated Video
SUO
keeb
ebjject
Prompt: The inan takes from the kine plass.
Prompt: The ian putá the donul into his mouth.
Figure 3: Attention maps per token type. Noun tokens (subject, object) align with their respective regions (e.g., layer //); verb tokens aligns with the union of subject-object regions (e.g., layer 7).
Frome all
Frame #3t
Frame A00
Frame
Frame 404
Fremz b4t
Generned
Gevented
Query
Query
(A) Prepagation Attenton Map (Success Cise)
(b) Propegation. Propeggtion.Atemiiin Attention Map (Fuilure Case)
Figure 4: Visualization of attention maps. Propagation maps (e.g., layer 12) directly affect whether
interaction propagation succeeds or fails.
those layers via two terms: Semantic Grounding Alignment (SGA) loss, which aligns noun tokens with subject/object regions and verb tokens with unions in video-to-text attention, and Semantic Propagation Alignment (SPA) loss, which enforces video-to-video attention to preserve consistent instance tracks across frames. To align attention space and pixel space, introduce lightweight causal decoder that maps attention to frame-level mask tracks. Our approach applies to any Video DiT's that employ 3D full attention.
In addition, existing metrics (Huang et al.) 2023| Zheng et al. 2025a; Gu et al. 2025a) capture only global alignment and cannot localize subjects, verbs, objects, making interaction- evaluation unreliable. We introduce InterGenEval, an interaction-awarc evaluation protocol. Specifically, key interaction semantic alignment (KISA) checks the pre, during, and post conditions of key interaction. Semantic grounding integrity (SGI) measures whether the subject, object and verb are correctly grounded. Semantic propagation integrity (SPI) assess the temporal persistence of bindings and is applied alongside KISA and SGI. Interaction fidelity (IF) is reported as the mean of KISA and SGI.
In summary, our contributions are:
We construct MATRIX- 11K, 11K videos with aware captions for both analysis and training.
mask tracks and interaction-
We introduce the first systematic analysis of semantic grounding and semantic propagation in video DiTs, revealing how subject-object interactions emerge.
Motivated by our analysis, we propose MATRIX, an effective regularization composed of SGA and SPA, applied mask tracks via lightweight LoRA, improving grounding accuracy and consistency.
interaction-dominne layers, and conditioned on multi-instance
We design InterGenEval, a protocol for evaluating interaction-awareness of the video.
RELATED WORK
Interaction Representations Video DiTs. Prior works have examined internal representations in
UNet image diffusion (Hedlin et al. 2023 Jin et 2025 Nam al. 2024 Tang et al. 2023) image

---

2025) customize specific Published as a conference paper at ICLR 2026
(+) Interaction Identification and Instunce Assignment
(b) Interaction Scoring and Internction Filtering
Ground Truth Caption
(6) Instance Description Extraction
The video captunes a acene in modent barbersltog. A mun, with light
Determine whether GT Caption.
Avecs the degire of Contact, Dynumisto the
Provide detailed information unique
8
meticulousty styling the hair a man senned in from of him The mon, dressed in gray ishir, has a beand.....
A -dd man is everbo styling <id2> man
Contect
< with bellooe
light shitt
<idl> mus, mor
Dynamism :4
with beard
gryy
Ground Truth Video
mow
Mask Tracks
VIM Verification
Bpeman
Urpppet imaage
Propagation
Greuniinn DIN
Desciptionaa
T Tranes
Given isputs, decide which cropped images best moticees condidatc
Figure 5: Our dataset curation pipeline.
DiTs (Yu et al. 2025 Lee et al. 2025, and video DiTs (Nam et al., 2025 Zhang et al., 2025, but none formalize interaction representations. Since pixel-level reconstruction give little supervision for binding "who does what to whom" " maintaining it over time, an analysis of interactions in Video DiTs remains absent. We therefore define interactions as semantic grounding (token level binding) and semantic propagation (temporal binding), and analyze through attention.
Human-Object Interaction (HOI) Synthesis. Research in HOI synthesis has explored generating human motions conditioned on interaction prompts. Early works (Chao et al. 2018 Gkioxari al. 2018) focused on recognizing and localizing HOIs in 2D, while more recent studies (Pi et al. 2023 Soni al. 2025) synthesize 3D motions of single human multiple humans under verb condi- tioning. These methods demonstrate that interactions can be generated when instances are explicitly parameterized, but they remain restricted to motion-level synthesis. Importantly, they have not been integrated into video diffusion, where interaction modeling must directly govern pixel generation. Relation Customization. Recent methods (Wei et al., 2025 Tan al..
relations (e.g., pick up) via dulation-specffce adapters or motion priors. While effective in narrow cases, they rely on closed verb set, require per-relation tuning, decouple control from text ground- ing, and struggle with multiple instance pairs, limiting generalization to open-vocabulary verb set. Controllable Video Diffusion Models. Controllable video generation (Esser al.] 2023 /Zhang
al. 2023 Cai et al.) 2025 Li et al. 2025 Gu et al., 2025b) Geng et al. 2025 Feng et al. 2025a) introduces guidance signals such as depth, bounding boxes, optical flow, or trajectories to constrain scene geometry and motion. While such controls improve temporal consistency and enable user- defined dynamics, they remain agnostic to interaction semantics. Even multi-instance controls using bounding boxes or mask sequences operate independently of text, leaving subject-action-objecc re- lations under-specified. As a result, controllable methods support single-instance manipulation but fall short multi-instance interactions, which require explicit alignment with textual descriptions.
3 MATRIX- 1K DATASET
To systematically analyze and enhance semantic binding in 3D full attention of video DiTs, we in- troduce MATRIX-IIK a dataset of videos V paired with interaction-awart captions / and instance mask tracks M for each instance ID k. Prior datasets (Goyal et al. 2017 Ravi et al. 2024 Li al. 2021 Zhang et al.) (020) often suffer from low video fidelity. or semantically weak or misaligned captions and mask. MATRIX-1 addresses this by aligning instance mask tracks with interaction- aware captions, via an interaction-aware curation pipeline. We will release the curation pipeline and this dataset publicly. Sec. (.T describes LL-M-bsedd caption processing for interaction and II) extraction, while Sec. 3.2 details mask track construction with GroundingDINO, VLM verification and SAM2 propagation.
3.1 INTERACTION-AWARE CAPTIONING
We employ off-the-shelf LLM (et al. 2024 to process caption P in three steps. First, the LLM identifies whether an interaction verb is present (e.g.. hold, throw) and assigns an instance ID k to every participating noun while recording its base-noun class (e.g., man, cup). This yields interaction
4

---

Published as a conference paper at ICLR 2026
triplets (ksub. verb, kobj). where kaub and kobj denote the IDs bound to the subject and object nouns, and will later be tied to an instance mask track Mg. Second, to focus on physically grounded and temporally meaningful interactions, the LLM scores each interaction for Dynamism (degree of mo- tion or temporal change) and Contactness (physical contact or spatial proximity). Only interactions exceeding predefined thresholds retained, and any IDD l not linked to retained interaction is also removed. Third, for every retained k, the LLM extracts an appearance description (e.g., a man in gray shirt) to disambiguate same-class instances, which we later use for VLM verification.
3.2 MULTI-INSTANCE & INTERACTION MASK TRACKS
For each video and its instance set, we uniformly sample frames and use GroundingDINO (Liu al, 2024) to generate multiple bounding box candidates per instance ID k, each with confidence score. We begin with the highest-confidence candidate; if it fails VLM verification, we move to the next highest and continue until one verifies or all fail. A VLM (OpenAl & al. 2024 inspects each candidate as visual prompt together with the class label and the appearance description of l from Sec. 3.I and decides whether it matches the target instance. The first verified candidate becomes the anchor frame and box. From the anchor, we initialize SAM2 (Ravi et al.. 2024) and propagate masks through the clip obtain per-ID instance track Mg. If all candidates fail we remove At and drop any interaction that is linked to it. Videos with no remaining valid interactions are discarded.
Finally, human annotators manually inspect and filter residual errors, such mask drift, missing frames, or misaligned clips. Fig. 30 Fig. 31, Fig. 32 and Fig. 33 provide examples of the final dataset we curated. More details including data statistics are provided in Appendix
INTERACTION-AWARENT ANALYSIS IN VIDEO DITS
We present, to our knowledge, the first systematic analysis of how Video DiTs (Yang et al. 2024 Wan al. 2025: Kong et al. 2024) internally represent text-based interactions during generation. We ask whether DiTs encode (1) semantic grounding, where textual tokens (nouns, verbs) localize to the correct visual regions, and (ii) semantic propagation, where these bindings remain spatially co- herent over time so that instance identities and relations persist. These analyses determine whether models capture interactions end-to-end, both grounding roles ("who does what to whom") and prop- agating them throughout the sequence. This analysis motivates our regularization.
4.1 PRELIMINARIES- VIDEO DIFFUSION TRANSFORMERS
MM-DIT (Esse et al. 2024). the basic block of video DIT, stacks layers of 3D full attention that jointly process spatiotemporal and textual information. This design allows the model integrate text
video to-iooo--eee Allention keci-4o-skdee Allaniu
D vidao to-oo Allection todi-loeiodd Alleation
Kriboo
Kjest
and video tokens during generation. In the (-th layer, attention is:
Q.KT
Quides
A22v
A
Attn Q K,, Vi) = A.V where A Softmax
Here, Q1 K2, Vi are query, key, value matrices of the 1-th layer, and d is dimension of key. 3D full attention in DiT's operates unified sequence concatenating video latents and text embeddings:
Atte
A
Figure 6: Illustration of full 3D attention in video DiTs.
Q = Concat Qqidee
fext), K = Concootnn""" K(ext)
where Concat(-) indicates the concatenation operation along the token dimension. To summarize, the attention matrix of DiT can be divided into four distinct regions: video-to-video A v2v video-to-text A v2t text-to-video A 12v and text-to-text 1 shown in Figure 6. This unified formulation supports analysis of how Video DiTs bind visual and textual modalities into a coherent generative process and propagate across frames. In this work, we focus on A v2t and A v2v
5

---

Published as a conference paper at ICLR 2026
video-ionicxi attentio
video-lo-video atlention
vidco-to-test atlention
vidko-lo-vidce atiention
nubycctobjar
subyeetobjar
Sap0005
Fallurz
Stpon4
Sacons
(a) Influential Layer Candidates
(b) Dominant Layer Selection
Figure 7: Layer analysis. (a) Influential layers layers with high AAS that rank in the Top-10 for many videos. (b) Dominant layers: the influential layers whose mean AAS clearly separates successes from failures (Best viewed in zoom).
4.2 SEMANTIC GROUNDING
We ask whether video DiTs ground textual tokens to visual regions. We read A v as per-token heatmap: for text token (noun or verb), (1) Fxxx denotes attention over frames and H x W latent grid. We consider: (i) nouns, which align with subject/object spatial regions, and (ii) verbs, which capture interaction via joint attention to subject and object.
Noun Grounding. Nouns cover the roles of the instance, subject and object. For each role e (sub, obj}. we form a token set Te, containing the role's head noun and its modifiers. Since modifiers tend to attend to the instance region the head noun, aggregate heatmaps by mean:
221
1
Σ A (t),
(suubobjj]
Te
TE
Concretely, the sequence A, 22t E F indicates where the subject/object grounded.
Verb Grounding. Verbs express the interaction between the grounded subject and object. We obtain the verb map by averaging over the verb token set:
Σ A v2 (1),
worb
Tverbl
BETowhh
where Tverb contains the head verb and auxiliaries/partiies (e.g., "is". "up" in "is lifting up"). For evaluation, A and AX2) are compared to their respective instance mask tracks and A is compared to their interaction region, which is the per-frame union of subject and object mask tracks.
4.3 SEMANTIC PROPAGATION
Semantic propagation asks whether properly grounded bindings remain spatially coherent time. Specifically, the attention originating form a subject, or object region in the first frame should con- centrate on the same instance over time, and the interaction region should remain clustered without drift or duplication. To this end, we study A v2v which maps each video token to all others and reuse mask tracks Md (Sec. 3. For subject/object IDs kaub: kobj. we take first-frame masks M Mobjj downsample to latent grid H x W and denote the resulting masks as m mo E {0, 16 H w (we drop the frame superscript hereafter). The query sets are the latent locations where masks are one:
Q. = {(h, w) (h, w) = 1}
(sub, obj), Qverb Qsub U Qobj-
For any E Qe (e e (sub, obj, verb)). let A (42 E RFXWX be the attention from q to all spatiotemporal tokens. The propagation map is
Σ A v2v QEQ.
RxHXW E (sub, obj, verb)
Qu
producing per-frame heatmaps that trace ddentity-consiseee attention for subjects, objects and stable interaction region for verb. This produces the same canonical form as the grounding maps in Sec. 4.2 but shifts the focus from token alignment temporal consistency.

---

Published as a conference paper at ICLR 2026
DM
key
3D VALE
CE VAL
query
(c)
3D Attention Block
Av2
D
D
(b) Grounding Attention Map Extraction
The Sthaggz 11 ERY aal Bakei
Text Encoder
A porb
A Aven Avey
GATED I3E office
Jub
LSGA
LSPA
quary
visual token
subject token
object token
token
Mauh M. M.rrh Mabj
Muub M.vett Mobj
Attention Block
(a) Overall Architecture
(c) Propugution Attention Map Extraction
Figure 8: Main architecture of MATRIX.
4.4 EVALUATION METRIC: ATTENTION ALIGNMENT SCORE (AAS)
Each 2 A is per-frame heatmaps, where larger values indicate attention that location. Using the mask tracks Maub, Mobj (Sec. 3 we downsample to latent grid to obtain Maub; Mobj E {0, 1 Fx H and define the verb mask tracks mvverb by element-wise OR as Msub V Mobj- Given Ay2: A with e E (sub, obj, verb), we score alignment by the sum of attention maps mask-1 I locations, called attention alignment score (AAS):
AAS E
Σ (A T mm)(f, (h,,,,,) AAS = Σ (A o (f,) f.h.w
fhh,
where indicates the element-wise multiplication.
4.5 ANALYSIS
We analyze Cog CogVideoX-SB-I2V (Yan al. 2024 for semantic grounding and propagation of both nouns and verbs. For all analyses, we compute the Attention Alignment Scores (AAS) defined in Sec. Sec.Afrrom 3D full attention across 42 layers and 50 denoising timesteps. We consider four variants:
noun grounding (v21), verb grounding (v2t), noun propagation (v2v) and verb propagation (v2v).
Additional analyses other video DiT's [Wan et al. 2025 Kong et al. 2024) are in Appendix C
Layer Influence. For each video, we rank layers by their step-averaged AAS and mark the top-10. Aggregating across videos, each layer receives two statisties. Frequency counts in how many videos the layer appears in the top-10. Magnitude the mean AAS of that layer. As shown in Fig. (a), we combine the two by simple rank-sum and select the top-10 layers as influential for each variant. We find that the influence concentrates in few layers that repeatedly achieve high alignment across videos, indicating that alignment governed by specific layers rather than by outliers.
Among influential layers, we identify the dominant layer that directly govern outcomes. We split the generated video into equal-sized success and failure verification. For each influential layer, we compute its mean AAS on the success set, the failure set, and full set. The success gap is the difference between the success mean and the overall mean, and the failure gap is the difference between the failure mean and the overall mean. We call layer iorrractioodommiinnn when the success gap large and positive while the failure gap large and negative relative the overall mean; rank layers by this separation, as depicted in Figure (b).
Layer Dominance.
by human
Relevance to Interaction- Awareness Generated Videos. Fig. and 4 show when attention concentrates the corresponding instance/union regions, generations are correct and preferred: when diffused or mislocalized, failures are common. These observations support the defined AAS reliable proxy for interaction fidelity. As sanity check, we apply perturbation guidance (Ahn jet al. 2025) to these layers. As shown in Fig. 19 attention sharpens toward instance regions and interaction fidelity improves slightly. Detailed protocol and results are in Appendix B and
7

---

Published as a conference paper at ICLR 2026
Post
"At office entrance, the gray suit pushes the glass dook open ourward into the lobby."
ceur delivery nunoow ground
s-shart
Tpea Coo Cypok
Figure 9: Qualitative comparison.
MATRIX FRAMEWORK
Sec. identifies small set of interaction-dominnn layers whose attentions correlate strongly with semantic grounding and propagation. This analysis motivates MATRIX, which introduces Semantic Grounding Alignment (SGA) and Semantic Propagation Alignment (SPA) losses that directly align attention with ground-truth instance mask tracks.
Yang et al. 2024) with LoRA (Hu al. 2021), the model conditions on noise latent 21. the first RGB frame To. a first-frame multi-instance ID map Lo with stable IDs, and prompt /' whose tokens mark subject, verb and object. We extend the input projection to ingest Zo and To by channel-wise concatenation with the latent. Here To is the indette-indxxed aggregation of per-instance binary masks { {M²). so each ID k keeps a fixed color across the clip. This grounds identities at generation start, and gives users explicit control over targets at inference, since Io can be obtained by off-the-shelf segmentors (Ravi et al. 2024).
Baseline Architecture. Building on CogVideoX-5B-BB
Attention Alignment. We supervise attention directly with ground-truth instance mask tracks. We aggregate attentions at latent resolution, v v2v E [0. 1==xxx and compare them to pixel- space mask tracks M, E {0, 1) Hpix Wiix for E {sub, obj, verb). where Fpix Hpix- W Wpix denote the decoded video length and pixel resolution. To align scales, a lightweight causal decoder D.).) that mirrors 3D VAE (Yang et al. (024)) upsampling schedule maps attention RGB-space mask tracks at the correct spatiotemporal scale. Specifically, it expands time and space with the same strides as the 3D VAE with causal alignment of the first frame, so supervision applied the correct spatiotemporal scale. Specifically, let A Da(Ay xxt)) and y Do (AA v2v) denote decoder outputs at the pixel grid for e E {sub, verb, obj}. We compare the decoder outputs to the target mask tracks M.. Both SGA and SPA use the same composite loss f, weighted sum of BCE, soft DICE and L2 regression to the mask track. For prediction X and target Y, is formulated as
E(X,Y = BCE Y) + B1ice - Dice(X,, Y)) + -2 - Y
where Booes Baice and B2 are coefficients, respectively. The SGA and SPA losses defined ((Ay2v, Ma),
LSGA
Σ
{(A Ma), LSPA Σ
cE(sub,obj}
We apply these losses only to the interaction-dominatt layer found in the analysis, routing alignment where it is most effective, while leaving the other layer to preserve general video quality. Training minimizes simple objective that adds these losses to the denoising loss:
Liotal = LDM + ASGA + ASPACSPA.
updating the LoRA parameters, the input projection layer and the lightweight decoder D while keeping the remaining backbone frozen. Here LDM is the denoising loss, and λSGA ASPA are sealar weights of grounding and propagation, respectively. Additional details provided in Appendix D
8

---

Published as a conference paper at ICLR 2026
Table Quantitative comparison.
InterGenEval
Human Fidelity
Video Quality
Methods
KISA (1)
SGI (1)
IF (t)
HA (1)
MS (t)
1Q)
Cog VideoX-2DB -B-- Yang al. (2024
0,420
0,470
0.445
0.937
0,993
69.69
CogVideoX-5B-
Yang al.
2024
0.406
0.491
0.449
0,936
0.987
69.66
Open-Sora- 11B-12V Zheng ct 2024
0.453
0.508
0.480
0.891
0.992
63.32
TaVid (Kim Joo 2025
0.465
0.522
0.494
0.917
0.991
68.90
MATRIX (Ours)
0.546
0.641
0.593
0.954
0.994
69.73
Table 2: Ablation studies. All finetuning experiments were conducted on the MATRIX- 11K.
InterGenEval (TSA
Human Fidelity
Video Quality MS
Methods
SGI (t)
IF
IQ
(1)
Baseline VodoVooee----111 3B-12X)) Yang et al. 2024
0.406
0.491
0.449
0.936
0.987
69.66
(11)
TaVid Kim & Joo, 2025
0.465
0,522
0.494
0.917
0.991
68.90
(III)
(I) LoRA w/ MATRIX IIK dataset
0.445
0.526
0.486
0.940
0.994
69.77
(IV)
(III) + SPA loss
0.451
0.540
0.496
0.937
0.995
70.26
(III) SGA loss A thv
0.578
0.531
0.955
0.993
70.03
(VI)
(III) SGA loss in A
0.50y
0.592
0.550
0.952
0.994
69.62
(VII)
(III)
SGA (Ours)
0.641
0.994
69.73
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Dataset. We construct two evaluation sets, covering synthetic and real domains. The synthetic comprises 60 (image, prompt) pairs generated using (OpenAl & et al.) 2024) where each prompt describes interactions among distinct instances, corresponding images are generated to match. For real domain, curate 58 (image, prompt) pairs from open-source dataset (Nan al. 2025 Chao et al. 2018), selecting examples using our curation pipeline. Additional details are in Appendix
InterGenEval. We evaluate interaction-aware semantics with a structured QA protocol. For each key interaction we auto-generate 10 questions: six stage-wise checks (KISA) of the pre, during, and post states, and four grounding checks (SGI) of the subject, object, verb-conditioned union, all phrased with appearance cues and bounding boxes. We report KISA and SGI, each reweighted by factor SPI, which penalizes emergence and disappearance across frames. The overall score, IF, is the mean of KISA and SGL. Details are in the Appendix F
the
Additional Metric. Hallucination measured by HA (Human Anatomy) from VBench2.0 (Zheng et al. 2025a). For video quality, we adopt metrics from VBench (Huang et al., (023), including MS (Motion Smoothness) and IQ (Image Quality). Further details and results are in Appendix
6.2 COMPARISON AND ANALYSIS
Fig. 9 and Tab. compare method with open-source models (Yang al. 2024 Zheng al. 2024 Kim & Joo 2025). The 2B model (Yang et al.) [024] rarely completes the action (fails to open the door or lift the box; Fig. 99. yielding low KISA, SGI and IF, yet its conservative motion
produces clean frames with higher IQ and MS and fewer human anomalies, reflected by higher
HA. The 5B model (Yang et al., 2024) attempts actions more often and slightly raises interaction scores. but identity drift and contact violations (twisted torso, floating box; Fig. ()) reduce SGI and HA. Open-Sora-I2V (Zheng et al. 2024) follows prompt strongly and lifts KISA, while unstable grounding and propagation lower SGI and HA and degrade overall quality via extra or missing instances. Vid (Kim & Joo| 2025) benefits from an explicit target cue and improves grounding of one instance, but the lack of propagation signal limits temporal consistency and HA. In contrast, our method maintains correct bindings and tracks via SGA and SPA on the interaction-dominalt layers, achieving the strongest interaction fidelity in KISA. SGI and IF, highest HA, IQ and MS. Additional qualitative results in various scenarios, including non-con non-contact and non- hom-h scenarios, provided in Fig. of Appendix.

---

Published as a conference paper at ICLR 2026
loeal xdday dal Soeal ldkaa
UBS oopeaccti <ik lagitgg
"Th: SIGE RICEBR Gands AESRE SEROO Fay bank for pack."
Figure 10: Qualitative results of Wan2.1-14B-I2V WWn et al. 2025) with our MATRIX. While the bascline often ignores motion and produces nearly static videos, MATRIX helps preserve motion dynamics and enhances interaction fidelity.
6.3 ABLATION STUDIES
Tab. aligns with our analysis and isolates the effects of our curated data, layer selection, and each interaction-aware loss. (I) Vanilla Cog ViieooX-SSB-1Y (Yang et al.) 2024). without any finetuning, performs worst on interaction metrics since it lacks any interaction signal. (II) LoRA tuning with single-object conditioning improves over (I) but fails to enforce propagation and degrades overall quality. (III) Naive LoRA finetuning on our curated dataset without layer selection or auxiliary losses yields balanced yet middling performance and corresponds to the baseline that simply finetunes on our data. In particular, the improvement from (I) to (III) captures the benefit of finetuning on MATRIX-11K train set with minimal architectural changes. (IV) Adding SPA selected layers further enhances propagation, however, without explicit grounding it trades off noun/verb alignment, leading to higher smoothness and quality but lower grounding. (V) and (VI) correspond two SGA variants: (V) applies SGA on selected layers of A 12v while (VI) applies SGA on selected layers of v In practice, since a single text token can correspond to multiple locations, constraining text-to-video attention in (V) leads to unstable grounding. In contrast, (VI) supervises video-token queries directly, so the signal applied to the spatial regions being generated and leads more consistent gains. Therefore, adding SGA to (III) significantly boosts grounding (KISA, SGI, IF) by aligning noun/verb attentions, while keeping propagation. (VII) Combining SGA and SPA to (III) yields the best overall balance: strongest interaction fidelity (KISA, SGI, IF), the best human fidelity (HA) and improved video quality over the baselines, indicating that grounding first and then enforcing propagation offers complementary gains.
v2r
6.4 RESULTS WITH OTHER DIT BACKBONES
We also evaluate the MATRIX framework on another DiT-based backbone, Wan2.1-14B-122 (Wan jet al., 2025), making only minimal architectural modifications and finetuning solely with our SGA and SPA objectives. The procedures used to identify the dominant layers are described in Ap- pendix C As shown in Fig. 10 although Wan2. 1-14B-V- is already a strong backbone, simply applying MATRIX consistently improves interaction fidelity while preserving overall visual quality. These results demonstrate that MATRIX functions as plug-and aud--aay adapter: when applied on top of various DiT-based video models, it reliably enhances interaction fidelity without requiring exten- sive architectural redesign or sacrificing video quality. Additional qualitative results are presented in Fig. 42 and Fig. 43
CONCLUSION
We study how video DiTs represent multi-instance interactions. To answer this, we firstly curate MATRIX-IIK, video dataset pairing interaction-aware captions with multi-instance mask tracks. Using these tracks, we analyze 3D full attention and observe that semantic grounding and prop- agation concentrate in small set of ieoraccioo-dominann layers. Guided by this, we introduced MATRIX, a lightweight regularization that aligns attention in those layers to mask tracks via SGA and SPA loss. On InterGenEval (KISA, SGI, IF), MATRIX improves interaction fidelity, strength- ens noun and verb grounding. and reduces drift and duplication. Ablations confirm the critical role of layer selection and the complementary contributions of SGA and SPA.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) RRS-2019-11190075, RS-2024- 00509279, 00--002--I122000 2025-25441832 ASS-202--2544) 1838, 05-2025-022144499 RS-2025-02217221 and the Culture,
55-2023-00227592 RS-2025-0221444 RS-2024-00457882. RS-
Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism (RS-2024-00345025, RS-2024-0033306. RS-2023- 00222280, 2SS20023-00266509,, and National Research Foundation
Korea (RS-2024-003465997)
REFERENCES
Donghoon Ahn, Hyoungwon Cho, Jacwon Min, Wooscok Jang, Jungwoo Kim, SconHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attentoon guidance, 2025. jttta::
abs////b/////a// 2403. 1773
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding. local- ization, text reading, and beyond, 2023. URL http:httts:/araiii
abs/ / 2200 13966129666666
Romain Beaumont and Christoph Schuhmann. aesthetic-prediene hittps://githir com/
[ATON-AT/aesthetlcc c-teetii--peeictorr.. 2022. GitHub repository, MIT License.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint wrxiv:2311.15122 2023.
Daniel Bolya, Po- Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang. Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network, 2025. hRI.https:
13181
Yuanhao Cai, He Zhang, Xi Chen, Jinbo Xing, Yiwei Hu, Yugian Zhou, Kai Zhang, Zhifei Zhang, Soo Ye Kim, Tianyu Wang, Yulun Zhang, Xiaokang Yang. Zhe Lin, and Alan Yuille. Omniveus: Feedforward subject-driven video customization with multimodal control conditions, 2025. URL
https: wwt./as/////oooooooott.o.t..atrt /50///206.... .3361
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning detect human-object interactions, 2018. lttps://///
05448
Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheymin. Videojam: Joint appearance-mmotion representations for enhanced motion generation in video models, 2025. URL SRRL/ttpss//aaaai..rrraaaa/////
C2492
Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng. Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain- oh-thought for reasoning large language models, 2025. URL https: //ta:v///xxvvvv org/ abs/ 0553309566
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yold-world: Real-time voca-vocayulary object detection, 2024.
17270
Ziming Cheng, Binrui Xu, Lisheng Gong, Zuhe Song, Tianshuo Zhou, Shiqi Zhong, Siyu Ren, Mingxiang Chen, Xiangchao Meng, Yuxin Zhang, Yanlin Li, Lei Ren, Wei Chen, Zhiyuan Huang, Mingjie Zhan, Xiaojie Wang, and Fangxiang Feng. Evaluating milms with multimodal multi- image reasoning benchmark, 2025. hti.ttts:: //atxiii///xvv.../// org/ abs/abs550 04280
11

---

Published as a conference paper at ICLR 2026
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards general purpose purpoose vision-language models with instruction tuning, 2023.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models, 2023. URL
https:// /ttt:///axiv... www.aiii..ggaaaaaaaaaaaao abs/ 2302. 03011
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. URL httpss//arxiv.org//aa 2443.03205
Aaron Grattafiori et al. The llama herd of models, 2024. URL https://erxxv.orrg//2 0107.21782
Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In IEEE Conference Computer Vision and Putlern Recognition, pp- 3677-3686, 2020.
Sicong Feng, Jielong Yang, and Li Peng. Resource-efficient motion control for video generation via dynamic mask guidance, 2025a.
wwtp://wwwgg/////e///ooooo http://bbb//3..1.1 18386
Yao Feng, Hengkai Tan, Xinyi Mao, Guodong Liu, Shuhe Huang, Chendong Xiang. Hang Su. and Jun Zhu. Vidar: Embodied video diffusion model for generalist bimanual manipulation, 2025b.
URL https::/
org/ abw//501/7 12898
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, An- drew Owens, and Deqing Sun. Motion prompting: Controlling video generation with motion trajectories, 2025. httpstss::///// wttpvvarxvvvvvmm//////... org/ abs/ 02700
Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human- object interactions, 2018.
1104. 07333
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzyáska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, Flo- rian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something some- thing" video database for learning and evaluating visual common sense, 2017. URL https: // //arxiv.com org/ abaaabaaa// 0.2.06042666 04261
Jing Gu, Xian Liu, Yu Zeng, Ashwin Nagarajan, Fangrui Zhu, Daniel Hong, Yue Fan, Qianqi Yan, Kaiwen Zhou, Ming-Yo Yu Liu, et al. phyworldbench": A comprehensive evaluation of physical realism in text-to-video models. arXiv preprint arXiv:2507. 13428, 2025a.
Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, and Yuan Liu. Diffusion as shader: 3d-aware video diffiusion for versatile video generation control, 2025b. /tthtsss::/ //ar:i//aivivvia wwwooogg///.... abs 2501. 03847
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion, 2023. URL
https: //tp:////xvvvv.. http://wwwwva///aaa/a//a wrg//bs//205....... 15581
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
12

---

Published as a conference paper at ICLR 2026
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si. Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models, 2023. URL URL https::tpss....m.........tm ptactts://ariiie http://awwwwww.a.aaa//o
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang. Chenyang Si. Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- fion, 2024.
Boyu Jia, Junzhe Zhang, Huixuan Zhang, and Xiaojun Wan. Exploring and evaluating multimodal knowledge reasoning consistency of multimodal large language models, 2025. URL https:
Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, and Seungryong Kim. Appearance matching adapter for exemplar based semantic image synthesis in-the-wild, 2025.
httttp::/aarxiv.cx/// abs/ / 2412 03150
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. Musiq: Multi-scale image quality transformer, 2021. URL htthh://///aat http://aaxvvvv.. org/ http/abs/2110 05997.
Taeksoo Kim and Hanbyul Joo. Target-aware video diffusion models, 2025.
URL https://
arxiv. wrg/ abs/ 250325531111011 18950
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprini arXiv:2412.03603, 2024.
Jaa- Yeon Lee, Byunghee Cha, Jeongsol Kim, and Jong Chul Ye. Aligning text to image in diffusion models is easier than you think, 2025, https:::::////tttt /tattts:///riivii wawiiv.ivvvv..//vv///aaaaaiaaaiv org/ /bwg0////////.../// 220./5553.000 08250
Feng Li, Renrui Zhang. Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-nextiimterleav:: Tackling multi-image, video, and 3d in large multimodal models, 2024. URL httht:tt///sxxttt www.ooggg//////oeeeeee.eeeeeee/e
abs wbw.///.. 07895
Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, and Zuxuan Wu. Magicmotion: Control-
lable video generation with dense-to-sparse trajectory guidance, 2025. URL https://arxiv.
org/abs/2503.
Shuang Li. Yilun Du Antonio Torralba, Josef Sivic, and Bryan Russell. Weakly supervised human- object interaction detection in video via contrastive spatiotemporal regions, 2021. URL https:
arxiv. org.
Zhen Li, Zuo-Liang Zhu, Ling- Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi field transforms for efficient frame interpolation, 2023. URL org/abs/ 2304. 09790
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, 2018. BtLhhtts::/ arxii.iiiv..//.i. hllgg/bbs/11 1108 02002
Kun Liu, Qi Liu, Xinchen Liu, Jie Li, Yongdong Zhang, Jiebo Luo, Xiaodong He, and Wu Liu. Hoigen- Im: A large-scale dataset for human-object interaction video generation, 2025. URL https: /tmpss//aaze /arrvvaivv.//..////v//aaa///aaav wrg.
hrgg/bbs.. abs / 01553.23777
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024. URL https: http::/ss//a.x...rr/.o..oo wtgggbg//abb....
05499
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Ticyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprint arXiv:23 10. 14440, 2023a.
13

---

Published as a conference paper at ICLR 2026
Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine- grained evaluation of open-domain text-to-video generation. arXiv preprint arXiv: 2311.01813, 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
171100
Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense- based benchmark for video generation. arXiv preprint arXiv:2410.0566, 2024.
Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang. Dream- matcher: Appearance matching self-attention for semanticalll--consisteen text-to-image personal- ization, 2024. URL
Jisu Nam, Soowon Son, Dahyun Chung, Jiyoung Kim, Siyoon Jin, Junhwa Hur, and Seungry- ong Kim. Emergent temporal correspondences from video diffusion transformers, 2025. URL
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang. Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-Im: A large-scale high-quality dataset for text-to-video generation, 2025.
URL httpss:/aaxii org. abs//40//77 02371
OpenAI. Introducing gpt-5, August 2025. August 2025.
OpenAl and Josh Achiam Gpt-4 technical report, 2024. URL URL/ttpss//axxvv.rrr///// 2333308774
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023. URL https:
//axxiv.org/abs.to... 09748
Huaijin Pi, Sida Peng, Minghui Yang. Xiaowei Zhou, and Hujun Bao. Hierarchical generation of hujen- interactions with diffusion probabilistic models, 2023. httpps/::///xx
org/abs/2300 22100 02242
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
https: http:///xxxxxxx./..aa... 2203. 00020
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao-Yuse Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam Segment anything in images and videos, 2024. URL bttps://axxiv..
orq/ bxq/abs//20 240..2448.000 00714
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersscionn over union: A metric and loss for bounding box regression, 2019. URL
https: http:://arxxv..rr/aabb/ wwwp:://///..........099... 09630
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL https://arxiv.. wttp//////aabaa/a// 02502
Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, and Sherry Yang. Videoagent: Self celf-improving video generation, 2025. URL https: arxiv.
org/ 24 10076
14

---

Published as a conference paper at ICLR 2026
Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M. Jorge Cardoso. Gener- alised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations,
pp. 240-248. Springer International Publishing, 2017. ISBN 9783319675589. doi: 10.1007 978-3-319-67558- 28. UtI.http://dxx /ttt////////..........//00000110
-
28
Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v- compbench: A comprehensive benchmark for compositional text-to-video generation. arXiv preprint htth:2407.11500. 2024.
Shuai Tan, Biao Gong, Yujie Wei, Shiwei Zhang, Zhuoxin Liu, Dandan Zheng, Jingdong Chen, Yan Wang, Hao Ouyang, Kecheng Zheng, and Yujun Shen. Synmotion: Semantic- visual adaptation for motion customized video generation, 2025. URL htttp:://arxa wwwp:/v.rrga/b//2500000 23690
Luming Tang, Menglin Jia, Qianqian Wang. Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion, 2023. URL htt.::ttt:://xxw..orrr////////// ab...r////bb....... 03881
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, 2020. URL 12039
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric challenges. arXiv preprint arXiv:1812.01117, 2018.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative 2025.
models. arXiv preprint arXiv:2503.20311
Yujie Wei, Shiwei Zhang, Hangjie Yuan, Biao Gong, Longxiang Tang, Xiang Wang, Haonan Qiu, Hengjia Li, Shuai Tan, Yingya Zhang, and Hongming Shan. Dreamrelation: Relation-centric video customization, 2025, URL htttps://arxiv wttp:r///ab//.....a///// 07602
Lehan Yang, Lu Qi, Xiangtai Li. Sheng Li, Varun Jampani, and Ming Hswan Yang. Unified dense prediction of video diffusion, 2025. URL / http:///aaaxvv.o/aa/////000000
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint +xXxx:2408.06072 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models, 2024. URL
04840
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang. Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think, 2025. URL /ttps://aate www.aiiiirg..iiooooooooooooooort abs /2100 06940
Xiangdong Zhang, Jiaqi Liao, Shaofeng Zhang, Fanqing Meng, Xiangpeng Wan, Junchi Yan, and Yu Cheng. Videorepa: Learning physics for video generation through relational alignment with foundation models, 2025. URL https::/ http://wwww..rr//////////522//./ 23656
org/ http////s//255...05
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training free controllable text-to-video generation, 2023. URL bttps://:///xxi org/ org/aba/2305 13077
Zhu Zhang, Zhou Zhao, Yang Zhao, Qi Wang. Huasheng Liu, and Lianli Gao. Where does exist: Spatio-temporal video grounding for multi-form sentences, 2020. URL https: //arsiv.orrg/ab//200.cooo 06891
15

---

Published as a conference paper at ICLR 2026
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng. Yu Qiao, and Ziwei Liu. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness, 2025a. http::///aaai...oo///b..st./// 21755
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. VBench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint arXi::2003.27755, 2025b
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint arXiv:2412.20400 2024.
Haiyang Zhou, Wangbo Yu, Jiawen Guan, Xinhua Cheng, Yonghong Tian, and Li Yuan. Holo- time: Taming video diffusion models for panoramic 4d scene generation, 2025. URL https:
16

---

Published as a conference paper at ICLR 2026
APPENDIX
In this material, Sec. A provides details of our MATRIX- dataset curation pipeline. Sec.
expands our analysis with additional visualizations and discussions and Sec. Sexeend our analysis to additional DiT-based video models, including Hunyuan Video-12V (Kong et al. 2024) and Wan2.1
14B-I2V (Wan et al. 2025). Sec. D and Sec. describe details of our proposed model and guidance strategy. Sec. F introduces details our novel interaction-awes -ware evaluation protocol, while Sec. G reports evaluation results across metrics, datasets, and human evaluation studies. Sec. [H presents additional qualitative and quantitative results with analysis. Finally, Sec. mentions the limitations of our work Sec. disscusssss the future direction of our work.
A DATASET CURATION DETAILS
As illustrated in Fig. of the main paper, our MATRIX-IIK dataset was curated and filtered through a step-wise process. We describe the detailed prompt design used when leveraging a large language model (LLM) (et al., 2024) and dision-langues model (VLM) (OpenAI & et al. 2024), along with the examples the resulting filtered data.
A.1 DETAILS FOR INTERACTION-AWARI CAPTION PROCESSING DETAILS
Interaction Identification and Instance Assignment. Fig. 26 illustrates prompt design for in- teraction identification and instance assignment. Given a natural- -lnguage prompt for video V, the goal of this stage is to extract the II) set K. and interaction triplets R. The first turn is validator that counts only active interaction linking a living subject to a distinct object via an ex- plicit action verb, rejecting self-directed actions, vague verbs, and internal states. Then it outputs A(P), which is the valid actions, or null none exist. Second turn then enumerates all instance mentions that participate in some E A(P), assigns stable instance index ie and semantic class clsy, forming the ID set IC = {kt cls²) le participates in some a}, and record role-type rela- tion as R. {(a, ksub; Kobj) normalized output used in practice: one JSON object per interaction containing the surface form (idX verb idY), subject and object IDs, an interaction-type label (multi-subject relation or func- tional action), and the exact source sentence span. These outputs IC and R serve as the formal supervision for all subsequent interaction-awart curation and evaluation steps.
E A(P), (Kiuhh (Kohj, -) E KJ. The figure also shows the
Interaction Scoring and Filtering. Fig. 27 presents the prompt design for interaction scoring and filtering. For each extracted interaction triplet (a, kaub, kobj) E R from the prompt of video V, LLM rater (et al., 2024) consumes the full textual context including prompt P, {idX verb idY). and noun descriptors of each IDs, and returns two integer scores E {1.... 5). Contactness quantifies physical contact or tight spatial coupling implied by the action (1 = no contact, 3 indirect/uncertain, 5 = direct/certain contact). Dynamism measures degree of motion or temporal change (1 static relation, 3 = low/moderate movement or readiness, = strong action/state change). addition, the rater also outputs a concise natural-language justification for its judgment and a self-reported confidence, which we use for auditability and to discard uncertain cases. Interactions judged to exhibit sufficient contact and motion are retained, and instances not appearing in any retained triplet are pruned.
Instance Description Extraction. Fig. 28 shows the prompt design for instance description ex- traction. Given the prompt P. a selected interaction triplet (a, kaub kobj). and the base nouns cls; for the participating IDs, the L.M rater al. 2024) produces, (noun, app, spatial). Here "noun" is short, visually discriminable noun phrase (e.g., "a man in a blue shirt"), "app" is a one-sentence summary salient appearance or physical attributes, and "spatial" is one-sentence statement of location role in the scene. De- scriptors are canonicalized, coverage-complete (one per ID), and linked to (k, clsı), redefining ID set as IC {kk, clst, desc;)), }. We this set to support grounding and to verify detected bound- ing boxes or masks by matching appearance and spatial cues, improving disambiguation among same-class entities.
every referenced instance k E K,,
a compact descriptor desc;
17

---

Published as a conference paper at ICLR 2026
A.2 DETAILS FOR INTERACTION-AWAL MULTI-INSTANCE MASK TRACKS WITH VERIFICATION
Why multi-instance mask tracks? While recent video generation methods leverage additional modalities such as optical flow (Chefer et al. (025) and depth (Yang et 2025), we adopt instance- level segmentation and tracking (mask tracks) as our core interaction signal. In our setting, the reference modality must (i) provide instance-level semantic precision to verify whether a specific object correctly grounded, (ii) exhibit temporal continuity so that the grounding can be tracked across frames, and (iii) disambiguate multiple instances that share the same class (e.g., two people). Among common cues such as optical flow, depth, and segmentation, only instance mask tracks naturally satisfy all three conditions: optical flow offers dense motion but instance IDs, and depth encodes geometry but neither instance grouping nor same-class disambiguation. Mask tracks, in contrast, yield clear instance regions and persistent IDs over time, which is exactly what we need to analyze and supervise "who does what to whom" in subject-object-verh interaction structures.
Interaction-aware multi-instance tracks with verification. Fig. 29 illustrates the prompt de- sign for rision-languen verification, which checks the consistency between bounding-boo visual prompts and instance appearance. We generate tracks in four steps.
(1) Class-only proposals. Given video V. its prompt P and its ID set = [(k, clsk, desc )} we uniformly sample T frames and run Grounding (Liu et al. 2024) with clss, only. For each frame i, it returns up to J candidates boxes (1)) , (.)), where Uy is box coordinate and d. E [o, 1] is the class-conditioned confidence for the given class clst. Thus, for each id k; the video yields most JT candidates across the T sampled frames. This class-only setting provides high recall but cannot disambiguate same-class instances and may still miss the intended target on difficult frames.
(2) Anchor selection and VLM verification. For each noun ID k we collect at most J x T candidates ((b%) , " over the T sampled frames. We sort them by confidence and define the anchor as the gighest-scoring pair as: (i*, ((* arg max;,, de with bj b5'"" We then query a vision- language model (VLM) (OpenAl & et al., 2024) with for inputs, including frame i", the crop from by.. the class name clsy and the descriptor descc and ask whether the crop matches the description of ID k. If the VLM verifies the match, we accept bf. as the final box for ID k and initialize SAM2 (Ravi ctt al. (224) propagation from that frame to obtain the mask track of the ID. If not, we move to the next candidate in descending ddd and repeat the aforementioned process. When no candidate is verified, the ID is dropped; if both subject and object dropped, the clip is excluded.
When multiple IDs share the same class (e.g.. two persons), verification is one-to-one: once a can- didate box is accepted for an ID, it is removed from the pools of the other IDs of the same class. This mutual-exclusion pruning prevents duplicate assignments and reduces verification cost from a naive J(I sean much smaller set of checks in practice, while keeping recall high and disambiguation accurate.
(3) Human verification. As final check, we run a lightweight but explicit quality-control pass on the verified tracks. For each clip, annotators review 10 frames, including the first verified frame, the last valid frame and eight uniformly spaced interior frames. They view the RGB, instance mask tracks and boxes, the union mask tracks and the triplet descriptor. Each track is labeled Accept (clean and consistent), Fix (minor boundary/jiterr; quick snap/smooth), or Drop (identity drift/swap, duplication, hallucination, or clear temporal gaps). A clip is used for supervision only if both subject and object are Accept after any minor fixes; otherwise it is excluded. For same-class IDs we enforce one-to-one assignment by dropping the worse of any substantially overlapping tracks. About 5% subset of accepted clips is spot-checked by a second annotator.
Effect of the proposed VLM verification. As described in Sec. of the main paper, we employ a VLM (OpenAl & et al. 2024) to verify and refine the error of GroundingDINO (Liu et al. (024). Fig. II illustrates why this step is necessary. With only a class name (e.g., person, man, cake), GroundingDINO frequently returns multiple same-class instances over pre-defined threshold and cannot single out the intended target. In Fig. II (a) captures both the person inside and outside the shop. (b) captures the photographer, the person being photographed and even reflection in phone. (c) captures every cake in view and (d) captures both the stylist and the client. A straightforward solution is add appearance phrases (e.g., the man outside the shop) to figure out the intended
18

---

Published as a conference paper at ICLR 2026
class: person appearance: the man outside the shop
class: person uppearance: the person being photographed
class: cake appearance: the cake the table with blue
class: person
appearunce: the person styling someone's hair
GroundingDINO with Class
GroundingDINO with Appeurance
DroOndingDINO with Class + Appearance verification with VLM (Ours)
(14)
(c)
(d)
Figure 11: Effects of Our VLM Verification. The first row (GroundingDINO with class) and the second row (GroundingDINO with appearance) often pick a wrong same-class instance. The third row (ours) verifies candidates with VLM and keeps exactly one box per noun, resolving (a) to (d). Best viewed in zoom.
target. However, is unreliable since GroundingDINO often latches onto partial tokens and ignores modifiers. For instance, in (a) it selects the man inside the shop by focusing on "man" and "shop" while missing "outside", and in (b) it selects the person taking the photo instead of the intended person being photographed. In (c) it selects wrong cake rather than the blue cake the table, and in (d) it still captures both people, failing to disambiguate the stylist from the client.
Rather than injecting appearance phrases into GroundingDINO, we use it purely as class-consisten proposal generator, since with class names alone, it reliably enumerates candidate boxes but cannot disambiguate same-class instances. Motivated by recent results (Cheng et al., (025 Jia et al.) 2025 Chen et al. 2025) showing that VLMs (Bai et al.] 2023 Li et al.) 2024 Ye et al. 2024 OpenA introduce VLM verification stage that cross-checks each candidate against descriptors, including appearance cues, and selects exactly one box per noun. If no candidate satisfies the verifier, we drop that instance and we remove the clip from the supervision. This preserves high recall from GroNndingDINO while delegating fine-grained disambiguation to the VLM, yielding cleaner per-instance tracks. As presented in the last row of the Fig. VLM evaluates candidates against the provided appearance descriptor and selects the final bounding box that matches the cue.
& al. 2024 Dai al. 2023) excel image and multi-image reasoning,
A.3 DATASET EXAMPLES AND STATISTICS
We provide more dataset examples in Fig. 30 Fig. 31 shows the overall statistics of our curated dataset.
Fig.
32 and Fig. 33 Additionally, Fig.
In Fig. 12 (a) summarizes the distribution of video-text sources used in our study. Our primary source is HOIGen (Liu et al. 2025), whose captions explicitly describe humans, human-object in- teraction, human action and scene descriptions. Therefore, the text strongly interaction- aware and provides dense cues for extracting interactions. Since HOIGen collects videos from diverse sources, spans everyday to highly specific scenarios and offers abundant interaction instances. To improve generalization and ensure data quality, we further incorporate PE- Video (Bolya et al.) 2025). a high- quality, carefully annotated collection that covers a wide range of categories. (b) reports the joint distribution of contact and dynamism score in our curated corpus. We score contact on a 1-5 scale (none contact-rich) and dynamism on a 1-5 scale (static highly dynamic). While the corpus in- cludes static non-contact cases, it is enriched for dynamic, contact-rich interactions. Crucially, within each contact level(from 1 to 5), dynamism spans a broad range, ensuring diverse motion intensities conditioned contact level. Additionally, (c) summarizes the distribution of per-video
19

---

Published as a conference paper at ICLR 2026
Dynamism
Internction
ID
15K
5K
PlVVddo
12K
4K
HORGen
6K
2K
3K
IK
Contact
5
Num
(4) Source dalaset distribution
(b) Conlact, Dynamism scure
(e) Interaction, ID count
(d) Top-50 frequent Interaction distribution
6.6
Top-50 most frequent distribution
Figure 12: Dataset Statistics.
counts of interactions (1-8) and identities (1-10). The mass concentrates in the 1-5 range for both, with clear modes at two interactions and two identities, indicating that pairwise subject-object set- tings dominate. Motivated by this distribution, we cap instance identities at |K[ = per clip: the annotator collects up five tracks and the model predicts up five instance mask tracks. This choice balances coverage and compute while remaining extensible, raising [AC] only increases the number of track slots without altering the rest of the pipeline. Moreover, clips with more than five interactions or instances are empirical outliers in Fig. 12 (c), providing evidence that such highly crowded cases are rare. When they do occur, we either split the video into shorter sub-clips or retain the top-k salient instances and aggregate metrics at the original-vide level. Considering with (a) and (b), these statistics indicate an interaction-denss yet not overcrowded corpus, aligned with our modeling in Sec. and evaluation design in Sec.
Finally, the dataset exhibits strong linguistic coverage. In Fig. 12 (d), we plot the top-50 interaction verbs. Since contact frequently entails "hold", that verb dominates. Excluding "hold", the remaining verbs follow a comparatively balanced distribution, indicating broad action diversity rather than reliance handful of predicates. Fig. 12 (e) shows the top-50 identity nouns. As interaction typically involves at least one human subject, nouns such as "man", "person", and "woman" are frequent. Nevertheless, object nouns broadly distributed, reflecting diverse targets and scenes. Together, (d) and (e) indicate wide linguistic coverage over actions and instances, supporting robust training and evaluation of interaction-aware models.
B ANALYSIS DETAILS
B. ANALYSIS EVALUATION DATASET
To faithfully evaluate interaction-awarr video generation, we curate dedicated analysis evaluation dataset rather than relying on real-world videos. Using real videos for reconstruction is problematic due to inversion errors (Song al. 2022), imperfect prompt-video alignment, and distributional drifts, making it difficult to isolate model behavior. To circumvent these issues, we curate a con- trolled analysis evaluation dataset designed simulate the generation process itself. By fixing random seeds during synthesis, we approximate near cear-eerfect reconstruction conditions. Human an- notators further verify the outputs, ensuring that only videos with high overall fidelity and consistent
20

---

Published as a conference paper at ICLR 2026
Prompe: collesgue in nany suit shakes
Genented
hands with the collengue gray suit the office."
Enitial
class: colleegue,
description: in navy suit and dark navy tic
[id]]
class: collengue,
description: in a gray suit and mediumm-gra tie
(1E2)
aclion : stakes hand,
Loction
instance :idl) id2)
Saction]
Prompt Construction
(b) Pscudo GT Generation
Figure 13: Analysis Dataset Pairs Example.
interactions are retained. Each video in the benchmark has a resolution of 480 720, contains 49 frames, and the final dataset consists of 50 carefully validated prompt-video pairs.
Scenario Design. The curation process begins with scenario design proposed by (OpenAl al. 2024), where we systematically specify the conditions of interaction to ensure diversity and cover- age. Specifically, we distinguish between unidirectional interactions, where subject acts upon an object (e.g., person pushing box), and bidirectional interactions, where both subject and object mutually influence each other (e.g., two people shaking hands). We then vary the number of partici- pating instances, ranging from simple subject-object pairs to multi-party settings with three more instances, which introduce additional ambiguity in role assignment. Interactions are further catego- rized into contact (e-g., touching, holding), force (e.g., pushing, pulling), transport (e.g., handling over, carrying), manipulation (e.g.. cutting, opening) and social (e.g., hugging, waving), thereby covering broad spectrum of physical and social dynamics. Finally, ensure class diversity by in- cluding human-object, human-human human-animal, human-nature interactions, encouraging gen- eralization beyond human-centric scenarios. Together, these design choices allow construct structured prompts that specify the instances, their roles, and their relations, ultimately yielding balanced of interaction scenarios for evaluation.
Prompt Construction. Given scenario, we then construct prompts that specify instance identi- ties (IDs), class labels, concise descriptors, along with the intended interaction, following the same principles
our dataset curation process described in Sec. 3.1 We first compose an im- age prompt that captures the static scene and instance attributes. Next, we derive a motion-award aware video prompt by adding action and relation clauses -subject-verb-obeeet with temporal qualifiers (e.g., contact). To improve synthesis stability and phrasing consistency, we apply VLM (OpenAl et al., (224))224aaaa -based prompt enhancement while preserving instance IDs and interaction roles. For controlled synthesis and verification, we generate videos with fixed random seeds and standardized rendering settings, holding resolution and length constant. Human annotators review each prompt- video pair for overall visual quality, semantic fidelity to the prompt. and interaction plausibility. Only pairs passing all criteria are retained in the analysis dataset. Fig. B3 presents an example produced by our prompt-cone construccntttttionn procedure.
Pseudo Ground-truth Generation. Finally, to quantitatively evaluate semantic grounding and semantic propagation, we produce pseudo ground-truth mask tracks for each instance, since syn- thesized videos do not contain ground-truth supervision. Following the same grounding- verification procedure used in dataset curation Sec. 3.2 in the main paper, we first extract candi- date bounding boxes using GroundingDINO (Liu et al. 2024), verify them with a vision-language model (OpenAl & et al. 2024) to eliminate irrelevant detections, and propagate the validated boxes using SAM (Ravi et al."
2024) to obtain ier-instance mask tracks. A final human verification step
ensures correctness of both instance identity and mask track quality, yielding high-quality mask
21

---

Published as a conference paper at ICLR 2026
tracks that serve supervision for interaction analysis. Fig. 13 (b) shows example constructed by our pseudo ground-trune truth generation.
As a result of the above systematic and precise procedure, we obtain images, prompts and per- instance mask tracks for each instance ID. We use this analysis evaluation dataset to evaluate se- mantic grounding and semantic propagation, as presented in Sec.
Real-domain A Analysis Evaluation Set. To validate whether our findings hold beyond controlled synthesis, we additionally curate real-domain using PE- PEdVideo (Bolya al. 2025) and Open- Vid (Nan et al. 2025). As discussed in Sec. reconstructing real videos via prompt inversion is prone to inversion errors (Song et al. 2022) since accurate text prompts are difficult to recover and the real-video distribution differs from the training distribution. Accordingly, we select video-text pairs whose captions instantiate our interaction schema, extract the captions our ID, role, and action format, and reconstruct each clip with an image-to-video (Yang et al. (0224) using the paired caption. Human annotators then verify that the generated clip preserves roles, and overall appearance; only verified pairs are retained. The same rater used in scoring and filtering provides contactness, dynamism and brief justification with confidence, and pseudo mask tracks are produced with the grounding and verification pipeline (Sec. 3) and checked for instance identity and mask track quality.
intended interaction,
B.2 ADDITIONAL. ANALYSIS
Frame
Franc
Frume
Σ ACEEC
Aubb
11
11
Fasl
V2t Attention
V21
Qc
q E Q+
( E BRNHK
A
Averaged attention of
(a) Grounding Attention Map Extraction
(b) Propagation Attention Map Extraction
Figure 14: Attention Map Details for Grounding and Propagation.
Fig. 14 visualizes the procedures used in the main paper to extract grounding attention maps from video-to- attention and propagation attention maps from vidco-to-video attention.
Metric Choice. Sec. 4 in the main paper, we introduce the Attention Alignment Score (AAS) the primary analysis metric. Our goal is to test whether the model's 's attention encodes "who does what to whom" " at the level of targeted instances and whether the targeted spatial region for each instance is preserved consistently over time. In other words, attention should be concentrated on the instances (e.g., subject, object) along their mask tracks, providing spatial binding within frames and temporal persistence across frames. We define AAS as the spatio-temporal inner product of y A v2v E {sub, obj, verb} and mask track, which measures how much attention mass is placed on the exact support of the instance over space and time.
This formulation is driven by the evaluation goal and by the normalization behavior of 3D full attention. Queries attend over the union of visual and text keys, and the softmax normalization taken across that union. In our setting, the text stream contributes roughly 226 tokens, whereas the video stream contributes about 1350 x 13 = 17550 visual tokens (1350 spatial locations per latent frame across 13 latent frames). Even when video-to-text attention is correctly localized, the relative scale of attention to text tokens might be compressed by this large cardinality imbalance. Preserving raw magnitude is therefore informative since it quantifies how much attention mass is allocated on the instance's 's track versus how much is allocated to non-target tokens and regions, rather than merely indicating whether there is any overlap with the binary mask track. AAS integrates attention
22

---

Published as a conference paper at ICLR 2026
on the mask track without thresholding or calibration, so it remains comparable across layers and is robust to this token-count imbalance.
A straightforward alternative is to treat A A
{sub, obj, verb) a soft segmenta-
tion sequence and measure overlap with its corresponding mask track using standard segmentation scores (Rezatofighi et al., 2019; Lin et al. 2018 Sudre al. 2017). For example, one threshold to obtain a binary sequence and compute loU (Rezatofighi et al.] 2019) against mask track, or use threshold-free scores such as BCE or Dice. However, these options either introduce sensitivity an arbitrary threshold or discard absolute magnitude and retain only shape overlap. The loss of mag- nitude particularly limiting under 3D full attention, where cross- mosdl competition suppresses text-side scales. At the opposite extreme, simply aggregating raw attention over the whole scene preserves magnitude but no longer tests whether attention lies on the intended instance trajectory.
AAS provides direct measure of what we seek to evaluate. Accordingly, we use AAS in Sec. 4 locate
layers and to link attention concentration with semantic grounding and semantic propagation.
Qualitative Link between Attention Alignment and Generation Quality. We provide qualita- tive evidence that attention-mask alignment relates to interaction fidelity. In the Fig. (b), video- to-text (v20) grounding improves generation when noun and verb attentions align with the subject, object ad union regions, whereas misalignment coincides with failures. Fig. 4 visualizes video-to- video (v2v) propagation maps. For each instance, (c.g., boy, girl), first frame mask pixels serve as query points. For each query, read its v2v attention to all spatial tokens across frames, reshape the result into an x x W map, and overlay on the video. In successful examples, attention initialized within the instance mask remains compact, follows the same instance through time, and yields clean, consistent clips. In failure cases, even with accurate first-frame grounding, propagation diffuses within the mask, leaks outside, or jumps to other regions. producing identity drift and hal- lucinated parts. These observations indicate that both v2t grounding and v2v propagation alignment matter for generation quality, which motivates supervising them explicitly with SGA and SPA.
Liyer-wise Analysis. Fig. 34 compares, for each noun and verb token, the attention maps from all 42 layers of naive CogVi VideoX--BBBBBB. -2- (Yang et al.) [024)). Two patterns emerge. First, small sub- set of layers shows strong alignment with the instance mask region for nouns and with the subject- object union for verbs. Second, many other layers exhibit grid-like responses consistent with posi- tional embedding effects rather than semantic binding. This heterogeneity implies that layers play distinct roles, so finetuning every layer can dilute or damage the layers that carry useful semantics. Notably, even vanilla Cog VideoX already displays alignment in several layers highlighted by our analysis (e.g., layers 7, 8, 9, 10 and 11), further motivating our focus on ineeractiondooinna layers.
C MATRIX ON OTHER VIDEO DITS
MATRIX is designed to be model-agnostic and can be plugged in any off-the-shelf video DiT ar- chitecture. To demonstrate this, we extend our analysis to two additional DiT-based image-to- -o-ieeeo models, including Hunyuan Video-12V (Kong et al. 2024) and Wan14B-12V (Wan et al. (025), and analyze their behavior under our framework.
V2deo-22.. In Fig. 15 we present additional analysis of Hunyuan Video- 12V using MATRIX framework. The backbone model consists of 60 layers, and for this experiment, we use 50 sampling steps. We observe a clear pattern that a small number of layers carry most of the interaction signals. For Hunyuan Video-12V, the dominant video-to-text layers are concentrated in the early stage (layers 6,7,9,10), whereas the dominant video-to-video layers appear in middle layers (layers 11, 26, 27, 29). This behavior is consistent with our findings on Cog VeeViicoo----1222 where video-to-text dominant layers are located at 7 and 11 and video-to-video dominant layers at 12 and 19.
MATRIX on
MATRIX on Wan2.1-14B-I2V. We present further analysis of Wan2. 11..--11B-111 using our frame- work. The backbone consists of 40 layers and we use 50 sampling steps in this experiment. How- ever, the architecture of Wan slightly differs from the other DiT-based models we analyze. Instead
23

---

Published as a conference paper at ICLR 2026
video-to-text attention
video-to-video attention
Mean
Mean
verb subject/objeet
Freq
Freq
Mean
Mean
Proq
Freq
(a) Influential Layer Candidates
video-to-text attention
video-to-video attention
10
20
.00022
26
-1000
27
verb subject/object
Success
25
Success
510G4
12333
Failure
Failure
JU8044
50001
03007
35
B0115
21
52
000035
Laye Isdox
Layrr lade
0
-50033
1003
11137
Success Failure
3.009
Success Failure
00155
35
21
SLATIS
m01
32
Laya Isdex
Lost Jokc
(b) Dominant Layer Selection
Figure 15: Layer Analysis of Mlnyuanniido--P1 (a) Influential layers, (b) Dominant layers.
of a single 3D full-attention block that jointly processes visual and textual tokens, it employs self- attention to process visual tokens and separate cross-attention modules inject textual informa- tion. Due to this architectural separation, the distribution of our interaction ciciiionawwees -aware evaluation differs from the 3D full-attention cases. Vog
(Yang et al., 2024) 12V (Kong et al. 2024), textual tokens occupy only small fraction of the joint token space (Treu out of x H x + Tiex), whereas in Wan2.1-14B-222 the video-to-text branch operates purely on the Text tokens. As a result, we observe layers whose video-to-text scores are almost constant across different prompts and success/failure cases. In other words, they are effectively insensitive to the interaction content. To quantify this behavior, across prompts and identify a subset of near- near-constant, prompt-insensitii layers, which we treat as "dummy" layers). As shown in Fig. 16 these dummy layers exhibit extremely low variance, espe-
Hunyuan Video-
analyze the variance of each layers' score
24

---

Published as a conference paper at ICLR 2026
Variance (log scale)
10
10
v2v Noun
10 26
v2t Noun
4
8
10
12
14
16
18
20
22
24
26
28
30
32
34
36
38
Layer
(a) Noun Variance by Layer
Variance (log scale)
11
10
v2v Verb
10 25
v2t Verb
12
16
18
20
22
26
30
32
34
38
Layer
(b) Verb Variance by Layer
Figure 16: Filtering Dummy Layers of Wan2.1-14B-122 (Wan et al. 2025). (a) Variance of noun- token cases across layers for video-to-text versus video-to- vi-ddo attention. (b) The analysis for verb-token cases. In both plots, several video-to-text layers exhibit almost zero variance (10 (10-2)), meaning their attention responses remain nearly constant regardless of prompt. These layers represented as gray points and are treated as dummy layers. In contrast, no such dummy layers are observed in video-to-video attention, so dummy layer filtering applied only on the video-to-text before selecting dominant layers.
cially in the video-to-text branch. We therefore filter them out before selecting influential layers, focusing our analysis on layers that genuinely respond interaction signals.
After removing these dummy layers, Wan2.1-14B-I27 still shows clear dominance structure. Video-to-video dominant layers are concentrated in the early part of the network (layers 6-8), while video-to-text dominant layers emerge much later (layers 20-24). Compared to CogVideoX: 12V or Hunyuan Video-12V, this shift is consistent with Wan't self-then-cross design, where earlier self-attention layers visual tokens mainly handle propagation across frames, while later cross- attention layers bind these interactions to the text. Fig. [42 and Fig. 43 provide additional qualitative comparisons between baseline (Wan 2025 and MATRIX.
D MATRIX FRAMEWORK DETAILS
D. ARCHITECTURAL DETAILS
We build on the pretrained CogVideoX-5B... (Yag et al. 2024) and retain its transformer blocks and 3D VAE except for the input pathway and small set parameter-efficiee adapters. Our network requires (i) the noise latent, (ii) first RGB frame and (iii) instance masks that supervise attention alignment. These signals are concatenated along the channel dimension and projected by the input projection layer of the backbone. To preserve the pretrained capability at initialization, copy the original weights into the slice that corresponds to the original channels and zero-initialinn the newly added channel kernels. This keeps the base behavior unchanged at step 0, while allowing the added channels to learn during finetuning. We attach adapters to the query, key, value and output projections inside the corresponding attention modules while leaving all other weights frozen.
To manipulate internal attentions without overfitting, we adopt LoRA (Hu et al. [02T]) on a minimal set of layers identified by our analysis. Layer 7 and layer I1 used for semantic grounding based
25

---

Published as a conference paper at ICLR 2026
vidco-to-text attcntion
vidco-to-video attention
Mean
vcrb subject/objeet
Freq
Mesa
60
50 100
Mean
Freq
Mean
Free
04
12
(a) Influential Layer Candidates
video-to-text attention
video-to-video attention
24
vcrb subject/objeet
28
10
21000
33988
411044
Success
14
Failure
Success Failure
20
53338
30
-1.0015
18
50old
31
TagerTadex
Taper 7v0e
20
24
14
Success Failure
39
11424
70
Success
Failure
Stood
18
Layer Isdo
Experiddec
(b) Dominant Layer Selection
Figure 17: Layer Analysis of Wan 14B-I2V. (a) Influential layers, (b) Dominant layers.
on video-to-text attentions and layer 12 used for semantic propagation based on video-to-video attentions. These are the only transformer weights that receive trainable LoRA parameters and the rest of the backbone remains fixed.
With these adapters, supervise attention directly rather than supervising proxy features. We add two lightweight decoders, a grounding head and a propagation head that read the query-key product scores from the targeted layers, such as layer 11 and 12, and convert them into alignment scores trained against binary ground truth mask tracks in RGB space while the generator remains unchanged.
For semantic grounding, uses the video-to-text attention where video tokens act as queries and instance token in the text keys. For semantic propagation, it uses the video-to-video attention
26

---

Published as a conference paper at ICLR 2026
that links each location in one frame to matching locations in the next few frames and checks whether the same instance persists over time. After computing the query-key product we reshape the result to the backbone spatiotemporal token grid so that each value aligns with a patch and frame. We then take a simple mean across attention heads and feed the resulting map to a lightweight decoder. The decoder serves as supervised readout that turns token space attention into dense alignment scores against binary mask tracks. This separation lets the alignment loss update only the query and key projections in the adapted layers preserves the pretrained behavior at initialization and allows the grounding and propagation heads to be removed at inference when only generation is needed.
Both heads follow the upsampling strategy and the time causality used 3D VAE YYang et al. 2024) while remaining lightweight. A standard 3D VAE temporally compresses several frames into one latent which places attention on a shorter temporal lattice than the ground truth instance mask tracks. In Cog VideoX, the VAE temporally compresses frames from 1 + 4F to 1 + F, which reduces the effective frame rate of the latent sequence by a factor of 4. This places attention on a shorter temporal lattice than the ground truth binary mask tracks. In our setup, the latent attention sequence spans 13 steps, whereas the ground-truth instance mask tracks span 49 frames. The most straightforward solution to address this gap is to compress supervision by taking element-wiss, OR over every 4 consecutive frames so that each group maps to one latent step. However, this ignores temporal ordering and inflates foreground regions which weakens alignment under motion and degrades identity precision. Instead, we upsample the attention to the mask frame rate. The lightweight decoder mirrors the VAE temporal up path and causally expands the 13 step attention sequence to 49 frames without using future frames. Supervision is then applied at the original frame rate against the binary instance mask tracks. This preserves temporal ordering and sharp instance boundaries, avoids foreground inflation, and leaves the generator unchanged while confining updates to the query and key projections in the adapted layers.
D.2 IMPLEMENTATION DETAILS
We use the Cog -SBe0X- (Yang et al. 2024) as our base image-to-video diffusion model, and generate output videos at a resolution of 480 720 with a total of 49 frames. The trainable parameters are limited to the selected LoRA (Hu et al. 202T) layers (layer 7, 11, 12), the input projection layer and lightweight decoder heads for grounding and propagation heads. For model finetuning, we adopt LoRA (Hu et al. 2021) with rank of 128 and a 64. We optimized only the selected LoRA layers, input projection layer and lightweight decoders while keeping the other parts of the model frozen. Training was conducted on our curate dataset, MATRIX-11K using an AdamW (Loshchilov & Hutter, (019) optimizer with a cosine learning rate decay schedule. The model is trained for 4, 000 steps, which takes approximately 32 hours on a single NVIDIA A6000 GPU.
We apply Semantic Grounding Alignment (SGA) loss and Semantic Propagation Alignment (SPA) loss selectively. The SGA loss supervises video-to-text attention in blocks and 11. The SPA loss supervises video-to-video attention in block 12. This selective strategy concentrates updates on the query and key projections of the adapted layers, stabilizes optimization under motion and preserves the pretrained generator at initialization and at inference.
E TRAINING-FREE CROSS-MODAY GUIDANCE DETAILS
Our analysis shows that semantic grounding (video-to-text) and semantic propagation (video-to- video) are concentrated in small subset of interaction-dmminad layers. To validate whether these layers provide effective handles for improving interaction fidelity, we design a zero-shot guid- ance strategy applied only at the identified layers. Specifically, we introduce Cross-Modal Guid- ance (CMG), our novel approach for enhancing grounding, and adopt Cross-Attention Guidance (CAG) NNam et al. 2025) for propagation. CMG perturbs token-to-entity attention maps at domi- nant video-to- ditoo-to-text layers to simulate degraded grounding and then guides the model away from these perturbed predictions, reinforcing semantic alignment. In parallel, CAG applies the same perturb- and-guide principle to cross-frame attention, reinforcing temporal consistency without additional training. Fig. shows the architectural details of CAG and CMG.
27

---

Published as a conference paper at ICLR 2026
2u
Luideott11
Ap
V1,
Koloeree Rinn..) (Rrahean Recons))
Kej
Ke
V2:
Lydcot
Zeroing Cruss Frome Attention
Zyiddo.t
Zeroing video-to to-to-t--teed-- Altention
(a) CAG
(b) CMG
Figure 18: Guidance Details.
ARCHITECTURAL DETAILS
Cross-Attention Guidance (CAG). Inspired by PAG (Ahn al. 2025). which enhances image fidelity by transforming selected selffattention maps into identity matrices, we extend this idea to the video DIT architecture. PAG, identity matrices are created by multiplying a diagonal mask into the attention map before the softmax operation, setting diagonal elements to O and off-diagonal to -OG, which yields an identity matrix after softmax. A naive extension to video assigns -oo to cross- frame positions, but this undesirably suppresses self-frram and text-frame scales. To address this, DiffTrack (Nam et al. 2025) zero out only the cross- frame values after softmax in A producing modified maps A v2v that preserve other interactions.
Cross-Modal Guidance (CMG). Analogous to CAG, CMG applies the perturb-and-guide strat- egy video-to-text attention. At interaction-Dominant layers, we simulate degraded grounding by zeroing out instance alignments after softmax. For noun tokens, attention weights to instance regions are suppressed; for verb tokens, attentions capturing subject-object unions are removed. This produces modified maps A where semantic grounding intentionally weakened, while other attentions remain intact. The diffusion model then guided away from these degraded pre- dictions, reinforcing correct grounding without retraining or auxiliary conditions.
Both can be formulated as:
Ep (Zvidon,, Zexxt, 1) = eg Zvidoo,t Ztext; 1) + (ee (Zvidon,,s Zeeot, 1) - Epl (Zviden,ts Zeextt ());
where cp(-) is the noise prediction from a standard pass at timestep t, conditioned on the text, and (-))) indicates the noise prediction from a perturbed forward pass. is the perturbation guidance scale and the final prediction (-) is guided away from the degraded predictions.
E.2 IMPLEMENTATION DETAILS
video-to-video layers (e.g.. layer 12 in CogVideoX: 5B-I2VV identified by our analysis, and apply guidance across all sampling steps.
For CAG, adopt the I interaction-domiin
For CMG, similarly select the 2 interaction-dominant video-to-text layers (e.g., layer 7 and II CogVideoX-5B. -2V) and apply zero-shot guidance at every timestep. Both guidance scales are set following PAG (Ahn al. 2025), and additional parameters training are introduced.
E.3 EXPERIMENTAL RESULTS
In Fig. 19 we diagnose the failures through attention. the first row of (a), for "woman cuts cake", noun attention for woman leaks onto the man and the verb focuses him rather than the union of the woman-cake region, so the action is assigned to the wrong agent. In the second row of (a), noun attention to the subject man weak and diffuse and video-to-video attention does not carry
28

---

Published as a conference paper at ICLR 2026
(a) Cog VSgViooXX----I2V (Baseline)
(b)
Pross-Modal Guidance (CMG)
"woman"
"cuts"
Generated
"woman"
"cuts"
Generated
First Frame
Prompt:
woman white dress cuts wedding cake with knife she holding.
"man"
"receives"
Generated
"man"
"receives"
Generated
First Frame
Prompt: A man red floral shorts receives falling volleyball.
Figure 19: Cross-Modal Guidance Visualization.
Table Comparison of evaluation protocols. Existing benchmarks assess quality, compositional-
ity, or physics, but only InterGenEval targets interaction-level semantic alignment.
Protocol
Target
Semantic Granularity Temporal Semanties
Semantic Aligument
VBench
Visual Quality
Global (frame/clip)
Global appearance
VBench.2.0
Fanthfolness
Global Semantic
Human, controllability, physics
EvalCrafter
Quality & Alignment
Global (entity cues)
Basic visual-tent alignment
FETV
Attributes
Entity (attributes)
x Partial
Attribute level alignment
T2V-CompBench
Compositionality
Relation (multi-object)
Multi-object relations
PhyGenBeuch
Physics
Event (physics)
Physical plausibility
Phy WorldBench
Physics
Event (physics)
Physical plausibility
InterGenEval (ours)
Interaction Fidelity
Interaction-Iveel
Interaction-Iewe alignment
stable subject track forward, so the motion does not start. These cases show that when grounding is weak, propagation also breaks.
We then apply perturbation guidance only the interaction-dominant layers identified by our anal- ysis and leave all other layers unchanged. The guidance biases video-to-text attention A v2t toward the intended subject, object and their union and stabilizes the carry-over in video-to-video attention with a small weight to avoid appearance drift. Under this setting, many borderline cases flip from failure to success. In the first row of (b), this sharpening results in the woman executing the cut with contact maintained across frames and in the second row of (b), the man is cleanly localized from the first frame and the motion initiates and proceeds without drift. The fact that a lightweight in-layer perturbation cleans up video-to-text and andeo-to-vide attention and improves plausibil- ity, frequently turning failures into successes, shows that these layers are the dominant handles for attention sharpening well for grounding and propagation.
thee critical limitations remain. CMG is zero-shot guidance that amplifies existing atten- tions selected layers, but does not inject region-level ID-level supervision. When the initial noun map is severely ambiguous, when the verb is not well approximated by the subject-object union, or under heavy occlusion, sharpening may be insufficient or may over-concentrate atten- tion and subtly degrade appearance. Moreover, increasing the guidance scale to compensate often saturates attention and collapse diversity. Therefore, these observations motivate our mask-track alignment losses that provide explicit grounding and propagation signals, as depicted in the Sec. 5 in the main paper.
F EVALUATION PROTOCOL DETAILS
F.1 RELATED WORKS
Early evaluations of video generation primarily relied on Inception Score (IS) (Salimans et al., 2016 Fréchet Inception Distance (FID) (Heusel et al. 2017). and Fréchet Video Distance (FVD) (Un- terthiner et al. 2018), which measure distributional fidelity and diversity but fail to capture mantic correctness. To address this, recent benchmarks introduced multi dilei--imensional protocols.
VBench (Huang et al. 2024) decomposes generation quality into 16 dimensions, including frame
29

---

Published as a conference paper at ICLR 2026
Fist fram:
MAURUX(
Cog -B-22-
Fint frame
MALRUXiours) 30th framse MAL MALKIX REX 40 Eracte
E
CLIPScore
CLiPScore ILIP
30,94
32.55
BLIP-BLU
0.019
(.0599
0,0001
0.0816
Prompt: The pualass ansther en
Prompt Winmant in back ahin
BBS thesed book library
Score misa misaligneeen between modals
(b) Score missligmment between frames
Figure 20: Limitations of Existing Semantic Alignment Metries using BLEU and CLIP. (a) Cross-model comparison: despite clear human preference for one model, BLEU and CLIP favor the other, assigning high scores implausible or semantically misaligned results. (b) Within-model frames: frames preferred by humans receive lower scores than other frames from the same clip, showing insensitivity instance-level grounding and temporal consistency. This gap motivates EnterGennval, an interaction-aves evaluation.
aesthetics, temporal consistency, and prompt adherence, and validates alignment with human judg- ments. EvalCrafter (Liu al., 2023a) further integrates large prompt suite and combines multiple automatic metrics with human preference weighting. FETV (Liu et al. 2023b) emphasizes attribute- level evaluation, scoring static and temporal quality as well as ane-grained alignment. These works broaden coverage beyond single-number scores but remain global or attribute- these benchmarks rely heavily on -Lseebased models to assess semantic similarity. However, as shown in Fig. 20, CLIP score and the BLIP-BLEU score from EvalCrafter fail capture interaction level granularity, highlighting their limitations in evaluating the interaction modeling capabilities of generated videos.
Many of
Other benchmarks target narrower capabilities. 12V-Compennch (Sun et al. 2024) measures com- positionality over relations, attributes, and actions through VLM-based and detection-based metrics. PhyGenBench (Meng et al. 2024) and Phy WorldBench (Gu et al. 2025a) evaluate physical com- monsense and causal plausibility with structured protocols, while V VBench-2.0 (Zheng al. 2025b) expands toward "intrinsic faithfulness", covering human fidelity, controllability, and physics. These efforts highlight compositional and physical reasoning. but still do not capture whether models real- ize prompt-specified interactions.
In particular, as compared Tab. 3 existing protocols assess global semantics or object attributes but lack inferaction aware semantic alignment: whether the correct subject acts on the correct object, contact occurs, and causal unfolding matches the prompt. Our proposed InterGenEval addresses this gap by treating interactions the evaluation unit and introducing grounded criteria for role- and time-sensitive alignment.
Vidco Frames
(a) KISA
(b) SGI
(c) SPI
Generate questions key intersction Video Frames following:
10
Q1. Pre-interaction state
Q7. Action by Agent
QJ8. Currect Agont
Q2-05. Progresvive interaction
Recipiist
Q6, Port-isterac tion state
Q10, Reciptow confinmation
Temporal Consistency expert
Q1 Is A not touching before interaction?
QZ. Is move hishher hunds toward B?
Q7. A taking action?
Evaluste Video Frames relative ancher
Q3.
contact B?
Q8. A the one liftnng
whether
Q4. Dozs A hold B with hisher hand?
B being lifled A?
1. entity emerged without a bounding box
Q5 Does start to lift B?
Q6. vaitl off groud being by
Q10, boing lifle??
2. Any previuusly makked instance disappeured frumes
Interaction aware Semantic Alignment expert
Evalunta Video Frnntes deternine
questioes answer Yis"
Emergence ratio ( (5b)
Disappearance ratio (%)
KISA
raw
SGI
SPI [((1 2- Emergence ratio)
SPI
x SPI
+ (1 - Disappearance ratio))
KISA
IF
SGI
Figure 21: Evaluation Protocol Pipeline.
30

---

Published as a conference paper at ICLR 2026
F.2 OVERVIEW
InterGenEval focuses on interaction aware semantic alignment between the video and the prompt, measured by two metries: Key Interaction Semantic Alignment (KISA) and Semantic Grounding Integrity (SGI). Specifically, after extracting key interactions from the prompt, KISA verifies step- by-step whether the subject actually performs the specified action on the object, while SGI assesses the grounding accuracy of the subject and object. When multiple key interactions are present in a prompt, each key interaction is evaluated to obtain its corresponding KISA and SGI, which are then averaged across all interactions to produce the final KISA and SGI. This enables evaluation in multi-interaction and multi- multii-nsstance scenarios. Fig. 21 providees the details of the evaulation protocol pipeline.
Meanwhile, maintaining temporal consisteney of interaction and grounding is also crucial. To ac- count for this, introduce Semantic Propagation Integrity(SPI) a sub-metric. SPI captures whether any instance suddenly appears or disappears throughout the video, providing measure of temporal consistency. SPI then applied to KISA and SGI, injecting temporal consistency into both metrics by penalizing inconsistent instance propagation overtime. The mean of KISA and SGI is then defined as the final Interaction Fidelity (IF) score. For clarity, we denote the unadjusted scores as KISAraw and SGI and SPI applied scores as KISA and SGI.
Setup. InterGenEval leverages multimodal foundation model GPT-5 (OpenAl 2025). utilizing its strong visual understanding and reasoning capabilities throughout the evaluation process. KISA and SGI are computed through question-answering framework, which verifies whether the subject actually performs the intended action and whether the subject and object are correctly grounded. Additionally, SPI is derived through an instruction- evaluation procedure, where GPT-5 is used detect the emergence and disappearnace of each instance across frames assessing temporal consistency.
InterGenEval uses sequence of frames where each instance involved in the interaction is visu- ally annotated with a bounding box in a distinct color. We generate these annotated frames using SAM2 (Ravi et al. 2024), which allows to extract precise bounding boxes for each instance. This visual representation enables GPT-5 to clearly identify each instance and focus on fine-grai fine-grained in- teraction details. Each evaluation frame sequence includes the first and last frames of a video, while intermediate frames are uniformly sampled using fixed stride. In this paper, the stride is set to 5.
F.3 EVALUATION METRICS
Question Generation. Since KISA and SGI are derived from guestion-answering framework, it is important to construct a well-structured set of questions that reflects whether each step of the interaction is performed and whether all instances are correctly grounded. To this end, we use GPT-5 to automatically generate 10 yes/no questions per key interaction, guided by a task-specific instruc- tion. As input, GPT-5 receives the text prompt, a list of instances, their corresponding appearance descriptions, and assigned bounding box colors. Based on this input, GPT-5 first identifies key in- teractions described in the prompt. Then for each key interaction, it generates aligned with the evaluation goals of KISA and SGI. Each question explicitly refers to instances us- ing both appearance and bounding box color(e.g. woman in a green jacket (red bbox)). The first six questions (Q1-Q6) are used to compute KISAraw- as they assess whether the interaction progresses through its expected stages. The remaining four questions (Q7-Q10) focus on verifying instance grounding and are used to compute SGT Further details on the structure of these questions and the computation of KISA and SGI are provided in the following section.
questions that are
Key Interaction Semantic Alignment (KISA). KISA evaluates an interaction by decomposing into three temporal stages: pre, during, and post interaction. Question corresponds to the pre- interaction stage and checks whether the subject and object are in the expected initial state prior to any engagement. Question 2 through 5 cover the during-interaction stage, where the model verifies the progression of the action across multiple steps. Finally, Question 6 focuses post-interaction stage, assessing whether the expected outcome of the interaction has been visibly achieved. For example, in the case of the interaction "A lifts B", the six questions will be constructed as follows. Q1. Is A not touching B before interaction? Q2. Is A move his/her hands toward B?, Q3. Does A make contact with B? Q4. Does A hold B with his/her hand? Q5. Does A start lift B?
the
31

---

Published as a conference paper at ICLR 2026
Q6. Is B visibly off the ground and being held by A? KISAraw is then computed the proportion of "Yes" responses among these six questions, indicating how successfully the interaction is executed across all expected stages.
Semantic Grounding Integrity (SGI) SGI evaluates whether the subject and object are correctly grounded within the interaction. To this end, it comprises four questions. Question 7 verifies whether the subject is correctly identified as the actor of the interaction. Question 8 checks whether the subject performs the specified action object is being acted upon by the specified subject. Question 10 assesses whether the object is indeed the correct recipient of the action. For example, in the case of the interaction "A lifts B", the four grounding questions will be constructed as follows. Q7. Is A laking an action? Q8. Is A the one tisting B? Q9. Is being lifted by A? Q1O. Is being lifted? SGI is then computed as the proportion of "Yes" responses among these four questions, capturing the accuracy of instance level semantic grounding within the interaction.
the intended object. Question 9 evaluates whether the
Semantic Propagation Integrity (SPI). SPI measures the temporal consistency of each instance throughout the video. The first frame is used anchor, and the remaining frames are compared against it to detect any changes. We provide GPT-5 with a list of instances, their bounding box colors, and the bounding box visualized frame sequence as input. GPT-5 then outputs the detection results for emergence and disappearance for each frame. Specifically, emergence is defined as the appearance of a new instance that does not appear in the anchor frame but emerges in later frames. Disappearance occurs when an instance annotated with bounding box in the anchor frame is no longer visible in subsequent frames. To compute the SPI score, we first calculate the ratio of frames in which emergence or disappearance is detected. Each of these ratio is multiplied by a penalty weight X, and the result subtracted from I to obtain the emergence score and disappearance score, respectively. The final SPI score is defined as the average of these two scores. In this paper, we set A=
Overall Scoring. As previously mentioned, we use SPI to incorporate temporal consistency into KISA and SGI. SPI ranges within (-4,1], with higher values indicating better temporal consistency of instances. To strongly penalize videos with poor temporal consistency, we multiply SPI with both KISAraw and 1GI to obtain their reweighted final values.
KISA KISA SPI,SGI = SGI SPI.
The final Interaction Fidelity (IF) score is then computed as the average of the reweighted KISA and SGI.
IF KISA + SGI
IF combines KISA, SGI, and SPI to provide a quantitative score that reflects interaction aware se- mantic aligmment with temporal consistency. This formulation offers an interpretable and consistent metric for assessing interaction quality. As a result, InterGenEval functions as a practical evaluation framework that gives precise feedback on the quality of interaction-awan video generation.
G EVALUATION
G.1 COMPARISON MODELS
We compare our approach against several recent open-source image-to-video diffusion mod- els including Cog VideoX-2B-12V (Yang et al.] 2024), Cog (Yang et al.) 2024). VaVid (Kim & Joo 2025)and Open-Sora (Zheng et 2024). CogVideoX-2B-12V is lightweight version with approximately billion parameters, designed for efficient video synthesis. In con- trast, Cog VideoX-SB-12V scales to 5B parameters and offers stronger generative capacity through larger model size and broader training coverage. Finally, we include Open-Sora (11B) as fully open-source alternative, widely adopted as community benchmark. Collectively, these compar- ison models span a spectrum scales, training regimes, and accessibility levels, enabling us to evaluate both the absolute quality of our method and its relative efficiency against existing models.
32

---

Published as a conference paper at ICLR 2026
G.2 ADDITIONAL METRICS
In addition to proposed protocol, adopt several metrics from VBench (Huang al., 2024) and VBench-2.0 (Zheng et al., 2025b) to provide broader evaluation of video quality. VBench decomposes video quality into temporal and frame-wise aspects.
For temporal quality, Subject Consistency measures whether the main subject maintains a stable appearance across frames, computed via DINO (Caron al. 202T) feature similarity. Background Consistency evaluates the stability of the background using CLIP (Radford et al., (021)) feature similarity. Motion Smoothness quantifies whether motion physically plausible and continuous, using motion priors derived from a video interpolation model (Li et al. (202). Dynamic Degree measures the amplitude of motion in the generated video, estimated with RAFT (Teed & Deng. 2020)-based optical flow.
For frame-wise quality, Aesthetic Quality captures perceptual attractiveness such as composition and color harmony, evaluated with the LAION aesthetic predictor (Beaumont & Schuhmann, 2022) Imaging Quality assesses low-level fidelity by detecting distortions such as blur, noise, or over- exposure using MUSIQ (Ke et al. 2021) trained on the SPAQ (Fang et al.) (2200) dataset.
From VBench-2.0, we additionally include Human Anatomy, which evaluates whether human in- stances are consistently maintained without abnormal merging, splitting, deformation across frames. This is achieved by detecting humans, hands, and faces with COLO-World (Cheng et al.) 2024), and applying anomaly detectors trained large-scale dataset of real and generated human samples. The final score is defined as the proportion of frames not flagged as abnormal.
G.3 EVALUATION DATASET
Fig. illustrates the benchmark we used for evaluated, consisting of 118 image-prompt pairs. These pairs were constructed by selecting images with varying number of instance IDs (2, 3 or 4), and by categorizing motions from simple to complex based on levels of contact and dynamism, such as "walking along the street" (low contact and low dynamism) or "hands over the cup" (hight contact and hight dynamism). Each prompt was designed include (1) main subjects and objects involved in the interaction, (2) the interaction or motion descriptions between the main subjects and objects, and (3) a scene description specifying the appearance of the main instances. For all images, we used a large language model (LLM) to generate prompts that satisfy these conditions, following the same guidelines used during our dataset curation process in Sec. 3 of the main paper and analysis evaluation dataset curation process in Sec of Appendix.
G.4 ADDITIONAL ANALYSIS
To further validate the reliability our evaluation, we visualize the full per-clip distributions of the normalized KISA. SGA, and IF scores for all models on our interaction-aware evaluation set, as presented in Fig. 22 Each violin is computed from all clips in the benchmark, and summarizes the distribution of scores for given model- metric pair.
We observe that the distributions for MATRIX are consistently skewed toward higher scores with less mass the low-score region across all three metrics, while baseline and other comparison models exhibit longer tails toward low values. This indicates that our model not only improves the mean performance reported in the main paper but also reduces variance and the frequency of severe failures. In other words, MATRIX achieves more reliable interaction- aware generation, leading to produce high-quality, interaction-consiiten videos, rather than relying on few outlier successes.
G.5 HUMAN EVALUATION
Human evaluation details. We adopt Two-Alternative Forced Choice (2AFC) proto-
col Blattmann et al., 2023 Chefer et al. 2025), where raters compare two videos bid and select the better one. Two models are uniformly sampled from (CogVideoX-SB-I2) (Yang et al. 2024), Open-Sora -2V (Zheng et al. 2024), TaVid (Kim & Joo 22025. Ours), yielding all six model pairs. For each sampled pair, randomly select text-image prompt from the InterGenEval
33

---

Published as a conference paper at ICLR 2026
0.6
0.4
1.0
Normalized Score (0.0
08-
04
0.-
0.0
0.8
0.6
0.4
0.0
CopVideoX-8--12
Coglideox-5B-22
Figure 22: Per-Clip Distributions of Interaction-aware Metrics. Violin plots of normalized KISA, SGA, and IF scores for MATRIX and four comparison models( Yang et al. (2024), iZhng al. (2024), Kim & Joo (2025)) Each violin summarizes the distribution per-clip scores over all evaluation videos, and the black vertical line indicates the overall range observed for that model- metric pair.
evaluation and generate video per model using the same prompt. The left/right presentation order is randomized to avoid positional bias, and raters are not allowed to skip or assign ties.
Each trial consists of five evaluation questions: (1) Interaction Accuracy correctness of the spec- ified interaction (who interacts with whom and what they are doing); (2) Semantic Grounding inclusion of objects indicated in the image prompt as instructed by the text prompt; (3) Semantic
Propagation temporal consistency and absence of hallucinated objects: (4) Semantic Alignment overall fidelity and naturalness of the interaction; (5) Overall Quality perceptual realism and visual plausibility.
Each participant evaluated six model pairs with two prompts per pair, resulting video comparisons (12 pairs) per participant. This design ensured equal comparison frequency across models, providing balanced and fair evaluation protocol.
12
Participants. We recruited 31 participants, each responding to multiple trials cover pairwise comparisons under diverse prompts. Results are aggregated using the win rate across pairwise comparisons and criteria, fol- lowing standard practice in perceptual evalua- tion. Fig. 39 illustrates the 2AFC setup.
Interaction
CogVidenX-59
Accuracy
Open-Sora
TaVid
Overall Quality
Somantic
Groending
Human evaluation results. Fig. 23 sum- marizes the win rates. Our model (MA- TRIX) consistently exceeds 0.9 across all cri- teria, while its backbone CogVideoX-5B-INY remains around 0.36-0.44. This demonstrates substantial improvements in interaction-aware
Semantic Alignment
Semantic Propagation
Figure 23: Human Evaluation Results.
34

---

Published as a conference paper at ICLR 2026
Table 4: Additional Quantitative Comparison.
Human Fidelity
Video Quality
Methods
HA (1)
SC
BC (1)
MS
DD
AQ
IQ (t)
Vig ---112 Yang et al. 2024
0.937
0.969
0.962
0.993
0.152
0.602
69.69
Cog VedeieooX-SB -B-12X- Yane 2024
0.942
0.556
Open-Sora-I2V Zheng 2024
0.893
0.926
0.937
0.992
0.762
0.495
63.32
TaVid Kim & Joo 2025
0.919
0.942
0.939
0.991
0.727
0.568
68.90
MATRIX (Ours)
0.954
0.962
0.956
0.994
0.492
0.587
69.73
semantic alignment, covering interaction accu- racy, grounding, propagation, and alignment well as perceptual quality. Other baselines, such as Open-Sora and Vid, show even lower per- formance. Overall, MATRIX not only inherits the strengths of Cog VideoX but also delivers robust interaction fidelity and perceptual realism, validating the core contribution of our approach.
H ADDITIONAL RESULTS
H. ADDITIONAL QUALITATIVE RESULTS
Fig. 40 and Fig. 41 present the additional qualitative results comparing method with oth-
ers, including Cog Videoo VideoX-5B-12V YYnn et al. 2024), Open-Sora-12V (Zheng et al.] (202) and
Ta Vadid (Kim & Joo 2025). To further verify that our improvements are limited human-object contact, Fig. 14sreports results on dedicated evaluation subset consisting soiely on non-human and non-contact interaction scenarios.
in gray hoodic painting finger anoahie Ila
a
robutics was vallapooe
Bobal picks up cenamic cup
tray using
stand by nean siges."
its two-Engerol gripa
molling billiand ball plides the table toward another
small golden retrient gently bumpi with Blakkamd robiite bonder wagging excitedly as they greet."
brunch neck bells, White
moving only gold beil."
Worman between lamp green lamp clearty pointi
Bliding glass kitehen counter slowly soward lightly tapsia
linger loward only the ged lamg
cousing the shaker till.
(a) Noo Contact Secturio Results
(b) Non Humann Scenario Results
Figure 24: Generalization to non-contact and non-human interactions.
H.2
ADDITIONAL QUANTITATIVE RESULTS AND ANALYSIS
Tab.
reports additional quantitative comparisons across CogVideoX-255 --2-12 (Yang et al. 2024 AidexX-5B-12% (Yang et al. 2024). Open-Sora-12V (Zheng et al.] 2024). TaVid (Kim & Joo and MATRIX (Ours), across the standard metrics of VBench (Huang al. 2024).
Cog
2025
SC (Subject Consistency) and BC (Background Consistency) are highest for CogVideoX-288 but this stems from its near-static outputs rather than stronger modeling. As shown in Fig. 25 little
35

---

Published as a conference paper at ICLR 2026
CogVideoX
Promptt The
silver mug sip.
finmme (ATRIX(uus))
Frime
Frame
Last frame
quality 0.555
Subject consitency:
Baciggruud consistency:: 0.961
Modion smoothness 0.594
Prompt The Wocmam in the wide brimmeed hat miises her silver travel mug to take sip.
First firme
Frame 10
Framo 20
Frim: 30
Framo 40
1et framo
Image quality 25,042
quality
Subject consisency: 0.882
Bockkgrund consisency" 9.913
Motion smoothness
Figure 25: Limitations of VBench metries.
Table 5: Training Cost and Efficiency Comparison.
InterGenbEaal SGI (T)
Human Fidelity
Video Quality
Methods
Timeloooch
KISA (t)
IF
HA
MS (T)
(7)
(
Baxellin Yong al. bro4
0.406
0,491 0.525 (+7.1%)
0.44
0.936
0.987
69
LoRA wo nager
br
0.455 (-9.6%)
0.486 (+8.2%)
0.400 (+0.19)
0.996 (-0.7%)
69.77 (+1.6%)
Ours
hr
0.546 (+34.5%)
0.641 (+30.5%
(+32.1)))))
0,954 (+1.%)
0.994 (+0.7%)
69.73 (+10%%
or no motion inflates inter-frame consistency and keeps AQ and IQ high because there minimal motion-induced degradation. The motion metric, Dynamic Degree (DD), confirms this with very low values. Very high DD is not always desirable either, since excessive motion increases off-track drift and hallucination risk. In Fig. 25 when comparable motion is introduced, SC, BC and AQ drop sharply, while human raters still prefer results that express the intended motion with correct bindings. Thus, SC, BC, AQ and IQ do not reliably track human preference in these settings. These metrics should be integrated alongside motion-aware and interaction-awant measures such as DD, KISA, SGI and IF Our method maintains moderate DD and higher interaction fidelity, which aligns better with human judgments.
H.3 ADDITIONAL ABLATION RESULTS
We compare the training cost of different variants under near-iso-FLOPPs setting. All fine-tuning experiments are conducted on single A6000 GPU at resolution of 480 720 with the same batch size, so the wall-clock time per epoch serves as a practical proxy for the total FLOPs used. We con- sider three variants, (I) the naive Cog VideoX-5B-12V model, which is the original pretrained model without any additional finetuning, (II) a LoRA-only baseline, which we term "LoRA w/o layer se- lection". and (III) our full method. All three variants evaluated under the same configuration, in- cluding resolution, number of sampling steps and guidance setting. For variants (II) and (III, Tab. 5 reports the per-epoch training time. Specifically, the LoRA-only baseline requires about 24.43 hours per epoch, whereas our full method requires 31.09 hours per epoch, corresponding to an additional 6.7 hours of training time. Despite this moderate overhead, the improvements on interaction-centric metrics, including KISA, SGI and IF, are substantially larger. Compared to the naive Cog VideoX. SB-I2V model. our method improves KISA from 0.406 0.546 (+34.5%), SGI from 0.491 to 0.641 (+30.5%). and IF from 0.449 to 0.593 (+32.1%). Even when compared directly to the LoRA-only baseline, our method still yields relative gains of +22.7% (KISA), +2 .9% (SGI), and +-22.0% 000% (IF), while holistic appearance metrics (HA/MS/IQ) remain essentially unchanged. This efficiency is particularly notable when contrasted with recent video generation approaches (Chefer et al. 2025) that jointly train a video backbone from scratch, which typically incur substantially higher training cost. In our case, a lightweight LoRA-base fine-tuning recipe on top of Cog VideoX-5B- yields 30 30-35% relative improvements core interaction metrics at essentially identical inference cost and only a modest increase in training time, resulting in a favorable trade-off.
I LIMITATIONS
Instance Scalability. Our current framework supports up to five instance mask tracks per scene. While this upper bound appear restrictive, it is well aligned with the distribution observed in
36

---

Published as a conference paper at ICLR 2026
dataset Appendix As illustrated in Fig. 12 Appendix, scenes containing more than five interact- ing instances are rare, and most examples contain two to four distinct objects involved in interaction. This design choice thus reflects practical tradeoff between generality and simplicity. allowing the model to remain effective without introducing unnecessary architectural complexity. Nevertheless, extending support larger number of instances remains feasible direction for future work.
Small Mask Sensitivity. Another limitation arises when the instance mask occupies very small spatial region. In such cases, the visual grounding signal may become weak or even ambiguous, potentially degrading the model's ability to generate accurate motion. Future improvements could involve strategies such as hierarchical mask encoding, spatially adaptive attention or resolution- aware learning to enhance robustness against object size variations. We leave these directions for future exploration.
J DISCUSSION AND FUTURE WORK
Although we focus on instance mask tracks as the core supervision and analysis modality, our for- mulation is modality-agnosti on the analysis side, and naturally extends multi- modti settings. Additional cues such as optical flow and depth are highly complementary. Optical flow can pro- vide stronger supervision for fine-grained motion continuity and temporal consistency, while depth can help disambiguate occlusions and 3D proximity in contact-rich interactions. In fact, the same semantic grounding and propagation framework depth-based consistency terms, or by conditioning the video DiT on flow or depth features during training. A systematic integration of such multi-modal cues (e.g., mask tracks flow + depth) therefore promising direction for future work.
incorporate these signals by adding flow- or
Beyond flow and depth, point tracking can provide another complementary modality. Mask tracks offer instance-level grouping and are well suited for supervising semantic grounding and propaga- tion the level of each instance ID, but they inherently provide relatively coarse supervision fine-grained local motion (i.e., exactly where each local detail moves in the next frame). Point tracks are almost the opposite: they capture very fine local motion but do not induce clear instance- level grouping role semantics (e.g.. subject object). This suggests natural trade-off and complementarity. A compelling extension is to use mask tracks to supervise instance-level roles and interaction structure subject-object-verb), while using point tracks to refine fine motion and local detail propagation across frames. We leave such multi-modal extensions as valuable future directions for interaction-aware video generation.
K THE USE OF LARGE LANGUAGE MODELS (LLMs)
In accordance with the ICLR 2026 submission policy, disclose that Large Language Models used to assist in grammar correction, polishing of the writing in this paper and caption processing in our dataset curation pipeline.
37

---

Published as a conference paper at ICLR 2026
First turn
Given the following video caption, determine whether there any active and meaningful interactions involving living subject and another distinct entity (another person, object, animal).
Video caption: /caprion/
Valid interactions must involve:
-A living subject
A scparate target cntity (another person, an object)
A clear relationship or action connecting them
Do NOT count:
Self-directed actions (c.g., 'a man gesturing". "a person walking', 'somcone raising their hand')
Vague verbs with larget (e.g., e.. 'a moves', 'a person acts')
Emotional internal with external relation (c.g., 'a boy thinking', girl smiling')
Only interactions that involve:
Clear action verbs between two entities (e.g., "hugging', "pointing at', "talking to', "giving something")
Respond exactly one of following:
mull - no such interaction exists
an integer (e.g., 1, 2, 3, 4, ...) representing the exact count of interactions
# Second lurn
You are Al that extracts valid and meaningful interactions from a video caption.
Vidco caption: [caption]
Follow these rules:
First, identify all unique, living subjects and distinct entities mentioned in the caption. Assign consistent ID (<id1>, <id2>, etc.) each unique entity. A single entity must have only ID, even if is part multiple interactions.
the caption describes multiple entities of same type (e.g.. 'two men') you must use descriptive details (like "on left', 'on the right', , or clothing) to assign them distinct IDs. Do not use a single for multiple individuals.
Extract all active and meaningful interactions described in the caption. Do not omit any valid interactions, even if they seem less dynamic than others.
-A valid interaction must meet all of the following conditions: a living subjecl (src l), separate target (igil # srel), and a clear action verb. Valid examples include both highly active actions like *<id>> gives <id2>' and less dynamic actions like ><dd>> holds <id2>'
Second, classify each interaction its type: 'multi subject relation' 'functional action based interaction'.
Third, each interaction, provide the exact sentence from original caption where found.
NOT include self-directed actions, vague verbs, internal states.
Your output must a JSON array of interaction objects, with extra text or explanation.
Output format:
"interaction"" *xidX> <action verbo <idY>",
"srel": "<idX="
"tgtl": "<idY>",
"type": "..."
"source "enttene"""..""""
Figure 26: Prompt Design for Interaction Identification and Instance Assignment.
You are strict rater that evaluates un interaction triplet itself (e.g., '<id1> is holding <id2=").
Use the full available contexa(caption, ids) to determine CONTACT and DYNAMISM.
Scoring rules (integers 1-5);
-Contact: = contact; 3=uncertain/indiiee indirect: 3=direct/fim contact implied the interaction
-Dynamism: 1=static relation; 3-loo//moderae movement/redinnes;; 5=strong action/state change
Video caption: /caption]
Interaction: finteraction triplet)
Noun of <id>: fhase nouns of <id>s in interaction]
Detailed information of noun: [detailed information of base nouns]
Oulpul Format: ("Contact': <int 1-5>, "Dynamism": <int 1-5>, "Explanation": <short reasun>)
Figure 27: Prompt Design for Interaction Scoring and Filtering.
38

---

Published as a conference paper at ICLR 2026
Provide detailed information each unique used above.
Make sure to include detailed cntry for every ID (e.g. <idl>, <id2>, <id3>) mentioned carlier.
each ID, include:
"noun", short and visually distinguishable noun phrase (e.g.. "a man in a blue shirt", "a dog with brown fur")
This should be specific and concise to help an object detection model localize the entity.
"app": appearance or physical description (I sentence)
"sputial"" their spatial location or role the scene (I sentence)
Vidco caption: [caption]
Interaction: finteruction triplet]
Noun of <id>: /base
<id> in interaction]
Output Format:
<<idd>> ("noun": ... "upp": ... "spatial""
idd>>" 3
Figure 28: Prompt Design for Instance Description Extraction.
You are given one image crop (JPEG) of detected object and a list of candidate IDs.
Each candidate has fields: id, noun, appearancc.
Decide which ID best matches the crop.
If none of the candidate IDs elearly match, or if the object appears to be something clse not described in the candidates, then you MUST return null.
Be conservative when uncertain.
Return STRICT ISON only:
"assigned id": string/null, "confidence": number, "rationale": string }
The detection label for this crop is: fbhox label] (may help disambiguation)."
Candidate IDs (JSON array): [id descriptions]
Image to classify: simage]
Figure 29: Prompt Design for Vision-Language Verification.
39

---

Published as a conference paper at ICLR 2026
Source
the person preparing meal kitchen. person wearing black apron black glove, indicating they middle coeking. They using knife green cacumber wooden cutting board. cucumber placed the board. and the person holding the knife in their right hand, using slice through the cucumber. The background the video blurred, but appears kitchen setting with counter sink visible. The focus the video actions suggest that they confidently described. The relative position of the objects such that the person is standing the counter, with the cutting hoard and cucumber front them. The knife being used the eucumber, which the cutting board. The sink located to the right the person and cutting board. The counter visible in the buckgruund, behind the and the cutting hoard. The video does contain any other objects people"
Original Prenpe
person's hands and cucumber, with preparing dish that involves
other objects people the frame. The additional objects that
eucumber. The video does contain
Interaction: <idl> is using <id2> to zut <id3s
Contact: 5, Dynamism 4
(elass: person, app a person wearing a apron black gloves, spatial person standing at counter, with
cutting board and cucumber in front them)
<id2> (class knife, app the knife being used cut the cacumber, spatial: the knife the person's 's right hand }
<idi> (class: eucumber, app green oucumber, spatial cucumber placed cutting board person ]
Mask Tracks
Figure 30: Dataset Examples.
40

---

Published as a conference paper at ICLR 2026
Original Prompt
The video depicts a woman in a kitchen, preparing a drink. standing granite countertop, which is adorned with various kitehen items, including blender, microwave, and a spice rack. The woman wearing a colorful floral dress and engaged the process of blending ingredients. She starts by placing a black on the blender, then proceeds to add a red straw into a glass. The kütchen well-fit, with natural light streaming in from a window adorned with purple flowers. The woman appeurs to be the midst a tutorial demonstration, as indicated by the text "LiweLoveaa.coom displayed in the top right comeer of the video. The video captures the woman's actions Iluid, continuous motion, emphasizing the process of b'eending and serving a drink.".
Interaction <
places sid2> <idl> adds <id]>.
Interaction
Contact 5, Dynamism
<idl>: (class woman, app
colorful floml spatial She standing countertop the kitchen.)
<dd2>; (class blender, app blender standard kitchen appliance, spatial It located the granite countertop the kitehen.)
<d3>>
(class straw, app red spatial: being added glass countertop.
Figure 31: Additional Dataset Example.
41

---

Published as a conference paper at ICLR 2026
Original Pronpe
person uses power grind top black metal barre. person wearing, brown jacket blue jeans; the background a brick wall. The sparks from tool flying everywhere, The person holds tool their right hand.
Interaction: : <id holds <id2>, <idl>s <idll>grinds <id3>,
Contact 5. Dynamism 5
<id|>:
(class person, app person wearing brown jucket and blue jears, spatial The person standing in front of brick wall.)
<id2>:
(cluss tool, power tool is producing sparks, spatial 'he power tool being held the person's right hund..)
<id3:: (class harrel, app black metal hurrel, spatial The barrel is in front of the person and against the brick wall.)
Tracks
Figure 32: Dataset Examples.
Source video
Original Prompt
video shows person standing front of sink with a silver finect white countertop. The person's hand holding
toothpaste in one hand toethbrush in other, they are carcfully squeezing the toothpaste onto bristics of the toothbrush.
The video is clese-up shot, focusing on the person's hands and toothbrush. The lighting is bright and clear, making easy to see the details of person's hands and the toothbrush.
Interaction: : <idl> bolding <idZ>, <idl> bolding sidlo, <idl> is squeezing sid2>
Interaction Info.
Contact: 5, Dynamism :
<idl>: (class: person, app the person's hand has fingers, spatial: wripped around the toothpuste and toothbrush.]
<id2>:
(class toothpaste, cylindrical object with nozzle, spatial heing held in the persons's 's hand.)
<id3>: (class: toothbush, long handle and head with bristles, spatial being held in the person's hand next to the toathpaste.,
Tracks
Figure 33: Dataset Examples.
42

---

Published as a conference paper at ICLR 2026
Prompt: "The man in a black shirt swings the red paddle and hits the orange ping pong ball." "
"man"
18
L9
L10
L11
L12
L11
LI5
L16
L17
L19
1.21
122
1.23
124
1.26
1.27
1.28
1.29
L30
1.31
1.32
L.33
1.34
1.36
1.40
"swing"
14
L.7
LS
L.U
L10
C11
1112
L11
L.21
1.22
1.25
1.24
1.24
1.26
1.27
L.28
L,29
L50
E.32
1.33
1.34
1.35
1.36
L.37
1.3%
1.39
1.40
1.11
Figure 34: Layer-wise Visualization Comparison.
43

---

Published as a conference paper at ICLR 2026
Task answer yes/no questions about video frames Inputs
An ordered list of frames (indexed from 0). list of yes'no questions. Each question specifies entities with their colored bboxes (e.g.. "a man (yellow bbox) touches cup (blue bbox)").
Rules
Judge visible evidence only. Do not infer beyond what is clearly the frames. Color disanbiguation. Because alone may uniquely identify instance frame, the specified colored bbox as the reference pinpoint the intended entily, and base your judgment on that entity's visible evidence.
Por-Question Procedure Select the decisive frame.
Scan frames and choose the single frame that gives the clearest evidence for "yes" or "no".
If multiple frames are equally decisive, pick the earliest index. If no frame provides clear evidence, answer "no" and sel frame_ind index to null.
Answer (yes/no).
Based solely what is visible in the decisive frame (and color-tagged entities), answer "yes" "no".
Visual plausibility check the decisive frame).
the decisive frame shows visually implausible analomy/geomettry override the answer with "no (visually
implausible)".
Plausibility flags include (not exhaustive):
Human anatomy anomalics: missina hands/fcct/armss, impossihle joint bends, detached limbs.
Object/bodd fusion fusion/splitting artifacts within a bbex, or severe distortions that break physical continuity. Self-intersection or impossible penetration (e.g.. hand passes through a solid object) that invalidates the observation.
Purpose: reject interactions that "occur" but are visually nonsensical.
Output (JSON)
Retum amay of objects:
"question id": 1, "answer":: "yes", "frame index":
"question id": 2, "answer": "no (visually implausible)", "frame_inder
"question id": 3, "answer": "no", "frame index": null
answer ("yes", "no", "no (visually implausible)")
frame index is the decisive trame used to judge the answer (or null if none was decisive).
Figure 35: Prompt design for KISA and SGI in Evaluation Protocol.
44

---

Published as a conference paper at ICLR 2026
Role
You are a ballucination detection expert.
Your task is to evaluate a sequence of video frames relative to fixed anchor frame (frame 0) and determine
whether:
Any known entity class emerged without bounding box,
Anyy previously tracked instance disappeared in later frames.
## Inputs
You are given:
'frame_ 0%: anchor'reference frame
"fammes k':
frames where [1. N]
Each frame contains bounding boxes, and every bbox defined by (class, color)' identity
-A color-to-class mapping JSON;
"json
"entities:: ["person", "cup", "paper"]. "colors": ["green", "red", "blue"]
The arruys eetities' and 'colors' are index-aligned, e.g., "rred" "ccup" Use this mapping identify track instances consistently across all frames.
## Detection Rules
Emergence
Mark "emergence "yes"" any frame *k contains:
An unboxed object of class listed in entities', and
That class had no visual instance (boxed or unboxed) in frame
This includes cases where:
The object appeurs fully unboxed in the background
The object appears embedded inside another bbox (c.g., ball inside person)
Track all frame indices where emergence occurred.
### 2.
Disappearance
set (class, color)' pairs from 'frae define the complete instance each frame *k*, there must bbox the same (class, color) every such instance. any original instance is missing in frame *k*, mark 'dsappearancee "yes" include:
The frame index *k*
A description of which instances were lost (by (class, color)" pair or class count)
#* Output Format
Produce single JSON object that summarizes emergence and disappearance across all frames:
"json
"emergence": "yes"| "no".
"emergenee frames": [<frame idx <framm idx 2>;,..... ...]. "emergence reason": "bricf explanation empty string no", "disappearance". "yes") "no",
'diappeearance frames": [<frame idx 1>, <frame 2>, -.,, disappearance_rese reason": "list missing instances as (class,color) and/or class-level count deltas"
## Evaluation Notes
You must compare all frames after frame against the instance roster from frame O.
Ignore objects not listed in the 'entities' array.
Emergence is class-based: second instance of class (without bbox) can be emergent if not present in frame 0.
[f no emergence disappearance occurs in any frame, all values should default to ""no", 'O" and 190.
Figure 36: Prompt design for SPI in Evaluation Protocol.
45

---

Published as a conference paper at ICLR 2026
The in blue shirt with rolled-up sleeves pushes the wooden chair toward the table.
The boy in blue hoodie with curly hair presses the round elevator button.
The girl in yellow dress with a ponytail taps the black tablet screen.
The student in a gray hoodic with glasses places rod book on the desk.
The bay in red jersey with short blond hair kicks the white soccer ball on the field.
The friend in white T-shirt hugs his friend in black jacket in the park.
A girl with glasses touches a framed painting in quict art gallery with soft lighting and white walls while man is walking behind.
in busincss suit vacuum cleancr on beige carpet, and golden retriever runs toward the open deor at the back of the living room.
A woman in a red coat pushes stroller along a park path with fallen leaves scattered
in husiness suit shaking hands with another man in front of a glass office building, while woman nearby walking across the plaza with folder in her hand.
around. Nearby, a child in a green hoodic
jumps with excitement on the grassy field.
A buy wearing headphones throws a
In a museum, a man in glasses touches sculpture with curiosity, while a young girl walks slowly past row of paintings on the wall.
basketball toward hoop in a quict
neighborhood court, while unother boy
waves from the sideline.
A man feeds baby in high chair while woman holds a baby bottle a cozy kitchen with warm lighting and wooden cabinets.
man in workout clothes pushes a shopping cart in a parking lot, while a woman next him picks up a grocery bag from the ground
A girl a pink sweater opens a refrigerator, and her brother pulls chair toward the kitchen table in a modem home interior.
boy reads a picture book beside fireplace, while cat on the windowsill touches toy mouse with its paw.
A woman lifts chair in a classroom, while a
A girl pushes another girl on tire swing park, while man in the background is shaking hands with boy near picnic tables.
boy pats dog sitting calmly near the desk.
Figure 37: Generated Evaluation Dataset Pairs Example.
The woman in black sports jacket hands over the sealed tea packet in front of the woman to the man in blue shirt.
The woman slices the zucchini with the kitchen knif's placed on the weoden counter.
The soccer player exchanges high five with the coach near the sideline after being substituted.
The man checkered shirt gently holds bowl of prepped vegetables, his hands steady ready to transfer them into a pan.
A man walking past with yellow towel wipes the front panel or windshicld of the red SUV.
The female nurse taps on the tablet screen start recording the man's gait pattern.
The man in green shirt walks and sits down on green bench, settling next to the woman.
The man in striped sweater and beanie gently pats the head of the wearing glasses and dark shirt.
The person tilts the frying pan slightly to spread the egg mixture evenly across the surface.
The woman in the wide-brimmed hat raises her silver travel mug to take sip.
Figure 38: Sampled Evaluation Dataset Pairs Example.
46

---

Published as a conference paper at ICLR 2026
Comportson Video totai)
Comparison Set
Prompt:
man blie with rolinc-up slenves poshee the warden trwoed the
FIRST FRAME
MODEL
MODEL
0:007 0:06
Pleaze watch the video completely before making your cholce bai ove.
The vidso shows. First Frame: Medill Modil
watching completely,
muliipe
different aspecis.
1. [Dnteraction Accuracy] Which video more accurataly represente the interaction specified in the bext prompt (who is interacting with whom and what they are doing)?
Model
Model
2. [Semantic Grounding] Which videa more accurately includes the abjects Indicated In the image prompt as instructed by the text prompt the interaction (-e., better grounding)?
Model
Model
3. [Semantio Propagation] Which video more tomporally consistent, without sudden apporanees/ispppprrree (hallucination)?
Model
C
Model
[Bemantic Aligraman]] Overall, which video shows prompt?
faithfully and naturally according
Model
C
Model
[Overall Quality)
higher quality?
Model
PREVIOLS VIDEO
NEXT VIDEO
Figure 39: An example of human evaluation.
47

---

Published as a conference paper at ICLR 2026
"At kitchen counter, student in white shirt places notebook wooden table. Cups and plates are scattered nearby, and sunlight comes in through the window. The action demonstrates placing the notebook the flat surface." "
Cog
Open 12V Cog 58
MATRIX
"A woman in red coat pushes stroller along park path with fallen leaves scattered around. Nearby, a child a green hoodie jumps with excitement on the grassy field."
Open-Sora Cog Vidoox 28
laVid
MATRIX
Figure 40: Additional Qualitative Results.
48

---

Published as a conference paper at ICLR 2026
Frami
"The man with a sushi t-shirt puts the bitten donut into his mouth to eat it."
Open 12V Cog 58 Cogg Video
TAVid
MATRIX
Frame
"The man in a blue shirt feeds a strawberry to the woman in a white chef coat."
VigVidaaX ConVideoX 28
Open-Sora 12V
laVid
MATRIX
Figure 41: Additional Qualitative Results.
49

---

Published as a conference paper at ICLR 2026
MMTTRIX 1-148-112
"Inside cozy kitchen with white walls and wooden shelves, wenring blue shirt stands front of the fridge the counter. He
carefully holds Silver his right hand while light from the window shines the tiled floor."
+MATRIX Waed 1-1 1188-EE
front house, yellow wooden
taps
lightly, producing sound. simple action shows initiating
Want. 1-1 48-12V
-MATRIX
"In library filled with wooden shelves, a wurman in a black skirt stands neur a study desk under a bright lamp. She bends slightly
she lifls a (lack red book with both hands. The woman carefully raises the heavy book from desk surfisee, preparing to move it to
another spot in the library."
Figure 42: Qualitative Comparisons between Wan-14B-I2V (Wan et al. 2025 and MATRIX (Ours).
50

---

Published as a conference paper at ICLR 2026
Wind 1-1
MMTTRI
"The mani the ramp pushes the furmiture toward the track while the inside the truck pulls @ inward."
~MATRIX Winc. 1-1-8--22 22V
'The wearing black t-shirt and sharts gets the bike and riding forward.
-MATRIX Wan2 -188-12
"The lands Skateboard and performs jump
Figure 43: textbfQualitative Comparisons between Wan-14B-12% (Wan et al. 2025 and MATRIX (Ours).
51


Reviews:
[{"review_id": "fTkwQoIZLA", "reviewer": "Reviewer_TLG6", "summary": "The paper investigate why t2v diffusion transformers struggle with multi instance, subject object interactions and proposes a targeted fix. The authors proposed MATRIX-11K, a dataset of ~11K videos with interaction-aware captions and multi instance mask tracks. And analyzed where interaction semantics emerge inside 3D full attention of video DiTs, finding that semantic grounding and semantic propagation concentrate in a small set of layers, and introduced the lightweight regularization that aligns attention in those layers to the mask tracks via SGA and SPA loss. On their benchmark, MATRIX improves interaction fidelity and reduces drift and duplication.", "questions": "see weakness", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. A deep and comprehensive clear failure analysis, and the MATRIX fixed that.\n2. The data pipeline of the seg data generation is well designed, pragmatic and reproducible.\n3. Simple, effective objective. The SGA and SPA losses are standard but cleverly applied to attention maps rather than pixels alone, brings lower computational cost and easy to adapt into Dit.", "weaknesses": "1. It mainly compared and analyse with CogVideoX, as this work is a 2025 work, it probably need to do more with the more recent models like wan and hunyuan.\n2. 11K dataset may not that a good amount for the video generation tasks, especially for those larger and new models, the society may be benifit if the dataset could be larger\n3. Like the other related work videojam or udpdiff, they used optical flow, seg, depth to help with the video generation, while in this work only segmentation is used, consider to include discussion on involving others would beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsykUpqH2p", "reviewer": "Reviewer_payz", "summary": "This paper analyzes a common problem in video models, incorrect multi-object interactions. It identifies key concepts that allow analyzing two aspects of this problem (semantic grounding and propagation), and uses these concepts and their corresponding metrics to understand how these interactions are represented inside a typical DiT model. Based on this analysis and metrics, the paper introduces 1) a method for regularization (Matrix), 2) an evaluation protocol (InterGenEval), and  3) a dataset with 11k videos (Matrix-11k) with properly labeled interactions and instances.\n\nThe method for effective regularization is based on locating the layers that most impact on semantic grounding and propagation, and fine-tuning those using LoRA on the corresponding losses for those two concepts: Semantic Grounding Alignment and Semantic Propagation Alignment.", "questions": "* What are your insights regarding which layers are the most influential? do you find that they are consistently placed around the same locations in different kinds of networks?\n\n* Have you considered how general this method is; would you expect similar results in non-DiT based networks for example?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper is a strong work in all the important dimensions:\n\n* originality: to the best of my knowledge this is novel work\n* quality: the problem is analyzed from an interesting perspective, starting with an analysis that provides a new understanding on how the interactions are represented in particular layers of the network, and the solution (introducing new concepts, a regularization method, a dataset, and an evaluation benchmark) presented properly deals with the problem\n* clarity: the paper is well written and easy to follow\n* significance: this is where I doubted the most, because while the problem is important for video generation, it was unclear to me that per se it would justify the added complexity in the training system (even if based on something as lightweight as LoRA). However, the use of this method is also beneficial for overall quality and other important metrics, which is intriguing, impactful, and to me brings final justification to this additional complexity.", "weaknesses": "* There is no analysis about how this method impacts training time. Especially important would be to understand whether the benefits of fine-tuning are as strong as presented, when compared to a training run with the same amount of flops used (iso-FLOPS comparison)\n\n* The dataset used for eval, based on only 60 synthetic and 60 real pairs appears too small. It would be useful to include analysis on what to expect, in terms of noise, from this dataset size\n\n* While ablations for most important aspects of this work are provided, there's little justification for why Av2t  and Av2v are the best proxies for semantic grounding and propagation. Could one use At2v for the former, or even reuse the Av2t for the latter? What other alternatives were considered? While it's not reasonable to expect a completely exhaustive ablation for all possible alternatives, even a small explanation would improve on this.\n\n* this work focuses on actions that require contact between different objects in the videos, such that the fine-tuning process only includes such actions in the dataset. This is a reasonable assumption for this particular paper (but as a minor improvement, it could be interesting to consider how the work could be expanded to better support contact-less interactions). However it would be useful to understand how this choice impacts other model properties, more specifically:\n  * is the fine-tuned model more prone to generate contact actions even when provided with a contactless prompt?\n  * minor: it would be interesting to understand, also, whether a fine-tuned model improves on contactless actions as well, even if those aren't included in the expanded loss\n\n* similarly, since the primary source HOIGen is described to be mostly focused on human-based interaction,  it seems useful to ask what's the impact on non-human video generation\n\n* minor: given the strong results using LoRA, the paper could benefit from some discussion about whether using the same method on large scale pretraining", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4K4YYdiWb9", "reviewer": "Reviewer_NstU", "summary": "The current work investigates how the object interactions are represented inside the video diffusion transformers and proposes a training strategy to refine them. More concretely, it investigates the attention map inside the CogVideoX-5B-I2V model in terms of text/visual tokens interactions, and observe that there is some object grounding (some spatial visual tokens have higher attention scores to the object/subject text prompt) and interaction grounding (the interaction verb attends to both the object and the subject). The curate a dataset of 11K videos and annotate instance-level segmentation mask tracks to quantify this grounding. This allows to locate the layers where grounding is the highest, and also to evaluate the interaction quality later. To improve the interaction modeling, the authors propose extra regularizations on attention maps pushing them closer to the correctly grounded segmentation maps.", "questions": "- Typo: \"suject-object\" (the first sentence in the abstract)", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The paper is well written and easy to follow, and the illustrations are high-quality\n- The proposed dataset could be useful for simply fine-tuning to improve the human-object and human-human interactions\n- The included attention visualizations could be useful for future references\n- The overall topic is quite important to the community", "weaknesses": "- The fact that text tokens are grounded to visual tokens in attention to some extent is a trivial observation and dates back to StableDiffusion-V1 editing pipelines for UNet+Attention architectures. The same observation was made for convolution-free architectures as well (i.e., DiT/UViT). One of the recent references is https://arxiv.org/abs/2505.04320 where it's visualized for Flux. I do not think that it's a surprising observation that similar mechanics happens for video DiTs. I would be more surprised if it would be something opposite. That's why the analysis is not insightful.\n- The paper uses outdated baselines (e.g., CogVideoX-5B) instead of the modern ones (e.g., Wan 2.1/2.2)\n- The paper does not compare to the baseline of simply fine-tuning on their curated data. I believe it's a critical ablation\n- The submission does not include any samples from the dataset, making it hard to assess its quality\n- The overall visual quality is far behind modern open-source models (e.g. Wan, Hunyuan, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)