TEMPURA: Temporal Event Masked Prediction and Understanding for
  Reasoning in Action
Authors: name: Jen-Hao Cheng; name: Vivian Wang; name: Huayu Wang; name: Huapeng Zhou; name: Yi-Hao Peng; name: Hou-I Liu; name: Hsiang-Wei Huang; name: Kuang-Ming Chen; name: Cheng-Yen Yang; name: Wenhao Chai; name: Yi-Ling Chen; name: Vibhav Vineet; name: Qin Cai; name: Jenq-Neng Hwang
Published: 2025-05-02 21:00:17

Abstract: Understanding causal event relationships and achieving fine-grained temporal
grounding in videos remain challenging for vision-language models. Existing
methods either compress video tokens to reduce temporal resolution, or treat
videos as unsegmented streams, which obscures fine-grained event boundaries and
limits the modeling of causal dependencies. We propose TEMPURA (Temporal Event
Masked Prediction and Understanding for Reasoning in Action), a two-stage
training framework that enhances video temporal understanding. TEMPURA first
applies masked event prediction reasoning to reconstruct missing events and
generate step-by-step causal explanations from dense event annotations, drawing
inspiration from effective infilling techniques. TEMPURA then learns to perform
video segmentation and dense captioning to decompose videos into
non-overlapping events with detailed, timestamp-aligned descriptions. We train
TEMPURA on VER, a large-scale dataset curated by us that comprises 1M training
instances and 500K videos with temporally aligned event descriptions and
structured reasoning steps. Experiments on temporal grounding and highlight
detection benchmarks demonstrate that TEMPURA outperforms strong baseline
models, confirming that integrating causal reasoning with fine-grained temporal
segmentation leads to improved video understanding.