VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning
Authors: Han Lin
Tushar Nagarajan
Nicolas Ballas
Mido Assran
Mojtaba Komeili
Mohit Bansal
Koustuv Sinha
Venue: ICLR 2025 Poster (2025)
Keywords: Procedural Learning from Videos
Representation Learning
Diffusion Transformer
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Procedural video representation learning is an active research area where the objective is to learn an agent which can anticipate and forecast the future given the present video input, typically in conjunction with textual annotations. Prior works often rely on large-scale pretraining of visual encoders and prediction models with language supervision. However, the necessity and effectiveness of extending compute intensive pretraining to learn video clip sequences with noisy text supervision have not yet been fully validated by previous works. In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR. Instead of learning representations from pixel space, our method utilizes the latent embedding space of publicly available vision encoders. By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our prediction model is able to learn robust representations for forecasting through iterative denoising —leveraging the recent advances in diffusion transformers (Peebles & Xie, 2023). Empirical studies over a total of five procedural learning tasks across four datasets (NIV, CrossTask, COIN and Ego4D-v2) show that our model advances the strong baselines in long-horizon action anticipation (+2.6% in Verb ED@20, +3.1% in Noun ED@20), and significantly improves the SoTA in step forecasting (+5.0%), task classification (+3.8%), and procedure planning tasks (up to +2.28% in success rate, +3.39% in mAcc, and +0.90% in mIoU).

Published as a conference paper at ICLR 2025
VEDIT: LATENT PREDICTION ARCHITECTURE FOR PROCEDURAL VIDEO REPRESENTATION LEARNING
Han Lin"!, Tushar Nagarajan', Nicolas Nicolls Mohit Bansal' & Koustuv Sinha? UNC Chapel Hill, ?FIIR, Meta
Ballas?, Mido Assran,, Mojtaba Komeili",
{hanlincs, mbansal Jocs uncceddu koustuvs@meta.ccom
ABSTRACT
Procedural video representation learning is an active research area where the ob- jective is to learn an agent which can anticipate and forecast the future given the present video input, typically in conjunction with textual annotations. Prior works often rely on large-scale pretraining of visual encoders and prediction models with language supervision. However, the necessity and effectiveness of extend- ing compute intensive pretraining to learn video clip sequences with noisy text supervision have not yet been fully validated by previous works. In this work, we show that strong off-the-shelf frozen pretrained visual encoder, along with well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR. Instead of learning representations from pixel space, our method utilizes the latent embedding space of publicly available vision encoders. By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our predic- tion model is able to learn robust representations for forecasting through iterative denoising leveraging the recent advances diffusion transformers (Peebles & Xie, 2023). Empirical studies over total of five procedural learning tasks across four datasets (NIV, CrossTask. COIN and Ego4D-v2) show that our model advances the strong baselines in long-hone hongzhorizon action anticipation (+2.6% in Verb ED@ 20, +3.1 % in Noun ED @ 20), and significantly improves the SoTA in step forecasting (+5.0%), task classification (+3.8%), and procedure planning tasks (up to +2.28% in success rate, +3.39% in mAcc and +0.90% in mloU). Project page: https: // /gi thub. com HL- hanlin/veii......
INTRODUCTION
Humans regularly perform complex, multi-step procedural activities with ease (e.g., cooking a recipe, assembling piece of furniture). This ability stems from our capacity recognize, reason about and plan for these activities, which is crucial for developing effective embodied Al systems to perform similar tasks. Towards this, designing systems that can understand procedural activities and predict the next logical steps is an active research problem (Brohan et al., 2023; Chang et al., 2020; Tellex et al., 201 1). On the one hand, a large body of prior work on visual representation learning demonstrates the importance of large-scale image or video pretraining for single-step activity understanding (Oquab al., 2024; Bardes et al., 2024; Zhai al., 2023; Wang et al., 2023c; Chen et al., 2021; Assran et al., 2023; Xu et al., 2024). On the other hand, encoding sequences of steps (i.e., building a prediction model) for future step prediction in videos is a relatively new area of research. Existing procedural video representation learning approaches (Lin et al., 2022; Zhong et al., 2023) typically inherit the same methodology as traditional activity understanding from single short video clip -ap---ch extending single-clip pretraining to large-scale pretraining on video clip sequences (e.g., in HowTo 100M (Miech et al., 2019)) with generic objectives, such as masked step prediction supervised by noisy ASR annotations obtained from narrated videos (Shvetsova et al., 2024) or fixed text knowledge bases like wikiHow (Koupace & Wang, 2018).
Work done during internship Meta.

---

Published as a conference paper at ICLR 2025
However, the necessity and effectiveness of pretraining the prediction model on video clip sequences have not yet been fully validated in these works for two main reasons. First, the dominant single- clip pretraining objectives (e.g., masked token prediction) were designed for feature learning of a short single clip, and are not well aligned to the breadth of downstream procedural tasks (e.g.. step forecasting, task classification, procedural planning). Second, pretraining for sequences rather than single steps demands scale of data beyond what is currently available. As result, current approaches (Lin et al., 2022; Zhong et al., 2023) fall back on text annotations that are often noisy and poorly temporally aligned with the video content (e.g.. ASR narrations). Therefore, in this work, we investigate how far an approach can go without requiring extensive pretraining on video clip sequences. Our hypothesis is that learning an efficient prediction model (i.e., transition function) over strong abstract representations from frozen visual encoders offers a compelling alternative to extensive large-scale video clip sequences pretraining for procedural learning tasks.
To this end, we propose our framework VEDIT Video Embedding Diffusion Transformer a scalable diffusion transformer (DiT, Peebles & Xie (20ss))-based prediction model to encode multi- step procedural videos. VEDIT inherits both the diffusion-style training objective and architecture. Specifically, during training, we utilize the latest Flow Matching technique (Esser et al., 2024; Lipman et al., 2023; Goodfellow et al., 2016) for iterative denoising from random Gaussian noise into video clip embeddings. Unlike DiT-based models designed for fine-grained image/video generation (Yang et al., 2024; Esser et al., 2024) which operate at the patch level, our prediction model works as the step/state transition function, utilizing the abstract frame-level representations from frozen visual encoders, operating in latent space (LeCun, 2022). This abstraction allows our model to capture the temporal aspects of the procedural learning task, resulting in the ability learn an efficient transition function. Crucially, our method does not require pre-training as it utilizes existing pre-trained representations, nor does it rely on additional supervision (from text or ASR).
We evaluate our model on five diverse procedural learning tasks across four datasets. (1) On the COIN (Tang et al., 2019) dataset, our model outperforms previous SoTA by a large margin (+5.0% for step forecasting and +3.8% for task classification), and demonstrates scalable learning as we increase the model size. (2) Our newly proposed VEDIT significantly enhances the overall performance of previous SoTA (Niu et al., 2024) procedure learning tasks NIV (Alayrac et al., 2016), CrossTask (Zhukov et al., 2019), and COIN (up to +2.28% % in success rate, +3.39 % in mean accuracy, and +0.90% % in mean IoU), as well as the strong baseline for the Ego4D-v2 (Grauman et al., 2022) long-horizon action anticipation task (+2.6% % in Verb ED@ 20, +3.1% in Noun ED)20). (3) Finally, we conduct detailed ablation studies on the choice of visual encoders, architecture ablations and large-scale pretraining on video clip sequences, and confirm the effectiveness of each component of our framework design.
In nutshell, our main contributions can be summarized as follows:
We propose a procedural video representation learning framework (VEDIT) which leverages diffusion transformers to predict visual representations entirely in the embedding space.
By combining strong pretrained visual encoders on single video clips with a simple prediction model design, our framework is designed to be trained effectively for downstream tasks, eliminating the need for large-scale pretraining on video clip sequences and additional supervision from actions labels or language for learning the prediction model.
single cross-entropy loss
We evaluate VEDIT on five downsteam tasks, including step classification, step forecasting, task classification, procedure planning, and long-term action anticipation across four widely-used benchmark datasets. Our framework outperforms previous SoTAs and baselines by a large margin.
2 RELATED WORKS
Procedural Video Understanding. Learning procedural knowledge from videos has become an active research area, driven by recent large-scale datasets (Miech et al., 2019; Sener et al., 2022; Afouras et al., 2024; Song et al., 2024) and models trained on them (Niu et al., 2024; Wang et al., 2023a;b; Zhao et al., 2022; Lin et al., 2022; Zhong et al., 2023). These models often rely heavily on large-scale scale text supervision. For example, DistantSup (Lin et al., 2022) creates text supervision by linking step descriptions from textual knowledge base (wikiHow) (Koupace & Wang, 2018) 8) to text narrations from ASR in videos. ProceduralVRR, (Zhong et al., 2023) aligns ASR narration embeddings to video representations using strong pretrained language models (Radford et al.,
2

---

Published as a conference paper at ICLR 2025
[a] Step Forecasting /Action Anticipation Task
Cross Entropy Loex
Flow Matching Noise Scheduler
Class Labela:
VEDIT
Attentive Classifier
pour
axcumber
X Staps
Vector Embedding DIT (VEDIT)
(b) Procedure Planning Task
Class Lebebs:
VEDIT
Attentive Classifler
rudiin pieas Suntcn
Rerel tsta iop cu
Random Gaursian Noise
Visuel Encoder
Visual Encoder
(c) Procedural Activity Classification Task
Gross Entropy
Class Labels:
VEDIT
Attentive Classifler
maka
make barser
Video Clip1
Vidao Clip N
Figure I: Overview of our VEDIT training pipeline. Model architecture (left): We introduce masked clip-level latent prediction our training objective, where train Vector Embedding DiT (VEDiT) to iteratively denoise I steps from random gaussian noise with flow matching noise scheduler. Downstream tasks (right): We train VEDiT with a light-weight attentive classifier (Bardes et al., 2024) with tross-entrope loss for the following tasks. (a) Step forecasting / action anticipation task: predict the embeddings of next unseen clip from observed clips with VEDiT. (b) Procedure planning task: predict the embeddings of intermediate unseen clips from observed starting and goal clips with VEDiT. (c) Procedural activity classification task: given a sequence of observed video clips, predict the label of the procedural video.
2021). Moreover, prior work uses architectures that necessitate large-scale pretraining on video clip sequences (c.g., self-attention transformers in ProceduralVRL). In contrast, propose efficient architecture that learns directly from video, side-stepping the requirement for large-scale language annotations pretraining on video clip sequences.
Diffusion Transformers and Flow Matching. Diffusion models (Song et al., 2020a; Ho et al., 2020; Sohl-Dickstein et al., 2015; Song al., 2020b) have emerged new state-of-the-art architecture
for deep generative models. Compared with UNet-based diffusion models (Rombach et al., 2022;
Blattmann et al., 2023), recent architectures (Ma al., 2024c; Chen al., 2024; Esser al., 2024;
Gao et al., 2024; Yang et al., 2024) designed based on diffusion transformers (Peebles & Xie, 2023) have achieved significant success and scalability in image and video generation. On the other hand, flow matching (Lipman et al., 2023; Ma et al., 2024b; Karras et al., 2022; Nichol & Dhariwal, 2021) has shown great potential as an alternative to DDPM (Nichol & Dhariwal, 2021) for diffusion model noise scheduling, Inspired by these advances, our work introduces novel modification of the DiT architecture and flow matching for procedural activity learning, that leverages pretrained visual features on single video clips and explicitly models the temporal order of steps.
Procedural Activity Learning with Diffusion Models. Prior work has incorporated diffusion into procedural learning in various ways. Some works (Soucek et al., 2024; Black et al., 2024) propose using image- and text-conditioned diffusion- bbatud generation or editing models to generate images of actions and object state changes while preserving the input image scene. These methods primarily use diffusion models as off-the-shelf tools for generating intermediate steps in the pixel space. In contrast, other works (Fang et al., 2023; Shi al., 2025; Wang al., 2023b; Zhong al., 2023) integrate diffusion as a training objective within their model design. predicting the embeddings of unseen target clips based on the embeddings of observed clips. Our work follows this second approach. treating diffusion (flow matching) as noise scheduler that denoises video embeddings from random noise, and we have designed a procedural learning framework based on the latest DiT architecture. Our framework differs from previous works in three folds: (1) training supervision: our model is designed to be trained directly with cross-entropy loss on downstream tasks, without the need for extra language supervision; (2) prediction model architecture: instead of using vanilla transformer blocks as the denoising model, we introduce new Vector Embedding DiT architecture for procedural learning from videos, which is proved to be more effective; (3) latent embedding generation: Our model generates unseen video embeddings from a frozen encoder, thereby operating in the latent embedding space.
3

---

Published as a conference paper at ICLR 2025
XM Blocks
VEDiT Block
Gate
Y3
7
FFN
FFN
Scale Shift
Scale & Shift
71
Gate
2
Gate
Joint Attention
RoPE
RoPE
Q'
Concal
Concat
Concat
Q'
SiLU & Linear
2 P
a
Scale & Shift
Scale & Shift
Timestamp
Random Gaussian Noise
Visual Encoder
Visual Encoder
Video Clip 1
Video Clip
Figure 2: Vector Embedding Diffusion Transformer (VEDIT) architecture. During training, our model first uses frozen visual encoders to convert observed video clips into corresponding video embeddings. Then random Gaussian noises are generated the initial video embeddings of unseen target clips. The DiT-based prediction model processes both seen and target video embeddings in two separate branches, and fuses their information via joint attention blocks where Q' = Concat Q", K' = Concat KK, K'), V' Concat (V", V*j. To enable temporal modeling of clips, Rotary positional embeddings (RoPE) is applied to Q' and K' before being input to the attention module. The denoised target clip embeddings are then given input to the attentive classifier in downstream tasks.
APPROACH
In this work, look at three canonical tasks from the procedural representation learning literature, Step forecasting, Procedure Planning, and Task classification, following the setup from Zhong et al. (2023); Niu al. (2024). Given series of video clips (or states) from procedural event (i.e. cooking a dish), the objective is to a) predict the label of the unseen future event (or state) to occur (step forecasting), b) predict the label of unseen events that happened in-between (procedure planning) and c) predict the label of the entire set of events, from a list of probable classes (task classification). Generally, given a sequence of N observable video clip representations (v;). we aim to learn procedural state representation 0, which can either capture the information of unseen clips (step forecasting or procedure planning) or summary information
the task (task classification) using conditional predictor é Fo ( E S)), where S is a set of observable clips. Using this representation, we then aim to learn a classifier to predict the class labels (C) for the given task h: x D + C. where & denotes set of output embeddings for representation of dimension D.
Learning a predictor to predict an unseen clip representation typically requires an extensive training process to learn rich visual representation and temporal information. In this work, we bypass the need learning visual information by leveraging existing pre-trained encoders v, = T (u;) E & where v being the clip in pixel space. Therefore, focus on learning the temporal transition among clips by operating over the encoder embeddings, = Fo([T* (v:)i E S)), where the predictor needs to generate latent representation x embeddings. To generate embeddings with rich visual signals,
4

---

Published as a conference paper at ICLR 2025
draw inspiration from the diffusion model literature, particularly recent diffusion transformers (DiT) (Peebles & Xie, 2023; Esser et al., 2024). Given their powerful text-conditioned image and video generation capabilities (Yang et al., 2024; Ma et al., 2024a), we adapt their strong conditional generation architecture into a sequential step prediction model for procedural activities. Thus, learning strong predictor would allow generate unseen clip embeddings, enable to perform the tasks in procedural representation learning.
3.1 PRELIMINARY: LATENT DIFFUSION MODEL AND RECTIFIED FLOWS
Given an image æ E B with caption c, image Latent diffusion models (LDMs) (Rombach et al., 2022) first use an atoencoder & to encode the image into latents 2g = E(x) RCxH xww where C' represents the number of latent channels, and H' H/pp W Wpp represent the spatial dimension of the latents, with p denotes as patch size. The forward diffusion process is a fixed diffusion process which adds random noise to the latent variable 2q. For example, forward process with Rectified Flows (RFs) (Liu et al., 2023; Lipman et al., 2023; Albergo & Vanden Vanden-Eijnden, 2023) is defined as a straight path between the data distribution zo and a standard normal distribution e (i.e., Ze = (1 =) + le, where /. E [0, 1). The reverse process in RFs, on the other hand, gradually produces less noise samples starting from 21 z in T denoising steps through learnable wansformer-basee denoiser model Fe parameterized by 0 and conditioned on caption c. In our setup, we transform such denoiser model conditioned on caption/text into a sequential step prediction model for procedural activities conditioned on observed video clips, with explicit temporal order modeling via RoPE (Su et al., 2024).
3.2 OUR APPROACH: VECTOR EMBEDDING DIFFUSION TRANSFORMERS (VEDIT)
Overall Training Pipeline of VEDIT. The overall training pipeline of our VEDIT is illustrated in Fig. 1 left. Given a set S that contains N observable (seen) video clips (U1, U2,-., Ux ). where each video clip v E R xxx3xHx contains K frames, first apply frozen visual encoder (") to derive the corresponding video embeddings for each clip v. = -" T((:). Next, random Gaussian noises are generated as the initial video embeddings of unseen clips {ūylū T), where T represents the target set. Then we design VEDIT as the learnable prediction model Fo which predicts the unseen target video clip embeddings S T. This prediction model is then trained using iterative denoising (Ho et al., 2020) over T steps, with diffusion timestamps sampled from the Flow Matching Fuler Discrete Scheduler (Esser et al., 2024). Unlike DIT, which is based-on pixel-level and text-conditioned generation, our model is designed to predict abstract video features in the procedural activity, based on observed video clips. Additionally, unlike previous works (Lin et al., 2022; Zhong et al., 2023) for procedural activity understanding, our method does not use extra language supervision such as ASR or textual knowledge base (e.g., wikiHow) that aligns visual embeddings with text.
(0) conditioned on all seen video embeddings:
i = Fo((()() R,),
i (,))
Training Objective of VEDIT. A key innovation and distinction of our method, compared with previous approaches video embedding prediction for downstream tasks (Zhong et al., 2023; Lin et al., (922)-which typically rely on a combination of multiple loss functions (such as video embedding reconstruction loss and video-language matching loss) for supervision-is that our pipeline can be effectively trained with a single cross-entropy loss. Unlike previous works that enforce alignment of the predicted video embeddings with noisily annotated language descriptions, using single cross-entropy loss allows the optimization target to align more effectively with downstream datasets.
Choice of Visual Encoder. Previous works on procedural learning (Lin et al., 2022: Zhong et al., 2023; Niu et al., 2024) typically use clip-level features as abstracted visual representation for each video clip, which can result in a loss of detail. Conversely, DIT models for image and video generation (Yang et al., 2024; Esser et al., 2024; Peebles & Xie, 2023) are designed for patch- level generation with focus on fine-grained visual details, but this comes at the cost of higher computational demands. To ensure that our model can process videos with multiple clips, encode sufficient visual information, and avoid excessive computational costs, we explore diverse CLIP- and SSL-based encoders that output visual features at the clip-, frame-, and patch-levels. We found empirically that using the [CL.S] tokens of SIgL.I (Zhai al., 2023) from 16 uniformly
model with
5

---

Published as a conference paper at ICLR 2025
sampled frames in each clip stands out as the strongest visual representation. An ablation study of visual encoders is discussed in Sec. 4.3.
VEDIT Model Architecture Design. Fig. 2 visualizes the components of each VEDIT block. Our architecture is derived from DiT (Peebles & Xie, 2023), which has a two-branch architecture with one query branch (typically text) tasked to condition the target branch (vision) through adaptive layernorm (Perez et al., 2018). In our work, we utlize the query branch to process the observable (or seen) video encoder embeddings {veli E $}. which conditions the targel branch that operates on unseen, noisy embeddings {ē, j E } through adaptive layernorm, which gets iteratively updated through denoising. The query branch further conditioned by using the timestamp t sampled from the noise scheduler, along with its usual application of determining the scale of noise. Unlike DIT, the information of the query branch and the target branch are fused together using joint attention before being processed independently through ferd-forward layers without weight sharing. Lastly, to enable temporal modeling of clips, Rotary positional embeddings (RoPE) (Su et al., 2024) is applied to the input immediately prior to the joint attention module. By utilizing these mechanisms, VEDIT allows us to learn unseen video embeddings, starting from noise, by conditioning on the observed video encoder representations. More ablations design choices provided in Appendix A.2.
4 EXPERIMENTS
In this section, we first introduce the evaluation datasets and the implementation details of our model in Sec. 1. We then compare our method with SOTAs on five downstream tasks across four datasets in Sec. 4.2. Finally, we present ablation studies on the visual encoders, as well as the necessity of pretraining on video clip sequences in Sec. 4.3. More ablations on VEDIT architecture design is provided in Appendix A.2.
4.1 EXPERIMENTAL SETUP
Evaluation Tasks and Datasets. We evaluate our method on five downstream tasks across four datasets. COIN (Tang et al., 2019) contains 476 hours of YouTube videos covering 180 tasks and 778 unique steps of daily activities. Following (Zhong et al., 2023; Lin al., 2022), we evaluated our model on two tasks: step forecasting and task classification (see Fig. 1 for details). For Ego4D- v2 (Grauman et al., 2022), we focus on the long-term action anticipation benchmark, which aims to predict the next Z- N /. future action classes [(verb), noun;). ). (verb2, noung), .... (verbz, nounz)] given an input video up to timestamp This forecasting benchmark contains 243 hours of videos with a total of 3472 annotated clips. In addition, we utilize NIV (Alayrac et al., 2016), CrossTask (Zhukov et al., 2019), and COIN datasets to evaluate the procedure planning task (Chang et al., 2020), which can be seen as variant of step forecasting task that aims at predicting intermediate action steps given the observed start and goal video clips. Specifically, CrossTask dataset contains 2750 videos covering 18 tasks and 133 actions, and NIV dataset contains 150 videos with 5 tasks and 48 actions. Following (Niu et al., 2024), we report the results with prediction horizon T E (3,4). 4).
Evaluation Metrics. For COIN step forecasting and task classification tasks, we use top-1 clas- sification accuracy of the predicted step/task as the evaluation metric following DistantSup (Lin al., 2022). For Ego4D-v2 long-horizon anticipation task, we use the default edit distance (ED) metric, which is computed as the Damerau-Levenshteer distance (Damerau, 1964) over sequences of predicted verbs or nouns. Following (Grauman al., 2022), we report the minimum edit distance at Z = 20 (ED@) @ 20) for K = 5 predicted sequences on the validation set. In addition, for procedure planning tasks on NIV, CrossTask, and COIN, we evaluate the models on three metrics, including success rate (SR), mean Accuracy (mAcc) and mean Intersection over Union (mIoU) following previous works (Chang et al., 2020; Niu et al., 2024; Zhao al., 2022).
Implementation Details. Our default VEDIT architecture contains 12 transformer blocks, with a hidden size of 2048 and attention head dimension of 64. During training, we apply classifier-free guidance with scale of 7 and denoise the diffusion model for 24 steps using the Flow Matching Euler Discrete Scheduler (Esser et al., 2024). For COIN step forecasting and task classification tasks, scheduled learning rate linearly increases from 5 10 6 to 5 10-5 during the first 3 epochs, and then decays to 5 x 10 following a cosine schedule, with a total of 30 epochs. For long- horizon anticipation and the procedure planning tasks, following the same training setting in previous

---

Published as a conference paper at ICLR 2025
works (Grauman al., 2022; Niu et al., 2024), we train the model for 100 and 500 epochs respectively. Together with VEDIT, we train task specific attentive classifiers : R - C' (Bardes et al., 2024), which is attentive pooler over & output embeddings followed by single linear layer.
4.2
MAIN RESULTS
Step Forecasting and Task Classification. Table 1, we demonstrate the effectiveness of our VEDIT design the COIN step forecasting and task classification tasks. Firstly, use the pre- trained TimeSforeer (Bertasius et al., 2021) visual encoder as T", as used in ProceduralVRL. (Zhong et al., 2023), the previous state-of-the-art in these tasks. We combined the TimeSformer encoder with VEDIT designed with joint attention, to arrive at the model TimeSformer+ VEDTT. This model achieves improvements of 2.2% and 0.6% in top-1 accuracy on the step forecasting and task classifi- cation tasks. Next, using SigLIP (Zhai et al., 2023) as T' with VEDIT yields additional gains of 3.1 % and 3.2% on these two tasks. It is worth noting our methods does not require any large-seale pretraining on video clip sequences, which proves the effectiveness of using strong gegguugg--lignee single-clip pretrained representations. Additionally, our method does not require explicit text supervi- sion (i.e., unsupervised) compared to baselines DustantSup and ProceduralVRL which are trained with explicit language matching loss (iee, ASR or ASk+wikiHow) Furthermore, we observe linear scalability of VEDIT on Step Forecasting task, leading to improved numbers with increasing number of model parameters (Appendix A.2.4).
Procedure Planning Task. We further eval- uate VEDIT on procedure planning results on the NIV, COIN, and CrossTask datasets with horizons T E (3.4) 4} in Table 2. Specifi- cally, we build upon the previous So'T model, SCHEMA (Niu et al., 2024), by replacing their vanilla transformer blocks in the state decoder and step decoder with our VEDIT blocks. To ensure a fair comparison, we use the same number of transformer blocks (i.e., 2 blocks) with identical hidden dimensions, attention heads, and we strictly adhere to their training and evaluation hyperparameters and setups without any changes. We report the mean and standard deviation of SR, mAce, and mloU for results as well
74
309
SCHEMA
SC1EMA
CoasTaac
46
replication of SCHEMA averaged over 10 runs.
16
SCHEMA
As shown in Table 2, SCHEMA with VEDIT consistently outperforms the original SCHEMA method NIV (gains of 0.97% 2.28% for SR, 1.92%-3.399 for mAcc, and 0.49% for mloU) and COIN (gains of 2.21%-5588%% for SR, 0.86%-2.579 for mloU). Additionally, our VEDIT achieves better average performance on the Task dataset. Moreover, as plotted in Fig. 3, we also observe better stability of VEDIT over vanilla transformer blocks as we scale up the number of transformer blocks.
Figure 3: SCHEMA w/ VEDIT is more stable than SCHEMA w/ vanilla transformer we increase the number of transformer blocks.
for mAce, and
Long-Horien Horizon Action Anticipation. In Table 3, we evaluate our model on the Ego4D-v2 long- horizon action anticipation task. We introduce new baseline by replacing the SlowFast (Feicht- enhofer et al., 2019) visual encoder in the Ego4D bascline model with SigLIP, while keeping the prediction model (i.e., s lowfast. f v2) unchanged. For fair comparison, we initialize VEDIT with the same number of transformer blocks and hidden dimension size as the Ego4D baseline transformer prediction model. Using SigLIP as visual encoder, VEDIT outperforms the Ego4D baseline in both Verb ED @ 20 (Ours: 0.697 v.s. Ego4D Baseline: 0.718, lower is better) and noun ED@20 (Ours: 0.711 v.s. Ego4D Baseline: 0.742, lower is better). In addition, while not directly comparable, PaMsEgoAI (Ishibashi al., 2023), achieves lower ED metrics by introducing several enhancements, including an ensemble of SlowFast and SlowFast CLIP models, label smoothing to relax order constraints for future actions, and constraining the (verb, noun) classes based on
7

---

Published as a conference paper at ICLR 2025
Model
Pretiaining Supervision
Preirain Data
Step Forecasting
Task Classification
Random Guess
N/A
N/A
0.1
SlewFast (Peichtenboter et al., 2019)
Supervised: action labels
Kinetics
25.6
71.6
S3D (Kie ot 2018)
Unsupervised: ASR w. MIL-NCE
HTT00M
28.1
70.2
ClipBERT (Lei et al., 202.)
Supervised: captions
COCO+VG
VideoCLIP (Xu et al,, 2021)
Unsupervised ASR
HT100M
72.5
Flow) (Tang
2019)
Supervised:
TimeSformer (Bertasius et ol., 2021)
Supervised:: action lahels
Kinetics
34.7
81.5
TimeSlotmer (Bertasius
2021)
Unsupervised:
HTTOOM
DistantSup (Lin ot al, 2022)
Unsupervised: ASR + wikiHow
HTT0CM
39.4
88.9
ProssduraVRRL (Zhong 2023)
Unsupervised:
HT100M
46.8
Ours: TimeSformer VEDIT
N/A
N/A
48.7
91
SP (Zhui al, 2023) VEDIT
N/A
N/A
51.8
94.6
Table 1: Step forecasting and task classification results on COIN (Tang et al., 2019) dataset. We compare our method with set of strong baselines as well SOTA methods. Top-1 accuracies are reported. We bold and underline the best and the second best models in each task respectively.
T =
T=
Datasets
Models
(1)
mAce (1)
mloU (1)
(f)
mAce (1)
mioU (f)
Random
2.21
4,07
6,09
1.12
2.73
5.84
DDN (Chang et al., 2020)
18.41
32,54
$6.56
15.97
27.09
$3.84
LAL-GAIL (Bi al., 2021)
22.11
42.20
65.93
19.91
36.31
53.84
PIV (Zhao et 2022)
24.68
49.01
74.29
20.14
38.36
67.29
EGPP (Wang et al., 2023a)
26.05
$1.24
75.81
21.37
41.96
74.90
SCHEMA (Niu al., 2024)
27.93
41.64
76.77
23.26
39.93
76.75
SCHEMA! SCHEMA w/ VEDIT
26.66-+-2.27
39.94422.79
75.58+1147
22.32++1.15
36.96+1.844
74.39-1.11
28.94++107
43.3340.90
76.48=0062
23.29=#.44
38.8841.19
74.88=0.89
(Ours)
(2,281)
(3,391)
(0.901)
(0.977)
(1.927)
(0.49f)
Random
<0.01
<0.01
2.47
< 0.01
< 0.01
2.32
Retrieval
2,71
36.97
DDN (Chang al., 2020)
13.90
20.19
64.78
11.13
17.71
68.06
P'IV (Zhan et 2022)
2022)
15.40
21.67
76.31
11.32
18.85
70.53
COIN
EGPP (Wang et al,, 2023a)
19.57
31.42
84,95
13.59
26.72
84.72
SCHEMA (Niu et al., 2024)
32.09
49.84
83.83
22.02
45.33
83.47
SCHEMA
26.38.1.366
43.08:14.28
8149:11.70
21.00:12.56
43.37:1..6
82.70=1.00 1.08
SCHEMA w/ VEDIT
32.27:10444
40.15
84.07
23.11
45.68 10.52
83.56-00455
(Ours)
(5.891)
(7.071)
(2,571)
(2.11-)
(2.311)
(0. 86f))
Retrieval
8.05
23.30
32.06
3.95
22.22
36.97
DDN (Chang 2020)
12.18
31.29
47.48
5,97
27.10
48.46
Est GAIL (Bi 2021)
21.27
49.46
61.70
16.41
43.05
60.93
CrssTTsk
PIV (Zhao et 2022)
23.34
49.96
73.89
13:40
44.16
70.01
PPDP (Wang 20236)
26.38
18.69
52.44
62.38
EGPP (Wang et al,, 2023a)
26.40
$3.02
74.05
16.49
48.00
70.16
SCHEMA (Niu al., 2024)
31.83
$7.31
78.33
19:71
$1.85
74.46
SCHEMAT SCHEMA w/ VEDIT
30.57+00.38
56.02+0.32
77.60=2025
20.26:00.3
51.93=0.17
74.51=0.25
31.08:10.31
56.15:10.57
77.5420.35
20.42:10.24
52.2620.51
74.76-0.29
(Ours)
(0.51
(0.131)
(0.06.)
(0.16")
(0.337)
(0.25t)
Table 2: Procudure planning results NIV (Alayrac al., 2016), COIN (Tang al., 2019), and CrossTask (Zhukov et al., 2019) datasets with prediction horizon T (3.4]. SCHEMA': our replication their method averaged over 10 runs. The best numbers are bolded. Our improvement over SCHEMA baseline is colored in blue.
word co-occurrence. Some studies (Zhao et al., 2024; Pei et al., 2024; Huang et al., 2023) have found that combining vision models with the strong planning capabilities of LLMs can achieve good performance, particularly for long-horizon action anticipation tasks. Therefore, integrating our method with an LLM could be promising future direction.
4.3
ABLATIONS
4.3.1 WHICH VISUAL ENCODER WORKS BEST?
To test the impact of visual encoders on procedural activity understanding from instructional videos, train VEDIT with blocks for 10 epochs with different visual encoders. We include strong CLIP-based and self-supervised (SSL) encoders, including SigLIP (VIT-S0400M/14G) 384) (Zhai al., 2023), V-JEPA (Bardes et al., 2024), DINOv2 (Oquab et al., 2024), and VideoMAE Tong et al.
8

---

Published as a conference paper at ICLR 2025
Method
Encoder
Prediction Model
EDe5 (1)
EDES (1)
ED@ (1)
ED@20
Verb
Noun
Verb
Noun
Ego4D Baseline (Grauman et al., 2022)
SlowFast
Transformer
0.745
0.779
Egu4D Baseline (Gruuman et al., 2022)
SigLIP
Transformer
0.703
0.736
0.718
0.742
PaMsEgoAI (Ishibashi al., 2023)
SlowFast CLIP
Transformer
0.670
0.629
Ours
SigLIP
VEDIT
0.677
0.711
0.697
0.711
Table 3: Comparison of methods on the validation set of Ego4D (Grauman et al., 2022) long-term action anticipation challenge. Edit distance (ED) metrics are reported prediction horizon and 20.
Model
Architecture
Pretrain Data
CLIP/SSL
Token
Step Forecusting
Task Classification
DINO>2
ViI-Gian/|14ee 224
LVD- LVD-142M (Oquab et al. 2024)
SSL
[018]
47.03
90,
VIEPA
VTT Vig-Huuee/668 224
VideoMix2M (Bardes et al., 2024)
SSL
Parch
47.24
$6, 13
VJEPA
Vil-Huge/16e 184
VideoMix2M (Bardes al. 2024)
SSI.
Parch
48.23
14
VideoMAE
VIT-Huge/1 16
K400 (Kay al, 2017)
SSL
Patch
44.78
83.40
SigLIP (default)
VIT S040000000/ 584
WebLI (Chen et al., 2023)
CLIP
[C1S]
50.05
94.38
Table 4: Ablation frozen video encoders COIN (Tang et al., 2019) step forecasting and procedural activity classification tasks. 1 accuracies are reported. We bold and underline the best and the second best models in each task respectively.
(2022). For V-JEPA and VideoMAE, we provide patch tokens to VEDIT, while for DINOv2 and SigLIP, we provide [CLS] ] tokens. We evaluate the model trained with different encoders on COIN for step forecasting and task classification tasks.
As we observe from Table 4, SigLIP outperforms SSL-based encoders. Among the SSL-based encoders, VJEPA ViT-H 384 and DINOv2 performs comparably than the baselines for step forecasting and task classification task respectively. SigLIP outperforms both on a large margin, especially in Task classification, highlighting the need of language-aligned rich visual representations for stronger procedural activity understanding.
4.3.2 IS PRE-TRAINING NECESSARY?
Instead of training directly on downstream datasets (i.e., COIN), previous works (Zhong et al., 2023; Lin et al., 2022) undergo large-scale pretraining on vidco clip sequences from publicly available video datasets, such as How 100M (Miech et al., 2019). In this section, we question whether such clip-sequence pretraining is necessary. Specifically, we compare our model trained directly dataset.
on COIN with variant that includes additional clip- clip-sequence pretraining on HowTool0 Following the reiope of Zhong et al. (2023), we set the total number of clips to 9, randomly mask out the video embedding of one clip, and use the masked clip embedding reconstruction as the training objective. Additionally, instead of relying on the noisy automatic speech recognition (ASR) annotations, use the starting and ending timestamps of each video clip processed and filtered by HowToCaption (Shvetsova et al., 2024). During pretraining, we employ the Adam (Loshchilov & Hutter, 2019) optimizer, with the learning rate linearly increasing from the first 0.5 training epochs, and then remaining constant for a total of 30 epochs. The pretraining is conducted on 128 H100 GPUs with total batch size of 1024, and takes days and 4.5 days for the 165M and 1.77B VEDIT models respectively (see Table 9 for model architecture details).
10-5 to 1 10-4 during
Surprisingly, observe only marginal improvement with significant clip-sequence pretraining (Table 5). Pretraining on video clip sequences only provides an additional 0.3% boost in top-1 accuracy for the step forecasting task. We hypothesize this limited effectiveness of clip-sequence pretraining may stem from two factors: (1) pretraining dataset is relatively noisy, leading to distribution gap with the downstream COIN dataset, and (2) the pretraining objective in (Zhong et al., 2023) may not be optimal. Exploring better clip-sequence pretraining objectives that can generalize well across different downstream tasks is left for future work.
5
LIMITATIONS
One potential limitation of our model with multi-step denoising is that it sacrifices efficiency for performance, Additionally, it is not specifically designed for real-time inference, which is parallel

---

Published as a conference paper at ICLR 2025
Model
Pretraining Supervision
Pretrain Data
Step Forecasting
Task Classification
SipLIP VEDIT
N/A
51.8
94.6
SigLIP VEDIT
Unsupervised: HowToCaption
HTT00M
52.1
94.6
Table 5: Effect of large-scale clip-sequence pretraining. We compare VEDIT without large-scale pretraining with variant that's 's pretrained on 1.16M videos from HowToo00M (Miech et al., 2019) dataset, using temporal information from TooToCaptoon (Shvetsova et al., 2024). Top-1 accuracies reported.
Denoising Steps
8
12
20
24
Training Clock Time (sec.)
0.41
0.49
0.63
0.82
0.94
1.09
1.24
Inference Clock Time (sec.)
0.40
0.47
0.56
0.66
0.75
0.84
0.93
Training GPU Memory (GB)
21.8
21.9
22.0
22.0
22.1
22.2
22.3
Inference GPU Memory (GB)
15.7
15.7
15.8
15.8
15.8
15.9
Table 6: Training and inference clock time and GPU memory of VEDIT as we increase the number of denoising steps.
topic to our paper and typically involves techniques such as model distillation, quantization, and hardware-level optimization. We leave the exploration of this direction for future work.
Here, we provide some analysis by evaluating the training and inference time as well as GPU memory usage of our model with varying numbers of denoising steps. Specifically, we conducted this experiment using the VEDiT architecture with 696M trainable parameters. We measured the clock time and GPU memory required to inference on the model using COIN video consisting of 8 clips, with gradient checkpointing enabled.
As shown in the table below, increasing the number of denoising steps from 1 to 24 results in only 1.32x increase in inference time and a 2.02x increase training time. This efficiency is partially because of the adoption of efficient scalable dot product attention in each VEDiT block. Moreover, because we employ gradient checkpointing to optimize GPU memory usage, GPU memory usage remains nearly constant without significant variation.
CONCLUSION
In this work, we demonstrate that carefully designed predictive models learned on top of single-clip pretrained visual representations achieve state-of-thes of-the-art performance on procedural learning tasks across the COIN, CrossTask, NIV, and Ego4D datasets, including step forecasting, procedural activity classification, procedure planning, and long-term action anticipation. Notably, we achieve these results without pretraining the prediction module, instead learning it directly from the end tasks. This contrasts with previous works, which often require computationally expensive pretraining of the predictor, sometimes with additional supervision. Our findings suggest that further research is needed to improve clip-sequence pretraining for procedural activities. Specifically, exploring ways to better align pretraining tasks with downstream tasks could help fully leverage the benefits of pretraining. Moreover, the data distribution gap, as well as differences in the timestep boundaries of clips between the large-scale pretraining dataset and the downstream dataset, could be potential bottleneck that hinders the effectiveness of large-scale pretraining on noisy videos. Strategies aimed at improving robustness to distribution shifts (Sun et al., 2020) represent another promising direction for exploration.
REFERENCES
Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagarajan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos. Advances in Neural Information Processing Systems, 36, 2024.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In Proceedings of the
10

---

Published as a conference paper at ICLR 2025
IEEE conference computer vision and pattern recognition, pp. 4575 45735-588300 2016.
Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023.
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vetit, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang. Xiaodong Wang, William Wen, Shunting Zhang. Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume (ASPLOS '24). ACM, April 2024. doi: 10.114553620065.34443666 URL https: /ttp/rchtccccccccccooo.oocccccct org/ assets wwwesettsspttttteeeeeeeeee httppyytttrccheeeeo pdf.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with beibddembedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 15619-15629, 2023.
Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: // ///peereevew. wttp://.eet/footuen//eeeee
forum? SerrummiddaaCuuD FBK2. Featured Certification.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
Jing Bi, Jiebo Luo, and Chenliang Xu. Procedure planning in instructional videos via contextual modeling and model-based policy learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2021.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. In The Twelfth International Conference on Learning Representations, 2024. URL https: http://eeeeeeeeeeeeeeeeeee..eeee net/ / forum?id=coa ddcumlld=cochhTsbe...
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
22563-2255 22575,, 2023.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do i can, not as i say: Grounding language in robotic affordances. In Conference on robot learning, pp. 287-318. PMLR, 2023.
Chien-Yi Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles. Procedure planning in instructional videos. European Conference on Computer Vision, 2020.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Se- bastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol
11

---

Published as a conference paper at ICLR 2025
Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-imal -magee model. In The Eleventh International Conference Learning Representations. 2023. URL https:
//openreview net/ forum? ddmWWoB22400
Xinlei Chen, Hao Fang, Tsung-Yl Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. 2015.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9640-9649, 2021.
Fred J Damerau. A technique for computer detection and correction of spelling errors. Communica- tions of the ACM, 703):71-177,, 1964.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Fen Fang, Yun Liu, Ali Koksal, Qianli Xu, and Joo-Hwee Lim. Masked diffusion with task-awareness for procedure planning in instructional videos. arXiv preprint arXiv:2309.0744 07499, 2023.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In P Proceedings of the IEEE/CVF international conference computer vision, pp. 6202-6211, 2019.
Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, al. Lumina-t2x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint arXiv:2405.05945 2024.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Daoji Huang, Otmar Hilliges, Lue Van Gool, and Xi Wang. Palm: Predicting actions through language models @ ego4d long-term action anticipation challenge. arXiv preprint arXiv:2306. /6545, 2023.
Tatsuya Ishibashi, Kosuke Ono, Noriyuki Kugo, and Yuji Sato. Technical report for ego4d long term action anticipation challenge 2023, 2023. URL. httpss:///aaiiiii orgg abss /237 01 677.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green. Trevor Back, Apostol Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. ArXiv, abs/ 71070005.9990 2017.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, pp. 2. Minneapolis, Minnesota, 2019.
12

---

Published as a conference paper at ICLR 2025
Mahnaz Koupaee and William Yang Wang. Wikihow: A large scale text summarization dataset. arXiv preprint arXiv: 2018.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62, 2022.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan. Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and- lan-a-aa-g learning via sparse sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7331-7341, 2021.
Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13853-13863, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.
Xingchao Liu, Chengyue Gong, al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models. arXiv preprint 1rXiv:240.11887 2024a.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnde and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv:240l.08744 2024b.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.0304. 2024c.
Antoine Miech, Dimitri Zhukov, Cean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the TEEE/CVF international conference on computer vision, pp. 2630-2640, 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Yulei Niu, Wenliang Guo, Long Chen, Xudong Lin, and Shih-Fu Chang. SCHEMA: State CHanges
MAtter for procedure planning in instructional videos. In The Twelfth International Conference on Learning Representationss 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research Journal, pp. 1-31, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang. Yali Wang, Tong Lu, Limin Wang, and Yu Qiao. Egovideo: Exploring egocentric foundation model and downstream adaptation. arXiv preprint arXiv:2406. 18070, 2024.
13

---

Published as a conference paper at ICLR 2025
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAl conference on artificial intelligence, volume 32, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Fadime Sener, Dibyadip Chatterjee, Daniel Shelepov, Kun He, Dipika Singhania, Robert Wang, and Angela Yao. Assembly 101: A large-scale multi-view video dataset for understanding proce- dural activities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21096 21 106, 2022.
Lei Shi, Paul-Christian Bürkner, and Andreas Bulling. Actiondiffusion: An action-aware diffusion model for procedure planning in instructional videos. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025.
Nina Shvetsova, Anna Kukleva, Xudong Hong, Christian Rupprecht, Bernt Schiele, and Hilde Kuehne. Howtocaption: Prompting Iims to transform video annotations at scale. ECCV, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2256-2265 PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yale Song, Eugene Bymne Tushar Nagarajan, Huiyu Wang. Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. Advances in Neural Information Processing Systems, 36, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020b.
Tomas Soucek, Dima Damen, Michael Wray, Ivan Laptev, and Josef Sivic. Genhowto: Learning to generate actions and state transformations from instructional videos. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR): CVPR, Institute of Electrical and Electronics Engineers (IEEE), 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with sele-supervision for generalization under distribution shifts. In International conference machine learning, pp. 9229-9248. PMLR, 2020.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprchensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation.
Proceedings of the -ient--Fifth AAAI Conference on Artificial Intelligence, AAAI'11, pp. 1507-1514. AAAI Press, 2011.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data- efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078-11000 10093, 2022.
14

---

Published as a conference paper at ICLR 2025
An-Lan Wang. Kun- Yu Lin, Jia-Run Du, Jingke Meng, and Wei-Shi Zheng. Event-guided procedure planning from instructional videos with text supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13565-13575 2023a.
Hanlin Wang, Yilu Wu, Sheng Guo, and Limin Wang. Pdpp: Projected diffusion for procedure planning in instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023b.
Limin Wang. Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang. Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14549-14560, 2023c.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: apeer-accuracy trade-offs in video classification. In Proceedings the European conference on computer vision (ECCV), pp. 305-321, 2018.
Hu Xu, Gargi Ghosh, Po- Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoerr and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2021. Association for Computa- tional Linguistics.
Hu Xu, Saining Xie, Xiaoqing Tan, Yao Huang, Russell Howes, Vasu Sharma, Shang-Wan Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. In The Twelfth International Conference on Learning Representations, 2024.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cogvideox: with an expert transformer. arXiv preprint arXiv:2408. 06072, 2024.
diffusion models
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11966119866, 2023.
He Zhao, Isma Hadji, Nikita Dvomik, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. P3iv: Probabilistic procedure planning from instructional videos with weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2948,9948, 2022.
Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do. Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Antgpt: Can large language models help long-term action anticipation from videos? In The Twelfth International Conference on Learning Representations, 2024.
Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, and Yin Li. Learning procedure- aware video representation from instructional videos and their narrations. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14825-14835, 2023.
Dimitri Zhukov, Jean- Baat--tttttte Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivie. Cross-task weakly supervised learning from instructional videos. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3537-3545, 2019.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
In this appendix, we first present additional results on the step classification task (Appendix A.1.1) and procedure planning task (Appendix A.1.2). Then we discuss ablations of our model design, including the choice of attention mechanism (Appendix A.2.1), denoising steps (Appendix A.2.2), classifier-free guidance scales (Appendix A.2.3), and the downstream task performance as we scale up the VEDIT model size (Appendix A.2.4). Furthermore, we provide the PyTorch (Ansel et al., 2024) implementation of VEDIT in Algorithm 1.
A.1 ADDITIONAL RESULTS
A.1.1 STEP CLASSIFICATION TASK
In this section, we present additional results on the COIN step classification task, which aims to predict the class labels of single-clip videos. In other words, this task tests only the capability of visual encoders, as the prediction model is not involved. Specifically, for step forecasting and task classification described in the main paper, we design the attentive pooler as lightweight single cross- attention block with one query token to pool the video clip embedding (e.g., the predicted frame-level [CLS] tokens Lig (Zhai et al., 2023)) into a single vector. For the COIN step classification task, we increase the depth of the attentive pooler by adding three additional self-attention blocks before the cross-attention block to aggregate information from the visual features, which we find further improves classification accuracy.
We compare our method, which uses off- shethh frozen visual encoders with a trainable attentive classifier, against baseline methods reported in previous works (Lin et al., 2022; Zhong et al., 2023). As shown in Table 7, our method with all five frozen encoders outperforms previous baselines. V-JEPA performs best among the two SSL-based video encoders (i.e., V-JEPA and VideoMAE). Increasing the resolution from 224 to 384 on V-JEPA further boosts accuracy. Additionally, due to the rich information encoded in patch-level tokens, V-JEPA achieves the best performance on the step classification task among all encoders. Moreover, SigLIP, pretrained on both image and text data, outperforms all other encoders except for V-JEPA, demonstrating the effectiveness of using visual-text aligned encoders for procedural activity understanding in instructional videos. However, as the prediction model not involved, we do not put primary focus on this task in our paper.
Model
Pretraining Supervision
Pretrain Data
Top-1 Acc. (5)
Baselines
SlowFast (Feichtenhofer 2019)
Supervised: action labels
Kinetics (Kay et 2017)
32.9
TineSformer (Bertasius al,,,_2211
Supervised: action labels
Kineties (Kay 201777
2017)
45.3
ClipBERT (l.ei 2021)
Supervised: captions
COCO VG (Chen
Krishna al., 2017)
30.8
VideoCLIP eX 2021)
Unsupervised: ASR
HIT00M (Miseh et al., 2019)
39.4
TimeStormer (Bertasius et all,.2222)) 2021)
Unsupervised: ASR w. MIL-NCE
HITDOM (Miech et al,, 2019)
46.5
CLIP (Radford 2021)
Unsupavised: captions
CLIM400M (Radford ct 2021)
15.9
Distant
ol., 2022)
Unsupervised: ASR wikiHow
HTT00M (Miceli 20199
ProceduralVRL (Zbong et al. 2023)
Unsupervised: ASR
HTT00M (Micch et 2019)
56.9
Ours (Prozen encoder lightweight trainahle attentive classifier)
DINOw2 (Qquah ol., 2024)
Self-supervised
IVD-142M (Oquab et 2024)
$7.9
V-JEPA or 224 (Bardes al., 2024)
Sell-supervised
VideaMi (Bardes et al, 2024)
61.4
V-JEAA @ 381 (Bardes et 202()
Solf- supervised
VideoMis2M (Bardes et al., 202()
62.7
VidooMAB (Tong et 2022)
Scll-superrised Image+ Pairs
Kinetics400 (Kay all,, 2017)
58.5
SigLIP et al., 2023)
WebLI (Chen 2023)
Table 7: Step classification on COIN dataset. We bold and underline the best and the second best models in each task respectively. Our strategy of using strong frozen visual encoder with trainable attentive classifier outperforms all baseline methods.
A.1.2 PROCEDURE PLANNING TASK
In addition to the main results presented in Table 2, we show in Fig. 3 the comparison of our VEDIT and the vanilla transformer model (Niu et al., 2024) as we increase the number of transformer blocks. We report success rate (SR), mean accuracy (mAcc), and mean loU (mloU) as evaluation metrics on the NIV, COIN, and Cross Task datasets. For a fair comparison, we keep all hyper- parameters the same, with the only change being the number of blocks. We observe that our VEDIT exhibits significantly better stability compared to the vanilla transformer blocks as we scale up the model size, without overfitting the training set. This finding is consistent with results in Fig. 7.
16

---

Published as a conference paper at ICLR 2025
A.2 VEDIT MODEL ABLATIONS
1.2.1 CHOICE OF ATTENTION MECHANISM
In Fig. 4, we illustrate the differences between our default joint attention in each VEDIT block and self-attention and cross-attention. We denote the observed and unseen video clip embeddings as v or v'. In self-attention, we concatenate v² or ve along the sequence dimension as single input to the attention module. In contrast, cross-attention does not utilize self-attention within o" or v'. Here we conduct ablation study of these attention mechanisms with a prediction model of VEDIT blocks. As shown in Fig. 5, our joint attention outperforms self-attention in step forecasting (50.3 vs. 49.7) and task classification (94.4 94.3) the COIN dataset. This proves the usefulness of processing v and o' differently through adaptive normalization layers before inputing to the attention module. In addition, due to the absence of self-attention within v o', cross-attention performs the worst.
(a) Joint Attention (default)
(b) Self Self-Altention
(c) Cross- Altention
Attenttttttioon
Self Attention
Cross Attenton
Cress Attenion
K
K
Q
K
Top- Acc. (%)
(a)
(b)
(c)
Step Forecasting
50.3
49.7
47.2
Coneel
Task Classification
94.4
94.3
93.8
K
Scale & Shift
Scale & Shitt
Scale Shift
Scale Shitt
Scale Shift
Figure 5: Top-1 classification ac- curacy on COIN dataset with dif- ferent attention mechanisms.
Cuncaly r']
Figure 4: Ablation of attention mechanisms, including our de- fault joint attention, self-attention, and cross-attention. We denote seen and target video clip embeddings as v" and e' respectively.
A.2.2 CHOICE OF DENOISING STEPS
Previous work on masked token prediction, such as BERT (Kenton & Toutanova, 2019) for language and MAE (He et al., 2022) for images, can be considered single-step denoising, while diffusion models typically perform single-step denoising during training and multi-step denoising during inference. In this context, we conduct an ablation study on different denoising steps using diffusion timestamps sampled from the Flow Matching Euler Discrete Scheduler (Esser et al., 2024) in VEDIT training on the COIN step forecasting task. The ablation study here is conducted with a prediction model of VEDIT blocks, and we report the top-1 classification accuracy averaged over three independent runs in Fig. 6. Our results show that applying 20 to 40 denoising steps achieves better accuracy compared to single-step denoising (51.01 for 36 denoising steps v.s. 50.77 for single denoising step). This multi-step denoising allows us to reuse the same VEDIT architecture, with observed and target embeddings scaled and shifted by adaptive normalization layers at different timestamps, without drastically increasing the model's 's trainable parameters. Additionally, we observe that timestamps that are too sparse (e.g., denoising steps of 4) or too dense (e.g., denoising steps greater than 44) make the model difficult to optimize. We default to use 24 denoising steps in our main paper as it achieves a good balance between computational cost and accuracy.
A.2.3 CHOICE OF CLASSIFIER-REE GUIDANCE (CFG) SCALE
We conduct experiments to compare the effect of different CFG scales on the COIN step forecasting and task classification tasks. Specifically, we performed this experiment using the VEDIT architecture with 12 layers and hidden dimension of 1280, reporting the same top-1 classification accuracy metric used in our main paper.
As shown in the table below, CFG-7 achieves the best step forecasting accuracy, while CFG=3 achieves the best task classification accuracy. Additionally, the accuracy differences across the various CFG scales are relatively small. This is because best fit the given CFG scale. Therefore, in the experiments presented in our main paper, we did not fine-tune the CFG parameter and instead used the default CPG value of 7, in Stable Diffusion 3.
model effectively learns parameters that
17

---

Published as a conference paper at ICLR 2025
CFG Scale
3
5
7
11
Step Forecasting Acc. (%)
50.73
50,43
51.30
51.85
51.74
51.28
Task Classification Acc. (%)
94.70
94.84
94.71
94.67
94.47
94.42
Table 8: Ablation of different classifier-free guidance (CFG) scales.
51.50
51.25
Top-1 Acc (%)
51.00
50.75
50.50
50.25
50.00
8
20
32
44
Number Denoising Steps
Figure 6: Ablation of denoising steps. We report the top-1 accuracy COIN step forecasting task. A Denoising steps of 24 achieves good balance between computational cost and accuracy. The numbers averaged over 3 independent runs.
A.2.4 SCALABILITY OF VEDIT
Table 9 presents the model architecture details for 10 different scale models implemented in scalability results shown Fig. Specifically, we examined two different sets hidden dimensions (i.e., 1280 and 2048), with varying numbers of VEDIT blocks ranging from 18. These parameter settings effectively cover model parameters from 62M to 1.77B (up to x28 larger in scale). We evaluated these models on the COIN step forecasting task, and the results are averaged over 5 runs. As shown in Fig. 7, with the same number of training epochs, a larger VEDIT model achieves lower top-1 validation error compared smaller VEDIT models. This demonstrates that VEDIT is scalable as we increase the model size.
Model
# Train Params
Layers
Hidden Dim.
# Attn. Heads
Head Dim.
Hidden Dim. 1280
VEDiT-Single
62M
1280
20
VEDIT-Tiny
165M
1280
20
VEDiT-Small
342M
1280
20
VEDIT-Large
696M
12
1280
20
VEDIT-XI
1.05B
18
1280
20
Hidden Dim. = 2048
VEDIT-Single
132M
2048
32
VEDIT- Tiny
418M
2048
32
64
VEDIT-Small
871M
2048
VEDIT-Medium
1.34B
2048
VEDIT-Large
1.77B
12
2048
32
Table Details of VEDIT models. We introduce models of different number of transformer blocks (i.e., layers) with two hidden dimension settings.
18

---

Published as a conference paper at ICLR 2025
62M
165M
342M
418M
Top-1 Val Eir
54
696M
871M
1.058
1.348
1.778
12
18 Training Epochs
15
21
24
27
30
Figure 7: Ablation of different VEDIT model sizes. We report the top-1 validation error on COIN step forecasting task. Our VEDIT demonstrates good scalability as we scale up the model size up to 28 times from 62M to 1.77B. The numbers are averaged over 5 independent runs.
19

---

Published as a conference paper at ICLR 2025
Algorithm 1 Simplified PyTorch Implementation for Each VEDiT Block
import torch
2
from torch import mn
class VEDiT (nn. Modull)
5
def init
self, dim, num_attention_heads, attention_headddi max_len)
adaptive layernorm
self. melf.norml_seen AdalayerNozmZero (dim)
self.nozmi_targes Adalaye:NozmZero (dim)
normalization layers
self.nomm2_scen In. (dim, elementwise_affinn
15
self.nozm2_target nn LayerNorm (dim, elementwise_afflne
PPW
sel f. ff_seen reedForward (dim-dim, dim_out-dim)
self. tfrgget FeedForward (dim=dim, dim sim_out=dim)
20
joint attention self.atth JointAttention dim, numLattention_headd, attention_headdding max_len)
24
def forward(seff, target_emb, seen_emb, temb, target_mask);
scale shift
norm_target, t_gate_msa _sa, t_shift_mlp, t_scale_mip, t_gate_mlp self.t (target_eemt emb, emb=temb)
29
norm_seen, s_gate_msad s_shift_mip, s_scale_mip, s_gate_mlp self.norml (seen emb=temb)
2. joint attention
seen_atth_ouue atttttouuuuu,, target_attn_outpu: self.attn
hidden seateen_statss-nnrtt target, encoder encoder_hidden hiddendddddddd http:_hhdde__sstseeeeeee states-norm seen, target_ market_mask-taaget mask)
38
$ 3.1. target: gate, scale & shift
target_emks target_emb t_gate_mas targot_atth_outpur norm_ emm self nelff (target_emb) norn_target_eem
norm_target_emb emb (1 + _sscle__nlp) + t_shift_nip
$ 3.2. target: feed foreward
if output self.ff target (norm (tworntttrretteeeeeneeae target_enm emb target_emb ga of_target_output
8 4.1. seen: gate, scale shift seen_emb seen_emb + s_gate_msa seen_attn_output
emb self. (seen _mbb norm seem_seen_ee emb norm_seem_enb (1 + s_scale_rip) + s_shift_nlp
$ 4.2. seen: feed foroward [f [f_sccn_output self. ff_scen (wwrm_ssen_eeeeeeee seen_emb sebn_emb s_gate_nlp if ff_seen_output
return seen_emb, target_enb
20

---

Published as a conference paper at ICLR 2025
Simplified PyTorch Implementation for JointAttention
import torch
from torch import
3 import torch. nn functional as
4
5 class JointAttention (nn. Module)
def _init_
8
self, dim, nun hedd, attn_
dim, max len)
9
din num attn heads, attn aett_hhead__ dim self. inner_mme__di nun_attn_heaad hetd_head_dim
self.heads,
self.max_len max_len max number of clips procodural video
self.to_g nn Ln.Linear (dim, self.inner_dim, bias-True) self.to_k nn. (dim, self.inner_dii bias-True) self.to_v Linear (dim, self.inner_dim, bias-True)
self. add_ q nn an.Iinear (dim, self.ade k nn. nn.Linear (dim, selt selt.inner_dee bias-True) sel self.add_y nn.Linear (dim,
bias-True)
selff.iner_dim,, bias-True)
22
self.to_out out nn. in. (se1 inneer_dim, dim, bias-True) self.add out = nn. In. (seif seseef.innerrdfieeeeeee dim, dim, bias-True)
23
self.rotary_ent Rotaryšmbeddiing (dim=dim) $ rope
27
def forward hidden_states, secodor_hidder_ttaee target_mask)
residual = hidden_stated bs hi sden_sstates.shape
32
1. concat q k, and v from projected embeddings
query torch. Casch.cat/ssslttt/o.. (lsellltto (fseelttto_gguidcddeetttttttt dim-1)
self steee statess)) dim=1)
value torch. self aet.ad/_/eenccderr.ssseeeetttes
dim=1)
$ 2. get positional indices of seen and target embeddings indices = torch.
colffmax_len) repeat ([bs, 11)
seen_pos = indices [ "target_maskl wtt.aaassa....app....... reshape [bs, -11) target_pos input_pos torch. corcat [[target_pos, sean_pos], axis-1)
indices [target_mask] reshape [bs, -1l)
$ 3. apply rope to query and key query self, hal..rotary_ebe rotate (query, input_pos) key self.rotary www.ottayybbbbttttttee....eeeeee rotate (key, input_post
$ 4. apply attention
hidden_states F. sesaleeeLoo__prrddccttttttttttte (query, key, value)
hidden_states hidden_stated reshae (bs, -1, self inter_nmerdim
hidden_states, encoder.hh.ddensttrre hidden_states residual..hapell hidden_statesl: susiidull.happppp [1] :])
5. linear projection
hidden_state states telf.to_out (nidden_states)
39
= add out hicccodser_dcdde_sstttteeee
return hidden_states, encoder hideee_reiddeesttttees
21


Reviews:
[{"review_id": "Ew7Y0FbIhs", "reviewer": "Reviewer_yPQ6", "summary": "This paper introduces a novel pipeline,VEDIT, for procedural classifciation task using diffusion transformers without large-scale pre-training or additional language-based supervision. VEDIT leverages a diffusion transformer-based model to operate on latent embeddings derived from frozen, pretrained visual encoders. It is designed to predict the future steps in multi-step tasks (e.g., cooking, assembly) by iteratively denoising embeddings in the latent space, which are generated from observed steps. Authors conducted adequate experiments to demonstate that VEDIT achieves state-of-the-art (SoTA) results on various tasks like step forecasting, task classification, and procedure planning across multiple datasets (e.g., COIN, Ego4D-v2, CrossTask, and NIV).", "questions": "I also have a variety of questions that did not significantly impact the rating for the paper:\n- The paper demonstrated the advantage of using VEDiT on chunk-level classification tasks. Can the approach be extended to other temporal understanding tasks that predict start/end time? e.g., action grounding\n- The paper entirely leverages flow-matching sampling with a fixed number of sampling steps. How's the model's performance with different scheduler/sampling parameters?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Generally, the primary strength of this paper is the introduction of the novel diffusion-based representation learning method into the video understanding field. VEDIT achieves significant performance gains across multiple procedural tasks, such as step forecasting and task classification. \n\n**Clarity**: Most of this paper is well-written, with clear explanations and a solid grounding in relevant literature. \n\n**Originality**: The main idea of training the VEDiT using the CrossEntropyLoss from the downstream task for video representation learning is novel and has the potential to be applied to various temporal grounding tasks beyond classification. \n\n**Versatility**: VEDIT stands out for its versatility across tasks. VEDIT successfully handles multiple procedural learning tasks, including step forecasting, task classification, and procedure planning, proving its robustness in diverse scenarios.\n\n**Quality**: The overall quality of the paper is high. The empirical results are well-supported by ablation studies, which explore the impact of different model components, and the paper presents a thorough comparison against state-of-the-art methods, showcasing the strengths of the proposed model.", "weaknesses": "There are some major concerns:\n- While the approach of training VEDiT using CrossEntropyLoss from downstream tasks is novel, it remains challenging for the audience to understand why the diffusion model can learn with this design and the training pipeline. From the destination in L239-L241 and  L301-L303,  it seems the sampled Gaussian noise will be passed to $12 \\times 24= 288 $ transformer blocks in a single forward pass, where $12$ is the number of transform blocks and $24$ is the denoising steps. It seems that the computing needs are much higher than baseline methods, and it may not support other inference parameters, including denoising steps and classifier-free guidance scales, since they are fixed in training. It isn't clear how well it works when reducing the sampling steps to lower steps to balance the computing cost and performance boost.\n- Using diffusion models for video embedding prediction is time-consuming since it requires $N$ times longer inference time. However, authors didn't discuss the limitation in this paper.\n\n\nIn conclusion, these are concerns and weaknesses that could further improve the paper. However, the weaknesses outweigh the strengths. Consequently, I am leaning towards suggesting rejection of this paper. I am looking forward to the authors' rebuttal and clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AAfJZzYyTv", "reviewer": "Reviewer_Xzeh", "summary": "This paper aims to advance video representation learning by proposing a diffusion transformer specifically for procedural video representation. It demonstrates that a frozen, pretrained visual encoder paired with a prediction head can achieve promising performance across five procedural learning tasks.", "questions": "See details in 'Weakness' section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method avoids the need for compute-intensive pretraining of the prediction model. It uses a pretrained, frozen visual encoder to capture robust visual representations, eliminating the need for additional supervision (e.g., language or ASR) commonly required in prior methods\n\n2.The model's empirical evaluation demonstrates significant performance improvements on procedural video benchmarks like COIN, CrossTask, and Ego4D, showing gains in task classification accuracy and long-horizon action anticipation.", "weaknesses": "1.Lack of Sufficient Innovation: The VEDIT model's components lack distinct novelty. (1) VEDIT’s architecture is primarily a combination of existing techniques—pretrained visual encoders and diffusion transformers—rather than introducing new modules. (2) The method’s adaptation of diffusion transformers to work in the latent embedding space of video representations has been explored previously, as in PPDP.\n\n2.Omission of Computational Cost Analysis: The paper does not address the computational cost of the method. Due to the iterative nature of the diffusion process, VEDIT incurs slower inference times than models that make predictions in a single forward pass. This characteristic may limit its use in real-time or low-latency applications.\n\n3.Lack Disccusion of Multimodal Integration: By not incorporating textual data or language models, the method overlooks semantic information that could enrich the understanding of procedural steps, especially in instructional videos that include narration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJloCpwRac", "reviewer": "Reviewer_ajEB", "summary": "This paper deals with  Procedural Video Representational Learning, in which a model predicts steps in a underlying process (eg cooking a dish). The goal is to learn a representation that enables prediction and understanding of the semantics of step-by-step processes through videos, often egocentric or how-to videos. Most works in this space pre-train models on data from HowTo100M with extracted ASR captions which is time-consuming and expensive. This paper's contribution is (1) a novel architecture for the procedural video task based on diffusion / flow matching in latent state-action space and (2) a method that relies only on frozen pre-trained image encoders rather than explicit procedural weakly supervised pretraining. In doing so, they achieve SoTA results.\n\nIn general, I think this is a reasonably strong paper, and the only weakness to me is that the architecture has slightly limited novelty; it's not very clear to me exactly how this differs from a standard diffusion transformer. The use of a frozen encoder, while simple, works well and avoids having to do expensive ASR-based pre-training. The models are also scaled to high parameter counts and thus will be useful to the community.", "questions": "1) It's stated the task is either predicting the label of unseen events in between two input events (procedure planning), or predicting the label of a future event.  In the case of procedure planning, how does the model know how many clips to generate?  It's not obvious from the system figure (Fig 2) how multi-clip generation works - does that require multiple runs of the entire diffusion model? Is this number fixed?\n\n2) From an intuition standpoint, why is diffusion the right choice for generating an embedding given a set of input embeddings? Are there no faster alternatives? What was the reasoning behind this choice compared to other alternatives?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed architecture (diffusion in latent space between steps) is interesting. I think doing diffusion on the latent embeddings is a smart way to generate new embeddings that can be used for predicting final class embeddings. \n2) The results surpass existing SOTA and the ablations are comprehensive. The method also avoids relatively expensive pre-training on direct step data, making use of easily available visual encoders like SigLIP. The appendices are thorough and explain relevant design decisions for the proposed architecture.\n3) The models trained are relatively large and will be helpful for the community if made available.", "weaknesses": "1) The models trained are quite large, with the largest having 1.77B parameters. Given that the improvement over SOTA is relatively minor in most cases, it's not clear that scaling this method really works - could you comment on why this might be the case?\n\n2) The architectural novelty is a little limited. The idea of doing diffusion to produce intermediate embeddings isn't new (as is stated in the paper) and this paper differs from prior work mostly by using a frozen SigLIP encoder rather than one pre-trained on HowTo100M (if i am understanding correctly). \n\n3) It's not super clear how the proposed VeDIT differs from standard diffusion transformer blocks except for the embedding concatenation - am I missing something? If so, could you explain further?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B8OMVvPrzj", "reviewer": "Reviewer_czP8", "summary": "This work questioned the necessity and effectiveness of pretraining by develop a SoTA method for forecasting and procedural planning without the need for pretraining the prediction model. The proposed model, VEDIT, leverage diffusion transformer and attentive classfier directly trained for downstream tasks. Empirical results showed that VEDIT outperform all baselines.", "questions": "All of my questions are listed in the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Nice presentation and well-designed figures.\n\n2. Strong empirical performance: VEDIT outperform all baselines quantitatively on all three benchmarks.", "weaknesses": "1. Definition of pre-training: The most confusing point for me is that the authors claim the results of this work could challenge the necessity of pretraining. However, first of all, VEDiT itself uses a pretrained encoder. How is this different from what the authors define as pretraining? Secondly, isn’t the training of the diffusion part considered pretraining? I think the authors should make a stricter distinction and definition of pretraining to help readers clearly understand what they really want to refer to by \"pretraining.\"\n\n2. Training objective of DiT: If the training of the diffusion part isn’t considered pretraining, then what is the role of diffusion here? The authors should use mathematical formulas to clarify exactly what their loss function is; the current explanation is too vague. Additionally, starting from line 233, the description of diffusion training seems to suggest a possible misunderstanding of diffusion training.\n\n3. Test-time training [1]:  This is just a suggestion, but based on the authors' model, I think that applying test-time training exclusively to the diffusion component could further enhance the model's performance during inference.\n\n\n\n \n\n[1]. Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. and Hardt, M., 2020, November. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning (pp. 9229-9248). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1xQM9vCyMT", "reviewer": "Reviewer_2SCK", "summary": "This paper proposes a procedural video representation learning framework that utilizes diffusion transformers to predict visual representation in embedding space. This framework can be effectively transferred to various downstream tasks.", "questions": "Please answer the Weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-motivated and easy to follow.\n\n2. VEDiT does not require pre-training the representation space but consumes multiple clips by using pre-trained representation, which is more efficient and applicable, as better representation can improve VEDiT further.\n\n3. Extensive experimental results on various downstream tasks demonstrate the effectiveness of the proposed method.", "weaknesses": "1. Question about fair comparison: Without pre-training, I think VEDiT seems to be a large header for the downstream task. I cannot find how the baseline TimeSFormer is evaluated, but if it is evaluated with a simple linear classifier, i.e., linear probing, it seems that VEDiT's performance gain is just from \"training more parameters on the pre-trained representation\".\n\n2. Typo on page 2 (\"fall back on on text annotations\" -> \"fall back on text annotations\")", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)