VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning

## Abstract

Abstract Procedural video representation learning is an active research area where the objective is to learn an agent which can anticipate and forecast the future given the present video input, typically in conjunction with textual annotations.
Prior works often rely on large-scale pretraining of visual encoders and prediction models with language supervision. However, the necessity and effectiveness of extending compute intensive pretraining to learn video clip sequences with noisy text supervision have not yet been fully validated by previous works.
In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR.
Instead of learning representations from pixel space, our method utilizes the latent embedding space of publicly available vision encoders.
By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our prediction model is able to learn robust representations for forecasting through iterative denoising —leveraging the recent advances in diffusion transformers (Peebles & Xie, 2023 ) .
Empirical studies over a total of five procedural learning tasks across four datasets (NIV, CrossTask, COIN and Ego4D-v2) show that our model advances the strong baselines in long-horizon action anticipation (+2.6% in Verb ED@20, +3.1% in Noun ED@20), and significantly improves the SoTA in step forecasting (+5.0%), task classification (+3.8%), and procedure planning tasks (up to +2.28% in success rate, +3.39% in mAcc, and +0.90% in mIoU).

## 1 Introduction

Humans regularly perform complex, multi-step *procedural* activities with ease (e.g., cooking a recipe, assembling a piece of furniture).
This ability stems from our capacity to recognize, reason about and plan for these activities, which is crucial for developing effective embodied AI systems to perform similar tasks.
Towards this, designing systems that can understand procedural activities and predict the next logical steps is an active research problem (Brohan et al., ; Chang et al., ; Tellex et al., ).
On the one hand, a large body of prior work on visual representation learning demonstrates the importance of large-scale image or video pretraining for single-step activity understanding (Oquab et al., ; Bardes et al., ; Zhai et al., ; Wang et al., ; Chen et al., ; Assran et al., ; Xu et al., ).
On the other hand, encoding sequences of steps (*i.e*., building a *prediction model*) for future step prediction in videos
is a relatively new area of research. Existing procedural video representation learning approaches (Lin et al., ; Zhong et al., ) typically inherit the same methodology as traditional activity understanding from single short video clip
— extending pretraining to large-scale video clip sequences (e.g., in HowTo100M (Miech et al., )) with generic objectives, such as masked step prediction supervised by noisy ASR annotations obtained from narrated videos (Shvetsova et al., ) or fixed text knowledge bases like wikiHow (Koupaee & Wang, ).

However, the necessity and effectiveness of pretraining the prediction model have not yet been fully validated in these works for two main reasons.
First, the dominant pretraining objectives (e.g., masked token prediction) were designed for single-clip feature learning, and are not well aligned to the breadth of downstream procedural tasks (e.g., step forecasting, task classification, procedural planning).
Second, pretraining for *sequences* rather than single steps demands a scale of data beyond what is currently available. As a result, current approaches (Lin et al., ; Zhong et al., ) fall back on on text annotations that are often noisy and poorly temporally aligned with the video content (*e.g*., ASR narrations).
Therefore, in this work, we investigate how far an approach can go without requiring extensive pretraining.
Our hypothesis is that learning an efficient prediction model (*i.e*., transition function) over strong abstract representations from frozen visual encoders offers a compelling alternative to extensive large-scale pretraining for procedural learning tasks.

To this end, we propose our framework
VEDiT - Video Embedding Diffusion Transformer – a scalable diffusion transformer (DiT,  Peebles & Xie ())-based prediction model to encode multi-step procedural videos.
VEDiT inherits both the diffusion-style training objective and architecture. Specifically, during training, we utilize the latest Flow Matching technique (Esser et al., ; Lipman et al., ; Goodfellow et al., ) for iterative denoising from random Gaussian noise into video clip embeddings.
Unlike DiT-based models designed for fine-grained image/video generation (Yang et al., ; Esser et al., ) which operate at the patch level, our prediction model works as the step/state transition function, utilizing the abstract frame-level representations from frozen visual encoders, operating in latent space (LeCun, ).
This abstraction allows our model to capture the temporal aspects of the procedural learning task, resulting in the ability to learn an efficient transition function.
Crucially, our method does not require pre-training as it utilizes existing pre-trained representations, nor does it rely on additional supervision (from text or ASR).

We evaluate our model on five diverse procedural learning tasks across four datasets. (1) On the COIN (Tang et al., ) dataset, our model outperforms previous SoTA by a large margin (+5.0% for step forecasting and +3.8% for task classification), and demonstrates scalable learning as we increase the model size.
(2) Our newly proposed VEDiT
significantly enhances the overall performance of previous
SoTA (Niu et al., )
on procedure learning tasks on NIV (Alayrac et al., ), CrossTask (Zhukov et al., ), and COIN (up to +2.28% in success rate, +3.39% in mean accuracy, and +0.90% in mean IoU), as well as the strong baseline for the Ego4D-v2 (Grauman et al., ) long-horizon action anticipation task (+2.6% in Verb ED@20, +3.1% in Noun ED@20).
(3) Finally, we conduct detailed ablation studies on the choice of visual encoders, architecture ablations and large-scale pretraining, and confirm the effectiveness of each component of our framework design.

In a nutshell, our main contributions can be summarized as follows:

- •
We propose a procedural video representation learning framework (VEDiT) which leverages diffusion transformers
to predict visual representations entirely in the embedding space.
- •
By combining strong pretrained visual encoders with a simple prediction model design, our framework is designed to be trained effectively on a single cross-entropy loss for downstream tasks, eliminating the need for large-scale pretraining and additional supervision from actions labels or language for learning the prediction model.
- •
We evaluate VEDiT on five downsteam tasks, including step classification, step forecasting, task classification, procedure planning, and long-term action anticipation across four widely-used benchmark datasets. Our framework outperforms previous SoTAs and baselines by a large margin.

Figure: Figure 1: Overview of our VEDiT training pipeline. Model architecture (left): We introduce masked clip-level latent prediction as our training objective, where we train a Vector Embedding DiT (VEDiT) to iteratively denoise $T$ steps from random gaussian noise with flow matching noise scheduler. Downstream tasks (right): We train VEDiT with a light-weight attentive classifier (Bardes et al., ) with cross-entropy loss for the following tasks. (a) Step forecasting / action anticipation task: predict the embeddings of next unseen clip from observed clips with VEDiT. (b) Procedure planning task: predict the embeddings of intermediate unseen clips from observed starting and goal clips with VEDiT. (c) Procedural activity classification task: given a sequence of observed video clips, predict the label of the procedural video.
Refer to caption: /html/2410.03478/assets/x1.png

## 2 Related Works

##### Procedural Video Understanding.

Learning procedural knowledge from videos has become an active research area, driven by recent large-scale datasets (Miech et al., ; Sener et al., ; Afouras et al., ; Song et al., ) and models trained on them (Niu et al., ; Wang et al., ; ; Zhao et al., ; Lin et al., ; Zhong et al., ).
These models often rely heavily on large-scale text supervision. For example,
DistantSup (Lin et al., ) creates text supervision by linking step descriptions from a textual knowledge base (wikiHow) (Koupaee & Wang, ) to text narrations from ASR in videos.
ProceduralVRL (Zhong et al., ) aligns ASR narration embeddings to video representations using strong pretrained image-language models (Radford et al., ).
Moreover, prior work uses architectures that necessitate large-scale pretraining (e.g., self-attention transformers in ProceduralVRL). In contrast, we propose an efficient architecture that learns directly from video, side-stepping the requirement for large-scale language annotations or pretraining.

##### Diffusion Transformers and Flow Matching.

Diffusion models (Song et al., ; Ho et al., ; Sohl-Dickstein et al., ; Song et al., ) have emerged as a new state-of-the-art architecture for deep generative models. Compared with UNet-based diffusion models (Rombach et al., ; Blattmann et al., ), recent architectures (Ma et al., ; Chen et al., ; Esser et al., ; Gao et al., ; Yang et al., ) designed based on diffusion transformers (Peebles & Xie, ) have achieved significant success and scalability in image and video generation.
On the other hand, flow matching (Lipman et al., ; Ma et al., ; Karras et al., ; Nichol & Dhariwal, ) has shown great potential as an alternative to DDPM (Nichol & Dhariwal, ) for diffusion model noise scheduling.
Inspired by these advances, our work introduces a novel modification of the DiT architecture and flow matching for procedural activity learning, that leverages pretrained visual features and explicitly models the temporal order of steps.

##### Procedural Activity Learning with Diffusion Models.

Prior work has incorporated diffusion into procedural learning in various ways. Some works (Soucek et al., ; Black et al., ) propose using image- and text-conditioned diffusion-based generation or editing models to generate images of actions and object state changes while preserving the input image scene. These methods primarily use diffusion models as off-the-shelf tools for generating intermediate steps in the pixel space. In contrast, other works (Fang et al., ; Shi et al., ; Wang et al., ; Zhong et al., ) integrate diffusion as a training objective within their model design, predicting the embeddings of unseen target clips based on the embeddings of observed clips.
Our work follows this second approach, treating diffusion (flow matching) as a noise scheduler that denoises video embeddings from random noise, and we have designed a procedural learning framework based on the latest DiT architecture.
Our framework differs from previous works in three folds: (1) training supervision: our model is designed to be trained directly with cross-entropy loss on downstream tasks, without the need for extra language supervision; (2) prediction model architecture: instead of using vanilla transformer blocks as the denoising model, we introduce a new Vector Embedding DiT architecture for procedural learning from videos, which is proved to be more effective;
(3) latent embedding generation: Our model generates unseen video embeddings from a frozen encoder, thereby operating in the latent embedding space.

Figure: Figure 2: Vector Embedding Diffusion Transformer (VEDiT) architecture. During training, our model first uses frozen visual encoders to convert observed video clips into corresponding video embeddings. Then random Gaussian noises are generated as the initial video embeddings of unseen target clips. The DiT-based prediction model processes both seen and target video embeddings in two separate branches, and fuses their information via joint attention blocks where $\text{Q}^{\prime}=\text{Concat}[\text{Q}^{\text{s}},\text{Q}^{\text{t}}]$, $\text{K}^{\prime}=\text{Concat}[\text{K}^{\text{s}},\text{K}^{\text{t}}]$, $\text{V}^{\prime}=\text{Concat}[\text{V}^{\text{s}},\text{V}^{\text{t}}]$. To enable temporal modeling of clips, Rotary positional embeddings (RoPE) is applied to $\text{Q}^{\prime}$ and $\text{K}^{\prime}$ before being input to the attention module. The denoised target clip embeddings are then given as input to the attentive classifier in downstream tasks.
Refer to caption: /html/2410.03478/assets/x2.png

## 3 Approach

In this work, we look at three canonical tasks from the procedural representation learning literature, Step forecasting, Procedure Planning, and Task classification, following the setup from Zhong et al. (); Niu et al. ().
Given a series of video clips (or states) from a procedural event (i.e. cooking a dish), the objective is to a) predict the label of the unseen future event (or state) to occur (step forecasting), b) predict the label of unseen events that happened in-between (procedure planning) and c) predict the label of the entire set of events, from a list of probable classes (task classification).
Generally, given a sequence of $N$ observable video clip representations $(\bm{v}_{i})$, we aim to learn a procedural state representation $\bm{\hat{v}}$, which can either capture the information of unseen clips (step forecasting or procedure planning) or a summary information of the task (task classification) using a conditional predictor $\bm{\hat{v}}=\bm{\mathcal{F}_{\theta}}(\{\bm{v}_{i}|i\in\mathcal{S}\})$, where $\mathcal{S}$ is a set of observable clips.
Using this representation, we then aim to learn a classifier to predict the class labels ($C$) for the given task $h:\mathbb{R}^{k\times D}\xrightarrow{}\mathcal{C}$, where $k$ denotes a set of output embeddings for representation $\bm{\hat{v}}$ of dimension $D$.

Learning a predictor to predict an unseen clip representation typically requires an extensive training process to learn a rich visual representation and temporal information. In this work, we bypass the need of learning visual information by leveraging existing pre-trained encoders $\bm{v}_{i}=\bm{\tau}^{*}(v_{i})\in\mathbb{R}^{k\times D}$,
where $v_{i}$ being the clip in pixel space. Therefore, we focus on learning the temporal transition among clips by operating over the encoder embeddings, $\bm{\hat{v}}=\bm{\mathcal{F}_{\theta}}(\{\bm{\tau}^{*}(v_{i})|i\in\mathcal{S}\})$, where the predictor needs to generate latent representation $\mathbb{R}^{k\times D}$ embeddings. To generate embeddings with rich visual signals, we draw inspiration from the diffusion model literature, particularly recent diffusion transformers (DiT) (Peebles & Xie, ; Esser et al., ). Given their powerful text-conditioned image and video generation capabilities (Yang et al., ; Ma et al., ), we adapt their strong conditional generation architecture into a sequential step prediction model for procedural activities.
Thus, learning a strong predictor would allow us to generate unseen clip embeddings, to enable us to perform the tasks in procedural representation learning.

### 3.1 Preliminary: Latent Diffusion Model and Rectified Flows

Given an image $\bm{x}\in\mathbb{R}^{3\times H\times W}$ with caption $\bm{c}$, image Latent diffusion models (LDMs) (Rombach et al., ) first use an atoencoder $\mathcal{E}$ to encode the image into latents $\bm{z}_{0}=\mathcal{E}(\bm{x})\in\mathbb{R}^{C\times H^{\prime}\times W^{\prime}}$, where $C$ represents the number of latent channels, and $H^{\prime}=H/p$, $W^{\prime}=W/p$ represent the spatial dimension of the latents, with $p$ denotes as patch size.
The forward diffusion process is a fixed diffusion process which adds random noise to the latent variable $\bm{z}_{0}$. For example, forward process with Rectified Flows (RFs) (Liu et al., ; Lipman et al., ; Albergo & Vanden-Eijnden, ) is defined as a straight path between the data distribution $\bm{z}_{0}$ and a standard normal distribution $\bm{\epsilon}$ (*i.e*., $\bm{z}_{t}=(1-t)\bm{z}_{0}+t\bm{\epsilon}$, where $t\in[0,1]$). The reverse process in RFs, on the other hand, gradually produces less noise samples
starting from $\bm{z}_{1}$ to $\bm{z}_{0}$
in $T$ denoising steps through a learnable transformer-based denoiser model $\bm{\mathcal{F}_{\theta}}$ parameterized by $\theta$ and conditioned on caption $\bm{c}$.
In our setup, we transform such denoiser model conditioned on caption/text into a sequential step prediction model for procedural activities conditioned on observed video clips, with explicit temporal order modeling via RoPE (Su et al., ).

### 3.2 Our Approach: Vector Embedding Diffusion Transformers ( VEDiT )

##### Overall Training Pipeline of VEDiT .

The overall training pipeline of our VEDiT is illustrated in [Fig. 1](#S1.F1) left.
Given a set $\mathcal{S}$ that contains $N$ observable (seen) video clips $\{{v}_{1},{v}_{2},...,{v}_{N}\}$,
where each video clip ${v}_{i}\in\mathbb{R}^{K\times 3\times H\times W}$ contains $K$ frames, we first apply a frozen visual encoder $\bm{\tau}^{*}(.)$ to derive the corresponding video embeddings for each clip $\bm{v}_{i}=\bm{\tau}^{*}({v}_{i})$.
Next, random Gaussian noises are generated as the initial video embeddings of unseen clips $\{\bm{\tilde{v}}_{j}|j\in\mathcal{T}\}$, where $\mathcal{T}$ represents the target set.
Then we design VEDiT as the learnable prediction model $\bm{\mathcal{F}_{\theta}}$ which predicts the unseen target video clip embeddings ($\bm{\hat{v}})$ conditioned on all seen video embeddings: $\bm{\hat{v}_{j}}=\bm{\mathcal{F}_{\theta}}(\{\bm{v}\}_{i},\bm{\tilde{v}}_{j}),i\in\mathcal{S};j\in\mathcal{T}$.
This prediction model is then trained using iterative denoising (Ho et al., ) over $T$ steps, with diffusion timestamps sampled from the Flow Matching Euler Discrete Scheduler (Esser et al., ).
Unlike DiT, which is based-on pixel-level and text-conditioned generation, our model is designed to predict abstract video features in the procedural activity, based on observed video clips.
Additionally, unlike previous works (Lin et al., ; Zhong et al., ) for procedural activity understanding, our method does not use extra language supervision such as ASR or textual knowledge base (*e.g*., wikiHow) that aligns visual embeddings with text, and is trained directly with the cross-entropy loss on downstream tasks.

##### Choice of Visual Encoder.

Previous works on procedural learning (Lin et al., ; Zhong et al., ; Niu et al., ) typically use clip-level features as abstracted visual representation for each video clip, which can result in a loss of detail. Conversely, DiT models for image and video generation (Yang et al., ; Esser et al., ; Peebles & Xie, ) are designed for patch-level generation with a focus on fine-grained visual details, but this comes at the cost of higher computational demands.
To ensure that our model can process videos with multiple clips, encode sufficient visual information, and avoid excessive computational costs, we explore our model with diverse CLIP- and SSL-based encoders that output visual features at the clip-, frame-, and patch-levels. We found empirically that using the [CLS] tokens of SigLIP (Zhai et al., ) from 16 uniformly sampled frames in each clip stands out as the strongest visual representation. An ablation study of visual encoders is discussed in [Sec. 4.3](#S4.SS3).

##### VEDiT Model Architecture Design.

[Fig. 2](#S2.F2) visualizes the components of each VEDiT block.
Our architecture is derived from DiT (Peebles & Xie, ), which has a two-branch architecture with one query branch (typically text) tasked to condition the target branch (vision) through adaptive layernorm  (Perez et al., ).
In our work, we utlize the query branch to process the observable (or seen) video encoder embeddings $\{\bm{{v}}_{i}|i\in\mathcal{S}\}$, which conditions the target branch that operates on unseen, noisy embeddings $\{\bm{\tilde{v}}_{j}|j\in\mathcal{T}\}$ through adaptive layernorm, which gets iteratively updated through denoising.
The query branch is further conditioned by using the timestamp $t\in T$ sampled from the noise scheduler, along with its usual application of determining the scale of noise.
Unlike DiT, the information of the query branch and the target branch are fused together using joint attention before being processed independently through feed-forward layers without weight sharing.
Lastly, to enable temporal modeling of clips, Rotary positional embeddings (RoPE) (Su et al., ) is applied to the input immediately prior to the joint attention module.
By utilizing these mechanisms, VEDiT allows us to learn unseen video embeddings, starting from noise, by conditioning on the observed video encoder representations. More ablations on design choices are provided in [Sec. A.2](#A1.SS2).

## 4 Experiments

In this section, we first introduce the evaluation datasets and the implementation details of our model in [Sec. 4.1](#S4.SS1). We then compare our method with SOTAs on five downstream tasks across four datasets in [Sec. 4.2](#S4.SS2).
Finally, we present ablation studies on the visual encoders, as well as the necessity of pretraining in [Sec. 4.3](#S4.SS3).
More ablations on VEDiT architecture design is provided in [Sec. A.2](#A1.SS2).

### 4.1 Experimental Setup

##### Evaluation Tasks and Datasets.

We evaluate our method on five downstream tasks across four datasets. COIN (Tang et al., ) contains 476 hours of YouTube videos covering 180 tasks and 778 unique steps of daily activities. Following (Zhong et al., ; Lin et al., ), we evaluated our model on two tasks: step forecasting and task classification (see [Fig. 1](#S1.F1) for details). For Ego4D-v2 (Grauman et al., ), we focus on the long-term action anticipation benchmark, which aims to predict the next $Z=N-t$ future action classes [(verb1, noun1), (verb2, noun2), …, (verbZ, nounZ)] given an input video up to timestamp $t$. This forecasting benchmark contains 243 hours of videos with a total of 3472 annotated clips. In addition, we utilize NIV (Alayrac et al., ), CrossTask (Zhukov et al., ), and COIN datasets to evaluate the procedure planning task (Chang et al., ), which can be seen as a variant of step forecasting task that aims at predicting intermediate action steps given the observed start and goal video clips. Specifically, CrossTask dataset contains 2750 videos covering 18 tasks and 133 actions, and NIV dataset contains 150 videos with 5 tasks and 48 actions. Following (Niu et al., ), we report the results with prediction horizon $T\in\{3,4\}$.

##### Evaluation Metrics.

For COIN step forecasting and task classification tasks, we use top-1 classification accuracy of the predicted step/task as the evaluation metric following DistantSup (Lin et al., ).
For Ego4D-v2 long-horizon anticipation task, we use the default edit distance (ED) metric, which is computed as the Damerau-Levenshtein distance (Damerau, ) over sequences of predicted verbs or nouns. Following (Grauman et al., ), we report the minimum edit distance at $Z=20$ (ED@20) for $K=5$ predicted sequences on the validation set. In addition, for procedure planning tasks on NIV, CrossTask, and COIN, we evaluate the models on three metrics, including success rate (SR), mean Accuracy (mAcc) and mean Intersection over Union (mIoU) following previous works (Chang et al., ; Niu et al., ; Zhao et al., ).

##### Implementation Details.

Our default VEDiT architecture contains 12 transformer blocks, with a hidden size of 2048 and attention head dimension of 64.
During training, we apply classifier-free guidance with a scale of 7 and denoise the diffusion model for 24 steps using the Flow Matching Euler Discrete Scheduler (Esser et al., ).
For COIN step forecasting and task classification tasks, we use a scheduled learning rate linearly increases from $5\times 10^{-6}$ to $5\times 10^{-5}$ during the first 3 epochs, and then decays to $5\times 10^{-7}$ following a cosine schedule, with a total of 30 epochs. For long-horizon anticipation and the procedure planning tasks, following the same training setting in previous works (Grauman et al., ; Niu et al., ), we train the model
for 100 and 500 epochs respectively.
Together with VEDiT, we train task specific attentive classifiers $h:\mathbb{R}^{k\times D}\rightarrow C$ (Bardes et al., ), which is an attentive pooler over $k$ output embeddings followed by a single linear layer.

### 4.2 Main results

##### Step Forecasting and Task Classification.

In [Table 1](#S4.T1), we demonstrate the effectiveness of our VEDiT design on the COIN step forecasting and task classification tasks. Firstly, we use the pretrained TimeSformer (Bertasius et al., ) visual encoder as $\bm{\tau}^{*}$, as used in ProceduralVRL (Zhong et al., ), the previous state-of-the-art in these tasks. We combined the TimeSformer encoder with VEDiT designed with joint attention, to arrive at the model TimeSformer+VEDiT.
This model achieves improvements of 2.2% and 0.6% in top-1 accuracy on the step forecasting and task classification tasks.
Next, using SigLIP (Zhai et al., ) as $\bm{\tau}^{*}$ with VEDiT yields additional gains of 3.1% and 3.2% on these two tasks. It is worth noting our methods does not require any large-scale pretraining, which proves the effectiveness of using strong language-aligned pretrained representations.
Additionally, our method does not require explicit text supervision (*i.e*., unsupervised) compared to baselines DistantSup and ProceduralVRL, which are trained with explicit language matching loss (*i.e*., ASR or ASR+wikiHow).
Furthermore, we observe linear *scalability* of VEDiT on Step Forecasting task, leading to improved numbers with increasing number of model parameters ([Sec. A.2.3](#A1.SS2.SSS3)).

Figure: Figure 3: SCHEMA w/ VEDiT is more stable than SCHEMA w/ vanilla transformer as we increase the number of transformer blocks.
Refer to caption: /html/2410.03478/assets/x3.png

##### Procedure Planning Task.

We further evaluate VEDiT on
procedure planning results on the NIV, COIN, and CrossTask datasets with horizons $T\in\{3,4\}$ in [Table 2](#S4.T2). Specifically, we build upon the previous SoTA model, SCHEMA (Niu et al., ), by replacing their vanilla transformer blocks in the state decoder and step decoder with our VEDiT blocks. To ensure a fair comparison, we use the same number of transformer blocks (*i.e*., 2 blocks) with identical hidden dimensions, attention heads, and we strictly adhere to their training and evaluation hyperparameters and setups without any changes. We report the mean and standard deviation of SR, mAcc, and mIoU for our results as well as our replication of SCHEMA averaged over 10 runs.

As shown in [Table 2](#S4.T2), SCHEMA with VEDiT consistently outperforms the original SCHEMA method on NIV (gains of 0.97%-2.28% for SR, 1.92%-3.39% for mAcc, and 0.49%-0.90% for mIoU) and COIN (gains of 2.21%-5.89% for SR, 2.31%-7.07% for mAcc, and 0.86%-2.57% for mIoU). Additionally, our VEDiT achieves better average performance on the CrossTask dataset. Moreover, as plotted in [Fig. 3](#S4.F3), we also observe better stability of VEDiT over vanilla transformer blocks as we scale up the number of transformer blocks.

**Table 1: Step forecasting and task classification results on COIN (Tang et al., ) dataset. We compare our method with a set of strong baselines as well as SOTA methods. Top-1 accuracies are reported. We bold and underline the best and the second best models in each task respectively.**
| Model | Pretraining Supervision | Pretrain Data | Step Forecasting | Task Classification |
| --- | --- | --- | --- | --- |
| Random Guess | N/A | N/A | 0.1 | - |
| SlowFast (Feichtenhofer et al., ) | Supervised: action labels | Kinetics | 25.6 | 71.6 |
| S3D (Xie et al., ) | Unsupervised: ASR w. MIL-NCE | HT100M | 28.1 | 70.2 |
| ClipBERT (Lei et al., ) | Supervised: captions | COCO+VG | - | 65.4 |
| VideoCLIP (Xu et al., ) | Unsupervised: ASR | HT100M | - | 72.5 |
| TSN (RGB + Flow) (Tang et al., ) | Supervised: action labels | Kinetics | - | 73.4 |
| TimeSformer (Bertasius et al., ) | Supervised: action labels | Kinetics | 34.7 | 83.5 |
| TimeSformer (Bertasius et al., ) | Unsupervised: ASR w. MIL-NCE | HT100M | 34.0 | 85.3 |
| DistantSup (Lin et al., ) | Unsupervised: ASR + wikiHow | HT100M | 39.4 | 88.9 |
| ProceduralVRL (Zhong et al., ) | Unsupervised: ASR | HT100M | 46.8 | 90.8 |
| Ours: TimeSformer + VEDiT | N/A | N/A | 48.7 | 91.1 |
| Ours: SigLIP (Zhai et al., ) + VEDiT | N/A | N/A | 51.8 | 94.6 |

##### Long-Horizon Action Anticipation.

In [Table 3](#S4.T3), we evaluate our model on the Ego4D-v2 long-horizon action anticipation task.
We introduce a new baseline by replacing the SlowFast (Feichtenhofer et al., ) visual encoder in the Ego4D baseline model with SigLIP, while keeping the prediction model (*i.e*., slowfast_trf_v2) unchanged.
For a fair comparison, we initialize VEDiT with the same number of transformer blocks and hidden dimension size as the Ego4D baseline transformer prediction model.
Using SigLIP as visual encoder, VEDiT outperforms the Ego4D baseline in both Verb ED@20 (Ours: 0.697 v.s. Ego4D Baseline: 0.718, lower is better) and noun ED@20 (Ours: 0.711 v.s. Ego4D Baseline: 0.742, lower is better).
In addition, while not directly comparable, PaMsEgoAI (Ishibashi et al., ), achieves lower ED metrics by introducing several enhancements, including an ensemble of SlowFast and SlowFast-CLIP models, label smoothing to relax order constraints for future actions, and constraining the (verb, noun) classes based on word co-occurrence. Some studies (Zhao et al., ; Pei et al., ; Huang et al., ) have found that combining vision models with the strong planning capabilities of LLMs can achieve good performance, particularly for long-horizon action anticipation tasks. Therefore, integrating our method with an LLM could be a promising future direction.

**Table 2: Procudure planning results on NIV (Alayrac et al., ), COIN (Tang et al., ), and CrossTask (Zhukov et al., ) datasets with prediction horizon $T\in\{3,4\}$. SCHEMA^†: our replication of their method averaged over 10 runs. The best numbers are bolded. Our improvement over SCHEMA baseline is colored in blue.**
| Datasets | Models | $T=3$ | $T=4$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SR ($\uparrow$) | mAcc ($\uparrow$) | mIoU ($\uparrow$) | SR ($\uparrow$) | mAcc ($\uparrow$) | mIoU ($\uparrow$) |  |  |
| NIV | Random | 2.21 | 4.07 | 6.09 | 1.12 | 2.73 | 5.84 |
| DDN (Chang et al., ) | 18.41 | 32.54 | 56.56 | 15.97 | 27.09 | 53.84 |  |
| Ext-GAIL (Bi et al., ) | 22.11 | 42.20 | 65.93 | 19.91 | 36.31 | 53.84 |  |
| P^3IV (Zhao et al., ) | 24.68 | 49.01 | 74.29 | 20.14 | 38.36 | 67.29 |  |
| EGPP (Wang et al., ) | 26.05 | 51.24 | 75.81 | 21.37 | 41.96 | 74.90 |  |
| SCHEMA (Niu et al., ) | 27.93 | 41.64 | 76.77 | 23.26 | 39.93 | 76.75 |  |
| SCHEMA^† | 26.66$\pm$2.27 | 39.94$\pm$2.79 | 75.58$\pm$1.47 | 22.32$\pm$1.15 | 36.96$\pm$1.84 | 74.39$\pm$1.13 |  |
| SCHEMA w/ VEDiT | 28.94$\pm$1.07 | 43.33$\pm$0.90 | 76.48$\pm$0.62 | 23.29$\pm$0.44 | 38.88$\pm$1.19 | 74.88$\pm$0.89 |  |
| (Ours) | (2.28$\uparrow$) | (3.39$\uparrow$) | (0.90$\uparrow$) | (0.97$\uparrow$) | (1.92$\uparrow$) | (0.49$\uparrow$) |  |
| COIN | Random | $<$0.01 | $<$0.01 | 2.47 | $<0.01$ | $<0.01$ | 2.32 |
| Retrieval | 4.38 | 17.40 | 32.06 | 2.71 | 14.29 | 36.97 |  |
| DDN (Chang et al., ) | 13.90 | 20.19 | 64.78 | 11.13 | 17.71 | 68.06 |  |
| P^3IV (Zhao et al., ) | 15.40 | 21.67 | 76.31 | 11.32 | 18.85 | 70.53 |  |
| EGPP (Wang et al., ) | 19.57 | 31.42 | 84.95 | 13.59 | 26.72 | 84.72 |  |
| SCHEMA (Niu et al., ) | 32.09 | 49.84 | 83.83 | 22.02 | 45.33 | 83.47 |  |
| SCHEMA^† | 26.38$\pm$3.66 | 43.08$\pm$4.28 | 81.49$\pm$1.70 | 21.00$\pm$2.56 | 43.37$\pm$3.64 | 82.70$\pm$1.08 |  |
| SCHEMA w/ VEDiT | 32.27$\pm$0.44 | 50.15$\pm$0.31 | 84.07$\pm$0.38 | 23.11$\pm$0.27 | 45.68$\pm$0.52 | 83.56$\pm$0.45 |  |
| (Ours) | (5.89$\uparrow$) | (7.07$\uparrow$) | (2.57$\uparrow$) | (2.11$\uparrow$) | (2.31$\uparrow$) | (0.86$\uparrow$) |  |
| CrossTask | Random | $<$0.01 | 0.94 | 1.66 | $<0.01$ | 0.83 | 1.66 |
| Retrieval | 8.05 | 23.30 | 32.06 | 3.95 | 22.22 | 36.97 |  |
| DDN (Chang et al., ) | 12.18 | 31.29 | 47.48 | 5.97 | 27.10 | 48.46 |  |
| Ext-GAIL (Bi et al., ) | 21.27 | 49.46 | 61.70 | 16.41 | 43.05 | 60.93 |  |
| P^3IV (Zhao et al., ) | 23.34 | 49.96 | 73.89 | 13.40 | 44.16 | 70.01 |  |
| PPDP (Wang et al., ) | 26.38 | 55.62 | 59.34 | 18.69 | 52.44 | 62.38 |  |
| EGPP (Wang et al., ) | 26.40 | 53.02 | 74.05 | 16.49 | 48.00 | 70.16 |  |
| SCHEMA (Niu et al., ) | 31.83 | 57.31 | 78.33 | 19.71 | 51.85 | 74.46 |  |
| SCHEMA^† | 30.57$\pm$0.38 | 56.02$\pm$0.32 | 77.60$\pm$0.25 | 20.26$\pm$0.33 | 51.93$\pm$0.17 | 74.51$\pm$0.25 |  |
|  | SCHEMA w/ VEDiT | 31.08$\pm$0.31 | 56.15$\pm$0.57 | 77.54$\pm$0.35 | 20.42$\pm$0.24 | 52.26$\pm$0.51 | 74.76$\pm$0.29 |
|  | (Ours) | (0.51$\uparrow$) | (0.13$\uparrow$) | (0.06$\downarrow$) | (0.16$\uparrow$) | (0.33$\uparrow$) | (0.25$\uparrow$) |

**Table 3: Comparison of methods on the validation set of Ego4D (Grauman et al., ) long-term action anticipation challenge. Edit distance (ED) metrics are reported at prediction horizon $5$ and $20$.**
| Method | Encoder | Prediction Model | ED@5 ($\downarrow$) | ED@5 ($\downarrow$) | ED@20 ($\downarrow$) | ED@20 ($\downarrow$) |
| --- | --- | --- | --- | --- | --- | --- |
| Verb | Noun | Verb | Noun |  |  |  |
| Ego4D Baseline (Grauman et al., ) | SlowFast | Transformer | - | - | 0.745 | 0.779 |
| Ego4D Baseline (Grauman et al., ) | SigLIP | Transformer | 0.703 | 0.736 | 0.718 | 0.742 |
| PaMsEgoAI (Ishibashi et al., ) | SlowFast + CLIP | Concat + Transformer | - | - | 0.670 | 0.629 |
| Ours | SigLIP | VEDiT | 0.677 | 0.711 | 0.697 | 0.711 |

### 4.3 Ablations

#### 4.3.1 Which visual encoder works best?

To test the impact of visual encoders on procedural activity understanding from instructional videos, we train VEDiT with 3 blocks for 10 epochs with different visual encoders. We include strong CLIP-based and self-supervised (SSL) encoders, including
SigLIP (ViT-SO400M/14@384) (Zhai et al., ), V-JEPA (Bardes et al., ), DINOv2 (Oquab et al., ), and VideoMAE Tong et al. (). For V-JEPA and VideoMAE, we provide patch tokens to VEDiT, while for DINOv2 and SigLIP, we provide [CLS] tokens. We evaluate the model trained with different encoders on COIN for step forecasting and task classification tasks.

As we observe from [Table 4](#S4.T4), SigLIP outperforms SSL-based encoders. Among the SSL-based encoders, VJEPA ViT-H 384 and DINOv2 performs comparably than the baselines for step forecasting and task classification task respectively. SigLIP outperforms both on a large margin, especially in Task classification, highlighting the need of language-aligned rich visual representations for stronger procedural activity understanding.

**Table 4: Ablation on frozen video encoders on COIN (Tang et al., ) step forecasting and procedural activity classification tasks. Top-1 accuracies are reported. We bold and underline the best and the second best models in each task respectively.**
| Model | Architecture | Pretrain Data | CLIP/SSL | Token | Step Forecasting | Task Classification |
| --- | --- | --- | --- | --- | --- | --- |
| DINOv2 | ViT-Giant/14@224 | LVD-142M (Oquab et al., ) | SSL | [CLS] | 47.03 | 90.89 |
| VJEPA | ViT-Huge/16@224 | VideoMix2M (Bardes et al., ) | SSL | Patch | 47.24 | 86.13 |
| VJEPA | ViT-Huge/16@384 | VideoMix2M (Bardes et al., ) | SSL | Patch | 48.23 | 87.14 |
| VideoMAE | ViT-Huge/16@224 | K400 (Kay et al., ) | SSL | Patch | 44.78 | 83.40 |
| SigLIP (default) | ViT-SO400M/14@384 | WebLI (Chen et al., ) | CLIP | [CLS] | 50.05 | 94.38 |

#### 4.3.2 Is pre-training necessary?

Instead of training directly on downstream datasets (*i.e*., COIN), previous works (Zhong et al., ; Lin et al., ) undergo large-scale pretraining on publicly available video datasets, such as HowTo100M (Miech et al., ).
In this section, we question whether such pretraining is necessary. Specifically, we compare our model trained directly on COIN with a variant that includes additional pretraining on HowTo100M dataset. Following the reicpe of  Zhong et al. (), we set the total number of clips to 9, randomly mask out the video embedding of one clip, and use the masked clip embedding reconstruction as the training objective.
Additionally, instead of relying on the noisy automatic speech recognition (ASR) annotations, we use the starting and ending timestamps of each video clip processed and filtered by HowToCaption (Shvetsova et al., ).
During pretraining, we employ the AdamW (Loshchilov & Hutter, ) optimizer, with the learning rate linearly increasing from $1\times 10^{-5}$ to $1\times 10^{-4}$ during the first 0.5 training epochs, and then remaining constant for a total of 30 epochs. The pretraining is conducted on 128 H100 GPUs with a total batch size of 1024, and takes 2 days and 4.5 days for the 165M and 1.77B VEDiT models respectively (see [Table 7](#A1.T7) for model architecture details).

Surprisingly, we observe only marginal improvement with significant pretraining ([Table 5](#S4.T5)). Pretraining only provides an additional 0.3% boost in top-1 accuracy for the step forecasting task. We hypothesize this limited effectiveness of pretraining may stem from two factors: (1) pretraining dataset is relatively noisy, leading to a distribution gap with the downstream COIN dataset, and (2) the pretraining objective in (Zhong et al., ) may not be optimal.
Exploring better pretraining objectives that can generalize well across different downstream tasks is left for future work.

**Table 5: Effect of large-scale pretraining. We compare VEDiT without large-scale pretraining with a variant that’s pretrained on 1.16M videos from HowTo100M (Miech et al., ) dataset, using temporal information from HowToCaption (Shvetsova et al., ). Top-1 accuracies are reported.**
| Model | Pretraining Supervision | Pretrain Data | Step Forecasting | Task Classification |
| --- | --- | --- | --- | --- |
| SigLIP + VEDiT | N/A | N/A | 51.8 | 94.6 |
| SigLIP + VEDiT | Unsupervised: HowToCaption | HT100M | 52.1 | 94.6 |

## 5 Conclusion

In this work, we demonstrate that carefully designed predictive models learned on top of pretrained visual representations can achieve state-of-the-art performance on procedural learning tasks across the COIN, CrossTask, NIV, and Ego4D datasets, including step forecasting, procedural activity classification, procedure planning, and long-term action anticipation. Notably, we achieve these results without pretraining the prediction module, instead learning it directly from the end tasks. This contrasts with previous works, which often require computationally expensive pretraining of the predictor, sometimes with additional supervision.
Our findings suggest that further research is needed to improve pretraining for procedural activity tasks. Specifically, exploring ways to better align pretraining tasks with downstream tasks could help fully leverage the benefits of pretraining.

## 6 Reproducibility

In this work, we aim to make VEDiT training and evaluation reproducibile, so that readers can assimilate the contributions in their own work.
By design, VEDiT is highly reproducible as it doesn’t require expensive pre-training, as it works on top of frozen vision encoders, specifically SigLIP(^1^11[https://huggingface.co/google/siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384)) for our main experiments. We ground the discussion of model development in [Sec. 3.2](#S3.SS2). We provide details of the training pipeline, training & evaluation data and metrics used in [Sec. 4.1](#S4.SS1).
Further results on step classification task and procedure planning tasks are provided in [Sec. A.1](#A1.SS1) for assiting replicability and generalization of VEDiT.
Detailed ablation results on the model design, including choice of attention mechanism and denoising steps, are provided in [Sec. A.2.3](#A1.SS2.SSS3).
Finally, we provide detailed results on the scalability of VEDiT to further shed light on its generalizability with increasing amount of model parameters.
PyTorch (Ansel et al., ) implementation of VEDiT blocks is provided in [Algorithm 1](#alg1).
We will release our code post peer review.

## 7 Acknowledgements

We would like to thank Mike Rabbat & Yann LeCun for their constant support & feedback in project and manuscript. We would also like to thank Peter Tong, Samuel Lavoie, Russ Howes, Adrien Bardes, David Fan, Quentin Garrido & Vlad Sobal for numerous feedback and discussions on various lifecycles of the project. We would like to thank Shubho Sengupta, Jacob Kahn, Ananya Saxena, Kalyan Saladi, Gabriel Synnaeve, Cody Ohlsen, Mack Ward, Maxwell Taylor, Adithya Kumar, Caleb Ho, Parth Malani & Xiaodong Ma for their dedicated support and troubleshooting on our GPU clusters. Finally, we would like to thank Carolyn Krol & Kathleen Roberts for their help in navigating policy and logistics.

## Appendix A Appendix

In this appendix, we first present additional results on the step classification task ([Sec. A.1.1](#A1.SS1.SSS1)) and procedure planning task ([Sec. A.1.2](#A1.SS1.SSS2)). Then we discuss ablations of our model design, including the choice of attention mechanism ([Sec. A.2.1](#A1.SS2.SSS1)), the choice of denoising steps ([Sec. A.2.2](#A1.SS2.SSS2)), and the downstream task performance as we scale up the VEDiT model size ([Sec. A.2.3](#A1.SS2.SSS3)). Furthermore, we provide the PyTorch (Ansel et al., ) implementation of VEDiT in [Algorithm 1](#alg1).

### A.1 Additional Results

#### A.1.1 Step Classification Task

In this section, we present additional results on the COIN step classification task, which aims to predict the class labels of single-clip videos. In other words, this task tests only the capability of visual encoders, as the prediction model is not involved.
Specifically, for step forecasting and task classification described in the main paper, we design the attentive pooler as a lightweight single cross-attention block with one query token to pool the video clip embedding (e.g., the predicted frame-level [CLS] tokens in SigLIP (Zhai et al., )) into a single vector. For the COIN step classification task, we increase the depth of the attentive pooler by adding three additional self-attention blocks before the cross-attention block to aggregate information from the visual features, which we find further improves classification accuracy.

We compare our method, which uses off-the-shelf frozen visual encoders with a trainable attentive classifier, against baseline methods reported in previous works (Lin et al., ; Zhong et al., ).
As shown in [Table 6](#A1.T6), our method with all five frozen encoders outperforms previous baselines. V-JEPA performs best among the two SSL-based video encoders (i.e., V-JEPA and VideoMAE). Increasing the resolution from 224 to 384 on V-JEPA further boosts accuracy. Additionally, due to the rich information encoded in patch-level tokens, V-JEPA achieves the best performance on the step classification task among all encoders. Moreover, SigLIP, pretrained on both image and text data, outperforms all other encoders except for V-JEPA, demonstrating the effectiveness of using visual-text aligned encoders for procedural activity understanding in instructional videos.
However, as the prediction model is not involved, we do not put primary focus on this task in our paper.

**Table 6: Step classification on COIN dataset. We bold and underline the best and the second best models in each task respectively. Our strategy of using strong frozen visual encoder with trainable attentive classifier outperforms all baseline methods.**
| Model | Pretraining Supervision | Pretrain Data | Top-1 Acc. (%) |
| --- | --- | --- | --- |
| Baselines |  |  |  |
| SlowFast (Feichtenhofer et al., ) | Supervised: action labels | Kinetics (Kay et al., ) | 32.9 |
| TimeSformer (Bertasius et al., ) | Supervised: action labels | Kinetics (Kay et al., ) | 48.3 |
| ClipBERT (Lei et al., ) | Supervised: captions | COCO+VG (Chen et al., ; Krishna et al., ) | 30.8 |
| VideoCLIP (Xu et al., ) | Unsupervised: ASR | HT100M (Miech et al., ) | 39.4 |
| TimeSformer (Bertasius et al., ) | Unsupervised: ASR w. MIL-NCE | HT100M (Miech et al., ) | 46.5 |
| CLIP (Radford et al., ) | Unsupervised: captions | CLIP400M (Radford et al., ) | 45.9 |
| DistantSup (Lin et al., ) | Unsupervised: ASR + wikiHow | HT100M (Miech et al., ) | 54.1 |
| ProceduralVRL (Zhong et al., ) | Unsupervised: ASR | HT100M (Miech et al., ) | 56.9 |
| Ours (Frozen encoder w/ lightweight trainable attentive classifier) |  |  |  |
| DINOv2 (Oquab et al., ) | Self-supervised | LVD-142M (Oquab et al., ) | 57.9 |
| V-JEPA@224 (Bardes et al., ) | Self-supervised | VideoMix2M (Bardes et al., ) | 61.4 |
| V-JEPA@384 (Bardes et al., ) | Self-supervised | VideoMix2M (Bardes et al., ) | 62.7 |
| VideoMAE (Tong et al., ) | Self-supervised | Kinetics400 (Kay et al., ) | 58.5 |
| SigLIP (Zhai et al., ) | Image+Text Pairs | WebLI (Chen et al., ) | 61.8 |

#### A.1.2 Procedure Planning Task

In addition to the main results presented in [Table 2](#S4.T2), we show in [Fig. 3](#S4.F3) the comparison of our VEDiT and the vanilla transformer model in (Niu et al., ) as we increase the number of transformer blocks.
We report success rate (SR), mean accuracy (mAcc), and mean IoU (mIoU) as evaluation metrics on the NIV, COIN, and CrossTask datasets.
For a fair comparison, we keep all hyper-parameters the same, with the only change being the number of blocks.
We observe that our VEDiT exhibits significantly better stability compared to the vanilla transformer blocks as we scale up the model size, without overfitting to the training set. This finding is consistent with our results in [Fig. 7](#A1.F7).

### A.2 VEDiT Model ablations

#### A.2.1 Choice of Attention mechanism

In [Fig. 5](#A1.F5), we illustrate the differences between our default joint attention in each VEDiT block and self-attention and cross-attention. We denote the observed and unseen video clip embeddings as $\bm{v}^{s}$ or $\bm{v}^{t}$. In self-attention, we concatenate $\bm{v}^{s}$ or $\bm{v}^{t}$ along the sequence dimension as a single input to the attention module. In contrast, cross-attention does not utilize self-attention within $\bm{v}^{s}$ or $\bm{v}^{t}$.
Here we conduct ablation study of these attention mechanisms with a prediction model of 3 VEDiT blocks.
As shown in [Fig. 5](#A1.F5), our joint attention outperforms self-attention in step forecasting (50.3 vs. 49.7) and task classification (94.4 vs. 94.3) on the COIN dataset. This proves the usefulness of processing $\bm{v}^{s}$ and $\bm{v}^{t}$ differently through adaptive normalization layers before inputing to the attention module. In addition, due to the absence of self-attention within $\bm{v}^{s}$ or $\bm{v}^{t}$, cross-attention performs the worst.

Figure: Figure 4: Ablation of attention mechanisms, including our default joint attention, self-attention, and cross-attention. We denote seen and target video clip embeddings as $\bm{v}^{\text{s}}$ and $\bm{v}^{\text{t}}$ respectively.
Refer to caption: /html/2410.03478/assets/x4.png

#### A.2.2 Choice of Denoising steps

Previous work on masked token prediction, such as BERT (Kenton & Toutanova, ) for language and MAE (He et al., ) for images, can be considered single-step denoising, while diffusion models typically perform single-step denoising during training and multi-step denoising during inference. In this context, we conduct an ablation study on different denoising steps using diffusion timestamps sampled from the Flow Matching Euler Discrete Scheduler (Esser et al., ) in our VEDiT training on the COIN step forecasting task. The ablation study here is conducted with a prediction model of 3 VEDiT blocks, and we report the top-1 classification accuracy averaged over three independent runs in [Fig. 6](#A1.F6). Our results show that applying 20 to 40 denoising steps achieves better accuracy compared to single-step denoising (51.01 for 36 denoising steps v.s. 50.77 for single denoising step). This multi-step denoising allows us to reuse the same VEDiT architecture, with observed and target embeddings scaled and shifted by adaptive normalization layers at different timestamps, without drastically increasing the model’s trainable parameters. Additionally, we observe that timestamps that are too sparse (*e.g*., denoising steps of 4) or too dense (*e.g*., denoising steps greater than 44) make the model difficult to optimize. We default to use 24 denoising steps in our main paper as it achieves a good balance between computational cost and accuracy.

Figure: Figure 6: Ablation of denoising steps. We report the top-1 accuracy on COIN step forecasting task. A Denoising steps of 24 achieves a good balance between computational cost and accuracy. The numbers are averaged over 3 independent runs.
Refer to caption: /html/2410.03478/assets/x5.png

#### A.2.3 Scalability of VEDiT

[Table 7](#A1.T7) presents the model architecture details for 10 different scale models we implemented in our scalability results shown in [Fig. 7](#A1.F7). Specifically, we examined two different sets of hidden dimensions (i.e., 1280 and 2048), with varying numbers of VEDiT blocks ranging from 1 to 18. These parameter settings effectively cover model parameters from 62M to 1.77B (up to $\times 28$ larger in scale).
We evaluated these models on the COIN step forecasting task, and the results are averaged over 5 runs. As shown in [Fig. 7](#A1.F7), with the same number of training epochs, a larger VEDiT model achieves a lower top-1 validation error compared to smaller VEDiT models. This demonstrates that VEDiT is scalable as we increase the model size.

**Table 7: Details of VEDiT models. We introduce models of different number of transformer blocks (*i.e*., layers) with two hidden dimension settings.**
| Model | # Train Params | Layers | Hidden Dim. | # Attn. Heads | Head Dim. |
| --- | --- | --- | --- | --- | --- |
| Hidden Dim. = 1280 |  |  |  |  |  |
| VEDiT-Single | 62M | 1 | 1280 | 20 | 64 |
| VEDiT-Tiny | 165M | 3 | 1280 | 20 | 64 |
| VEDiT-Small | 342M | 6 | 1280 | 20 | 64 |
| VEDiT-Large | 696M | 12 | 1280 | 20 | 64 |
| VEDiT-XL | 1.05B | 18 | 1280 | 20 | 64 |
| Hidden Dim. = 2048 |  |  |  |  |  |
| VEDiT-Single | 132M | 1 | 2048 | 32 | 64 |
| VEDiT-Tiny | 418M | 3 | 2048 | 32 | 64 |
| VEDiT-Small | 871M | 6 | 2048 | 32 | 64 |
| VEDiT-Medium | 1.34B | 9 | 2048 | 32 | 64 |
| VEDiT-Large | 1.77B | 12 | 2048 | 32 | 64 |

Figure: Figure 7: Ablation of different VEDiT model sizes. We report the top-1 validation error on COIN step forecasting task. Our VEDiT demonstrates good scalability as we scale up the model size *up to $28$ times from 62M to 1.77B*. The numbers are averaged over 5 independent runs.
Refer to caption: /html/2410.03478/assets/x6.png

Figure: Algorithm 1 Simplified PyTorch Implementation for Each VEDiT Block

Figure: Simplified PyTorch Implementation for JointAttention

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: I also have a variety of questions that did not significantly impact the rating for the paper:
- The paper demonstrated the advantage of using VEDiT on chunk-level classification tasks. Can the approach be extended to other temporal understanding tasks that predict start/end time? e.g., action grounding
- The paper entirely leverages flow-matching sampling with a fixed number of sampling steps. How's the model's performance with different scheduler/sampling parameters?
soundness: 3
strengths: Generally, the primary strength of this paper is the introduction of the novel diffusion-based representation learning method into the video understanding field. VEDIT achieves significant performance gains across multiple procedural tasks, such as step forecasting and task classification. 

**Clarity**: Most of this paper is well-written, with clear explanations and a solid grounding in relevant literature. 

**Originality**: The main idea of training the VEDiT using the CrossEntropyLoss from the downstream task for video representation learning is novel and has the potential to be applied to various temporal grounding tasks beyond classification. 

**Versatility**: VEDIT stands out for its versatility across tasks. VEDIT successfully handles multiple procedural learning tasks, including step forecasting, task classification, and procedure planning, proving its robustness in diverse scenarios.

**Quality**: The overall quality of the paper is high. The empirical results are well-supported by ablation studies, which explore the impact of different model components, and the paper presents a thorough comparison against state-of-the-art methods, showcasing the strengths of the proposed model.
summary: This paper introduces a novel pipeline,VEDIT, for procedural classifciation task using diffusion transformers without large-scale pre-training or additional language-based supervision. VEDIT leverages a diffusion transformer-based model to operate on latent embeddings derived from frozen, pretrained visual encoders. It is designed to predict the future steps in multi-step tasks (e.g., cooking, assembly) by iteratively denoising embeddings in the latent space, which are generated from observed steps. Authors conducted adequate experiments to demonstate that VEDIT achieves state-of-the-art (SoTA) results on various tasks like step forecasting, task classification, and procedure planning across multiple datasets (e.g., COIN, Ego4D-v2, CrossTask, and NIV).
weaknesses: There are some major concerns:
- While the approach of training VEDiT using CrossEntropyLoss from downstream tasks is novel, it remains challenging for the audience to understand why the diffusion model can learn with this design and the training pipeline. From the destination in L239-L241 and  L301-L303,  it seems the sampled Gaussian noise will be passed to $12 \times 24= 288 $ transformer blocks in a single forward pass, where $12$ is the number of transform blocks and $24$ is the denoising steps. It seems that the computing needs are much higher than baseline methods, and it may not support other inference parameters, including denoising steps and classifier-free guidance scales, since they are fixed in training. It isn't clear how well it works when reducing the sampling steps to lower steps to balance the computing cost and performance boost.
- Using diffusion models for video embedding prediction is time-consuming since it requires $N$ times longer inference time. However, authors didn't discuss the limitation in this paper.


In conclusion, these are concerns and weaknesses that could further improve the paper. However, the weaknesses outweigh the strengths. Consequently, I am leaning towards suggesting rejection of this paper. I am looking forward to the authors' rebuttal and clarification.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: See details in 'Weakness' section.
soundness: 2
strengths: 1. The proposed method avoids the need for compute-intensive pretraining of the prediction model. It uses a pretrained, frozen visual encoder to capture robust visual representations, eliminating the need for additional supervision (e.g., language or ASR) commonly required in prior methods

2.The model's empirical evaluation demonstrates significant performance improvements on procedural video benchmarks like COIN, CrossTask, and Ego4D, showing gains in task classification accuracy and long-horizon action anticipation.
summary: This paper aims to advance video representation learning by proposing a diffusion transformer specifically for procedural video representation. It demonstrates that a frozen, pretrained visual encoder paired with a prediction head can achieve promising performance across five procedural learning tasks.
weaknesses: 1.Lack of Sufficient Innovation: The VEDIT model's components lack distinct novelty. (1) VEDIT’s architecture is primarily a combination of existing techniques—pretrained visual encoders and diffusion transformers—rather than introducing new modules. (2) The method’s adaptation of diffusion transformers to work in the latent embedding space of video representations has been explored previously, as in PPDP.

2.Omission of Computational Cost Analysis: The paper does not address the computational cost of the method. Due to the iterative nature of the diffusion process, VEDIT incurs slower inference times than models that make predictions in a single forward pass. This characteristic may limit its use in real-time or low-latency applications.

3.Lack Disccusion of Multimodal Integration: By not incorporating textual data or language models, the method overlooks semantic information that could enrich the understanding of procedural steps, especially in instructional videos that include narration.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) It's stated the task is either predicting the label of unseen events in between two input events (procedure planning), or predicting the label of a future event.  In the case of procedure planning, how does the model know how many clips to generate?  It's not obvious from the system figure (Fig 2) how multi-clip generation works - does that require multiple runs of the entire diffusion model? Is this number fixed?

2) From an intuition standpoint, why is diffusion the right choice for generating an embedding given a set of input embeddings? Are there no faster alternatives? What was the reasoning behind this choice compared to other alternatives?
soundness: 3
strengths: 1) The proposed architecture (diffusion in latent space between steps) is interesting. I think doing diffusion on the latent embeddings is a smart way to generate new embeddings that can be used for predicting final class embeddings. 
2) The results surpass existing SOTA and the ablations are comprehensive. The method also avoids relatively expensive pre-training on direct step data, making use of easily available visual encoders like SigLIP. The appendices are thorough and explain relevant design decisions for the proposed architecture.
3) The models trained are relatively large and will be helpful for the community if made available.
summary: This paper deals with  Procedural Video Representational Learning, in which a model predicts steps in a underlying process (eg cooking a dish). The goal is to learn a representation that enables prediction and understanding of the semantics of step-by-step processes through videos, often egocentric or how-to videos. Most works in this space pre-train models on data from HowTo100M with extracted ASR captions which is time-consuming and expensive. This paper's contribution is (1) a novel architecture for the procedural video task based on diffusion / flow matching in latent state-action space and (2) a method that relies only on frozen pre-trained image encoders rather than explicit procedural weakly supervised pretraining. In doing so, they achieve SoTA results.

In general, I think this is a reasonably strong paper, and the only weakness to me is that the architecture has slightly limited novelty; it's not very clear to me exactly how this differs from a standard diffusion transformer. The use of a frozen encoder, while simple, works well and avoids having to do expensive ASR-based pre-training. The models are also scaled to high parameter counts and thus will be useful to the community.
weaknesses: 1) The models trained are quite large, with the largest having 1.77B parameters. Given that the improvement over SOTA is relatively minor in most cases, it's not clear that scaling this method really works - could you comment on why this might be the case?

2) The architectural novelty is a little limited. The idea of doing diffusion to produce intermediate embeddings isn't new (as is stated in the paper) and this paper differs from prior work mostly by using a frozen SigLIP encoder rather than one pre-trained on HowTo100M (if i am understanding correctly). 

3) It's not super clear how the proposed VeDIT differs from standard diffusion transformer blocks except for the embedding concatenation - am I missing something? If so, could you explain further?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: All of my questions are listed in the weakness part.
soundness: 2
strengths: 1. Nice presentation and well-designed figures.

2. Strong empirical performance: VEDIT outperform all baselines quantitatively on all three benchmarks.
summary: This work questioned the necessity and effectiveness of pretraining by develop a SoTA method for forecasting and procedural planning without the need for pretraining the prediction model. The proposed model, VEDIT, leverage diffusion transformer and attentive classfier directly trained for downstream tasks. Empirical results showed that VEDIT outperform all baselines.
weaknesses: 1. Definition of pre-training: The most confusing point for me is that the authors claim the results of this work could challenge the necessity of pretraining. However, first of all, VEDiT itself uses a pretrained encoder. How is this different from what the authors define as pretraining? Secondly, isn’t the training of the diffusion part considered pretraining? I think the authors should make a stricter distinction and definition of pretraining to help readers clearly understand what they really want to refer to by "pretraining."

2. Training objective of DiT: If the training of the diffusion part isn’t considered pretraining, then what is the role of diffusion here? The authors should use mathematical formulas to clarify exactly what their loss function is; the current explanation is too vague. Additionally, starting from line 233, the description of diffusion training seems to suggest a possible misunderstanding of diffusion training.

3. Test-time training [1]:  This is just a suggestion, but based on the authors' model, I think that applying test-time training exclusively to the diffusion component could further enhance the model's performance during inference.



 

[1]. Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. and Hardt, M., 2020, November. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning (pp. 9229-9248). PMLR.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please answer the Weakness part.
soundness: 3
strengths: 1. This paper is well-motivated and easy to follow.

2. VEDiT does not require pre-training the representation space but consumes multiple clips by using pre-trained representation, which is more efficient and applicable, as better representation can improve VEDiT further.

3. Extensive experimental results on various downstream tasks demonstrate the effectiveness of the proposed method.
summary: This paper proposes a procedural video representation learning framework that utilizes diffusion transformers to predict visual representation in embedding space. This framework can be effectively transferred to various downstream tasks.
weaknesses: 1. Question about fair comparison: Without pre-training, I think VEDiT seems to be a large header for the downstream task. I cannot find how the baseline TimeSFormer is evaluated, but if it is evaluated with a simple linear classifier, i.e., linear probing, it seems that VEDiT's performance gain is just from "training more parameters on the pre-trained representation".

2. Typo on page 2 ("fall back on on text annotations" -> "fall back on text annotations")

Meta-review: This paper receives final scores of 6,6,6,6,6, from five reviewers. Reviewers have raised several issues on the architectural novelty, scaling, and lack of detailed experimental analysis. The authors have addressed most of the concerns in the rebuttal. Thus, there is a consensus of acceptance. AC has checked the submission, the reviewers, and the rebuttal and agreed with the consensus of all five reviewers. Thus acceptance is recommended.

Decision: Accept (Poster)