VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning

Authors: Han Lin, Tushar Nagarajan, Nicolas Ballas, Mido Assran, Mojtaba Komeili, Mohit Bansal, Koustuv Sinha
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Procedural Learning from Videos, Representation Learning, Diffusion Transformer

Abstract:
Procedural video representation learning is an active research area where the objective is to learn an agent which can anticipate and forecast the future given the present video input, typically in conjunction with textual annotations. Prior works often rely on large-scale pretraining of visual encoders and prediction models with language supervision. However, the necessity and effectiveness of extending compute intensive pretraining to learn video clip sequences with noisy text supervision have not yet been fully validated by previous works. In this work, we show that a strong off-the-shelf frozen pretrained visual encoder, along with a well designed prediction model, can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning without the need for pretraining the prediction model, nor requiring additional supervision from language or ASR. Instead of learning representations from pixel space, our method utilizes the latent embedding space of publicly available vision encoders. By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our prediction model is able to learn robust representations for forecasting through iterative denoising —leveraging the recent advances in diffusion transformers (Peebles & Xie, 2023). Empirical studies over a total of five procedural learning tasks across four datasets (NIV, CrossTask, COIN and Ego4D-v2) show that our model advances the strong baselines in long-horizon action anticipation (+2.6% in Verb ED@20, +3.1% in Noun ED@20), and significantly improves the SoTA in step forecasting (+5.0%), task classification (+3.8%), and procedure planning tasks (up to +2.28% in success rate, +3.39% in mAcc, and +0.90% in mIoU).

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730620938138

thread_type:
official_review

<thread_object>
creator: reviewer_2sck

<content>
summary:
This paper proposes a procedural video representation learning framework that utilizes diffusion transformers to predict visual representation in embedding space. This framework can be effectively transferred to various downstream tasks.

strengths:
1. This paper is well-motivated and easy to follow.

2. VEDiT does not require pre-training the representation space but consumes multiple clips by using pre-trained representation, which is more efficient and applicable, as better representation can improve VEDiT further.

3. Extensive experimental results on various downstream tasks demonstrate the effectiveness of the proposed method.

weaknesses:
1. Question about fair comparison: Without pre-training, I think VEDiT seems to be a large header for the downstream task. I cannot find how the baseline TimeSFormer is evaluated, but if it is evaluated with a simple linear classifier, i.e., linear probing, it seems that VEDiT's performance gain is just from "training more parameters on the pre-trained representation".

2. Typo on page 2 ("fall back on on text annotations" -> "fall back on text annotations")

questions:
Please answer the Weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683130856

thread_type:
official_review

<thread_object>
creator: reviewer_czp8

<content>
summary:
This work questioned the necessity and effectiveness of pretraining by develop a SoTA method for forecasting and procedural planning without the need for pretraining the prediction model. The proposed model, VEDIT, leverage diffusion transformer and attentive classfier directly trained for downstream tasks. Empirical results showed that VEDIT outperform all baselines.

strengths:
1. Nice presentation and well-designed figures.

2. Strong empirical performance: VEDIT outperform all baselines quantitatively on all three benchmarks.

weaknesses:
1. Definition of pre-training: The most confusing point for me is that the authors claim the results of this work could challenge the necessity of pretraining. However, first of all, VEDiT itself uses a pretrained encoder. How is this different from what the authors define as pretraining? Secondly, isn’t the training of the diffusion part considered pretraining? I think the authors should make a stricter distinction and definition of pretraining to help readers clearly understand what they really want to refer to by "pretraining."

2. Training objective of DiT: If the training of the diffusion part isn’t considered pretraining, then what is the role of diffusion here? The authors should use mathematical formulas to clarify exactly what their loss function is; the current explanation is too vague. Additionally, starting from line 233, the description of diffusion training seems to suggest a possible misunderstanding of diffusion training.

3. Test-time training [1]:  This is just a suggestion, but based on the authors' model, I think that applying test-time training exclusively to the diffusion component could further enhance the model's performance during inference.



 

[1]. Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. and Hardt, M., 2020, November. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning (pp. 9229-9248). PMLR.

questions:
All of my questions are listed in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730952174257

thread_type:
official_review

<thread_object>
creator: reviewer_ajeb

<content>
summary:
This paper deals with  Procedural Video Representational Learning, in which a model predicts steps in a underlying process (eg cooking a dish). The goal is to learn a representation that enables prediction and understanding of the semantics of step-by-step processes through videos, often egocentric or how-to videos. Most works in this space pre-train models on data from HowTo100M with extracted ASR captions which is time-consuming and expensive. This paper's contribution is (1) a novel architecture for the procedural video task based on diffusion / flow matching in latent state-action space and (2) a method that relies only on frozen pre-trained image encoders rather than explicit procedural weakly supervised pretraining. In doing so, they achieve SoTA results.

In general, I think this is a reasonably strong paper, and the only weakness to me is that the architecture has slightly limited novelty; it's not very clear to me exactly how this differs from a standard diffusion transformer. The use of a frozen encoder, while simple, works well and avoids having to do expensive ASR-based pre-training. The models are also scaled to high parameter counts and thus will be useful to the community.

strengths:
1) The proposed architecture (diffusion in latent space between steps) is interesting. I think doing diffusion on the latent embeddings is a smart way to generate new embeddings that can be used for predicting final class embeddings. 
2) The results surpass existing SOTA and the ablations are comprehensive. The method also avoids relatively expensive pre-training on direct step data, making use of easily available visual encoders like SigLIP. The appendices are thorough and explain relevant design decisions for the proposed architecture.
3) The models trained are relatively large and will be helpful for the community if made available.

weaknesses:
1) The models trained are quite large, with the largest having 1.77B parameters. Given that the improvement over SOTA is relatively minor in most cases, it's not clear that scaling this method really works - could you comment on why this might be the case?

2) The architectural novelty is a little limited. The idea of doing diffusion to produce intermediate embeddings isn't new (as is stated in the paper) and this paper differs from prior work mostly by using a frozen SigLIP encoder rather than one pre-trained on HowTo100M (if i am understanding correctly). 

3) It's not super clear how the proposed VeDIT differs from standard diffusion transformer blocks except for the embedding concatenation - am I missing something? If so, could you explain further?

questions:
1) It's stated the task is either predicting the label of unseen events in between two input events (procedure planning), or predicting the label of a future event.  In the case of procedure planning, how does the model know how many clips to generate?  It's not obvious from the system figure (Fig 2) how multi-clip generation works - does that require multiple runs of the entire diffusion model? Is this number fixed?

2) From an intuition standpoint, why is diffusion the right choice for generating an embedding given a set of input embeddings? Are there no faster alternatives? What was the reasoning behind this choice compared to other alternatives?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731122059479

thread_type:
official_review

<thread_object>
creator: reviewer_xzeh

<content>
summary:
This paper aims to advance video representation learning by proposing a diffusion transformer specifically for procedural video representation. It demonstrates that a frozen, pretrained visual encoder paired with a prediction head can achieve promising performance across five procedural learning tasks.

strengths:
1. The proposed method avoids the need for compute-intensive pretraining of the prediction model. It uses a pretrained, frozen visual encoder to capture robust visual representations, eliminating the need for additional supervision (e.g., language or ASR) commonly required in prior methods

2.The model's empirical evaluation demonstrates significant performance improvements on procedural video benchmarks like COIN, CrossTask, and Ego4D, showing gains in task classification accuracy and long-horizon action anticipation.

weaknesses:
1.Lack of Sufficient Innovation: The VEDIT model's components lack distinct novelty. (1) VEDIT’s architecture is primarily a combination of existing techniques—pretrained visual encoders and diffusion transformers—rather than introducing new modules. (2) The method’s adaptation of diffusion transformers to work in the latent embedding space of video representations has been explored previously, as in PPDP.

2.Omission of Computational Cost Analysis: The paper does not address the computational cost of the method. Due to the iterative nature of the diffusion process, VEDIT incurs slower inference times than models that make predictions in a single forward pass. This characteristic may limit its use in real-time or low-latency applications.

3.Lack Disccusion of Multimodal Integration: By not incorporating textual data or language models, the method overlooks semantic information that could enrich the understanding of procedural steps, especially in instructional videos that include narration.

questions:
See details in 'Weakness' section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731200505388

thread_type:
official_review

<thread_object>
creator: reviewer_ypq6

<content>
summary:
This paper introduces a novel pipeline,VEDIT, for procedural classifciation task using diffusion transformers without large-scale pre-training or additional language-based supervision. VEDIT leverages a diffusion transformer-based model to operate on latent embeddings derived from frozen, pretrained visual encoders. It is designed to predict the future steps in multi-step tasks (e.g., cooking, assembly) by iteratively denoising embeddings in the latent space, which are generated from observed steps. Authors conducted adequate experiments to demonstate that VEDIT achieves state-of-the-art (SoTA) results on various tasks like step forecasting, task classification, and procedure planning across multiple datasets (e.g., COIN, Ego4D-v2, CrossTask, and NIV).

strengths:
Generally, the primary strength of this paper is the introduction of the novel diffusion-based representation learning method into the video understanding field. VEDIT achieves significant performance gains across multiple procedural tasks, such as step forecasting and task classification. 

**Clarity**: Most of this paper is well-written, with clear explanations and a solid grounding in relevant literature. 

**Originality**: The main idea of training the VEDiT using the CrossEntropyLoss from the downstream task for video representation learning is novel and has the potential to be applied to various temporal grounding tasks beyond classification. 

**Versatility**: VEDIT stands out for its versatility across tasks. VEDIT successfully handles multiple procedural learning tasks, including step forecasting, task classification, and procedure planning, proving its robustness in diverse scenarios.

**Quality**: The overall quality of the paper is high. The empirical results are well-supported by ablation studies, which explore the impact of different model components, and the paper presents a thorough comparison against state-of-the-art methods, showcasing the strengths of the proposed model.

weaknesses:
There are some major concerns:
- While the approach of training VEDiT using CrossEntropyLoss from downstream tasks is novel, it remains challenging for the audience to understand why the diffusion model can learn with this design and the training pipeline. From the destination in L239-L241 and  L301-L303,  it seems the sampled Gaussian noise will be passed to $12 \times 24= 288 $ transformer blocks in a single forward pass, where $12$ is the number of transform blocks and $24$ is the denoising steps. It seems that the computing needs are much higher than baseline methods, and it may not support other inference parameters, including denoising steps and classifier-free guidance scales, since they are fixed in training. It isn't clear how well it works when reducing the sampling steps to lower steps to balance the computing cost and performance boost.
- Using diffusion models for video embedding prediction is time-consuming since it requires $N$ times longer inference time. However, authors didn't discuss the limitation in this paper.


In conclusion, these are concerns and weaknesses that could further improve the paper. However, the weaknesses outweigh the strengths. Consequently, I am leaning towards suggesting rejection of this paper. I am looking forward to the authors' rebuttal and clarification.

questions:
I also have a variety of questions that did not significantly impact the rating for the paper:
- The paper demonstrated the advantage of using VEDiT on chunk-level classification tasks. Can the approach be extended to other temporal understanding tasks that predict start/end time? e.g., action grounding
- The paper entirely leverages flow-matching sampling with a fixed number of sampling steps. How's the model's performance with different scheduler/sampling parameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732302981136

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the Reviewers for their valuable feedback and suggestions. We have addressed all of the Reviewers' questions in the individual rebuttals. Below, we provide a summary of the revisions made in our updated PDF, a brief overview of the new experiments, and a general response to common questions raised by the Reviewers.

- - -
**1. Revised contents in the updated PDF:** 

Following the suggestions from Reviewer czP8 and Reviewer yPQ6, we have added and refined the following content in our **revised PDF (updated content is highlighted in blue)** to enhance the clarity and comprehensiveness of our paper:

- Distinguished “pretraining on single video clips” from “pretraining on video clip sequences” to avoid ambiguity when stating that our VEDiT does not require large-scale pretraining on procedural video datasets.
- Added a paragraph in the Method section (Sec. 3.1) to explicitly explain our training objective.
- Included a new section (Sec. 5) to discuss the computational cost associated with our multi-step denoising strategy.


- - -
**2. Brief summary of new experiments we conduct following Reviewers’ comments** 

In addition, based on the valuable suggestions and comments from the Reviewers, we have conducted several additional experiments that we believe effectively address the Reviewers' concerns and further strengthen our work. Below, we summarize the key experiments and kindly invite the Reviewers to refer to the corresponding individual replies for more details if needed:

- Experiment with different classifier-free guidance scales: see “W1 (part 3): Other inference/sampling parameters, including denoising steps and classifier-free guidance scales” under the response for Reviewer yPQ6.
- Computational cost of VEDiT: see “Q2: Computational cost of VEDiT” under this general response.
- Performance with different noise scheduler: see “Q2: Performance with different scheduler” under the response for Reviewer yPQ6.
- Multimodal Integration: see “W3: Multimodal Integration” under the response for Reviewer Xzeh.
- VEDiT with a simple linear classifier: see “W1: VEDiT with a simple linear classifier” under the response for Reviewer 2SCK.

- - -
**3. General response to common questions from Reviewers** 

Again, we would like to sincerely thank all Reviewers for the insightful suggestions. In the following, we address the common questions from the Reviewers as general responses:
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives final scores of 6,6,6,6,6, from five reviewers. Reviewers have raised several issues on the architectural novelty, scaling, and lack of detailed experimental analysis. The authors have addressed most of the concerns in the rebuttal. Thus, there is a consensus of acceptance. AC has checked the submission, the reviewers, and the rebuttal and agreed with the consensus of all five reviewers. Thus acceptance is recommended.