GenRec: Unifying Video Generation and Recognition with Diffusion Models
Authors: Zejia Weng
Xitong Yang
Zhen Xing
Zuxuan Wu
Yu-Gang Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: video understanding
video generation
diffusion
Primary area: machine_vision
TL;DR: Unifying Video Generation and Recognition with Diffusion Models

Abstract: Video diffusion models are able to generate high-quality videos by learning strong spatial-temporal priors on large-scale datasets. In this paper, we aim to investigate whether such priors derived from a generative process are suitable for video recognition, and eventually joint optimization of generation and recognition. Building upon Stable Video Diffusion, we introduce GenRec, the first unified framework trained with a random-frame conditioning process so as to learn generalized spatial-temporal representations. The resulting framework can naturally supports generation and recognition, and more importantly is robust even when visual inputs contain limited information. 
Extensive experiments demonstrate the efficacy of GenRec for both recognition and generation. In particular, GenRec achieves competitive recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. GenRec also performs the best on class-conditioned image-to-video generation, achieving 46.5 and 49.3 FVD scores on SSV2 and EK-100 datasets. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed. Code will be available at https://github.com/wengzejia1/GenRec.

GenRec: Unifying Video Generation and Recognition with Diffusion Models
Zejia ?e Xitong Yang', Zhen Xing 12, Zuxuan Wu 122 Yu-Gang Jiang" :2
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University 2 Shanghai Collaborative Innovation Center of Intelligent Visual Computing
3 Department of Computer Science, University of Maryland
Abstract
Video diffusion models are able generate high-quality videos by learning strong spatial spatial-tempora -tempora priors on large-scale datasets. In this paper, we aim to investi- gate whether such priors derived from a generative process are suitable for video recognition, and eventually joint optimization of generation and recognition. Build- ing upon Stable Video Diffusion, we introduce GenRec, the first unified frame- work trained with random-fraae frame conditioning process so as to learn generalized spatial-tempon representations. The resulting framework can naturally supports generation and recognition, and more importantly is robust even when visual in- puts contain limited information. Extensive experiments demonstrate the efficacy of GenRec for both recognition and generation. In particular, GenRec achieves competitive recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. GenRee also performs the best class-conditions image-to-videss generation, achieving 46.5 and 49.3 FVD scores on SSV2 and EK-100 datasets. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed. Code will be available at https//ggthub.oom/weggeejaa RGennRee.
Introduction
Diffusion models have achieved significant success in the field of image and video generation over the past few years. variety of generative tasks have been revolutionized by using diffusion models trained on Internet-scale data, such as text-to-image generation [33, 30], image editing [23], and more recently, text-to-video generation [15, 2, 52] and text&image-to-viiiee videe generation [56, 18, 21]. The excellent generative capabilities of diffusion models suggest that informative representation is learned during the generative training and strong visual priors are captured by the backbone models [9, 43, 6]. Therefore, recent work has explored leveraging the image diffusion models for image understanding tasks, including image recognition [9, 8]. object detection [7, 57], segmentation [55] and correspondence mining [39]. However, the capability of video diffusion models effectively capture spatial-tempores information is not fully understood, and their potential for downstream video understanding tasks remains under-explored.
In this paper, we study the potential of video diffusion models [28, 1, 27], particularly the uncondi- tioned or image-conditioned models, for video understanding by addressing the three key problems: (a) Does the backbone model trained for video generation extract effective spatial-temporaa represen- tations for semantic video recognition? (b) Can we retain the video generation capability by jointly optimizing generation and recognition? (c) Will such a unified training framework further benefit video understanding, especially in noisy scenarios where only limited frames are available [3, 25].
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Classification Loss
Generation Loss
Classification Loss
Generation
Loss
Classifier
Generator
GenRec
Full Videos
Noisy Videos
Masked Videos
Noisy Videos
V MO)
N(.o)
Video
Video V
Video V
(a) Classification
(b) Diffusion Generation
(c) GenRec
Comparison of classical pipelines for video classification and generation tasks with our proposed GenRec method. (a) Classification: Typical video classification focus understanding complete videos. (b) Diffusion Generation: Diffusion models learn the noise reduction trajectory from videos with varying levels of noise. These distinct training paradigms present challenges for task unification. To bridge this gap. we propose (c) GenRec: learning framework that processes mask frames VM using masking function M and noise videos with noise sampling ((,, aiming to simultaneously learn video understanding and content completion with the same partially observed visual content.
Figure 1:
While conceptually appealing, unifying video generation and recognition into a diffusion framework is non-trivial. Prior work either views the diffusion models as frozen feature extractors [9, 55, 39], deconstructs them for tasks while sacrificing their original generation capability [8]. One major challenge comes from their distinct training and inference processes. Diffusion models are typically optimized using corrupted inputs, optionally augmented with single conditioning frame, to achieve unconditioned or image-comee cinee--onditioned generation during inference [27, 1]. In contrast, video recognition models require access to multiple frames to reason about temporal relationships and expect clean inputs during inference [49, 51, 48]. Consequently, training recognition model using corrupted videos and imagle-image conditions tends suffer from inferior model optimization and more significant iiffenn-i gap-
To this end, we propose GenRec, unified video diffusion model that enables joint optimization for video generation and recognition. Our model is built upon the open-source, image-conditioned Stable Video Diffusion model (SVD) [1], which encodes strong patialltemporal priors by pretraining on large-scale image and video data. However, instead of conditioning on the same image across all video frames, we propose to condition on a random subset of frames while masking the remaining ones (see Figure 2). This simple random-framm conditioning process effectively bridges the gap between the learning processes the two tasks. On the one hand, the generation capability of SVD is extended to handle arbitrary frame prediction, which provides more flexible and unambiguous video generation. On the other hand, conditioning on random subset of frames allows the model to learn more discriminative and robust features for the recognition task. As shown in Figure 1, the model is jointly optimized using both generative supervision (i.e., noise prediction) and classification supervision.
We conduct extensive experiments to evaluate the performance of GenRec for both recognition and generation. Without sacrificing the generation capabilities, GenRec demonstrates competitive video recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed. For example, when only the front half of the video can observed, GenRec achieves the 57.7% accuracy, which corresponds to 76.6% of the accuracy (75.3%) when the entire video visible, emonstrating higher accuracy retention ratio than other methods. By leveraging the recognition model for classifier guidance [11], GenRec also achieves superior class-conditioned image-to-vides generation results, with FVD scores of 46.5 and 49.3 on the SSV2 and EK-100 datasets, respectively.

---

2 Preliminary
Representing the data distribution as Pduta (z) with standard deviation of data- we can obtain a family of smoothed distributions p(z; σ)) by adding independent and identically distributed Gaussian noise with standard deviation d. In the spirit of diffusion models, the generation process begins with a noise image Nn ~ (0,, man I) and iteratively denoises it at decreasing noise levels m > ON-1 > > do O. The final denoised result Zo thus distributed according the original data.
In the EDM [22] framework, the original Zn will be diffused as:
2
20
(),
and the corresponding PF-ODE [34] follows:
dz.
Va CogPrrZz, )do,
where vz log (2,) ) is the score function. Noise schedule o(t) is set as time step t. The training objective is to minimize the L2 loss with the denoiser network D9 for different o:
Exo--Pana ll Da(Zo) - 20112
(3)
with the relation between Dg and the score function V. log log o) follows:
V logg((2; (D(z.) z)/o²
SVD [31] utilizes the EDM framework to perform generative training on large-scale video datasets, resulting in a high-quality video generation model. An image-to-video generation model capable of forecasting future frames given the first frame has been released. Following SVD method, we also process videos in latent space. Given an input video V E RTx/XWxx x3, pretrained VAE encoder is used to project it into the latent space frame by frame, resulting in the latent representation zo I xxxxxx D. We then build GenRec based on SVD, inheriting its strong spatial-temporal priors as foundation for the subsequent generation and classification tasks.
3 GenRec
We now introduce GenRec, simple yet efficient framework, that not only generate temporally- coherent videos conditioned on an arbitrary number of provided frames but also is able to recognize actions and events with the help of encoded spatial-temporal priors. To this end, GenRec explores the strong spatial-temporal priors learned by video diffusion model. In this work, we instantiate GenRec with the powerful open-source Stable Video Diffusion model (SVD) [1], which is pretrained on large-scale video datasets and is able to produce a photo-realistic video when provided a single frame. Then, for generation, GenRee follows the classical EDM framework learn noise reduction trajectories. For recognition, on the other hand, GenRec operates on intermediate decoded features using a recognition head. Furthermore, to generate videos in a more free fashion, i.e. an arbitrary collection of frames used as condition, we design a latent masking strategy that "interpolates" masked frames. Such a strategy also benefits recognition by easing the training process. More importantly, by doing so GenRec supports multitude of downstream tasks, particularly when limited visual information provided.
3.1 Pipeline Overview
Latent diffusion and latent masking. During the diffusion process, the Gaussian noise with a certain noise level is added to the latent representation zo, creating a noisy latent representation ž; following Equation (I). Recall that while SVD contains powerful spatial-temporal priors, it can only perform generation when the first frame is provided. To allow a more "free" generation with an arbitrary number of frames as inputs, we design a latent masking strategy. More specifically, we apply random mask m to the latent representation Zo, producing a masked latent representation Zy. Such strategy encourages the model to reconstruct the original video content from incomplete frames, which is in a similar spirit to MAE [19]. Note that when only the first latent available, degrades to the same as SVD; if all latents are masked out, this degrades to unconditional generation. Furthermore, doing so also benefits recognition tasks when limited visual clues are available. For example, in scenarios with limited bandwidth leading reduced frame rates, the ability of video
3

---

Training Process
Classification
E
Diffusion
Z
Input Video
Pretrained Encoder
Video Latent
Noisy Latent
Spatial-Temporal LNeet
Reconstruction
D
Random Mask
1R
Mask
Masked Latent
Reconstructed Latent
Pretrained Decoder
Generated Video
The pipeline our proposed video processing method. The input video is first processed by pretrained encoder E to produce latent representation Zu, then undergoes diffusion generate a noisy latent 2, The random mask m is used to create the masked latent Zyj. During training, the noisy latent is concatenated with the masked latent as condition and fed into Spatial-Temport UNet, resulting in both reconstruction and recognition outputs. The reconstructed latent can be decoded by the pretrained decoder D to produce the final generated video.
Figure 2:
frame complementation enables the model better predict and perceive complete video information. In practice, we simulate such conditions by randomly erasing half to all video conditions, retaining on average only about one-fourth of the original video information. This technique allows the model to effectively fill in the missing information, enhancing its ability to recognize and understand the video content despite the reduced data availability.
Unifying generation and understanding. To unify generation with masked latents, GenRec predicts pixel-level and semantic-level contents with the combination of the noisy latent %, and the masked latent Σσ gained by the aforementioned latent diffusion and latent masking. The two latents are channel-wise concatenated [ž., zo] and are fed into Spatial-Temporaa UNet, together with features from observed frames, to learn spatial and temporal representations, following [1]. The weights of the UNet are initialized from [1] to obtain spatial and temporal priors, learned on large-scale video datasets.
For the generation task, the UNet aims to reconstruct the original latent representation from the combined noisy and masked inputs. Representing UNet as the mapping function F3, its goal is to predict clean latent, which, according the EDM framework, takes the form of representation mapping as follows:
Do (Z;; Zo. o) (a ( + (o) (o (o) zol):
which we set the same skip connection Caksps scaling factor Cout and Cin as [1].
For the recognition task, we break down the UNet mapping function as = Fioit Fhead- And we consider /Read [Cin (o)2, z)) E TT x xDD' as the compact video representation extracted from the intermediate layer of the UNet model, which is then fed into the classifier head dg. consisting of an attentive pooler and a fully connected layer to predict video categories:
rb(Fhend (Cin (a ) 2nl)).
3.2 Optimization
We train GenRec with both generation and classification objectives, encouraging the model to learn high-quality video generation and accurate video understanding.
The generative loss uses L2 loss to measure the difference between the original latent representation and the reconstructed output produced by the UNet, and defined as:
Lc(zo, ž., Z0 Do o) zoll²

---

where D,(2;; function based on the noise level referring to [1, 22]. While the classification loss uses cross- entropy loss to measure the discrepancy between the true labels and the predicted labels, and is defined as:
Zo, o) is the denoised output mentioned in Equation (5), and X(o) is weighting
L)( - Σ - log(ú.),
(8)
where y denotes the ground truth labels, and y represents the predicted labels referring to Equa- tion Equation (6).
To balance the learning of generative and recognition tasks, we set balancing weight yy to control the relative importance of each loss in the overall objective function. The total loss I. is given by:
L = Lp YLG,
3.3 Inference for Different Downstream Tasks
With the above training strategies, we now introduce how GenRee can flexibly support different types of generation and recognition tasks.
Video generation conditioned on frames. Once trained, GenRec is able to generate high-quality videos conditioned on an arbitrary number of given frames, thanks to the latent masking strategy. Particularly, following the EDM stochastic sampler framework and Equation (2), GenRec iteratively denoises the video conditioned on the masked latent Zo. as shown below:
d2
Zi = 2i + eD (2,; Zo) = 2, + (4:-1 - (2)
(10)
dt.
= 2, + -t,1 - (()-1- Do(2i; 20, σ) 2
(11)
where 2; is derived from ž; adding a perturbation. With an iteratively denoising process, we can finally obtain the denoised video latent Zu which can be decoded as a complete video.
Video generation conditioned on classes. When the number of visible frames is extremely limited, the motion trajectory becomes unpredictable and thus it would be hard to make reliable prediction of the future. To mitigate this issue, GenRec supports adding category information to guide video generation in the expected desired direction.
Formally, we simplify Equation (11) with Equation (4), and obtain:
co (2;; Zn) = (t:-1 - - Do(2;; Zo. o) 2,
(12)
&
= (-11)(1-1-
log pa (2:).
(13)
We substitute the score function 22, log (2();) with the conditional form Vg, log (2,,lv),, in which y denotes the conditional class. By applying Bayes' Theorem, the original score function can be replaced by p( 22; (p()))((() 22), and we can get the conditional version of residual, denoted as (2,; Zo):
< (2;; Zu) = (-1)(tt-1 - tt v2, log pl 2, )p()/z))
(14)
= (-1)((ti- - (V2 log p( (2) + V2, logp(y/2.)]
(15)
= =60 20) (4:11 - (.). v2, log log((y)2))
(16)
Considering the scaling factor of 2;: cin(a) Vo2+o (following [22], and a; = f2), that would
pre-scale the input as c(z:) = (l:) zz before model processing, the formulation can be further transferred as:
(t:11 - Ee) Vole.) logp(ylc(2.))
(Z(2;)0) = c0(2,; Z0)
(17)
Ve2 Odata
Following [11], we sharpen the distribution of p(y) (z) by multiplying scaling factor > 1, shown as 1zlogp(y(z) = Vz log )(()()) where Z is arbitrary constant. Larger scaling value would
5

---

bring more attention to the target category. Here, p(y]c((;)) comes from the classification branch in GenRec. Finally, we can use the same EDM sampling procedure with the derived class information to generate samples.
Standard video recognition. Based on Equation (6), GenRec can do the classical video recognition by setting constant no-mask, and thus Zu is replaced by Zo and the prediction follows:
= ob (b(Phena (o )z,, zol))-
(18)
Video recognition with partially observed frames. Based on Equation (6), GenRec can be applied to video recognition with partially observed frames, egg. early action prediction that aims to predict future events based on the initial frames, sparse video recognition where videos are sparsely encoded and transmitted due to bandwidth limitations. By masking the invisible frames to get 2,, and replacing the noisy latent with random noise ~ obeying Gaussian distribution, GenRee can do the prediction for partially visible videos, following:
y = Φ ([Phoaa((>> z0l))-
(19)
4 Experiments
4.1 Experimental Setup
Datasets. In our experiments, we use the following four datasets: Comething-Somethiinn V2
(SSV2) [17], Kinetics-400 (K400) [24], UCF-101 [35] and SpiccKKichenn-10 (EK-100) [10]. SSV2 dataset designed for fine-grained action recognition and it contains 174 action classes, 220,847
short video clips with average duration of 4 seconds. K400 contains 400 action classes, 306,245 video clips with an average duration of 10 seconds. The UCF-101 dataset comprises 13,320 videos from 101 action categories and is widely utilized for human action recognition. The EK-100 dataset focuses on egocentric vision. It contains a total of 90,000 annotated action segments, encompassing 97 verb classes and 300 noun classes.
Evaluation protocols. GenRec performs both generation and recognition tasks. For generation, use the Fréchet Video Distance (FVD) [41] metric to assess the quality of the generated videos. A lower FVD score indicates higher fidelity and realism. For recognition, we measure the top-1 I accuracy that reflects the portion of correctly classified videos. We validate model performance in formal video recognition, partial video recognition, class-conditioned image-to-viden generation and frame completion with the above metrics.
Implementation details. We initially set the learning rate to 1.0 x 10 and set the total batch size 32. Only generation loss will be retained for model adaptation specific datasets. We train 200k steps on EK- and UCF, and 300k steps on SSV2 and K400, respectively. Subsequently, we finetune GenRec with both generation and recognition losses. The learning rate is set to 1.25 x 10 and decayed to 2.5 x 10-7 using a cosine decay scheduler. We warm up models with 5 epochs, during which the learning rate is initially set as 2.5 x 10-7 and linearly increases the initial learning rate 1.25 x 10-5 The loss balance ratio y is set to 10, and the learning rate for the classifier head is ten times higher than the base learning rate. We drop out the conditions 10% of the time for supporting classifier-free guidance [20], and we finetune on K400 for 40 epochs and 30 epochs on other datasets. The training is executed on 8 A100s and each contains batch of samples. We sample 16 frames for each video.
4.2 Main Results
Comparison to state-of-the-ars in video recognition and video generation. We compare with tfate-of-the-art the-art methods in terms of their recognition accuracy and generation quality. The results are summarized in Table 1. The first two blocks of the table presents current advanced video recognition models, while the third block demonstrates the performance of the diffusion based class- duus-g image-to-video generation.
As shown in the table, GenRec achieves optimal results or performs on par with the state-of-the-ard approaches. In terms of video recognition, GenRec achieves 75.8% accuracy on SSV2 dataset,

---

Table 1: Performance of Video Recognition and Generation Methods. We evaluate on video recog nition and class-conditioned image-to-video generation tasks. SEERi predicts 16 frames, while others predict 12 frames. Top-1 accuracy and FVD scores are reported. Baseline adapts SVD to datasets with generative fine-tuning and then uses pitobiive-probing for classification. Baseline fully finetunes SVD with classification supervision only in traditional classification framework.
Classification Acc (f)
Generation FVD (L)
Method
Resolution
Param.
SSV2
K400
SSV2
EK-100
w/o multi-modal align.
VideoMAE-L [40]
224x224
305M
74.3
85.2
VideoMAE-H [40]
224x224
633M
86.6
OmniMAE-H [16]
224x224
650M
75.5
85.4
MVD-H [44]
224x224
633M
77.3
87.2
Hicra-L [32]
224x224
214M
75.1
87.3
Hiera-H [32]
224x224
673M
87.8
MaskFeat-L [47]
312x312
218M
75.0
86.4
w/ multi-modal align.
Vndco [45]
224x224
1.3B
77.2
91.1
Intern Video2 [46]
224x224
6B
77.4
92.1
Omni Vec [36]
85.4
91.1
Omni [37]
86.1
93.6
TATS [14]
128x128
428.1
920.0
MCVD [42]
256x256
3.5B
1407
4804
SimVP [13]
64x64
537.2
1991
VideoFusion [28]
256x256
1.8B
163.2
349.9
Tune-A- Video [50]
256x256
860M
291.4
365.0
SEER [18]
256x256
860M
112.9
271.4
SEER [18]
256x256
860M
355.4
Baseline
256x256
2.1B
63.7
82.0
50.3
53.6
Basline II
256x256
1.9B
75.9
86.6
GenRec
256x256
2.1B
75.8
87.2
46.5
49.3
surpassing the majority of current state-of-the-ar methods. On K400, GenRec achieves 87.2% accuracy, which is on par with the performance MVD-H (87.2%) and Hiera (87.3%, 87.8%), and surpasses other advanced methods. These results indicate the effectiveness of our approach in video recogmition. In addition, GenRec shows slight performance gap compared to the methods that these advanced methods benefit significantly from
in the second block. It important pretraining on large-scale multimodal alignment datasets, which provide extensive cross-modal supervision that enhances their ability capture semantic relationships across video frames.
We further construct two strong baselines. Baseline adapts SVD to the respective dataset through generative fine-tuning, followed by attentive-rrobing for classification, where the backbone is frozen and all frames are used as input. Baseline II involves fully fine-tuning the original SVD model with classification supervision only, ensuring that all frames are visible during training. Compared with them, GenRec performs on par or better. GenRec performs good in supporting not only classification but also generation, demonstrating its comprehensive capability in handling both tasks effectively.
terms of video generation, we evaluate the model on class-conditioned image-to-ve vi-video generation task following [18]. Comparing the FVD scores of SEER: 12.9 and SEER1:355.4 it can be inferred that generating longer videos with 16 frames is more difficult than generating 12 frames. GenRec generates vidcos with 16 frames and achieves much lower FVD scores than the other methods, demonstrating the effectiveness of our approach in vidco generation.
It is worth highlighting that, current research always treats video recognition and generation tasks in separate manner, and most of the advanced methods focus primarily on either recognition or generation tasks. For instance, SEER method excels in class-conde conditiond image-to-video generation, lacks the ability to video recognition. While current research on representation learning, shown as the first and second blocks in Table 1, lacks the ability to do video generation tasks. In contrast, GenRee not only unifies these tasks, but also achieves competitive results compared to the specialized methods.

---

Table 2: Early action prediction and limited interpolation problem on Soottiiinn-o V2 dataset, with one temporal crop. p denotes the visible ratio according the whole video. Acc denotes to the top-1 accuracy. Ratio metric represent the percentage of maximum performance that the model can maintain various frame rates. The 'w/o G' experiment refers the results obtained by removing the generative supervision from our method.
Early Action Prediction (p)
Limited Inter. Frames
Method
Metric
0.3
0.5
0.7
1.0
2
fs
Accuracy
20.5
28.6
41.2
47.1
66.3
TemPr [38]
Retention
30.9%
43.1%
62.1%
71.3%
100%
Accuracy
34,2
64.4
75.0
MVD [44]
Retention
45.6%
72.4%
85.9%
100%
Ascuracy
26.9
39.8
55.6
70.2
75.0
53.6
75.0
MVDi [44]
Retention
35.9%
53.1%
74.1%
93.2%
100%
71.5%
86.7%
91.7%
100%
w/o
Accuracy
27.3 11.6
39.6
56.3,14
71.6j08
75.0,0.
53.5
65.8g1.5
69.8.1.0
75.0
GenRee
Accuracy Retention
28.9
41.9
57.7
72.4
75.3
55.7
70.8
75.3
38.4% %
55.6 %
76.6%
96.1%
100%
74.0%
89.4%
94.0%
100%
50
45
98
EK100 Action Accuracy
40
96
35
UCF Accuracy
30
25
20
92
15
TemPr
TemPr
90
MVD+ RenRec
MVD+
5
GenRec
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9 1.0
0.0
0.1
0.2
0.3
0.4 0.5
0.6 0.7 0.8 0.9 1.0
Observation Ratio p
Observation Ratio
Figure Early action prediction EK-100 and UCF-100 datasets, with one temporal crop.
Comparison to state-of-the-art in video recognition with limited frames. GenRec supports video recognition when only partial frames can be observed. We evaluate this capability on the SSV2 and EK-100 datasets. Our evaluation includes two tasks: an early prediction task, where the model has access only to previous continuous frames following the setting of [38], and a recognition task where videos are sparsely sampled, and the model is expected to make correct predictions. For fair comparisons, we construct two strong baselines. We first apply MVD [44] to directly deal with the recognition task by constructing dense video through nearest neighbor interpolation. We also construct another baseline similar to our training pipeline, where we apply frame dropout in the training process of MVD [44] for better fitting on task with partial frames, and is named as MVD/. In all settings, the number of fully observed frames is 16.
Table 2 shows the results under these settings, in which p denotes the visible ratio (there are a total of 16 frames). In the early action prediction task, GenRec achieves the highest accuracy and ratio metrics at all observation levels. Notably, GenRec and MVD; exhibit similar performance when all frames are observed, but as number of observed frames decreases, GenRec demonstrates higher accuracy. GenRec also shows superior performance when videos are sparsely sampled, maintaining high accuracy even with fewer observed frames (c.g., 55.7% for frames and 70.8% for 4 frames), indicating its robustness in handling sparse data. Moreover, we compute the ratio metric representing the percentage of maximum performance that the model can maintain at various frame rates, mitigating the unfaimess caused by different backbone networks. In this scenario, GenRec still achieves the best performance.
We further investigate the contributions of generation supervision for recognition. As seen in Table 2, removing generation supervision results noticeable performance degradation across various tasks, especially when the number of visible frames get less. For example, in the early prediction task, the accuracy decreases by 0.3% at p = 1.0 and by 2.3% at p = 0.3. These results suggest that generation supervision is essential for maintaining high performance, particularly when the model has to make

---

Table 3: Relation between generation and recognition.
Table 4: Choice of UNet lay- ers for feature extraction.
Early Frames
Limited Inter. Frames
Method
Metric
f
8s
12fs
2
is
4 fs
Up Index
f
28.9
41,
57.7
72.4
55.7
67.3
70.8
SSV2
71.8
75.8 75.2
GenRec
FVD.
57.8
44.0
30.3
16.6
46,7
31.7
24.3
Table 5: Ablation study on different masking strategies.
Expectation of
SSV2
Method
masking ratio
Ace (t)
FVD (L)
75ur choice)
75.8
46.5
GenRec
50% 87.5%
+0.3
+0.5
-0,9
-0.3
predictions with limited visual information. By incorporating generation supervision, the model can better handle scenarios with incomplete data, improving robustness and accuracy.
We also evaluate the early action prediction on EK-100 and UCF-101. EK-100 is temporally sensitive dataset similar to SSV2, demanding in terms of the model's temporal modeling capability, while UCF-101 demands more appearance modeling. We conduct early prediction evaluation on them to further reveal the robustness of our GenRec. As shown Figure 3, GenRec clearly outperforms TemPr and MVDj. In particular, the improvement becomes more significant as the number observed frames decreases. More evaluation results can be seen in Appendix A.I.
These results collectively demonstrate that GenRec effectively handles missing video frames. The robustness and high accuracy of GenRec across different datasets and observation ratios highlight its potential for wallwwrldd applications where video data might be incomplete or sparsely sampled.
The relationships between generation and recognition. We further investigate the consistency between video generation and recognition, as shown in Table 3. We evaluate the performance of vidco recognition and generation with limited frames and find that the recognition accuracy not only depends on the number of visible frames but also significantly uniform sampling appears to facilitate video recognition better than dense sampling from the video prefix. Specifically, with the same number of frames, early prediction consistently shows lower accuracy compared to uniformly sampled frames (e.g.. 28.9% vs. 55.7% with 2 frames) and worse FVD scores (e.g., 57.8 vs. 46.7 with frames). When only three interpolated frames visible, the 31.7 FVD score is comparable to that of an eight-frame prefix (30.3), while achieving much higher recognition accuracy. These results highlight the importance of complete state observation for action recognition and also suggest that video generation performance can potentially reflect task difficulty.
the location of these frames. Interestingly,
Choice of UNet layers. As described in Section 3, the UNet mapping function F is decoupled into Finst Fhead, where Frend serves the feature extractor for video recognition. Our UNet model contains 4 main up-sampling blocks. We investigate which one is best suited for recognition. As shown in Table 4, using the second up-sampling block (Up Index 2) yields the best performance with an accuracy of 75.8%. The third block (Up Index 3) followed with 75.2%, while the first block (Up Index 1) has the lowest accuracy. As such, we choose the second block for feature extraction.
Explore the influence of the masking strategy. We also conduct an ablation study on the masking schemes using different expected masking ratios, shown in Table 5. The results show that the FVD scores remain similar across different ratios, and a larger masking ratio might be beneficial for generation, as it closely resembles our class-conditioned frame prediction scenario with one or two given frames. However, an excessively large masking ratio (87.5%) negatively impacts action recognition accuracy, leading to a 0.9% decrease compared our selected ratio.
Noise incorporation during inference for video recognition. In the inference stage for action recognition, GenRee applies a specific level of noise to video inputs before extracting visual features, as formulated in Equation 18. This added noise helps maintain consistency with the noisy training

---

Table 6: SSV2 action recognition accuracy with different random seeds.
Seed
2
3
4
5
AVG
SSV2 Acc (%)
75,83
75.82
75.83
75.83
75.86
75.84
75.835 + 0.0125
process. To further understand the influence of noise randomness classification accuracy, we conducted an experiment on the SSV2 dataset, using multiple random seeds to generate the noise, as presented in Table 6. The results demonstrate remarkable consistency in accuracy across different random seeds, with a standard deviation of only 0.0125%. This minimal variation highlights the model's robustness to noise fluctuations during inference, suggesting that the model's performance remains stable despite noise introduced by random sampling. If fully deterministic outcomes are desired, fixing the random seed for noise sampling will eliminate any remaining variability and guarantee consistent predictions across runs.
5
Related Work
Video diffusion models for generation. The great success of diffiusion models image generation has led to rapid advancements in video generation, including text-to-video generation [15, 2, 52, 54], image&text-to-video generation [56, 18, 21], and video editing [4. 5, 26, 29, 12, 53]. Many current works [52, 18] adapts the diffusion models from images to videos by incorporating temporal convolutions and attention mechanisms. One typical and excellent work, Stable Video Diffusion [1], follows the above description and has provided valuable foundations for generating high-quality, diverse, and temporally consistent videos. Different from the previous work, in our paper, we pursue only the quality of generation, but also the unity of model generation capability and classification ability.
Diffusion models for visual understanding. Recently, researchers start to uncover the significance of diffusion models for discrimination tasks. A notable approach involves utilizing pretrained visual diffusion models for various downstream tasks, such as image segmentation [55] and visual content correspondence [39]. Additionally, some studies treat diffusion learning as a self-supprvised method to acquire valuable feature representations [8]. However, most current works either use stable diffusion networks as pretrained backbones for downstream tasks or completely destroy their generative capabilities. Consequently, the potential benefits of integrating generation and classification abilities into a single model remain under-explored, which the primary focus of our paper.
6 Conclusion
In this work, we presented GenRec, a unified video diffusion model that enables joint optimization for both video generation and recognition. GenRec exploits the significant temporal modeling power embedded the diffusion model, allowing for mutual reinforcement between generation and recognition tasks. Extensive experiments demonstrate our approach contains strong generation and recognition capabilities at the same time in different kinds of scenarios, including normal or partial video recognition, video completion and image-to-video generation. Our findings highlight the potential of combining generation and classification tasks within single unified model, providing valuable insights into the development of more sophisticated and versatile video analysis models. Future work will focus on further refining this integration and exploring its applications across various real-world scenarios.
conducted to evaluate the performance of GenRec,
Acknowledgement
This work was supported in part by National Natural Science Foundation of China (#62032006). The authors would like to thank Rui Wang, Rong Bao, Rui Tian for their help and suggestions.
10

---

References
[1] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311. 2023.
[2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[3] Yu Cao, Daniel Barrett, Andrei Barbu, Siddharth Narayanaswamy. Haonan Yu, Aaron Michaux, Yuewei Lin, Sven Dickinson, Jeffrey Mark Siskind, and Song Wang. Recognize human activities from partially observed videos. In CVPR, 2013.
[4] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In ICCV, 2023.
[5] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency- aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23040-23050, 2023.
[6] Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang. Chengqi Duan, Hang Su, and Jun
Zhu. Robust classification via a single diffusion model. arXiv preprint arXiv:2305. 15241, 2023.
[7] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. ICCV, 2023.
[8] Xinlei Chen, Zhuang Liu, Saining Xie, and Kaiming He. Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint 1rXiv22401.14404, 2024.
[9] Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. In NeurIPS, 2024.
[10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, al. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- 100. IJC,, 2022.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurlPS, 2021.
[12) Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In ICCV, 2023.
[13] Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In CVPR. 2022.
[14] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang. David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In ECCV, 2022.
[15] songwei Ge. Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. In ICCV, 2023.
[16] Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and videos. In CVPR, 2023.
[17] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
11

---

[18] Xianfan Gu, Chuan Wen, Weirui Ye, Jiaming Song, and Yang Gao. Seer: Language instructed video prediction with latent diffusion models. arXiv preprint arXiv:2303. 14897, 2023.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:220..12598, 2022.
arXiv preprint
[21] Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXiv preprint arXiv:2206.07692 07696, 2022.
[22] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurlPS, 2022.
[23] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In CVPR, 2023.
[24] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 170s.:700.006900 2017.
[25] Tian Lan, Tsung-Chuan Chen, and Silvio Savarese. A hierarchical representation for future action prediction. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part III 13, 2014.
[26] Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng. Magicedit: High-fidelity and temporally coherent video editing, arXiv preprint 2444422008.4479, 2023.
[27] Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General-purpose video diffusion transformers via mask modeling. In The Twelfth International Conference Learning Representations, 2023.
[28] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In CVPR, 2023.
[29] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors. arXiv preprint MXXv:2302.01329 2023.
[30] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In JCML, 2021.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[32] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po- Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury. Omid Poursaeed, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the bells-and-whistee In ICML, 2023.
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic -xttte diffusion models with deep language understanding. In NeurIPS, 2022.
[34] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1rXiv:2011.133566, 2020.
[35] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[36] Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with cross modal sharing. In WACV, 2024.
12

---

[37] Siddharth Srivastava and Gaurav Sharma. Omnivec2-a novel transformer based network for large scale multimodal and multitask learning. CVPR, 2024.
[38] Alexandros Stergiou and Dima Damen. The wisdom of crowds: Temporal progressive attention for early action prediction. In CVPR, 2023.
[39] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In NeurIPS, 2023.
[40] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022.
[41] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: /81200177 2018.
[42] Vikram Voleti, Alexia Jolicoeur- Marreuur,,atnnaau, and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. In NeurIPS, 2022.
[43] Jinglong Wang, Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou, Qian Yu, Lu Sheng, and Dong Xu. Diffusion model is secretly training-fret free open vocabulary semantic segmenter. arXiv preprint arXiv: 2023.
[44] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengehen Liu, Lu Yuan, and Yu-Gang Jiang. Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning, In CVPR, 2023.
[45] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.
[46] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng. Jilan Xu, Zun Wang, et al. Intervvide22:: Scaling video foundation models for multimodal video understanding. arXiv preprint arXiv:2403. 11.4003.17777 15377, 2024.
[47] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-trninin In CVPR, 2022.
[48] Zejia Weng, Zuxuan Wu, Hengduo Li, Jingjing Chen, and Yu-Gang Jiang. Hems: Hierarchical and conditional modality selection for efficient video recognition. ACM TOMM, 2023.
[49] Zejia Weng, Xitong Yang, Ang Li, Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: Transforming clip to an open- open-vocabulary video model via interpolated weight optimization. In ICML, 2023.
[50] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023.
[51] Zuxuan Wu, Zejia Weng, Wujian Peng, Xitong Yang, Ang Li, Larry S Davis, and Yu-Gang Jiang. Building an open-vocabulary video clip model with better architectures, optimization and data. TPAMI, 2024.
[52] Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. arXiv preprint arXiv:2308.00 00770000 2023.
[53] Zhen Xing, Qi Dai, Zihao Zhang, Hui Zhang, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Vidiff: Translating videos via multi-modal instructions with diffusion models. arXiv preprint 2rXiv:2311.18833 2023.
[54] Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. ACM Computing Surveys, 2023,
[55] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023.
13

---

[56] Xi Ye and Guillaume-Alexame Alexandre Bilodeau. Stdiff: Spatio-temporal diffusion for continuous stochastic video prediction. In AAAI, 2024.
[57] Hui Zhang, Zheng Wang, Zuxuan Wu, and Yu-Gang Jiang. Diffusionad: Denoising diffusion for anomaly detection. arXiv preprint arXiv:2303.087300 2023.
14

---

A Appendix supplemental material
A.1 Early Prediction on EK-100 and UCF-101
More detailed evaluation results of the video recognition with limited frames on EK-100 and UCF-101
can be seen here.
Table 7: Early action prediction on EK-100.
Verb Obs. Ratio(p)
Noun Obs. Ratio(s)
Action Obs. Ratio(
Method
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
TemPr [38]
21.4
34.6
54.2
63.8
67.0
22.8
32.3
43.4
49.2
53.5
7.4
15.4
28.9
37.3
40,8
MVDF [44]
49.3
60.0
64.7
68.8
71.3
35.7
44.7
49.1
53.4
55.2
22.5
32.2
37.5
42.3
44.5
GenRee
55.8
63.6
67.9
71.7
73.1
40.1
47.8
52.0
55.3
56.7
28.1
36.1
40.8
45.0
46.6
Table 8: Early action prediction on UCF dataset.
Observation Ratio(p)
Method
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
TemPr [38]
88.6
93.5
94.9
94.9
95.4
95.2
95.3
96.6
96.2
MVD1 [44]
90.8
94.2
94.8
95.7
95.5
96.2
96.5
96.8
96.9
GenRec
94.4
96.0
96.5
96.6
96.9
97.4
97.8
97.9
97.9
A.2 Training and Inference Time Cost Compared with Previous Classification Methods
We compare the training and inference time cost between MVD-H, Baseline II and GenRec, fairly on the same hardware resource: 4-nodes V100s, and same batch size. Baseline II refers to fully fine-tuning the original SVD model with classification supervision only.
As shown in the Table 9, GenRec and the Baseline II consumes more testing time than MVD-H. The difference primarily arises from the varying number of parameters. Our model is derived from generative model, and the complexity of the generative task necessitates a larger number of parameters for effective learning. Compared with Baseline II, since GenRec requires additional decoder blocks for video generation training, the training time will get increased little bit. As the additional up-sampling blocks will not be used when doing action recognition, GenRec shares the same testing time with Baseline II. is worth noting that "f" in MVD-H is to highlight that MVD method would use repeated augmentation technique during training, but such augmentation will significantly increase the training time. Our approach does not need to use that augmentation. All the methods do the down-stream finetuning for 30 epochs.
Table 9: Comparison with recognition methods terms of parameters, training time, and test time.
Method
Trainable Params
Total Params
Training Time
Training Epochs
Test Time
(2 x 3 clips)
MVD-Hf Baseline II
633M
633M
3038 s/Epoch
30
441 s
1.3B
1.9B
2954 s/Epoch
30
1500 s
GenRec
1.5B
2.1B
3751 s/Epoch
30
1500
A.3 Ablation on Loss Balance Ratio
We conduct an ablation study on the loss balance ratio as shown Equation 9, Results presented in Table 10 show that varying the loss ratio has minor impact on action recognition accuracy. However, setting the ratio too low negatively affects the generation performance. In particular, when the ratio set to zero, significant forgetting occurs, severely compromising the model's generation ability.
15

---

Table 10: Ablation study on the effect of different loss balance ratios on SSv2 accuracy and FVD.
Higher SSV2 accuracy and lower FVD indicate better performance.
Balance Ratio
10
20
SSv2 Ace f SSv2 FVD +
75.6
75.5
75.6
75.8
75.5
1579.2
52.0
47.4
46.5
46.9
A.4 Comparision with SVD Baseline
Comparing with the open-source Stable Video Diffusion directly is meaningful. We conduct frame prediction tests using SVD model the SSV2 and Epic-Kitchen datasets, as shown in Table 11. Since the SVD model performs better at higher resolutions, we generated videos at 112x112 resolution and then downsampled them to 256x256 for FVD calculations. The results show that while SVD achieves competitive scores compared to previous state-of-the-an methods shown in Table 1, it is suboptimal compared to GenRec. This likely due SVD's design for general scenarios. Baseline represents our enhanced SVD baseline, which fine-tunes SVD on the target datasets for better results and fairer comparison with GenRec.
Table 11: Comparison of SSv2 FVD and Epic FVD scores frame prediction tests.
Method
SSv2 FVD
Epic FVD
SVD
99.7
180.8
Baseline I GenRec
50.3
53.6
46.5
49.3
A.5 Impact of Classification Training on Generative Ability
To demonstrate classification and generative can coexist well in our training without negatively impacting each other, we construct the comparison with "Baseline I (SVD baseline)", which adapts SVD on the target dataset without classification supervision, and then trains another classifier head freeze the generation backbone, ensuring no influence from classification supervision. We conduct comparisons on SSv2: FP (frame prediction) and CFP (class-condition frame prediction), as shown in Table 12. By comparing the FP results, the similar scores between the two methods conclude that classification loss does not degrade the model's generative ability our approach. Moreover, the CFP results indicate that our method, with its more accurate classification performance, can guide the model to achieve higher-quality video frame predictions.
Comparison of SSv2 Acc and SSv2 FVD scores for Baseline and GenRec methods in FP Table 12:
(frame prediction) and CFP (class-condition frame prediction) scenarios.
Method
SSv2 Acc +
SSv2 FVD 1
Baseline (FP)
55.5
GenRec (FP)
55.3
Baseline (CFP)
63.7
50.3
GenRec (CFP)
75.8
46.5
A.6 Case Study for Caas--ondittooed Emage-to-Vided Generation and Video Interpolation
We show the generated visualization of the GenRec. The model can support video generation given various numbers of frames, as well as category-guides generation. We show two of the most difficult generative scenarios, which are: (1) given the first frame and different action categories to guide the video generation, and (2) given the start and end frames, the model is expected to complement the video. We also compare our methods with SEER [18] with cases picked from its official website. The generation results can be seen in Figure 4, Figure 5 and Figure 6.
16

---

GT Video: Covering something with something
Covering something with something
Pulling something from left to right
Pulling something from right to left
Lifting up one end of something, then letting it drop down
Lifting up one end of something, without letting it drop down
Figure 4: Video generation case study. We generate videos given the first frame together with the classifier guidance for various categories.
A.7 Limitations and Broader Impacts
Limitations and Future Work The objective of our paper is unify the tasks of generation and recognition, achieving or even surpassing the state-of-the-arr experimental performance across various tasks, However, our method based on fine-tuning a pretrained video diffusion model, using more pretraining data and having a larger number of parameters compared to previous methods. This an issue need address in the future, and exploring the distillation of well-pretrained video diffusion model into a smaller model is worthwhile future endeavor.
Broader Impacts The broader impact of the GenRec framework extends into various fields, enhancing capabilities in content creation, security, and accessibility. In the media industry, it allows for the automated generation of tailored, high-quality videos, reducing production costs and fostering creativity. For surveillance, its robustness in limited information scenarios improves monitoring effectiveness, particularly in challenging environments. Additionally, advancements of GenRec in video prediction can aid in developing assistive technologies, making digital content more accessible and interactive, particularly individuals with visual impairments.
17

---

GT
Gen
GT
Gen
GT
Gen
GT
Gen
Figure 5: Video generation study. We generate videos given the first frame and the last frame.
18

---

Action: Pushing something from left to right.
GT
SEER
Ours
Action: Covering something with something.
GT
SEER
Ours
Action: Dropping something front of something
GT
SEER
Ours
Figure 6: Video generation case study. We compare our methods with SEER [18] in the setting of generating videos given the first frame together with the classifier guidance. Cases are picked from the official website of SEER [18].
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We properly claim our contribution in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations in Appendix.
20

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The paper has provided the proof in main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided all the details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release the code in the feature.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Souiides/CodeSumiisstte
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /Co//Soodemiiicceee Sooodesumisssioooooo
/ //nips... /tippscc/ppuit wttp:cccpppppppcccccceo/ococpcca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The training and test details have been reported in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The cost of training required to report error hars is excessively high,
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the computation resources in this paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:///n....peee cc/ cctppblic//eeeeeeccccecccccceeee httpo/lic//hiiieeeee ctticchuuellnnes
Answer: [Yes]
Justification: The paper conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provided the broader impacts in appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper did not release data or models for the main contribution.
Guidelines:
The answer NA means that the paper poses such risks.

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper follows the CC-BY 4.0 license in experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper did not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing experiments in this paper.
25

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing experiments in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
26


Reviews:
[{"review_id": "YOuBBOCcws", "reviewer": "Reviewer_caZh", "summary": "This work proposes a video diffusion model to unifies the video generation and video recognition tasks. The proposed GenRec, during training, jointly optimize the generation objective and the classification objective. During inference, GenRec can deal with both the video generation conditioned on frames or classes, and video classification task. The experiments demonstrate that GenRec shows superior performance in both tasks.", "questions": "Pleaser refer to the weakness part. I would like to modify my rating according to the responses from the authors.", "limitations": "The authors have already claimed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed GenRec unifies the video generation and video recognition tasks.\n- The experimental results validate the effectiveness of the proposed unified GenRec.", "weaknesses": "- Considering that previous works [8,37,46] have already demonstrated that diffusion models have the capability of extracting sufficient features for understanding tasks, the combination of generation and recognition tasks to boost both tasks is too straight-forward, limiting the novelty of the paper.\n- It is unclear for the inference stage for video recognition. Like why the noised $\\tilde{z}$ is still incorporated for classification, which may lead to the recognition result undeterministic. During inference, is the multi-step denoising procedure sill required?\n- There is the lack of training/inference time cost comparison with previous video classification methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will address your concerns point by point.\n\n**Q1: Concern about the novelty compared with [8,37,46].**\n\nThanks. Previous works [8, 37, 46] applied diffusion models to understanding tasks, but there are some limitation that mainly differs from our paper: \n- [8] abandons generation capabilities in the end for improved discriminative performance;\n- [37] uses a frozen image diffusion model and only focuses specifically on low-level geometric correspondence;\n- [46] uses a frozen image diffusion backbone only for mask location prediction;\n\nIn summary, they ignore visual generation, do not unify generation and recognition tasks, and also they are limited to the image domain. In contrast, our paper focuses on the video domain, addressing the following challenges:\n\n- We demonstrate that video diffusion models can leverage generative priors for analyzing high-level actions, especially for tasks requiring thorough temporal information analysis.\n- We demonstrate that video generation and understanding can coexist and enhance each other, improving recognition robustness with limited visual frames, and adherence to instructions for generation.\n\nWe believe these explorations are novel and significant as suggested by Reviewer 96MS.\n\n[8] Deconstructing denoising diffusion models for self-supervised learning. arXiv preprint arXiv:2401.14404, 2024\n\n[37] Emergent correspondence from image diffusion. In Neurips, 2023\n\n[46] Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, 2023\n\n**Q2: Why is noise incorporated during inference for video recognition, how does it affect result determinism, is multi-step denoising required during inference?**\n\nThank you for your feedback. Our unified models are trained on noisy inputs, so maintaining consistency during inference is necessary to ensure robustness. Works [37, 46], as you previous mentioned, also need to add a certain-level noise before extracting visual features. We further do the testing on SSv2 action recognition for multiple times with different random seeds, shown as the table below. The results show that the accuracy is consistent with only 0.0125\\% standard variation, indicating the model’s robustness to the noise. For fully deterministic results, the random seed for noise sampling should be fixed.\n\nDuring inference, only one single-step denoising is required for prediction. Thus, the inference process is efficient.\n\n|**Seed** | 0 | 1 | 2 | 3 | 4 | 5 | AVG|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|**SSv2 Acc** | 75.83\\% | 75.82\\% | 75.83\\%  |75.83\\%  |75.86\\%  |75.84\\%  |(75.835 $\\pm$ 0.0125)\\%  |\n| |\n\n**Q3: Training and inference time cost compared with previous classification methods.**\n\nThank you for your valuable feedback. We compare the training and inference time cost between MVD-H, Baseline II and GenRec, fairly on the same hardware resource: 4-nodes * 8 V100s, and same batch size. Baseline II refers to fully fine-tuning the original SVD model with classification supervision only.\n\nAs shown in the table, GenRec and the Baseline II consumes more time for testing than MVD-H. The difference in time primarily arises from the varying number of parameters. Our model is derived from a generative model, and the complexity of the generative task necessitates a larger number of parameters for effective learning. Compared with Baseline II, since GenRec requires additional decoder blocks for video generation training, the training time will get increased a little bit. As the additional up-sampling blocks will not be used when doing action recognition, GenRec shares the same testing time with Baseline II. It is worth noting that ''✝'' in MVD-H is to highlight that MVD method would use repeated augmentation technique during training, but such augmentation will significantly increase the training time. Our approach does not need to use that augmentation.\nAll the methods do the down-stream finetuning for 30 epochs.\n\n|**Method** | Trainable Params | Total Params | Train Time | Train epochs | Test Time (2$\\times$3 clips) |\n|:--:|:--:|:--:|:--:|:--:|:--:|\n|MVD-H ✝ |  633M | 633M  | 3038 s/Epoch| 30 | 441s |\n|Baseline II | 1.3B | 1.9B | 2954 s/Epoch | 30 | 1500s |\n| GenRec | 1.5B | 2.1B | 3751 s/Epoch | 30 | 1500s |\n| |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UXHg9sI6fZ", "reviewer": "Reviewer_PRtT", "summary": "Recently, diffusion models have been released that can generate high quality video clips, indicating that they strongly capture natural appearance and motion in video. This paper seeks to adapt one recent model, Stable Video Diffusion (SVD), to explore if what it has learned can be leveraged to improve video recognition performance, and investigate whether co-training on generation and recognition using this model can improve on both tasks.\n\nThe two tasks, video diffusion and recognition, have quite different setups: the first takes uncorrupted video as input, while the second starts with a single conditioning frame as well as noise-corrupted video. This paper proposes to unify the tasks, adapting the SVD model to take masked-uncorrupted video plus noised video, concatenated, as input. This unifies the tasks, allowing joint training on recognition and generation. It also permits new tasks: generation and recognition using a variety of partially-masked video configurations as input.\n\nThe paper shows competitive results on classic video clip classification tasks, and very strong results on generation FVD. Further they demonstrate strong classification performance on masked input scenarios including early action prediction, and recognition with limited frame-rates.", "questions": "The generative results in Table 1 demonstrate a really surprising decrease in FVD compared to the other methods. Can you explain why this is the case in more detail? Is the model really that good? Is it a typo? Is it because this benchmark is not particularly competitive? (I think FVD is more often presented on UCF and Kinetics-400 in the papers I've read, e.g. https://arxiv.org/pdf/2212.05199). If you double-check the FVD result, and convince me, then I'll gladly list this surprising decrease as one of the paper's strengths. \n\nWould it be useful to clarify to readers why the focus is \"particularly the unconditioned or image-conditioned models\" in line 29?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper addresses what is to me, a researcher in video recognition and generation, a really burning question: generative video models have clearly learned an incredible amount about the visual world, so how can we harness that to advance recognition performance?\n\nBut the setups of the problems, recognition and generation via diffusion, are really quite different. The authors show how to unify them, and co-train on the two tasks, in an original way that actually produces synergies for both. This is an exciting result to me.\n\nThe paper is very clearly written and easy to follow (although the math dump in section 3.3 could be a little clear).\n\nI appreciate the inclusion of the two strong baselines (I and II, described on line 213), that compare the unified model to models trained only on generative or only on discriminative tasks.", "weaknesses": "In the recognition results, Table 1 doesn't represent the current SOTA. E.g. InternVideo (https://arxiv.org/abs/2212.03191) from 2022 claims 77.2% on SSv2 and 91.1% on K400. It's fine if GenRec doesn't beat SOTA, but SOTA should be accurately represented in the table.\n\nThe generative experimental results show a surprising improvement on FVD on SSv2 and EK-100, which I don't feel the authors do enough to explain or analyze.  (See an associated \"Question\" below.)\n\nThere is no exploration of how different values of the generative/discriminative loss balance ratio, gamma, affects performance on the benchmarks.\n\nSimilarly, there is no exploration of different latent masking schemes or ratios during training.\n\nMinor weaknesses:\n* The method is described as \"analogous to a temporal extension of MAE\" in line 53. VideoMAE [38] does already exist. \n* On line 61, please define the accuracy retention ratio of include a citation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive feedback! We address your questions below.\n\n**Q1:SoTA results completion.**\n\nThank you for your valuable feedback. We have updated the Table 1 in the **uploaded PDF**, which includes more comparisons with methods like InternVid, InternVid2, OmniVec and OmniVec2. In addition, we have included the parameter counts for each method for a more comprehensive comparison.\n\nIt is important to note that, our initial version primarily focused on comparing with single-modality self-supervised methods. In contrast, the newly added methods, e.g, InternVid2, incorporate more modalities, introducing multiple modality encoders for multi-modal alignment on large data, and is built upon the previous self-supervised methods. Therefore, we have distinguished between these two types of methods in the updated table.\n\n**Q2:More analysis and explanation about the FVD scores.**\n\nThank you for your valuable feedback. We have double-checked our FVD results and confirmed their accuracy. The reduction in FVD can be attributed to two main factors: The use of the SVD model and the guidance of an accurate classifier. Among two of them, the application of the SVD pretrained model is the primary reason.\n\nTo illustrate this point, we have added the FVD scores for baseline I for frame prediction and class-conditioned frame prediction. Results are shown as below. It is evident that using the fine-tuned SVD of \"baseline I\" for frame prediction (FP, given the first two frames and predicting the subsequent video frames) yields good scores as shown in the first line of the table below, thanks to the rich priors learned during the SVD pretraining on large-scale video data. In contrast, other methods primarily rely on image-based stable diffusion models and fine-tune them on limited video data, making it more challenging to learn temporal dynamics and resulting in lower performance. In addition, the table below also shows that precise classifier guidance, corresponding to CFP (class-condition frame prediction), leads to better FVD results.\n\nIn summary, the major cause for the low FVD is the usage of pretrained SVD model, along with the significant impact of accurate classification guidance. We will include the analysis and explanation of the FVD scores in our revised version.\n\n| **Method** | **SSv2 Acc $\\uparrow$** | **SSv2 FVD $\\downarrow$** |\n| :--: | :--: | :--: |\n|Baseline I (FP) | - | 55.5 |\n|Baseline I (CFP) | 63.7 | 50.3 |\n|GenRec (CFP) | 75.8 | 46.5 |\n||\n\n**Q3:Ablation on the loss balance ratio and the masking schemes.**\n\nThank you for your valuable feedback. We conduct an ablation study on the loss balance ratio, and the results are presented below. The study shows that varying the loss ratio has a minor impact on action recognition accuracy. However, setting the ratio too low negatively affects the generation performance. In particular, when the ratio is set to zero, significant forgetting occurs, severely compromising the model’s ability to perform the generation task.\n\n| |  |  |  | |  |\n| :--: | :------: | :--: | :--: | :--: | :--: | \n| **Balance Ratio** |  0 | 1 | 5 | 10 | 20 |\n| **SSv2 Acc $\\uparrow$**| 75.6 | 75.5 | 75.6 | 75.8 | 75.5 |\n| **SSv2 FVD $\\downarrow$** | 1579.2 |52.0 | 47.4 | 46.5 | 46.9 |\n||\n\nWe also ablate the masking schemes with different masking ratio expectation. The results show that a larger masking ratio might be beneficial for generation, as it closely resembles our class-conditioned frame prediction with one or two given frames setting. However, too large masking ratio (87.5\\%) harms the action recognition accuracy, resulting in a 0.9% decrease compared to our chosen ratio.\n\nThese ablation results will be included in the final revised version.\n\n| **Expectation of Masking Ratio** | **SSv2 Acc $\\uparrow$** | **SSv2 FVD $\\downarrow$** |\n| :--: | :--: | :--: |\n|**75\\% (our choice)** | 75.8 | 46.5 |\n|50\\% | +0.3 | +0.5 |\n|87.5\\% | -0.9 | -0.3 |\n||\n\n**Q4:Why is the focus on “particularly the unconditioned or image-conditioned models?**\n\nThanks. This sentence highlights our focus is not on the type of text-conditioned video generation methods that requires video description as input. This is because, understanding videos requires the model to analyze video content, whereas text-conditioned generation relies on descriptions that predefine the content. Such conflict makes it harder to unify these two approaches.\n\n**Q5:Minor weakness about the expressions.**\n\nThank you so much for your carefully reading. We will definitely revise the expressions in the revision version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nr5Ax7hj8G", "reviewer": "Reviewer_JBof", "summary": "This paper investigates to unify the video generation and recognition tasks into the same diffusion model. The authors propose a conditional feature mask mechanism to unify the two learning tasks. Experiments are conducted on several tasks to validate the effectiveness of the proposed method.", "questions": "1. Why the loss weight of the recognition loss in Eq. 8 is not related with the noise level?\n2.  How about the training and testing efficiency compared with other recognition models?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of unifying generation and recognition process into the same diffusion model is interesting and necessary for the computer vision. \nThe modification to the SVD backbone is simple, including a masked conditional input and a classifier head.", "weaknesses": "1. The design of unifying generation and recognition tasks is not elegant. This paper directly introduces an extra classification head, and uses a multi-task learning loss for these two tasks. As verified in the Table 4, the trade-off between high-level semantic recognition task and low-level detail synthesis is not controlled well and may be conflict.\n2. The experimental results are not promising. Firstly, the model size and computational cost should be reported in Table 1. It will be clear to analyze the efficiency of the recognition model. Secondly, for the generation ability evaluation, since the model is initialized from the SVD, it’s better to compare with SVD on different datasets. A concern is that whether the classification loss training will degrade the model’s synthesis ability on natural scene. It’s over claim to say “surpassing the sota”.\n3. The mask mechanism in the proposed method shares some similarity with the paper [1]. The authors are suggested to add a discussion with this work.\n4. For the recognition task, the proposed method just uses the SVD backbone to extract features, since z0 is directly used. Such a design (i.e., extracting features from the input video and sending them to a classifier)  is conventional.\n\n\n[1] Masked Diffusion Transformer is a Strong Image Synthesizer, ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will address your concerns point by point.\n\n**Q1: Why the loss weight of the recognition loss is not related with the noise level?**\n\nThanks for your valuable feedback. Adding loss weighting related to noise is standard for diffusion training, but is unexplored for classification. Designing the dynamic classification loss weight brings additional hyperparameters, influenced by factors such as noise level and masking ratio. Note that in EDM generation, small loss ratio are set when the noise levels are large and small. However, for classification, we aim for robust partially video recognition with varing noise. Thus, we maintain a constant classification loss weight to ensure stability, and also to reduce the number of hyperparameters, making training more manageable. Experimental results demonstrate our design effectively balances both generation and classification.\n\n**Q2: Training and testing efficiency/computational cost, compared with other recognition models.**\n\nThanks for your valuable feedback. We compare the training and inference time costs between MVD-H, Baseline II, and GenRec, using the same hardware (4 nodes * 8 V100s) and batch size. Baseline II refers to fully fine-tuning the SVD with classification only.\n\nAs shown in the table, both GenRec and Baseline II consume more test time than MVD-H, primarily due to the higher number of parameters. GenRec requires additional decoder blocks for video generation, which slightly increases the training time compared to Baseline II. However, these blocks are not used during action recognition, so GenRec and Baseline II share the same test time. Note that the ‘‘✝’’ in MVD-H indicates the use of repeated augmentation, which significantly increases the training time. Our approach doesn't need to use that augmentation. All methods fine-tune downstream for 30 epochs.\n\n|**Method**|Trainable Params|Total Params|Train Time|Train epochs|Test Time (2$\\times$3 clips)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|MVD-H ✝|633M|633M|3038 s/Epoch|30|441s|\n|Baseline II|1.3B|1.9B|2954 s/Epoch|30|1500s|\n|GenRec|1.5B|2.1B|3751 s/Epoch|30|1500s|\n||\n\n**Q3: Model size.**\n\nThanks. We have updated Table 1 in the uploaded PDF to include parameter counts.\n\n**Q4: Concern about whether the classification training will degrade the model generation ability.**\n\nThank you for your valuable feedback. To demonstrate classification and generative can coexist well in our training without negatively impacting each other, we have constructed the comparison with \"Baseline I (SVD baseline)\", which adapts SVD on the target dataset without classification supervision, and then trains another classifier head freeze the generation backbone, ensuring no influence from classification supervision. We conduct comparisons on SSv2: FP (predicting future frames) and CFP (predicting future frames with class information).\n\nBy comparing the FP results, the similar scores between the two methods conclude that classification loss does not degrade the model’s generative ability in our approach. Moreover, the CFP results indicate that our method, with its more accurate classification performance, can guide the model to achieve higher-quality video frame predictions.\n\nWe will update the above discussion in our revised version.\n\n|**Method**|**SSv2 Acc $\\uparrow$**|**SSv2 FVD $\\downarrow$**|\n|:-:|:-:|:-:|\n|Baseline I (FP)|-|55.5|\n|GenRec (FP)|-|55.3|\n||\n|Baseline I (CFP)|63.7|50.3|\n|GenRec (CFP)|75.8 |46.5|\n||\n\n**Q5: Mask mechanism discussion with [A].**\n\nThank you for your valuable suggestions. In [A], the authors choose to mask on the noisy latents directly, while we choose to mask the condition latents. Masking noisy latents can break the mathematical theorem basis and complicate the task. Therefore, we opt not to influence the latent noise but to change the conditional latents instead. \n\nWe will add the discussion about [A] in the related work section of our revised version.\n\n[A] Masked Diffusion Transformer is a Strong Image Synthesizer, ICCV 2023.\n\n**Q6: Table 4 results show recognition and generation may be conflict.**\n\nThanks. Table 4 presents an ablation study on the layer index for feature extraction, showing proper layer selection is needed for preventing conflict. Shallow feature extraction decreases performance due to insufficient complexity, while late feature extraction conflicts with generation. Thus, we selected an optimal layer to balance performance and prevent conflicts between generation and understanding tasks.\n\nFor further demonstration, we **upload the PDF** and compare with baseline performance. In the table 1, Baseline I train the diffusion process without classification, and Baseline II trains classification without generation. The results show our method outperforms Baseline I in generation and matches or exceeds Baseline II in classification. These findings confirm our final GenRec model does not exhibit conflicts between generation and classification.\n\n**Q7: SVD to extract features is conventional, and the design of unifying generation and recognition is not elegant.**\n\nThanks. We would like to stress that applying diffusion models for video understanding is non-trivial due to the complicated nature of videos. Moreover, unifying generation and recognition through diffusion models is even more challenging, as conflicts can easily arise and mutual enhancement between the two tasks is difficult to achieve. **This is precisely where our GenRec stands out.**\n\nBy exploring the temporal modeling capabilities of SVD and integrating our novel random masking condition, GenRe effectively bridges the gap between generation and classification, and achieves significant mutual enhancement between the two types of tasks. The effectiveness confirms our design is innovative and pragmatic, and we believe GenRec represents an exciting direction for the computer vision community by combining generation and recognition tasks in a unified framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47tCGCbf0B", "reviewer": "Reviewer_96MS", "summary": "The paper proposes to unify video recognition and generation with a masked finetuning of SVD (diffusion video model). By training with both classification and generation objectives, the model learns to do both. Rather than training with first frame only for conditioning, the masking is employed so that the classifier can learn better. The resulting model has recognition performance that is actually enhanced by the generative part of the training, and results are given across various tasks and subtasks, including action prediction and generating videos given multi-frame conditioning.", "questions": "How did you select your SOTAs? One of the main things preventing me from giving a higher rating is that the model does considerably lag behind SOTAs for k400 classification. \n\nWhy do you not provide example videos for the competing methods, as reference points? It is difficult to assess the capabilities of the video generation based on FVD alone.\n\nWhy is there no generative baseline, similar to the classification baselines?", "limitations": "Yes.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "[S1] The method has good results as a unified model, and introduces a novel way to finetune video diffusion for both recognition and generation, performing very well on both tasks. Such a system is important because ideally it saves on storage cost (no need to store a model for each task, just store 1 for both), training time, makes better use of data (training with supervised text losses and unsupervised generative losses), and should synergize between the tasks to form a true world model.\n\n[S2] The fact that the generative training helps the classification performance, as shown in Table for example, offers critical evidence to support the paper's claim that it unifies generation and recognition (rather than them competing with each other).\n\n[S3] Baseline I and Baseline II are very helpful for contextualizing the performance of the model, showing what is possible with the CE training alone.", "weaknesses": "[W1] It is unclear why the paper compares with methods that pretrain on only kinetics and imagenet (like Hiera) when GenRec uses the SVD which is pretrained on massive amounts of external data. For the recognition task, appropriate SOTAs would be OmniVec, OmniVec2, InternVideo, InternVideo2, etc. \n\n[W2] FVD is not a very good metric. A generative paper ought to have generative results in the main paper itself, FVD results alone are not adequate, and such examples ought not be saved for supplementary. Additionally, the examples in the supplementary are not adequate, since there is no comparison to the competing methods.\n\n[W3] Paper does not cite enough related work. For example, [1] and [2] are omitted, as are many others that explore similar diffusion-for-recognition directinos.\n\n[1] Your Diffusion Model is Secretly a Zero-Shot Classifier, Li et al.\n[2] Diffusion Models Beat GANs on Image Classification, Mukhopadhyay et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive feedback! We address your questions below.\n\n**Q1: SOTA selection and completion.**\n\nWe greatly appreciate your suggestion. We agree that including more comparable methods is very meaningful. **We have updated the Table 1 in the uploaded PDF** to include more comparable methods, specifically InternVid, InternVid2, OmniVec, and OmniVec2, and categorized them into single modality and multi-modality for clearer comparison.\n\nOur initial version primarily focused on comparing with single-modality self-supervised methods. While the newly added methods incorporate multiple modalities, introducing multiple modality encoders for multi-modal alignment on large datasets, building upon previous self-supervised methods. Although SVD is a model that is pretrained on additional data, it does not explicitly perform multi-modal alignment during the generation training process, making it hard to surpass their performance. Indeed, unifying the generation task and visual-language multi-modal alignment is a promising research direction and can be achieved by extending our work.\n\n**Q2: Provide example videos for the competing methods, as reference points.**\n\nThanks for your helpful suggestions. We have added more detailed example comparison with competing methods, as shown in the **uploaded PDF, Figure 1 and Figure 2**. In the update, we compare examples provided by SEER official website. Also we add more comparison among the closed-source generation methods (PIKA, Gen-2, KLING) and open-source method (SEER) on action control cases. From the cases in the PDF, we find that: \nSEER suffers from distortion issues and lacks accuracy, e.g, easy to get wrong for the left-right direction control. PIKA maintains strong visual consistency but fails to generate accurately according to instructions. Gen-2 often has problems with object deformation. KLING shows very high visual consistency, and for some instructions it can generate high-quality correct videos, e.g, cases of ``covering something<the toy> with something<a blue towel>''. But sometimes it struggles with fine-grained action control. Our method demonstrates good performance, with higher visual quality and more accurate fine-grained action control, compared with other methods.\n\nMore comprehensive cases comparison will be added in our final revision version.\n\n**Q3: Need Generation baseline.**\n\nThank you for your insightful suggestions. We acknowledge the importance of including a generation baseline for SVD. Baseline I can serve as a robust generation baseline, and we have supplemented it with its generation scores to act as the generation baseline, shown as bellow table. We would update this in our revised paper version.\n\n| **Method** | SSv2 Acc  $\\uparrow$ | K400 Acc  $\\uparrow$ | SSv2 FVD  $\\downarrow$ | Epic FVD  $\\downarrow$ | \n| :--: | :--: | :--: | :--: | :--: | \n|Baseline I | 63.7 | 82.0  | 50.3 | 53.6 |\n|GenRec | 75.8 | 87.2 | 46.5 | 49.3 |\n||\n\n**Q4: Paper should cite more enough works**\n\n Thanks. We would cite your mentioned related works in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nzi5AVn4ak", "reviewer": "Reviewer_C3Pd", "summary": "This paper explores the use of video diffusion models for joint generation and recognition. The authors introduce a framework called GenRec, based on Stable Video Diffusion, that aims to unify video generation and recognition. GenRec employs a random-frame conditioning process to learn generalized spatial-temporal representations, making it robust even with limited visual inputs. Experiments show that GenRec excels in both video recognition and generation. Additionally, GenRec proves to be highly robust in scenarios with limited frame availability.", "questions": "Can you provide more details on the derivation of Eq.14-16?", "limitations": "The authors have mentioned the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The integration of MAE and Diffusion models is a practical and innovative approach to unify video generation and recognition tasks.\n\n2. The experiments are comprehensive, covering video generation, video prediction, and video recognition. The provided results demonstrate strong performance of the model across all tasks.", "weaknesses": "1. The notations used in the Preliminary Section are inconsistent. The authors should review and correct the formulations for clarity.\n\n2. In Section 3.3, the authors claim to have adopted the stochastic sampler in EDM, but it appears that the second-order correction is not included according to Algorithm 2 in EDM. The authors are encouraged to provide justifications for this omission.\n\n3. In the derivation on Line 156, it is unclear why such a replacement can be used for the original score function. Additionally, the derivation from Equations 14 to 16 lacks clarity. The authors should provide more detailed explanations.\n\n4. In Table 1, the generative performance of Baseline I should be included. Furthermore, information on parameter counts and computational overhead should also be presented.\n\n5. There are misspellings such as \"mensioned\" on Line 135 and \"interatively\" on Line 147. These should be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will address your concerns point by point.\n\n**Q1: Provide more details on the derivation of Eq.14-16.**\n\nThe derivation of Eqs. 14-16 is similar to Eq. 12 in **[A]**. Here are the details.\n\nFirstly, Eq. 13 in our paper shows the formulation of the unconditional noise reduction residual.\n$$\n\\epsilon_{\\theta}(\\hat{z_i}; \\overline{z_0})=(-1)(t_{i-1}-\\hat{t_i}) \\nabla_{\\hat{z_i}} \\log p_{\\theta} (\\hat{z_i})\n$$\nTo introduce the class control, we substitute the score function $\\nabla p(\\hat{z_i})$ with the conditional form $\\nabla_{\\hat{z_i}} \\log p_{\\theta} (\\hat{z_i}|y)$, aiming it to be the conditional distribution we want. It is worth noting that the substitution introduces class information and is not an equivalent replacement. Next, by applying the Bayes` Theorem, the score function becomes:\n\n$$\n\\nabla_{\\hat{z_i}} \\log p (\\hat{z_i}|y)=\\nabla_{\\hat{z_i}} \\log \\frac{p (y | \\hat{z_i}) \\cdot p(\\hat{z_i})   }{p(y)}\n=\\nabla_{\\hat{z_i}} \\log p(y | \\hat{z_i}) + \\nabla_{\\hat{z_i}} \\log p(\\hat{z_i}) - \\nabla_{\\hat{z_i}} \\log p(y)\n=\\nabla_{\\hat{z_i}} \\log p(y | \\hat{z_i}) + \\nabla_{\\hat{z_i}} \\log p(\\hat{z_i})\n$$\n\nEvidently, ${\\nabla_{\\hat{\\mathbf{z}}_i}} \\log p(y)=0$ and can be removed. As a result, the conditional noise reduction residual formulation will become, \n\n$$\n\\epsilon_{\\theta}^{*}(\\hat{z_i}; \\overline{z_0})=(-1)(t_{i-1}-\\hat{t_i}) \\nabla_{\\hat{z_i}} [\\nabla_{\\hat{z_i}} \\log p(y | \\hat{z_i}) + \\nabla_{\\hat{z_i}} \\log p(\\hat{z_i})]\n$$\nwhich corresponds to the Eq. 15 in our paper. We will update this section with the above detailed derivation process in the revised version.\n\n[A] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.\n\n**Q2: The authors are encouraged to provide justifications for the EDM second-order correction ommision**\n\nWe follow the setting of SVD implementation, which uses the EulerDiscreteScheduler within the EDM framework, as can be seen in the **``huggingface, stable-video-diffusion-img2vid-xt, scheduler/scheduler\\_config.json''**. EulerDiscreteScheduler do the denoising step without second-order correction, so in our previous version, we omit the second order correction. \n\nIn addition, with the Eq. 17 that reveals the calculating way for the class-condition denoising residual  $\\epsilon^{*}_{\\theta}(\\hat{\\mathbf{z}}_i;\\overline{\\mathbf{\\mathbf{z}}_0})$, \n\nwe can substitute it into the algorithm 2 in EDM to get the second-order expression as below:\n$$\n\\begin{align}\n    z_{i-1}^{corr}\n    & = z_i + \\frac{1}{2}{(t_{i-1} - \\hat{t_i})}(d_i + d_{i}^{'}) = z_i + \\frac{1}{2} {\\epsilon_{\\theta}^{\\*} (\\hat{z_i};\\overline{z_0})} + \\frac{1}{2}\\frac{t_{i-1} - \\hat{t_i}}{t_{i-2}-t_{i-1}} \\epsilon_{\\theta} ^{\\*} (\\hat{z_i} + {\\epsilon_{\\theta}^{*} (\\hat{z_i};\\overline{z_0})};\\overline{z_0})\n\\end{align}\n$$\n\nWe will add the above illustration and the derivation process in the final revised version. \n\n**Q3: In Table 1, the generative performance of Baseline I should be included. Furthermore, information on parameter counts and computational overhead should also be presented.**\n\nThank you for your insightful suggestions. We acknowledge the importance of including the generative performance of Baseline I. We have supplemented Baseline I with its generation scores to act as the generation baseline, shown as bellow. \n\n| **Method** | SSv2 Acc  $\\uparrow$ | K400 Acc  $\\uparrow$ | SSv2 FVD  $\\downarrow$ | Epic FVD  $\\downarrow$ | \n| :--: | :--: | :--: | :--: | :--: | \n|Baseline I | 63.7 | 82.0  | 50.3 | 53.6 |\n|GenRec | 75.8 | 87.2 | 46.5 | 49.3 |\n||\n\nAlso the parameter counts information has been added in the **updated PDF, Table 1**. \n\nAs for the computational cost/time analysis, we compare the training and recognition inference time cost between MVD-H, baseline II and GenRec, fairly on the same hardware resource: 4-nodes * 8 V100s, and same batch size. Baseline II refers to fully fine-tuning the original SVD model with classification supervision only.\nAs shown in the table, both GenRec and Baseline II consume more test time than MVD-H, primarily due to the higher number of parameters. GenRec requires additional decoder blocks for video generation, which slightly increases the training time compared to Baseline II. However, these blocks are not used during action recognition, so GenRec and Baseline II share the same test time. Note that the ‘‘✝’’ in MVD-H indicates the use of repeated augmentation, which significantly increases the training time. Our approach doesn't need to use that augmentation. All methods fine-tune downstream for 30 epochs.\n\n|**Method** | Trainable Params | Total Params | Train Time | Train epochs | Test Time (2$\\times$3 clips) |\n|:--:|:--:|:--:|:--:|:--:|:--:|\n|MVD-H ✝ |  633M | 633M  | 3038 s/Epoch| 30 | 441s |\n|Baseline II | 1.3B | 1.9B | 2954 s/Epoch | 30 | 1500s |\n| GenRec | 1.5B | 2.1B | 3751 s/Epoch | 30 | 1500s |\n\nThe above discussions will be included in our revised version.\n\n**Q4: misspelling words and notation inconsistent**\n\nThanks for your carefully reading. We will definitely check and revise, and make sure of the consistency in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their detailed, thoughtful and valuable feedback. We are grateful that the reviewers identified our work unifies the video generation and video recognition (Reviewer caZh), provides "exciting result" (Reviewer PRtT), are interesting and necessary for the computer vision (Reviewer JBof), are novel and helpful (Reviewer 96MS), are comprehensive and demonstrates strong performance (Reviewer C3Pd). We have tried our best to address all the questions and improve our paper following the guidance of the reviewers. If you have further questions, feel free to attach us by the official comment. The uploaded pdf contains a large table and some visualization cases, which are helpful for the following rebuttal responses.

Meta-review: This paper garnered overall positive reviews, with consensus on the merit of the proposed model that integrates video generation and recognition. The authors adeptly addressed concerns about novelty, experimental design, and efficiency. Given its innovative nature, compelling results, and satisfactory responses to feedback, I recommend the paper for publication.  It's a significant contribution that paves the way for future unified video generation and recognition research.
Decision: Accept (poster)