GenRec: Unifying Video Generation and Recognition with Diffusion Models

Authors: zejia  weng
xitong  yang
zhen  xing
zuxuan  wu
yu-gang  jiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:31:02

Abstract: Video diffusion models are able to generate high-quality videos by learning strong spatial-temporal priors on large-scale datasets. In this paper, we aim to investigate whether such priors derived from a generative process are suitable for video recognition, and eventually joint optimization of generation and recognition. Building upon Stable Video Diffusion, we introduce GenRec, the first unified framework trained with a random-frame conditioning process so as to learn generalized spatial-temporal representations. The resulting framework can naturally supports generation and recognition, and more importantly is robust even when visual inputs contain limited information. 
Extensive experiments demonstrate the efficacy of GenRec for both recognition and generation. In particular, GenRec achieves competitive recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. GenRec also performs the best on class-conditioned image-to-video generation, achieving 46.5 and 49.3 FVD scores on SSV2 and EK-100 datasets. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed. Code will be available at https://github.com/wengzejia1/GenRec.

Main paper: 
 introduction: 
 Diffusion models have achieved significant success in the field of image and video generation over the past few years. A variety of generative tasks have been revolutionized by using diffusion models trained on Internet-scale data, such as text-to-image generation [33, 30] , image editing [23] , and more recently, text-to-video generation [15, 2, 52] and text&image-to-video generation [56, 18, 21] . The excellent generative capabilities of diffusion models suggest that informative representation is learned during the generative training and strong visual priors are captured by the backbone models [9, 43, 6] . Therefore, recent work has explored leveraging the image diffusion models for image understanding tasks, including image recognition [9, 8] , object detection [7, 57] , segmentation [55] and correspondence mining [39] . However, the capability of video diffusion models to effectively capture spatial-temporal information is not fully understood, and their potential for downstream video understanding tasks remains under-explored.In this paper, we study the potential of video diffusion models [28, 1, 27] , particularly the unconditioned or image-conditioned models, for video understanding by addressing the three key problems: (a) Does the backbone model trained for video generation extract effective spatial-temporal representations for semantic video recognition? (b) Can we retain the video generation capability by jointly optimizing generation and recognition? (c) Will such a unified training framework further benefit video understanding, especially in noisy scenarios where only limited frames are available [3, 25] . To bridge this gap, we propose (c) GenRec: a learning framework that processes mask frames V M using a masking function M (•) and noise videos V σ with noise sampling N (•, σ), aiming to simultaneously learn video understanding and content completion with the same partially observed visual content.While conceptually appealing, unifying video generation and recognition into a diffusion framework is non-trivial. Prior work either views the diffusion models as frozen feature extractors [9, 55, 39] , or deconstructs them for new tasks while sacrificing their original generation capability [8] . One major challenge comes from their distinct training and inference processes. Diffusion models are typically optimized using corrupted inputs, optionally augmented with a single conditioning frame, to achieve unconditioned or image-conditioned generation during inference [27, 1] . In contrast, video recognition models require access to multiple frames to reason about temporal relationships and expect clean inputs during inference [49, 51, 48] . Consequently, training a recognition model using corrupted videos and single-image conditions tends to suffer from inferior model optimization and a more significant training-inference gap.To this end, we propose GenRec, a unified video diffusion model that enables joint optimization for video generation and recognition. Our model is built upon the open-source, image-conditioned Stable Video Diffusion model (SVD) [1] , which encodes strong spatial-temporal priors by pretraining on large-scale image and video data. However, instead of conditioning on the same image across all video frames, we propose to condition on a random subset of frames while masking the remaining ones (see Figure 2 ). This simple random-frame conditioning process effectively bridges the gap between the learning processes of the two tasks. On the one hand, the generation capability of SVD is extended to handle arbitrary frame prediction, which provides more flexible and unambiguous video generation. On the other hand, conditioning on a random subset of frames allows the model to learn more discriminative and robust features for the recognition task. As shown in Figure 1 , the model is jointly optimized using both generative supervision (i.e., noise prediction) and classification supervision.We conduct extensive experiments to evaluate the performance of GenRec for both recognition and generation. Without sacrificing the generation capabilities, GenRec demonstrates competitive video recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed. For example, when only the front half of the video can be observed, GenRec achieves the 57.7% accuracy, which corresponds to 76.6% of the accuracy (75.3%) when the entire video is visible, emonstrating a higher accuracy retention ratio than other methods. By leveraging the recognition model for classifier guidance [11] , GenRec also achieves superior class-conditioned image-to-video generation results, with FVD scores of 46.5 and 49.3 on the SSV2 and EK-100 datasets, respectively. 
preliminary: 
 Representing the data distribution as p data (z) with a standard deviation of σ data , we can obtain a family of smoothed distributions p(z; σ) by adding independent and identically distributed Gaussian noise with standard deviation σ. In the spirit of diffusion models, the generation process begins with a noise image z N ∼ N (0, σ 2 max I) and iteratively denoises it at decreasing noise levels σ N = σ max > σ N -1 > . . . > σ 0 = 0. The final denoised result z 0 is thus distributed according to the original data.In the EDM [22] framework, the original z 0 will be diffused as:EQUATIONand the corresponding PF-ODE [34] follows:EQUATIONwhere ∇ z log p t (z t ) is the score function. Noise schedule σ(t) is set as time step t. The training objective is to minimize the L2 loss with the denoiser network D θ for different σ:EQUATIONwith the relation between D θ and the score function ∇ z log p(z; σ) as follows:EQUATIONSVD [31] utilizes the EDM framework to perform generative training on large-scale video datasets, resulting in a high-quality video generation model. An image-to-video generation model capable of forecasting future frames given the first frame has been released. Following SVD method, we also process videos in latent space. Given an input video V ∈ R T ×H×W ×3 , a pretrained VAE encoder is used to project it into the latent space frame by frame, resulting in the latent representation z 0 ∈ R T ×h×w×D . We then build GenRec based on SVD, inheriting its strong spatial-temporal priors as foundation for the subsequent generation and classification tasks. 
genrec: 
 We now introduce GenRec, a simple yet efficient framework, that can not only generate temporallycoherent videos conditioned on an arbitrary number of provided frames but also is able to recognize actions and events with the help of encoded spatial-temporal priors. To this end, GenRec explores the strong spatial-temporal priors learned by a video diffusion model. In this work, we instantiate GenRec with the powerful open-source Stable Video Diffusion model (SVD) [1] , which is pretrained on large-scale video datasets and is able to produce a photo-realistic video when provided a single frame. Then, for generation, GenRec follows the classical EDM framework to learn noise reduction trajectories. For recognition, on the other hand, GenRec operates on intermediate decoded features using a recognition head. Furthermore, to generate videos in a more free fashion, i.e. an arbitrary collection of frames used as condition, we design a latent masking strategy that "interpolates" masked frames. Such a strategy also benefits recognition by easing the training process. More importantly, by doing so GenRec supports a multitude of downstream tasks, particularly when limited visual information is provided.Latent diffusion and latent masking. During the diffusion process, the Gaussian noise with a certain noise level is added to the latent representation z 0 , creating a noisy latent representation zi following Equation (1) . Recall that while SVD contains powerful spatial-temporal priors, it can only perform generation when the first frame is provided. To allow a more "free" generation with an arbitrary number of frames as inputs, we design a latent masking strategy. More specifically, we apply a random mask m to the latent representation z 0 , producing a masked latent representation z 0 . Such a strategy encourages the model to reconstruct the original video content from incomplete frames, which is in a similar spirit to MAE [19] . Note that when only the first latent is available, it degrades to the same as SVD; if all latents are masked out, this degrades to unconditional generation. Furthermore, doing so also benefits recognition tasks when limited visual clues are available. For example, in scenarios with limited bandwidth leading to reduced frame rates, the ability of video frame complementation enables the model to better predict and perceive complete video information.In practice, we simulate such conditions by randomly erasing half to all video conditions, retaining on average only about one-fourth of the original video information. This technique allows the model to effectively fill in the missing information, enhancing its ability to recognize and understand the video content despite the reduced data availability.Unifying generation and understanding. To unify generation with masked latents, GenRec predicts pixel-level and semantic-level contents with the combination of the noisy latent zi and the masked latent z 0 gained by the aforementioned latent diffusion and latent masking. The two latents are channel-wise concatenated [ zi , z 0 ] and are fed into a Spatial-Temporal UNet, together with features from observed frames, to learn spatial and temporal representations, following [1] . The weights of the UNet are initialized from [1] to obtain spatial and temporal priors, learned on large-scale video datasets.For the generation task, the UNet aims to reconstruct the original latent representation from the combined noisy and masked inputs. Representing UNet as the mapping function F θ , its goal is to predict clean latent, which, according to the EDM framework, takes the form of a representation mapping as follows:EQUATIONin which we set the same skip connection c skip , scaling factor c out and c in as [1] .For the recognition task, we break down the UNet mapping function asF = F tail • F head . And we consider F head ([c in (σ)z, z]) ∈ R T ×h ′ ×w ′ ×D ′as the compact video representation extracted from the intermediate layer of the UNet model, which is then fed into the classifier head ϕ θ , consisting of an attentive pooler and a fully connected layer to predict video categories:EQUATIONWe train GenRec with both generation and classification objectives, encouraging the model to learn high-quality video generation and accurate video understanding.The generative loss uses a L2 loss to measure the difference between the original latent representation and the reconstructed output produced by the UNet, and is defined as:EQUATIONwhere D θ ( zi ; z 0 , σ) is the denoised output mentioned in Equation ( 5), and λ(σ) is a weighting function based on the noise level σ referring to [1, 22] . While the classification loss uses a crossentropy loss to measure the discrepancy between the true labels and the predicted labels, and is defined as:EQUATIONwhere y denotes the ground truth labels, and ŷ represents the predicted labels referring to Equation Equation (6) .To balance the learning of generative and recognition tasks, we set a balancing weight γ to control the relative importance of each loss in the overall objective function. The total loss L is given by:EQUATIONWith the above training strategies, we now introduce how GenRec can flexibly support different types of generation and recognition tasks.Video generation conditioned on frames. Once trained, GenRec is able to generate high-quality videos conditioned on an arbitrary number of given frames, thanks to the latent masking strategy.Particularly, following the EDM stochastic sampler framework and Equation ( 2), GenRec iteratively denoises the video conditioned on the masked latent z 0 , as shown below:EQUATIONwhere ẑi is derived from zi adding a perturbation. With an iteratively denoising process, we can finally obtain the denoised video latent z 0 which can be decoded as a complete video.Video generation conditioned on classes. When the number of visible frames is extremely limited, the motion trajectory becomes unpredictable and thus it would be hard to make a reliable prediction of the future. To mitigate this issue, GenRec supports adding category information to guide video generation in the expected desired direction.Formally, we simplify Equation (11) with Equation (4), and obtain:EQUATIONWe substitute the score function ∇ ẑi log p θ (ẑ i ) with the conditional form ∇ ẑi log p θ (ẑ i |y), in which y denotes the conditional class. By applying Bayes' Theorem, the original score function can be replaced by p(ẑ i )p(y|ẑ i ), and we can get the conditional version of residual, denoted asEQUATIONEQUATIONConsidering the scaling factor of ẑi :c in (σ) = 1 √ σ 2 +σ data(following [22] , and σ i = t i ), that would pre-scale the input as c(z i ) = c in (t i ) • z i before model processing, the formulation can be further transferred as:EQUATIONFollowing [11] , we sharpen the distribution of p(y|z) by multiplying a scaling factor s > 1, shown as s • ∇ z log p(y|z) = ∇ z log 1 Z p(y|z) s where Z is an arbitrary constant. Larger scaling value would bring more attention to the target category. Here, p(y|c(ẑ i )) comes from the classification branch in GenRec. Finally, we can use the same EDM sampling procedure with the derived class information to generate samples.Standard video recognition. Based on Equation ( 6), GenRec can do the classical video recognition by setting constant no-mask, and thus z 0 is replaced by z 0 and the prediction follows:EQUATION)Video recognition with partially observed frames. Based on Equation ( 6), GenRec can be applied to video recognition with partially observed frames, e.g., early action prediction that aims to predict future events based on the initial frames, sparse video recognition where videos are sparsely encoded and transmitted due to bandwidth limitations. By masking the invisible frames to get zi , and replacing the noisy latent with random noise ∼ obeying Gaussian distribution, GenRec can do the prediction for partially visible videos, following:EQUATION4 Experiments 
experimental setup: 
 Datasets. In our experiments, we use the following four datasets: Something-Something V2 (SSV2) [17] , Kinetics-400 (K400) [24] , UCF-101 [35] and Epic-Kitchen-100 (EK-100) [10] . SSV2 dataset is designed for fine-grained action recognition and it contains 174 action classes, 220,847 short video clips with an average duration of 4 seconds. K400 contains 400 action classes, 306,245 video clips with an average duration of 10 seconds. The UCF-101 dataset comprises 13,320 videos from 101 action categories and is widely utilized for human action recognition. The EK-100 dataset focuses on egocentric vision. It contains a total of 90,000 annotated action segments, encompassing 97 verb classes and 300 noun classes.Evaluation protocols. GenRec performs both generation and recognition tasks. For generation, we use the Fréchet Video Distance (FVD) [41] metric to assess the quality of the generated videos. A lower FVD score indicates higher fidelity and realism. For recognition, we measure the top-1 accuracy that reflects the portion of correctly classified videos. We validate our model performance in formal video recognition, partial video recognition, class-conditioned image-to-video generation and frame completion with the above metrics.Implementation details. We initially set the learning rate to 1.0 × 10 -5 and set the total batch size as 32. Only generation loss will be retained for model adaptation on specific datasets. We train 200k steps on EK-100 and UCF, and 300k steps on SSV2 and K400, respectively. Subsequently, we finetune GenRec with both generation and recognition losses. The learning rate is set to 1.25 × 10 -5 and decayed to 2.5 × 10 -7 using a cosine decay scheduler. We warm up models with 5 epochs, during which the learning rate is initially set as 2.5 × 10 -7 and linearly increases to the initial learning rate 1.25 × 10 -5 . The loss balance ratio γ is set to 10, and the learning rate for the classifier head is ten times higher than the base learning rate. We drop out the conditions 10% of the time for supporting classifier-free guidance [20] , and we finetune on K400 for 40 epochs and 30 epochs on other datasets.The training is executed on 8 A100s and each contains a batch of 8 samples. We sample 16 frames for each video.Comparison to state-of-the-art in video recognition and video generation. We compare with state-of-the-art methods in terms of their recognition accuracy and generation quality. The results are summarized in Table 1 . The first two blocks of the table presents current advanced video recognition models, while the third block demonstrates the performance of the diffusion-based class-guided image-to-video generation.As shown in the table, GenRec achieves optimal results or performs on par with the state-of-the-art approaches. In terms of video recognition, GenRec achieves 75.8% accuracy on SSV2 dataset, surpassing the majority of current state-of-the-art methods. On K400, GenRec achieves 87.2% accuracy, which is on par with the performance of MVD-H (87.2%) and Hiera (87.3%, 87.8%), and surpasses other advanced methods. These results indicate the effectiveness of our approach in video recognition. In addition, GenRec shows a slight performance gap compared to the methods in the second block. It is important to note that these advanced methods benefit significantly from pretraining on large-scale multimodal alignment datasets, which provide extensive cross-modal supervision that enhances their ability to capture semantic relationships across video frames.We further construct two strong baselines. Baseline I adapts SVD to the respective dataset through generative fine-tuning, followed by attentive-probing for classification, where the backbone is frozen and all frames are used as input. Baseline II involves fully fine-tuning the original SVD model with classification supervision only, ensuring that all frames are visible during training. Compared with them, GenRec performs on par or better. GenRec performs good in supporting not only classification but also generation, demonstrating its comprehensive capability in handling both tasks effectively.In terms of video generation, we evaluate the model on class-conditioned image-to-video generation task following [18] . Comparing the FVD scores of SEER:112.9 and SEER †:355.4, it can be inferred that generating longer videos with 16 frames is more difficult than generating 12 frames. GenRec generates videos with 16 frames and achieves much lower FVD scores than the other methods, demonstrating the effectiveness of our approach in video generation.It is worth highlighting that, current research always treats video recognition and generation tasks in a separate manner, and most of the advanced methods focus primarily on either recognition or generation tasks. For instance, SEER method excels in class-conditioned image-to-video generation, but lacks the ability to do video recognition. While current research on representation learning, shown as the first and second blocks in Table 1 , lacks the ability to do video generation tasks. In contrast, GenRec not only unifies these tasks, but also achieves competitive results compared to the specialized methods. Comparison to state-of-the-art in video recognition with limited frames. GenRec supports video recognition when only partial frames can be observed. We evaluate this capability on the SSV2 and EK-100 datasets. Our evaluation includes two tasks: an early prediction task, where the model has access only to previous continuous frames following the setting of [38] , and a recognition task where videos are sparsely sampled, and the model is expected to make correct predictions. For fair comparisons, we construct two strong baselines. We first apply MVD [44] to directly deal with the recognition task by constructing a dense video through nearest neighbor interpolation. We also construct another baseline similar to our training pipeline, where we apply frame dropout in the training process of MVD [44] for better fitting on task with partial frames, and is named as MVD †. In all settings, the number of fully observed frames is 16.Table 2 shows the results under these settings, in which ρ denotes the visible ratio (there are a total of 16 frames). In the early action prediction task, GenRec achieves the highest accuracy and ratio metrics at all observation levels. Notably, GenRec and MVD † exhibit similar performance when all frames are observed, but as the number of observed frames decreases, GenRec demonstrates higher accuracy. GenRec also shows superior performance when videos are sparsely sampled, maintaining high accuracy even with fewer observed frames (e.g., 55.7% for 2 frames and 70.8% for 4 frames), indicating its robustness in handling sparse data. Moreover, we compute the ratio metric representing the percentage of maximum performance that the model can maintain at various frame rates, mitigating the unfairness caused by different backbone networks. In this scenario, GenRec still achieves the best performance.We further investigate the contributions of generation supervision for recognition. As seen in Table 2 , removing generation supervision results in noticeable performance degradation across various tasks, especially when the number of visible frames get less. For example, in the early prediction task, the accuracy decreases by 0.3% at ρ = 1.0 and by 2.3% at ρ = 0.3. These results suggest that generation supervision is essential for maintaining high performance, particularly when the model has to make We also evaluate the early action prediction on EK-100 and UCF-101. EK-100 is a temporally sensitive dataset similar to SSV2, demanding in terms of the model's temporal modeling capability, while UCF-101 demands more on appearance modeling. We conduct early prediction evaluation on them to further reveal the robustness of our GenRec. As shown in Figure 3 , GenRec clearly outperforms TemPr and MVD †. In particular, the improvement becomes more significant as the number of observed frames decreases. More evaluation results can be seen in Appendix A.1.These results collectively demonstrate that GenRec effectively handles missing video frames. The robustness and high accuracy of GenRec across different datasets and observation ratios highlight its potential for real-world applications where video data might be incomplete or sparsely sampled.The relationships between generation and recognition . We further investigate the consistency between video generation and recognition, as shown in Table 3 . We evaluate the performance of video recognition and generation with limited frames and find that the recognition accuracy not only depends on the number of visible frames but also significantly on the location of these frames. Interestingly, uniform sampling appears to facilitate video recognition better than dense sampling from the video prefix. Specifically, with the same number of frames, early prediction consistently shows lower accuracy compared to uniformly sampled frames (e.g., 28.9% vs. 55.7% with 2 frames) and worse FVD scores (e.g., 57.8 vs. 46.7 with 2 frames). When only three interpolated frames are visible, the 31.7 FVD score is comparable to that of an eight-frame prefix (30.3), while achieving much higher recognition accuracy. These results highlight the importance of complete state observation for action recognition and also suggest that video generation performance can potentially reflect task difficulty.Choice of UNet layers. As described in Section 3, the UNet mapping function F is decoupled into F tail • F head , where F head serves as the feature extractor for video recognition. Our UNet model contains 4 main up-sampling blocks. We investigate which one is best suited for recognition. As shown in Table 4 , using the second up-sampling block (Up Index 2) yields the best performance with an accuracy of 75.8%. The third block (Up Index 3) followed with 75.2%, while the first block (Up Index 1) has the lowest accuracy. As such, we choose the second block for feature extraction.Explore the influence of the masking strategy. We also conduct an ablation study on the masking schemes using different expected masking ratios, as shown in Table 5 . The results show that the FVD scores remain similar across different ratios, and a larger masking ratio might be beneficial for generation, as it closely resembles our class-conditioned frame prediction scenario with one or two given frames. However, an excessively large masking ratio (87.5%) negatively impacts action recognition accuracy, leading to a 0.9% decrease compared to our selected ratio.Noise incorporation during inference for video recognition. In the inference stage for action recognition, GenRec applies a specific level of noise to video inputs before extracting visual features, as formulated in Equation 18. This added noise helps maintain consistency with the noisy training Diffusion models for visual understanding. Recently, researchers start to uncover the significance of diffusion models for discrimination tasks. A notable approach involves utilizing pretrained visual diffusion models for various downstream tasks, such as image segmentation [55] and visual content correspondence [39] . Additionally, some studies treat diffusion learning as a self-supervised method to acquire valuable feature representations [8] . However, most current works either use stable diffusion networks as pretrained backbones for downstream tasks or completely destroy their generative capabilities. Consequently, the potential benefits of integrating generation and classification abilities into a single model remain under-explored, which is the primary focus of our paper. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We will release the code in the feature.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this work, we presented GenRec, a unified video diffusion model that enables joint optimization for both video generation and recognition. GenRec exploits the significant temporal modeling power embedded in the diffusion model, allowing for mutual reinforcement between generation and recognition tasks. Extensive experiments were conducted to evaluate the performance of GenRec, demonstrate our approach contains strong generation and recognition capabilities at the same time in different kinds of scenarios, including normal or partial video recognition, video completion and class-conditioned image-to-video generation. Our findings highlight the potential of combining generation and classification tasks within a single unified model, providing valuable insights into the development of more sophisticated and versatile video analysis models. Future work will focus on further refining this integration and exploring its applications across various real-world scenarios.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The training and test details have been reported in this paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The cost of training required to report error bars is excessively high.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper did not release data or models for the main contribution. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: This paper follows the CC-BY 4.0 license in experiments.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper did not introduce new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No crowdsourcing experiments in this paper.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Early Prediction on EK-100 and UCF-101More detailed evaluation results of the video recognition with limited frames on EK-100 and UCF-101 can be seen here. We conduct an ablation study on the loss balance ratio λ as shown in Equation 9. Results presented in Table 10 show that varying the loss ratio has a minor impact on action recognition accuracy. However, setting the ratio too low negatively affects the generation performance. In particular, when the ratio is set to zero, significant forgetting occurs, severely compromising the model's generation ability. Comparing with the open-source Stable Video Diffusion directly is meaningful. We conduct frame prediction tests using SVD model on the SSV2 and Epic-Kitchen datasets, as shown in Table 11 . Since the SVD model performs better at higher resolutions, we generated videos at 512x512 resolution and then downsampled them to 256x256 for FVD calculations. The results show that while SVD achieves competitive scores compared to previous state-of-the-art methods shown in Table 1 , it is suboptimal compared to GenRec. This is likely due to SVD's design for general scenarios. Baseline I represents our enhanced SVD baseline, which fine-tunes SVD on the target datasets for better results and fairer comparison with GenRec. To demonstrate classification and generative can coexist well in our training without negatively impacting each other, we construct the comparison with "Baseline I (SVD baseline)", which adapts SVD on the target dataset without classification supervision, and then trains another classifier head freeze the generation backbone, ensuring no influence from classification supervision. We conduct comparisons on SSv2: FP (frame prediction) and CFP (class-condition frame prediction), as shown in Table 12 . By comparing the FP results, the similar scores between the two methods conclude that classification loss does not degrade the model's generative ability in our approach. Moreover, the CFP results indicate that our method, with its more accurate classification performance, can guide the model to achieve higher-quality video frame predictions. We show the generated visualization of the GenRec. The model can support video generation given various numbers of frames, as well as category-guided generation. We show two of the most difficult generative scenarios, which are: (1) given the first frame and different action categories to guide the video generation, and (2) given the start and end frames, the model is expected to complement the video. We also compare our methods with SEER [18] with cases picked from its official website. The generation results can be seen in Figure 4 , Figure 5 and Figure 6 . Limitations and Future Work The objective of our paper is to unify the tasks of generation and recognition, achieving or even surpassing the state-of-the-art experimental performance across various tasks. However, our method is based on fine-tuning a pretrained video diffusion model, using more pretraining data and having a larger number of parameters compared to previous methods. This is an issue we need to address in the future, and exploring the distillation of a well-pretrained video diffusion model into a smaller model is a worthwhile future endeavor.Broader Impacts The broader impact of the GenRec framework extends into various fields, enhancing capabilities in content creation, security, and accessibility. In the media industry, it allows for the automated generation of tailored, high-quality videos, reducing production costs and fostering creativity. For surveillance, its robustness in limited information scenarios improves monitoring effectiveness, particularly in challenging environments. Additionally, advancements of GenRec in video prediction can aid in developing assistive technologies, making digital content more accessible and interactive, particularly for individuals with visual impairments. We compare our methods with SEER [18] in the setting of generating videos given the first frame together with the classifier guidance. Cases are picked from the official website of SEER [18] .• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: This paper garnered overall positive reviews, with consensus on the merit of the proposed model that integrates video generation and recognition. The authors adeptly addressed concerns about novelty, experimental design, and efficiency. Given its innovative nature, compelling results, and satisfactory responses to feedback, I recommend the paper for publication.  It's a significant contribution that paves the way for future unified video generation and recognition research.

Reviews: review: paper_summary: paper_summary: This work proposes a video diffusion model to unifies the video generation and video recognition tasks. The proposed GenRec, during training, jointly optimize the generation objective and the classification objective. During inference, GenRec can deal with both the video generation conditioned on frames or classes, and video classification task. The experiments demonstrate that GenRec shows superior performance in both tasks.
strength_weakness: strength_weakness: - Considering that previous works [8,37,46] have already demonstrated that diffusion models have the capability of extracting sufficient features for understanding tasks, the combination of generation and recognition tasks to boost both tasks is too straight-forward, limiting the novelty of the paper.
- It is unclear for the inference stage for video recognition. Like why the noised $\tilde{z}$ is still incorporated for classification, which may lead to the recognition result undeterministic. During inference, is the multi-step denoising procedure sill required?
- There is the lack of training/inference time cost comparison with previous video classification methods.
questions: questions: Pleaser refer to the weakness part. I would like to modify my rating according to the responses from the authors.
limitations: limitations: The authors have already claimed the limitations.
paper_summary: paper_summary: Recently, diffusion models have been released that can generate high quality video clips, indicating that they strongly capture natural appearance and motion in video. This paper seeks to adapt one recent model, Stable Video Diffusion (SVD), to explore if what it has learned can be leveraged to improve video recognition performance, and investigate whether co-training on generation and recognition using this model can improve on both tasks.

The two tasks, video diffusion and recognition, have quite different setups: the first takes uncorrupted video as input, while the second starts with a single conditioning frame as well as noise-corrupted video. This paper proposes to unify the tasks, adapting the SVD model to take masked-uncorrupted video plus noised video, concatenated, as input. This unifies the tasks, allowing joint training on recognition and generation. It also permits new tasks: generation and recognition using a variety of partially-masked video configurations as input.

The paper shows competitive results on classic video clip classification tasks, and very strong results on generation FVD. Further they demonstrate strong classification performance on masked input scenarios including early action prediction, and recognition with limited frame-rates.
strength_weakness: strength_weakness: In the recognition results, Table 1 doesn't represent the current SOTA. E.g. InternVideo (https://arxiv.org/abs/2212.03191) from 2022 claims 77.2% on SSv2 and 91.1% on K400. It's fine if GenRec doesn't beat SOTA, but SOTA should be accurately represented in the table.

The generative experimental results show a surprising improvement on FVD on SSv2 and EK-100, which I don't feel the authors do enough to explain or analyze.  (See an associated "Question" below.)

There is no exploration of how different values of the generative/discriminative loss balance ratio, gamma, affects performance on the benchmarks.

Similarly, there is no exploration of different latent masking schemes or ratios during training.

Minor weaknesses:
* The method is described as "analogous to a temporal extension of MAE" in line 53. VideoMAE [38] does already exist. 
* On line 61, please define the accuracy retention ratio of include a citation.
questions: questions: The generative results in Table 1 demonstrate a really surprising decrease in FVD compared to the other methods. Can you explain why this is the case in more detail? Is the model really that good? Is it a typo? Is it because this benchmark is not particularly competitive? (I think FVD is more often presented on UCF and Kinetics-400 in the papers I've read, e.g. https://arxiv.org/pdf/2212.05199). If you double-check the FVD result, and convince me, then I'll gladly list this surprising decrease as one of the paper's strengths. 

Would it be useful to clarify to readers why the focus is "particularly the unconditioned or image-conditioned models" in line 29?
limitations: limitations: Yes
paper_summary: paper_summary: This paper investigates to unify the video generation and recognition tasks into the same diffusion model. The authors propose a conditional feature mask mechanism to unify the two learning tasks. Experiments are conducted on several tasks to validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The design of unifying generation and recognition tasks is not elegant. This paper directly introduces an extra classification head, and uses a multi-task learning loss for these two tasks. As verified in the Table 4, the trade-off between high-level semantic recognition task and low-level detail synthesis is not controlled well and may be conflict.
2. The experimental results are not promising. Firstly, the model size and computational cost should be reported in Table 1. It will be clear to analyze the efficiency of the recognition model. Secondly, for the generation ability evaluation, since the model is initialized from the SVD, it’s better to compare with SVD on different datasets. A concern is that whether the classification loss training will degrade the model’s synthesis ability on natural scene. It’s over claim to say “surpassing the sota”.
3. The mask mechanism in the proposed method shares some similarity with the paper [1]. The authors are suggested to add a discussion with this work.
4. For the recognition task, the proposed method just uses the SVD backbone to extract features, since z0 is directly used. Such a design (i.e., extracting features from the input video and sending them to a classifier)  is conventional.


[1] Masked Diffusion Transformer is a Strong Image Synthesizer, ICCV 2023.
questions: questions: 1. Why the loss weight of the recognition loss in Eq. 8 is not related with the noise level?
2.  How about the training and testing efficiency compared with other recognition models?
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes to unify video recognition and generation with a masked finetuning of SVD (diffusion video model). By training with both classification and generation objectives, the model learns to do both. Rather than training with first frame only for conditioning, the masking is employed so that the classifier can learn better. The resulting model has recognition performance that is actually enhanced by the generative part of the training, and results are given across various tasks and subtasks, including action prediction and generating videos given multi-frame conditioning.
strength_weakness: strength_weakness: [W1] It is unclear why the paper compares with methods that pretrain on only kinetics and imagenet (like Hiera) when GenRec uses the SVD which is pretrained on massive amounts of external data. For the recognition task, appropriate SOTAs would be OmniVec, OmniVec2, InternVideo, InternVideo2, etc. 

[W2] FVD is not a very good metric. A generative paper ought to have generative results in the main paper itself, FVD results alone are not adequate, and such examples ought not be saved for supplementary. Additionally, the examples in the supplementary are not adequate, since there is no comparison to the competing methods.

[W3] Paper does not cite enough related work. For example, [1] and [2] are omitted, as are many others that explore similar diffusion-for-recognition directinos.

[1] Your Diffusion Model is Secretly a Zero-Shot Classifier, Li et al.
[2] Diffusion Models Beat GANs on Image Classification, Mukhopadhyay et al.
questions: questions: How did you select your SOTAs? One of the main things preventing me from giving a higher rating is that the model does considerably lag behind SOTAs for k400 classification. 

Why do you not provide example videos for the competing methods, as reference points? It is difficult to assess the capabilities of the video generation based on FVD alone.

Why is there no generative baseline, similar to the classification baselines?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper explores the use of video diffusion models for joint generation and recognition. The authors introduce a framework called GenRec, based on Stable Video Diffusion, that aims to unify video generation and recognition. GenRec employs a random-frame conditioning process to learn generalized spatial-temporal representations, making it robust even with limited visual inputs. Experiments show that GenRec excels in both video recognition and generation. Additionally, GenRec proves to be highly robust in scenarios with limited frame availability.
strength_weakness: strength_weakness: 1. The notations used in the Preliminary Section are inconsistent. The authors should review and correct the formulations for clarity.

2. In Section 3.3, the authors claim to have adopted the stochastic sampler in EDM, but it appears that the second-order correction is not included according to Algorithm 2 in EDM. The authors are encouraged to provide justifications for this omission.

3. In the derivation on Line 156, it is unclear why such a replacement can be used for the original score function. Additionally, the derivation from Equations 14 to 16 lacks clarity. The authors should provide more detailed explanations.

4. In Table 1, the generative performance of Baseline I should be included. Furthermore, information on parameter counts and computational overhead should also be presented.

5. There are misspellings such as "mensioned" on Line 135 and "interatively" on Line 147. These should be corrected.
questions: questions: Can you provide more details on the derivation of Eq.14-16?
limitations: limitations: The authors have mentioned the limitations.

Comments: title: Post-rebuttal comments
One more question
comment: I have carefully read the responses from the authors and other reviewers' comments. The response has addressed my concerns, I would like to raise my rating to 5.
Dear Reviewer JBof,

Hope this message finds you well. We wanted to kindly follow up on the feedback for our submission. Your insights are invaluable, and we would greatly appreciate any further comments you may have.

Thank you for your time and consideration.
Dear Reviewer caZh,

Hope this message finds you well. We wanted to kindly follow up on the feedback for our submission. Your insights are invaluable, and we would greatly appreciate any further comments you may have.

Thank you for your time and consideration.
Thank you for reviewing our work and providing valuable feedback. We are pleased that we have been able to address your concerns. 

- For the Eq. 3 and Eq. 4 in the preliminary section of our paper, the equations are from Eq. 2 and Eq. 3 in the EDM paper.

- For the Eq. 3 to Eq. 4 in our rebuttal, here is a detailed explanation:

Eq. 17 in our paper outlines the calculation method for the class-conditioned denoising residual $\epsilon^{*}_{\theta}(\hat{\mathbf{z}}_i;\overline{\mathbf{z}}_0)$. We can apply this iteratively as follows:

$$
z_{i-1} = \hat{z_i} + \epsilon_{\theta}^{*}( \hat{z_i};\overline{z_0} )
$$

$$
z_{i-2}= {z_{i-1}}  + \epsilon_{\theta}^{*}(z_{i-1};\overline{z_0})
$$

Following line 8 in Algorithm 2 of EDM, we can compute the terms:

$$
    d_{i} = \frac{{z_{i-1}} - \hat{z_{i}}}{t_{i-1}-\hat{t_{i}}} = \frac{ \epsilon_{\theta}^{*}( \hat{z_i};\overline{z_0})  }{t_{i-1}-\hat{t_{i}}}
$$

$$
\begin{align} 
    d_{i}^{'}  = \frac{z_{i-2}-z_{i-1}}{t_{i-2}-t_{i-1}} = \frac{\epsilon_{\theta}^{\*}(z_{i-1};\overline{z_0}) }{t_{i-2}-t_{i-1}}
=\frac{\epsilon_{\theta} ^{\*} (\hat{z_i} + {\epsilon_{\theta}^{\*} (\hat{z_i};\overline{z_0})};\overline{z_0})  }{t_{i-2}-t_{i-1}} 
\end{align}
$$

Using the second-order correction, and by substituting these two terms into line 11 of Algorithm 2 in EDM, we obtain:

$$
\begin{align} 
z_{i-1}^{corr} & = z_i + \frac{1}{2}{(t_{i-1} - \hat{t_i})}(d_i + d_{i}^{'}) = z_i + \frac{1}{2} {\epsilon_{\theta}^{\*} (\hat{z_i};\overline{z_0})} + \frac{1}{2}\frac{t_{i-1} - \hat{t_i}}{t_{i-2}-t_{i-1}} \epsilon_{\theta} ^{\*} (\hat{z_i} + {\epsilon_{\theta}^{*} (\hat{z_i};\overline{z_0})};\overline{z_0}) 
\end{align}
$$

Thank you again for your positive recognition and thoughtful feedback. We greatly appreciate the time and effort you have taken to review our work.
Many thanks for the authors' response. I've checked the authors' rebuttal and other reviewers' comments. I think the response has addressed my concerns. Therefore, I decide to raise the score to 5. Moreover, is it possible for the authors to provide the derivation from Eq. 3 to Eq. 4? The derivation is a bit confusing to me.
Dear Reviewers,

We are nearing the end of the period for discussion with the authors. I kindly request you to review the comments made by other reviewers, as well as the responses from the authors.

Given that we have a mixed and divergent score (9, 7, 4, 4, 3), it's crucial that we have a comprehensive understanding of each other's viewpoints to make a fair and informed decision on the paper.

In particular, I would like to request Reviewers caZh, JBof, and C3Pd to actively participate in the discussion with the authors. Your insights and perspectives are highly valued and can significantly contribute to the decision-making process.

Thank you for your timely attention to this matter.

Best regards,
Area Chair
Thank you so much. We apologize for the naming mistake in Table 1 of the rebuttal which should be "GenRec".
 
Thank you again for your recognition and thoughtful feedback. We greatly appreciate your time and effort in reviewing our work.
With these clarifications, I am inclined to increase my rating.

One quick question- what is OmniViD in table 1 of the rebuttal?
Thank you for thoroughly reviewing our work and responses. We appreciate your acknowledgment of the improvements, particularly with the inclusion of SOTA methods, parameter counts, and timing details. We’re glad you find our work interesting, and we share your enthusiasm about its potential implications.

Regarding the SVD comparison, we agree that comparing with SVD directly provides a meaningful baseline. Thus, we have newly added such comparison. We conducted frame prediction tests using the SVD model on the SSV2 and Epic-Kitchen datasets. Since the SVD model performs better at higher resolutions, we generated videos at 512x512 resolution and then downsampled them to 256x256 for FVD calculations.

The results show that while SVD achieves competitive scores compared to previous state-of-the-art methods, it is suboptimal compared to GenRec. This is likely due to SVD's design for general scenarios. Baseline I in Table 1 represents our enhanced SVD baseline, which fine-tunes SVD on the target datasets for better results and more a fairer comparison. 

Thank you again for your recognition and thoughtful feedback. We greatly appreciate your time and effort in reviewing our work. We will include the SVD baseline results in our revised version.

|Method| SSv2 FVD | Epic FVD |
|:--:|:--:|:--:|
| **SVD** | **99.7** | **180.8** |
| Baseline I | 50.3 | 53.6 |
| GenRec | 46.5 | 49.3 |
||
I have read through the other reviews, and all of the authors' responses, and believe that the majority of the concerns have been addressed. In particular, the inclusion of sota methods, parameter counts, and timing for training and test, significantly improves Table 1. Again, this paper remains very interesting to me even if it doesn't beat sota, in particular because it shows "mutual enhancement" on the two tasks, which is a much-sought-after result.

Thank you for double-checking the FVD computation. Your explanation makes sense. However, reviewer JBof suggests "for the generation ability evaluation, since the model is initialized from the SVD, it’s better to compare with SVD on different datasets". Should different baselines which are trained on significant video data be used for comparison instead?