Title: CV-VAE: A Compatible Video VAE for Latent Generative Video Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Variational Autoencoder

Video Generation

Image Generation
Abstract: Spatio-temporal compression of videos, utilizing networks such as Variational Autoencoders (VAE), plays a crucial role in OpenAI's SORA and numerous other video generative models. For instance, many LLM-like video models learn the distribution of discrete tokens derived from 3D VAEs within the VQVAE framework, while most diffusion-based video models capture the distribution of continuous latent extracted by 2D VAEs without quantization. The temporal compression is simply realized by uniform frame sampling which results in unsmooth motion between consecutive frames. Currently, there lacks of a commonly used continuous video (3D) VAE for latent diffusion-based video models in the research community. Moreover, since current diffusion-based approaches are often implemented using pre-trained text-to-image (T2I) models, directly training a video VAE without considering the compatibility with existing T2I models will result in a latent space gap between them, which will take huge computational resources for training to bridge the gap even with the T2I models as initialization. To address this issue, we propose a method for training a video VAE of latent video models, namely CV-VAE, whose latent space is compatible with that of a given image VAE, e.g., image VAE of Stable Diffusion (SD). The compatibility is achieved by the proposed novel latent space regularization, which involves formulating a regularization loss using the image VAE. Benefiting from the latent space compatibility, video models can be trained seamlessly from pre-trained T2I or video models in a truly spatio-temporally compressed latent space, rather than simply sampling video frames at equal intervals. To improve the training efficiency, we also design a novel architecture for the video VAE. With our CV-VAE, existing video models can generate four times more frames with minimal finetuning. Extensive experiments are conducted to demonstrate the effectiveness of the proposed video VAE.

CV-VAE: A Compatible Video VAE for Latent Generative Video Models

Abstract

Spatio-temporal compression of videos, utilizing networks such as Variational Autoencoders (VAE), plays a crucial role in OpenAI's SORA and numerous other video generative models. For instance, many LLM-like video models learn the distribution of discrete tokens derived from 3D V AEs within the VQVAE framework, while most diffusion-based video models capture the distribution of continuous latent extracted by 2D VAEs without quantization. The temporal compression is simply realized by uniform frame sampling which results in unsmooth motion between consecutive frames. Currently, there lacks of a commonly used continuous video (3D) VAE for latent diffusion-based video models in the research community. Moreover, since current diffusion-based approaches are often implemented using pre-trained text-to-image (T2I) models, directly training a video VAE without considering the compatibility with existing T2I models will result in a latent space gap between them, which will take huge computational resources for training to bridge the gap even with the T2I models as initialization. To address this issue, we propose a method for training a video VAE of latent video models, namely CV-VAE, whose latent space is compatible with that of a given image VAE, e.g., image VAE of Stable Diffusion (SD). The compatibility is achieved by the proposed novel latent space regularization, which involves formulating a regularization loss using the image VAE. Benefiting from the latent space compatibility, video models can be trained seamlessly from pre-trained T2I or video models in a truly spatio-temporally compressed latent space, rather than simply sampling video frames at equal intervals. To improve the training efficiency, we also design a novel architecture for the video V AE. With our CV-V AE, existing video models can generate four times more frames with minimal finetuning. Extensive experiments are conducted to demonstrate the effectiveness of the proposed video V AE.

1 Introduction

Video generation has gained significant public attention, especially after the announcement of OpenAI SORA [2]. Current popular video models can be divided into two categories based on the modeling space, i.e., pixel and latent space. Imagen Video [17], Make-a-video [29], and Show-1 [42] are representative video diffusion models that directly learn the distribution of pixels. On the other hand, Phenaki [33], MAGVIT [41], VideoCrafter [6], AnimateDiff [15], VideoPeot [20], and SORA, etc , are representative latent generative video models that are trained in the latent space formed using variational autoencoders (VAEs). The latter category is more prevalent due to its training efficiency.

将

SVD

(a) Uniform frame sampling

conv

conv

(b) 3D convolution for temporal compression latent constrant

）的

上

Time

(a)

(b)

(c)

Furthermore, latent video generative models can be classified into two groups according to the type of VAE they utilize: LLM-like and diffusion-based video models. LLM-like models train a transformer on discrete tokens extracted by a 3D V AE with a quantizer within the VQ-V AE framework [32]. For example, VideoGPT [40] initially trains a 3D-VQVAE and subsequently an autoregressive transformer in the latent space. The 3D-VQVAE is inflated from the 2D-VQVAE [32] used in image generation. TATS [14] and MAGVIT [41] use 3D-VQGAN for better visual quality by employing discriminators, while Phenaki [33] utilizes a transformer-based encoder and decoder, namely CViViT.

However, recent latent diffusion-based video models typically exploit 2D V AEs, rather than 3D V AEs, to generate continuous latents to train a UNet or DiT [25]. The commonly used 2D V AE is the image VAE [28] from Stable Diffusion, as training a video model from scratch can be quite challenging. Almost all high-performing latent video models are trained with the SD image model [28] as initialization for the inflated UNet or DiT. Examples include Align-your-latent [5], VideoCrafter1 [6], AnimateDiff [15], SVD [4], Modelscope [34], LaVie [35], MagicVideo [44], Latte [24], etc. Temporal compression is simply achieved by uniform frame sampling while ignoring the motion information between frames (see Fig. 1). Consequently, the trained video models may not fully understand smooth motion, even when FPS is set as a condition. When projecting a sampled latent sequence to a video using the decoder of the 2D VAE, the generated video exhibits a low FPS and lacks visual smoothness.

Currently, the research community lacks a commonly used 3D video VAE for generating continuous latent variables with spatio-temporal compression for latent video models. Training a high-quality video VAE without considering the compatibility with existing pretrained image and video models might not be too difficult. However, even though the trained video V AE exhibits low reconstruction errors, a gap exists between its learned latent space and the one used by pretrained models, such as the video VAE of Open-Sora-Plan [1]. This means that bridging the gap requires significant computational resources and extensive training time, even when using pre-trained models as initialization. One example is shown in Fig. 2. When training a video VAE independently without considering compatibility, the sampled latent of SVD [4] cannot be projected into the pixel space correctly due to the latent space gap, as shown in Fig. 2(a). After finetuning the SVD model in the new latent space on 16 A100 for 58K iterations, the quality of the generated video is still poor (see Fig. 2(b)). In contrast, our video VAE achieves promising results in the pretrained SVD even without finetuning the UNet as shown in Fig. 2(c).

In this work, we propose a novel method to train a video VAE to extract continuous latents for generative video models, which is compatible with existing pretrained image and video models, e.g. Stable Diffusion [28] and SVD [4]. We also inflate the SD image VAE to form a video VAE by adding 3D convolutions to both encoder and decoder of the 2D VAE, which allows us to train video models efficiently with the pretrained models as initialization in a truly spatio-temporally compressed

latent space, instead of uniform frame sampling for temporal compression (see Fig. 1). Consequently, the generated videos will be smoother and have a higher FPS than those produced using a 2D VAE.

To ensure latent space compatibility between 2D and 3D VAEs, we propose a latent space regularization to avoid distribution shifts. We examine the effectiveness of using either the encoder or decoder of the 2D VAE to form constraints and explore four types of mapping functions to design regularization. Moreover, to improve video VAE efficiency, we investigate its architecture and partially integrate 3D convolutions instead of exploiting 3D convolution in all blocks. The proposed video VAE can be used not only for training new video models with pretrained ones as initialization but also as a frame interpolator for existing video models with slight finetuning.

Our main contributions are summarized as follows: (1) We propose a video VAE that provides a truly spatio-temporally compressed continuous space for training latent generative video models, which is compatible with existing image and video models, greatly reducing the expense of training or finetuning video models. (2) We propose a latent space regularization to avoid distribution shifts and design an efficient architecture for the video V AE. (3) Extensive experiments are conducted to demonstrate the effectiveness of the proposed video V AE.

2 Related Work

Variational Autoencoder. Variational Autoencoders (VAEs), introduced by [19], have been widely used in two-stage generative models. The first stage involves compressing the pixels into a lowerdimensional latent representation, followed by a second stage that generates pixels from this latent space. VAEs can be divided into two groups according to the token, i.e., discrete and continuous latent. The difference between the two types of VAEs is the quantization. Continuous VAEs have no quantization, while discrete VAEs learn a codebook for quantization and use it to convert the continuous latent features to discrete indices, called VQV AE [32]. When training discrete VAEs, some methods exploit a discriminator to improve the visual image quality, called VQGAN [12].

In video generation, 2D VAEs are typically inflated into 3D ones by injecting 3D Conv or temporal attention. 3D Convs are for CNN-based VAEs, e.g., 3D-VQVAE [40], 3D-VAQGAN [14, 41]. Attentions are for transformer-based V AEs, e.g., CViViT [33]. Although there are several discrete 3D VAEs for video generation, there are no commonly used continuous 3D VAEs.

Video Generative models. Video generation has achieved remarkable progress in recent years. The announcement of Imagen Video [17] and Make-A-Video [29] made researchers see the hope of purely AI-generated videos. Then, the launch of OpenAI SORA [2] brought the enthusiasm of researchers in academia and industry to a climax. Many video generation models [17, 29, 42] directly learn the distribution of pixels while some others [5, 6, 44, 34, 35, 24, 41, 14, 33, 40, 4, 15] learn the distribution of tokens in a latent space. The tokens are always extracted by a variational autoencoder [11]. Latent video generation models can be categorized into two groups according to whether the token is discrete or continuous. TATS [14], MAGVIT [41], VideoGPT [40], and Phenaki [33] are representative models trained with discrete tokens extracted by a 3D V AE within the VQVAE framework [32]. A codebook is learned jointly with the VAE for quantization. SVD [4], AnimateDiff [15], VideoCrafter [6], etc. , are video models trained with continuous latent extracted by a 2D VAE without quantization, rather than a 3D VAE. SD image VAE is the commonly used 2D VAE. One reason is that video models are difficult to train from scratch and they are always initialized with the weights of a pretrained T2I model such as Stable Diffusion UNet [28]. Hence, the corresponding image VAE is used to extract latents from a video. Since the image V AE can only perform spatial compression, the temporal compression is realized by uniform frame sampling. This strategy ignores the motion between key frames.

There lacks a video VAE that is compatible with the pretrained T2I or video models. Though it is not difficult to train a video V AE (3D V AE) independently with high reconstruction accuracy; it will result in a latent space gap between the learned video V AE and existing pre-trained image and video models that are always used as initialization. The Open-Sora-Plan project [1] offers a video V AE; however, it is not compatible with existing image or video models. Large computational resources and a long training time are required to bridge the gap. In this work, we propose a latent space regularization method to train a video V AE whose latent space is compatible with pretrained models.

3 Method

3D

Encoder

3D

Decoder

Pretrained

2D Decoder

Reconstruction Loss

Reconstruction

Loss

GAN Loss

Mapping

3D

Discriminator

Input Video

Latent

Reconstructed Video

3D

Encoder

3D

Decoder

Pretrained

2D Encoder

3D

Encoder

3D

Decoder

Pretrained

2D Decoder

Text

(a) Encoder Regularization

(b) Decoder Regularization

(c)  Framework of CV-VAE with Decoder Regularization

w(X)

文

f

f

r

r

f

f

r

f

r

r

f

I

I

I

I

I

I

1

I

I

I

We propose a latent space regularization method for training a video V AE that is compatible with pretrained image and video models. We examine multiple strategies for implementing the regularization, focusing on either the encoder or the decoder of the image V AE. Additionally, we explore four types of mapping functions to develop the regularization loss. To enhance the efficiency of the video V AE, we introduce an architecture that employs different inflation strategies in distinct blocks, instead of incorporating 3D convolutions in all blocks.

3.1 Latent Space Regularization

We inflate a 2D VAE into a 3D VAE, initializing it with the 2D VAE's weights. The 3D VAE is designed to be capable of encoding both image and video (see details in Sec.3.2). The key of building a compatible video VAE is the latent space alignment between the video VAE and the image VAE.

Notations. Let x ∈ R H × W × 3 denote an image in RGB space and X ∈ R ( T +1) × H × W × 3 denote a video with T +1 frames. When T = 0 , X degrades into an image and the video VAE will process it with temporal padding. z ∈ R h × w × c denotes the latent tokens extracted by either an image V AE or a video VAE. Z ∈ R ( t +1) × h × w × c is latent tokens extracted by the video V AE. ρ s = H/h = W/w and ρ t = T/t are the spatial and temporal compression rates. Let E i and D i denote the encoder and decoder of the image VAE, respectively. While E v and D v are for the video V AE. Then, we have z = E i ( x ) , Z = E v ( X ) , and z = E v ( x ) . ˜ x = D i ( z ) = D i ( E i ( x )) , ˜ X = D v ( Z ) = D v ( E v ( X )) , and ˜ x = D v ( z ) = D v ( E v ( x )) are the reconstructed image and video from the latent tokens.

Regularization. We assume the latent of the image VAE follow a distribution, i.e, z ∼ p i ( z ) . The joint distribution of t +1 independent frames is p i ( Z ) = ∏ t +1 k p i ( z k ) . The latent distribution of the video VAE can be denoted as Z ∼ p v ( Z ) . To achieve the alignment between the latent spaces of the image and video VAEs, we have to build mappings between p i ( Z ) and p v ( Z ) . Since both distributions have no analytic formulation, distance metric for measuring differences between distributions is not applicable.

Here, we build the cooperation between the image VAE and the video one to construct reconstruction loss for space alignment. When exploiting the encoder of the image VAE for alignment, the latent extracted from the image encoder should be corrected decoded by the decoder of the video VAE, i.e., ˜ X v i = D v ( E i ( ψ ( X ))) . The illustration is shown in Fig. 3(a). For a given input video X ∈ R ( T +1) × H × W × 3 , we use a mapping function ψ to sample ψ ( X ) ∈ R ( T/ρ t +1) × H × W × 3 . Thus the reconstructed video ˜ X v i is the same as the shape of X . Then, the reconstruction loss of using the image encoder can be defined as

When exploiting the decoder of the image VAE, the latent extracted by the video encoder can be decoded by the decoder of the image VAE, i.e., ˜ X i v = D i ( E v ( X )) . The illustration is shown in Fig. 3(b). For a given input video X ∈ R ( T +1) × H × W × 3 , the reconstructed video is ˜ X v i ∈ R ( T/ρ t +1) × H × W × 3 . Then, the reconstruction loss of using the image decoder can be defined as

Mapping Functions. To bridge the dimension gap between ˜ X i v or ˜ X v i and X , we investigate four types of mapping functions ψ as follows. 1) First frame. We compare only the first frame of the input video and the reconstructed one. The regularization loss degenerates to measure the difference between the input and reconstruction of the image. 2) Slice. ψ samples one frame every ρ t frames to form a shorter video. It starts from the second frame and the first one is reserved. 3) Average. ψ computes the average of every ρ t frames, starting from the second frame. 4) Random. ψ randomly samples one frame from every ρ t frames, starting from the second frame.

Training Objective. Following the training of the 2D VAE in LDM [28], our basic objective is a combination of a reconstruction loss [43], an adversarial loss [12], and a KL regularization [19], i.e.,

where the first term is the reconstruction loss, the second and third are the adversarial loss, and the last is the KL regularization. D v is the discriminator that differentiates original videos from reconstructed ones. It is inflated from the image discriminator in LDM by injecting 3D convolutions. Then, for latent space alignment, our full training objective is:

where λ 1 and λ 2 are trade-off parameters. We explore different settings of λ 1 and λ 2 and find that using the decoder only achieves the best performance. The framework of CV-VAE is shown in Fig. 3(c) and evaluations between different regularization methods can be found in Tab. 6.

3.2 Architecture Design of Video VAE

We design the architecture of the video V AE according to the image V AE in LDM [28]. The detailed architecture is presented in the Appendix A.1. We explain the key modifications as follows.

Model Inflation. Considering the latent space compatibility and the convergence speed of the video VAE, we make full use of the pretrained weights of the image V AE for initialization, instead of training from scratch. We inflate the image V AE into the video V AE by replacing 2D convolutions with 3D convolutions. 3D convolutions are used to model the temporal dynamics among frames. To initialize the 3D convolutions, we copy the weights of the 2D Conv kernel to the corresponding positions in the 3D Conv kernel and set the remaining parameters to zero. We set the stride to achieve temporal downsampling and increase the number of 3D kernels by a factor of s to achieve s × temporal upsampling. To enable the video VAE to handle both image and video, given T +1 frames as input, we use reflection padding in the temporal dimension for the first frame. By initializing the video VAE using the above operations, we can reconstruct images without training, significantly accelerating the training convergence speed on video datasets.

Efficient 3D Architecture. Expanding 2D Convs to 3D Convs (e.g., k × k → k × k × k ) results in k × parameters and computational complexity. To improve the computational efficiency of the model, we adopt a 2D+3D network structure. Specifically, we retain half of the convolutions in the ResBlock as 2D Convs and set the other half as 3D Convs. We find that, compared to setting all Convs to 3D, the number of parameters and the computational complexity are reduced by roughly 30%, while the reconstruction performance remains nearly the same. See Sec. 4.2 for experimental comparisons.

Temporal Tiling for Arbitrary Video Length Existing image VAEs employ spatial tiling on large spatial resolution images to achieve memory-friendly processing, which cannot handle long videos. As a result, we introduce temporal tiling processing. During encoding, the video X is divided into [ X 1 , X 2 , ...X n ] , where X i ∈ R (1+ f · ρ t ) × H × W × 3 and f is a parameter controlling the size of each

̸

block. X i and X i +1 have a one-frame overlap in the temporal dimension. After encoding each X i to obtain Z i , we discard the first frame of Z i when i = 0 and concatenate all Z i in the temporal dimension to obtain Z . The decoding process is handled similarly to the encoding process. By combining our method with 2D tiling, we can encode videos with arbitrary resolution and length.

4 Experiments

4.1 Experimental Setups

Datasets and Metrics. We evaluate our CV-VAE on the COCO2017 [21] validation dataset and the Webvid [3] validation dataset which includes 1024 videos. Both images and videos are resized and cropped to a resolution of 256 × 256 . Each video is sampled with 33 frames and a frame stride of 3. We evaluate the reconstruction performance of CV-VAE on images and videos using metrics such as PSNR, SSIM [36], and LPIPS scores [43]. We employ 3D tiled processing to encode and decode videos with arbitrary resolution and length within a limited memory footprint. During inference, we allow a single video block size of 17 × 576 × 576 . We evaluate the video generation quality of our model using 2048 randomly sampled videos from UCF101 [30] and MSR-VTT [39]. Videos are resized and cropped to a resolution of 576 × 1024 to fit the SVD [4]. We use Frechet Video Distance (FVD) [31], Kernel Video Distance (KVD) [31], and Perceptual Input Conformity (PIC) [38] metrics to evaluate video generation quality. For evaluating image generation quality, we use 2048 samples from the COCO2017 validation dataset and employ FID [16], CLIP score [26], and PIC score metrics.

Training Details. We train our CV-VAE model using image datasets including LAION-COCO [9] and Unsplash [23], as well as the video dataset Webvid-10M [3]. For image datasets, we employ two resolutions, i.e. , 256 × 256 and 512 × 512 . In the case of video datasets, we use two settings of frames and resolutions: 9 × 256 × 256 and 17 × 192 × 192 . The batch sizes for these four settings are 8, 2, 1, and 1, with sampling ratios of 40%, 10%, 25%, and 25%, respectively. We employed the AdamW optimizer [22] with a learning rate of 1e-4 and cosine learning rate decay. To avoid numerical overflow, we trained CV-VAE using float32 precision, and the training was carried out on 16 A100 GPUs for 200K steps. To fine-tune the SVD on CV-VAE, we utilize in-house data with a frame rate and resolution of 97 × 576 × 1024 . We employ deepspeed stage 2 [27], gradient checkpointing [8] techniques, and train with bfloat16 precision. We used a constant learning rate of 1e-5 with the AdamW [22] optimizer, and only optimized the last layer of U-Net. The training was carried out on 16 A100 GPUs for 5K steps.

4.2 Image and Video Reconstruction

We evaluated the reconstruction quality of various VAE models on image and video test sets. The comparison group includes: (1) VAE-SD2.1 [28] which is widely used in the community for image and video generation models. (2) VQGAN [12] which encoding pixels into discrete latents. We use the f8-8192 version for comparision. (3) TATS [14]: a 3D VQGAN designed for video generation. (4) VAE-OSP [1]: a 3D VAE from Open-Sora-Plan which is initialized from VAE-SD2.1 and trained with video data. (5) Our CV-VAE (2D+3D): retains half of the 2D convolutions to reduce computational overhead. (6) Our CV-VAE (3D): utilizes only 3D convolutions.

As illustrated in Tab. 1, we present the parameter count (Params), Frame Compression Ratio (FCR), and compatibility with existing diffusion models (Comp.) for various VAE models. Thanks to the latent constraint, our model is compatible with current diffusion models, compresses videos by 4 × in the temporal dimension, and achieves top-tier image and video reconstruction quality. This enables the generation of longer videos under roughly the same computational resources. Reconstruction

SD2.1 + 2DVAE V.S  3DVAE + 3DVAE

TATS

VAE-OSP

CV-VAE(Ours)

VQGAN

VAE-SD2.1

CV-VAE(Ours)

96

.96

96

SOuth Tyne

SD2.1 + 2DVAE V.S  3DVAE + 3DVAE

quality improves as the number of latent channels increases. For comparison results with 16 latent channels, please refer to Appendx A.2.

We also conducted a qualitative comparison of the reconstruction results for different V AE models, as shown in Fig. 4. In the top row, we reconstructed images with a resolution of 512 × 512 and compared them with Image VAE models. All three models compressed the images to a latent size of 64 × 64 . Our results were close to those of V AE-SD2.1, while VQGAN had the worst performance. In the bottom row, we reconstructed videos with a resolution of 33 × 512 × 512 and compared them with Video VAE models. All three models compressed the videos to a latent size of 9 × 64 × 64 . Comparing the decoded videos at the same frames, our model achieved the best results. Check Appendx A.3 and A.4 for more reconstruction results.

4.3 Compatibility with Existing Models

Text-to-Image Models We tested the compatibility of our CV-VAE by integrating it into the pretrained SD2.1 [28], replacing the original 2D VAE without any finetuning. We evaluated it on

Table 3: Evaluation results of image-to-video generation. FCR denotes the frame compression rate.

the COCO-Val [21] dataset and compared the results with the SD2.1 model using PID, CLIP score, and PIC metrics. The data (see Tab. 2) suggest that both models perform similarly in text-to-image generation.

We also visualized the text-to-image generation results of both models in Fig. 5. In each pair, the left side depicts the results of SD2.1, while the right side shows the results generated by our CV-V AE, which replaced the original VAE, using the same random seed and prompt. The results show that both models generate images with almost identical content and texture, with only slight differences in color. This further validates the feasibility of building a compatible V AE via latent constraint.

Image-to-Video Models The primary objective of CV-VAE is to train a model that can compress both time and space, while also being compatible with the existing 2D VAE. In this section, we validate the compatibility of CV-V AE with existing video generation models. We integrate CV-VAE into SVD [4], replacing the original V AE, and decoded the generated video latents. CV-VAE offers the flexibility to decode either in image mode (CV-V AE-I) or video mode (CV-VAE-V); the former decodes n frames of latent into n frames of video, while the latter decodes n frames of latent into 1 + ( n -1) × 4 frames of video. We tested the video generation quality of both models. Furthermore, we fine-tuning the SVD for better alignment.

As shown in Tab. 3, incorporating 'CV-VAE-I' into a frozen SVD immediately yields video generation quality comparable to the original VAE. Using CV-VAE in video mode can also decode videos generated by SVD, and further improvements in video decoding quality can be achieved by finetuning only the output layer (approximately 12k parameters). One of the reasons for the noticeable gap in test metrics between 'SVD+CV-VAE-I' and 'SVD+CV-VAE-V' is that they use different numbers of frames, making a direct comparison challenging.

In Fig. 6, we also display the comparison results with SVD [4]. The top row shows the generated results by SVD, and the bottom row shows the generated results after inserting CV-V AE into SVD and fine-tuning the output layer. We use the first frame as a condition and generate with the same random seed. The U-Net generates 25 frames of latent, which are decoded by CV-VAE into a 97-frame video. As can be seen, compared to the original SVD, our results exhibit smoother motion. It is worth noting that both models have the same computational complexity during the diffusion process, which means that our model is more scalable.

By fine-tuning a small number of parameters, the image-to-video model can generate smoother and longer videos through CV-VAE, effectively serving as a frame interpolation method. Therefore, we compared CV-VAE with existing video interpolation models [18], conducting experiments on MSR-VTT [39]. We first used SVD to generate a video of size 25 × 576 × 1024 , and then applied the interpolation model to expand the video from 25 frames to 97 frames. For our results, we directly generated a video of size 97 × 576 × 1024 using CV-VAE and the fine-tuned SVD. The comparison results are shown in Tab. 4.3, where CV-V AE outperformed the FIRE [18] in two out of three metric, validating the potential of CV-V AE as an interpolation model.

Text-to-Video Models In this section, we integrate CV-V AE into the existing text-to-video model to futher validate the effectiveness. 'VC2' refers to decoding the results generated by VideoCrafter2 [7] using the original 2D V AE, 'VC2+CV-VAE-I' indicates decoding the results using CV-VAE in image mode, and 'VC2+CV-VAE-V' denotes decoding the results using CV-VAE in video mode, which generates videos of 4 × frames. We only fine-tuned a small number of parameters in U-Net, including

Table 4: Comparison between CV-VAE and frame interpolation model.

the first and last layers. We use captions from the validation set of MSR-VTT [39] for evaluation, with the resolution of 320×512. Following the approach taken by previous studies [37], we used the CLIP [26] metric to evaluate the generation quality of text-to-video models, including Frame Consistency (F.C.) and Textual Alignment (T.A.). The experimental results are shown in Tab. 4.3, where the 'VC2+CV-VAE-V' setting achieved the best generation performance through fine-tuning VideoCrafter2. Check Appendx A.5 for quantitative comparison.

4.4 Ablation Study

Influence of Regularization Type We evaluated the impact of three types of latent regularization, which are: (1) 2D Enc. , i.e. , λ 1 = 0 and λ 2 = 1 in Eq. 3; (2) 2D Dec. , i.e. , λ 1 = 1 and λ 2 = 0 in Eq. 3; (3) 2D Enc. + Dec. , i.e. , λ 1 = 1 and λ 2 = 1 in Eq. 3.

Table 5: Evaluation results of text-to-video generation.

Tab. 6 shows the impact of various latent regularization methods. Using the 2D decoder for latent regularization results in better reconstruction for both image and video test sets compared to the 2D encoder. This is likely because the gradient backpropagation through the 2D decoder provides better guidance for the 3D VAE's learning, while the frozen 2D encoder doesn't propagate gradients. The '2D Enc. + Dec.' method performs slightly better on image test sets but worse on video datasets compared to '2D Enc.' Since our main goal is video reconstruction and for simplicity, we use the 2D decoder for regularization.

Influence of Mapping Functions The 2D decoder decodes n frames of latents into n frames of video, while the 3D decoder decodes the same n frames of latents into 1 + ( n -1) × 4 frames of video. Therefore, we need to mapping the input video to n frames to calculate the regularization loss in Eq. 2. We evaluated four mapping functions mentioned in Sec. 3.1.

As shown in Tab. 7, the four methods have similar effects on image reconstruction, with the main differences being in video reconstruction. The '1st Frame' approach yields the worst video reconstruction results due to the lack of regularization and guidance for subsequent frames. The 'Slice' method results in poor reconstruction quality for the three unsampled middle frames. The 'Average' method is inferior to 'Random' in video reconstruction, primarily because calculating the mean for multiple consecutive frames leads to motion blur in the target.

5 Conclusion and Limitations

We propose a novel method to train a video VAE that is compatible with existing image and video models trained with SD image VAE. The video VAE provides a truly spatio-temporally compressed latent space for latent generative video models, as opposed to uniform frame sampling. Due to the latent space compatibility, a new video model can be trained efficiently with the pretrained image or video models as initialization. Besides, existing video models such as SVD can generate smoother videos with four times more frame using our video VAE by slightly fine-tuning a few parameters. Extensive experiments are performed to demonstrate the effectiveness of the proposed VAE.

Limitations. The performance of the proposed video VAE relies on the channel dimension of the latent space. A higher dimension may yield better reconstruction accuracy. Since we pursue the latent space compatibility with existing image and video models trained with SD image VAE, the channel dimension of our video VAE is limited to be the same as the image VAE. This can be improved if an image VAE with a higher channel dimension becomes available, e.g., the VAE of SD3 [13].

References

Open-sora-plan. Accessed May 15, 2024 [Online]. URL https://github.com/ PKU-YuanGroup/Open-Sora-Plan .

Openai sora. Accessed May 15, 2024 [Online]. URL https://openai.com/index/ video-generation-models-as-world-simulators/ .

M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision , 2021.

A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22563-22575, 2023.

H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q. Chen, X. Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512 , 2023.

H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047 , 2024.

T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.

S. Christoph, K. Andreas, V. Richard, C. Theo, and B. Romain. Laion-coco: 600m synthetic captions from laion2b-en, 2022. URL https://laion.ai/blog/laion-coco/ .

X. Dai, J. Hou, C.-Y. Ma, S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807 , 2023.

C. Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 , 2016.

P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y . Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 , 2024.

S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision , pages 102-118. Springer, 2022.

Y. Guo, C. Yang, A. Rao, Y. Wang, Y. Qiao, D. Lin, and B. Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023.

M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Z. Huang, T. Zhang, W. Heng, B. Shi, and S. Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision , pages 624-642. Springer, 2022.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, R. Hornung, H. Adam, H. Akbari, Y. Alon, V. Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125 , 2023.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

A. Z. Luke Chesser, Timothy Carbone. Unsplash. https://github.com/unsplash/datasets, 2023.

X. Ma, Y. Wang, G. Jia, X. Chen, Z. Liu, Y.-F. Li, C. Chen, and Y. Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 , 2024.

W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis , pages 1-16. IEEE, 2020.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 , 2018.

A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations , 2022.

J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023.

Y. Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y. Wang, C. Yang, Y. He, J. Yu, P. Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103 , 2023.

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7623-7633, 2023.

J. Xing, M. Xia, Y. Zhang, H. Chen, X. Wang, T.-T. Wong, and Y. Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190 , 2023.

J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5288-5296, 2016.

W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 , 2021.

L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y. Hao, I. Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10459-10469, 2023.

D. J. Zhang, J. Z. Wu, J.-W. Liu, R. Zhao, L. Ran, Y. Gu, D. Gao, and M. Z. Shou. Show1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818 , 2023.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 , 2022.

Conv 3x3x3, 3->128

ResBlock 128

Downsample 3x3x3,

128->128, stride 2x2x2

2x

ResBlock 128->256

ResBlock 256

Downsample 3x3x3,

256->256, stride 1x2x2

ResBlock 256->512

ResBlock 512

Downsample 3x3x3,

512->512, stride 2x2x2

ResBlock 512

3x

Attention

ResBlock 512

Conv 3x3x3, 512->8

Conv 3x3x3, 4->512

ResBlock 512

Attention

ResBlock 512

4x

Upsample 3x3x3,

512->1024, stride 1x1x1

ResBlock 512

3x

UPsample 3x3x3,

512->512, stride 1x1x1

ResBlock 512->256

ResBlock 256

2x

Upsample 3x3x3,

256->512, stride 1x1x1

ResBlock 256->128

ResBlock 128

2x

Conv 3x3x3, 128->3

Sample

Conv 3x3x3,

3->64, stride 2x2x2

ResBlock

64->128

ResBlock

128->256

ResBlock

256->512

ResBlock

512->512

Conv 3x3x3, 512->1

Encoder

Decoder

Discriminator

Conv 3x3x3, X->Y

Conv 3x3x3

or 3x3

Conv 1x1x1

X->Y

ResBlock X->Y

Conv 3x3x3

Conv 3x3x3

or 3x3

ResBlock

CV-VAE

A Appendix

A.1 CV-VAE Model Architecture

As illustrated in Fig. 7, we introduce the structure of the CV-V AE. The architecture of CV-V AE is primarily derived from the VAE in Stable Diffusion [28], with several notable differences: (1) Some or all 2D convolutions within the network are transformed into 3D convolutions, while retaining their weights. (2) Temporal downsampling is executed in the encoder through the use of strides. (3) Temporal upsampling is accomplished by increasing the output channel number of 3D convolutions by a specific factor. (4) A discriminator, comprising 3D convolutions, is utilized. The main differences are marked in red text in Fig. 7.

A.2 CV-VAE with more latent channels

More latent channels generally lead to better reconstruction performance in V AEs [13, 10], which is crucial for image and video editing tasks. In this subsection, I conducted experiments using a V AE with more latent channels (z=16). We used the 2D VAE from SD3 [13] as the baseline and trained the CVVAE-SD3 based on latent regularization. Testing was conducted under the same settings as in Tab. 1. The comparison results are shown in Tab. 8. CVVAE-SD3 outperforms VAE-SD3 in image reconstruction but is inferior to V AE-SD3 in video reconstruction. The main reason is that CVVAE-SD3 has a higher compression ratio in video compression, resulting in the loss of more information, though its reconstruction quality is significantly higher than that of V AE with z=4.

A.3 Qualitative Examples of Image Reconstruction

In Fig. 8, we showcase additional image reconstruction results using CV-V AE. we use the version of '2D + 3D'. These images are sourced from the COCO2017 [21] dataset with a resolution of 512 × 512 . The reconstructed image precisely shares the same colors and textures as the original, demonstrating the high fidelity of our CV-VAE in encoding and reconstructing images. Interestingly, in Fig. 5, slight color differences can be observed between the images decoded by the Image V AE and CV-V AE, given the same latent generated by the Image Diffusion Model. This suggests that there is still a minor discrepancy between the latent spaces of the video V AE trained with latent regularization and the Diffusion Model. This gap can be bridged with minimal additional training.

A.4 Qualitative Examples of Video Reconstruction

As shown in Fig. 9, we present the reconstruction results of 4 consecutive frames from a video clip ( 33 × 576 × 1024 ) using CA-VAE. The reconstructed video frames maintain consistency in color, structure, and motion with the ground truth. According to CA-VAE, these continuous frames are condensed into a single latent frame, signifying that even a single latent frame encapsulates motion information.

A.5 Compatibility with Existing Text-to-video Model

We tested the compatibility of CV-VAE with existing text-to-video diffusion models, such as Videocrafter2 [7], which also employs a 2D VAE from the SD as its first-stage model. We adopted a strategy similar to the training of 'SVD + CV-VAE' in Sec. 4.3, by fine-tuning the last layer of the U-Net in VC2 to adapt it to CV-VAE. We finetuned the model using in-house data at a resolution of 61 × 320 × 512 , which is equivalent to a latent size of 16 × 40 × 64 .

As shown in Fig. 10, compared to the original VC2, the 'VC2 + CV-VAE generates videos approximately four times longer, resulting in smoother motion. This further validates the feasibility of obtaining a compatible video VAE through latent regularization, thereby avoiding the massive computational power required to train a video diffusion model from scratch.

B Society Impacts

The CV-VAE can be seamlessly integrated into existing diffusion models, replacing the original 2D VAE for image or video generation, which may result in potential societal implications. While it proves beneficial in fields such as entertainment and advertising, by providing more realistic and immersive content, it also raises ethical and safety concerns. The ease of generating high-quality synthetic images and videos could lead to a surge in the production of harmful or misleading content, such as deepfakes, potentially exacerbating issues of misinformation and privacy invasion. We condemn the misuse of generative AI that harms individuals or spreads misinformation.

Real

Reconstructed

Real

Reconstructed

STOP

STOP

Real

Reconstructed

1st Frame

2nd Frame

3rd Frame

4th Frame

Real

Reconstructed

Real

Reconstructed

Real

Reconstructed

VC2

VC2+CV-VAE

1st Frame

2nd Frame

3rd Frame

4th Frame

5th Frame

VC2

VC2+CV-VAE

VC2

VC2+CV-VAE

VC2

VC2+CV-VAE

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a video V AE for latent generative video models. The Method section shows the technical contribution of the design and training of the V AE. The Experiment part provides comprehensive experiments to demonstrate the claim.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We use a paragraph to discuss the limitations in the manuscript.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The method part and experiment settings show the design of the model as well as the parameters for training.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Code and checkpoints will be released upon the acceptance of this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are in the experiment settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We use the commonly used protocol and metrics for evaluation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details are in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the instruction.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Societal impacts are in the supplementary.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the corresponding papers of the used assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CV-VAE, a 3D VAE that is trained through latent space regularization using an existing two-dimensional VAE decoder. This design facilitates seamless integration with 2D VAE-based diffusion models. The concept, while simple and straightforward, is notably efficacious and offers practical utility. Moreover, relative to conventional 2D VAEs, the proposed framework significantly compresses the latent space dimensionality by integrating temporal downsampling within the 3D VAE architecture.\n\nSTRENGTHS:\n1. The method is novel and delicate, which effectively compresses video latents and maintains latent distribution.  \n\n2. Experiments and visualizations demonstrate that CV-VAE performs equally to SD VAE on image-based/frame-based encoding and falls marginally behind on video encoding with little cost of finetuning SD models.\n\nWEAKNESSES:\n1. As presented in Figure 6, CV-VAE suffers from more severe flickering problems than the original SVD, which may possibly be due to the gap between the latent distribution of CV-VAE and SD-2.1. In the paper, the author also pointed out that text-to-image generation may experience differences in color. Authors may convince readers that such flaws can be overcome through further fine-tuning, e.g., fine-tuning full parameters of SVD, or fine-tuning for longer iterations.\n\n2. Missing details about training: The authors didn’t present details about training batch size, augmentations, video fps, etc. Also, whether the default CV-VAE displayed in the experiment section is trained on video and image data separately or jointly (as vae of Opensora) remains unknown.\n\nQUESTIONS:\n1. Please refer to weakness.\n\n2. In Table 3 and figure 6, authors may also present the metrics of SVD on 97 frames straightly or by using the frame interpolation model of SVD.\n\n3. Authors may specify whether the decoder-based regularization is still employed when training with images as inputs for CV-VAE.\n\n4. Authors may also present metrics comparison between SVD and SVD + CV-VAE on text-to-video generation.\n\n5. Can the method be supported by theoretical proof? For example, can we minimize the distribution distance between video latents of CV-VAE and one of corresponding image latents encoded by 2d VAE?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs a 3D VAE consistent with the 2D VAE, whose output mode can losslessly switch between 2D and 3D while retaining the characteristics of both the 2D VAE and 3D VAE. This allows for obtaining performance similar to the original 2D VAE while increasing the frame count of the 3D output.\n\nSTRENGTHS:\n1. Designing a 3D VAE with 2D latent consistent characteristics is very meaningful, and this paper successfully achieves this.\n\n2. The bidirectional VAE alignment method designed in this paper is very clever. A similar design is found in [1], but the scenario used in this paper is completely different.\n\n3. Achieving effective performance in both 2D and 3D simultaneously is of high practical value to the community, for example, supporting pre-training and frame interpolation in video Diffusion models.\n\n4. The writing is fluent, and the experimental design is excellent.\n\n[1] Christensen, Anders, et al. \"Image-free classifier injection for zero-shot classification.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\nWEAKNESSES:\n1. Figure 10 attempts to demonstrate the effect of the 3D VAE on frame count enhancement. I understand that the conclusion of this figure is actually the most critical value proof of this paper: retaining the 2D VAE latent consistency while having the embedding capability of 3D features, such as extending the frame count. This is quite challenging, as the 3D VAE might tend to completely collapse to perform similarly to the original 2D VAE, simply replicating the 2D results. The loss between replicated 2D frames is very small, making optimization difficult. While the distance of the horse's legs in Figure 10 might prove it's not direct replication, the other two cases are hard to distinguish differences. Can the authors provide more evidence to explain that the model is not just learning simple replication?\n\n2. In Table 1, when comparing, which pretrained 2D VAE is aligned with \"Ours\"?\n\nQUESTIONS:\n1. Would training the VAE with bf16 cause a performance collapse?\n\n2. In Lines 218-219, how are the images of 256 × 256 and 512 × 512, as well as the videos of 9 × 256 × 256 and 17 × 192 × 192 organized during the training cycle? Do you train on images first, then videos, or start with low resolution and then move to high resolution?\n\n3. What impact does training the VAE at low resolution or low frame rate have when performing SVD on high resolution and high frame rate?\n\nLIMITATIONS:\nAuthor added a limitation section in paper and I think there is no obvious potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new video VAE, starting with pretrained image VAE. It includes several techniques that are still capable of handling images and do not suffer from significant computational overhead. First, they use pretrained weights of 2D VAE (from Stable diffusion) by using their weights as initialization with model inflation. They also used efficient 3D architecture to only half of 2D convolutions as 3D convolutions to minimize the computation increase and also applied temporal tiling to handle long videos. They also use a pretrained frozen image decoder to regularize the latent space from the 3D encoder/encoder to ensure the latent space becomes similar to the stable diffusion latent space. The paper shows the proposed video encoder can have a compact latent space with a temporal compression factor (of 4) but is very similar to pretrained image latent space, which can be used for longer and smoother video generation by finetuning existing latent video diffusion models in this latent space.\n\nSTRENGTHS:\n- The paper is generally well-written and well-motivated. In particular, the paper tackles an important problem of constructing a compact latent space of videos that is similar to image space.\n- The performance drop from fine-tuning does not seem that large and shows a good potential for training a latent video generation model.\n- The paper shows a real use-case of this VAE by finetuning stable video diffusion.\n\nWEAKNESSES:\n- The paper lacks a comparison with recent video autoencoders. Specifically, the baselines that the paper provided are quite outdated (e.g., TATS and VQGAN), considering that there are many recent attempts to design a better video autoencoder, to name a few [1, 2, 3]. The authors should discuss what's the pros and cons of this approach compared with these approaches, and if possible, the authors should compare the performance as well. In particular, [2] provides very similar intuition to this paper because they also try to construct a latent space capable of jointly handling images and videos. \n- Lack of novelty: The proposed method is mainly composed of components that have already been widely used to extend image models to videos.\n- The paper does not mention the total training time (even though it includes the number of GPUs and the total number of training iterations).\n\n[1] Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.  \n[2] Language Model Beats Diffusion: Tokenizer is key to visual generation, ICLR 2024.  \n[3] Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition, ICLR 2024.\n\nQUESTIONS:\n- The paper only provides video generation results on image-to-video generation tasks; I wonder if one naturally achieves more smooth video generation model by fine-tuning existing text-to-video generation models in this latent space (such as modelscope). \n- Are you planning to release the code and the model parameters?\n\nLIMITATIONS:\nThe paper addresses the limitation appropriately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a compact video VAE suitable for both image and video generation tasks. The motivation stems from the absence of spatial-temporal compressed continuous latent 3D VAE models for video generation. To effectively utilize 2D VAE and seamlessly integrate image generation models, the authors propose a latent space regularization and initialization method to maximize the use of 2D VAE. A 3D CNN-based architecture is also proposed to compress both temporal and spatial dimensions efficiently. The evaluation of the proposed method is primarily based on reconstruction tasks and image/video generation tasks.\n\nSTRENGTHS:\nThe motivation is clear: the current video generation method may suffer from a lack of high-quality and high-efficiency spatial-temporal VAE if it can be used with the existing T2I model, which also makes it more general and effective. \nThe decoder loss seems a simple and novel design, which helps the model learn spatial latent efficiently and align with the original 2D VAE. Initialization of 3D VAE  with 2D ones seems to provide some insights in practice. \nThe qualitative results are good.\n\nWEAKNESSES:\n1. Improvements over the baseline are presented in Table 1, where the authors compare the main reconstruction results with different open-sourced VAE methods. However, I don’t think it is a fair comparison to claim superior performance. The training data is a key factor in this table. The authors use in-domain data, such as Webvid, for both training and evaluation, whereas other methods do not. Therefore, it is unclear if the proposed method can truly outperform other methods when trained on the same data.\n\n2. Many generation models have been released based on SD VAE. Since compactness does not require further fine-tuning, it is crucial to evaluate more models to strengthen the claim. Evaluating only one model in Table 2/3 is not convincing.\n\nQUESTIONS:\nThe term \"mapping function\" seems to require clarification, as it appears to be more of a simple sampling process from video frames. In Table 7, a higher CLIP score is better. Why does random sampling work best? Providing more exploration and explanation can help readers understand this better.\n\nLIMITATIONS:\nYes, the author discussed the limitations and potential social impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}