CV-VAE: A Compatible Video VAE for Latent Generative Video Models

Authors: sijie  zhao
yong  zhang
xiaodong  cun
shaoshu  yang
muyao  niu
xiaoyu  li
wenbo  hu
ying  shan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 06:45:18

Abstract: Spatio-temporal compression of videos, utilizing networks such as Variational Autoencoders (VAE), plays a crucial role in OpenAI's SORA and numerous other video generative models. For instance, many LLM-like video models learn the distribution of discrete tokens derived from 3D VAEs within the VQVAE framework, while most diffusion-based video models capture the distribution of continuous latent extracted by 2D VAEs without quantization. The temporal compression is simply realized by uniform frame sampling which results in unsmooth motion between consecutive frames. Currently, there lacks of a commonly used continuous video (3D) VAE for latent diffusion-based video models in the research community. Moreover, since current diffusion-based approaches are often implemented using pre-trained text-to-image (T2I) models, directly training a video VAE without considering the compatibility with existing T2I models will result in a latent space gap between them, which will take huge computational resources for training to bridge the gap even with the T2I models as initialization. To address this issue, we propose a method for training a video VAE of latent video models, namely CV-VAE, whose latent space is compatible with that of a given image VAE, e.g., image VAE of Stable Diffusion (SD). The compatibility is achieved by the proposed novel latent space regularization, which involves formulating a regularization loss using the image VAE. Benefiting from the latent space compatibility, video models can be trained seamlessly from pre-trained T2I or video models in a truly spatio-temporally compressed latent space, rather than simply sampling video frames at equal intervals. To improve the training efficiency, we also design a novel architecture for the video VAE. With our CV-VAE, existing video models can generate four times more frames with minimal finetuning. Extensive experiments are conducted to demonstrate the effectiveness of the proposed video VAE.

Main paper: 
 introduction: 
 Video generation has gained significant public attention, especially after the announcement of OpenAI SORA [2] . Current popular video models can be divided into two categories based on the modeling space, i.e., pixel and latent space. Imagen Video [17] , Make-a-video [29] , and Show-1 [42] are representative video diffusion models that directly learn the distribution of pixels. On the other hand, Phenaki [33] , MAGVIT [41] , VideoCrafter [6] , AnimateDiff [15] , VideoPeot [20] , and SORA, etc, are representative latent generative video models that are trained in the latent space formed using variational autoencoders (VAEs). The latter category is more prevalent due to its training efficiency. Furthermore, latent video generative models can be classified into two groups according to the type of VAE they utilize: LLM-like and diffusion-based video models. LLM-like models train a transformer on discrete tokens extracted by a 3D VAE with a quantizer within the VQ-VAE framework [32] . For example, VideoGPT [40] initially trains a 3D-VQVAE and subsequently an autoregressive transformer in the latent space. The 3D-VQVAE is inflated from the 2D-VQVAE [32] used in image generation. TATS [14] and MAGVIT [41] use 3D-VQGAN for better visual quality by employing discriminators, while Phenaki [33] utilizes a transformer-based encoder and decoder, namely CViViT.However, recent latent diffusion-based video models typically exploit 2D VAEs, rather than 3D VAEs, to generate continuous latents to train a UNet or DiT [25] . The commonly used 2D VAE is the image VAE [28] from Stable Diffusion, as training a video model from scratch can be quite challenging. Almost all high-performing latent video models are trained with the SD image model [28] as initialization for the inflated UNet or DiT. Examples include Align-your-latent [5] , VideoCrafter1 [6] , AnimateDiff [15] , SVD [4] , Modelscope [34] , LaVie [35] , MagicVideo [44] , Latte [24] , etc. Temporal compression is simply achieved by uniform frame sampling while ignoring the motion information between frames (see Fig. 1 ). Consequently, the trained video models may not fully understand smooth motion, even when FPS is set as a condition. When projecting a sampled latent sequence to a video using the decoder of the 2D VAE, the generated video exhibits a low FPS and lacks visual smoothness.Currently, the research community lacks a commonly used 3D video VAE for generating continuous latent variables with spatio-temporal compression for latent video models. Training a high-quality video VAE without considering the compatibility with existing pretrained image and video models might not be too difficult. However, even though the trained video VAE exhibits low reconstruction errors, a gap exists between its learned latent space and the one used by pretrained models, such as the video VAE of Open-Sora-Plan [1] . This means that bridging the gap requires significant computational resources and extensive training time, even when using pre-trained models as initialization. One example is shown in Fig. 2 . When training a video VAE independently without considering compatibility, the sampled latent of SVD [4] cannot be projected into the pixel space correctly due to the latent space gap, as shown in Fig. 2(a) . After finetuning the SVD model in the new latent space on 16 A100 for 58K iterations, the quality of the generated video is still poor (see Fig. 2(b) ). In contrast, our video VAE achieves promising results in the pretrained SVD even without finetuning the UNet as shown in Fig. 2(c) .In this work, we propose a novel method to train a video VAE to extract continuous latents for generative video models, which is compatible with existing pretrained image and video models, e.g. Stable Diffusion [28] and SVD [4] . We also inflate the SD image VAE to form a video VAE by adding 3D convolutions to both encoder and decoder of the 2D VAE, which allows us to train video models efficiently with the pretrained models as initialization in a truly spatio-temporally compressed latent space, instead of uniform frame sampling for temporal compression (see Fig. 1 ). Consequently, the generated videos will be smoother and have a higher FPS than those produced using a 2D VAE.To ensure latent space compatibility between 2D and 3D VAEs, we propose a latent space regularization to avoid distribution shifts. We examine the effectiveness of using either the encoder or decoder of the 2D VAE to form constraints and explore four types of mapping functions to design regularization. Moreover, to improve video VAE efficiency, we investigate its architecture and partially integrate 3D convolutions instead of exploiting 3D convolution in all blocks. The proposed video VAE can be used not only for training new video models with pretrained ones as initialization but also as a frame interpolator for existing video models with slight finetuning.Our main contributions are summarized as follows: (1) We propose a video VAE that provides a truly spatio-temporally compressed continuous space for training latent generative video models, which is compatible with existing image and video models, greatly reducing the expense of training or finetuning video models. (2) We propose a latent space regularization to avoid distribution shifts and design an efficient architecture for the video VAE. (3) Extensive experiments are conducted to demonstrate the effectiveness of the proposed video VAE. 
related work: 
 Variational Autoencoder. Variational Autoencoders (VAEs), introduced by [19] , have been widely used in two-stage generative models. The first stage involves compressing the pixels into a lowerdimensional latent representation, followed by a second stage that generates pixels from this latent space. VAEs can be divided into two groups according to the token, i.e., discrete and continuous latent. The difference between the two types of VAEs is the quantization. Continuous VAEs have no quantization, while discrete VAEs learn a codebook for quantization and use it to convert the continuous latent features to discrete indices, called VQVAE [32] . When training discrete VAEs, some methods exploit a discriminator to improve the visual image quality, called VQGAN [12] .In video generation, 2D VAEs are typically inflated into 3D ones by injecting 3D Conv or temporal attention. 3D Convs are for CNN-based VAEs, e.g., 3D-VQVAE [40] , 3D-VAQGAN [14, 41] . Attentions are for transformer-based VAEs, e.g., CViViT [33] . Although there are several discrete 3D VAEs for video generation, there are no commonly used continuous 3D VAEs.Video Generative models. Video generation has achieved remarkable progress in recent years. The announcement of Imagen Video [17] and Make-A-Video [29] made researchers see the hope of purely AI-generated videos. Then, the launch of OpenAI SORA [2] brought the enthusiasm of researchers in academia and industry to a climax. Many video generation models [17, 29, 42] directly learn the distribution of pixels while some others [5, 6, 44, 34, 35, 24, 41, 14, 33, 40, 4, 15] learn the distribution of tokens in a latent space. The tokens are always extracted by a variational autoencoder [11] . Latent video generation models can be categorized into two groups according to whether the token is discrete or continuous. TATS [14] , MAGVIT [41] , VideoGPT [40] , and Phenaki [33] are representative models trained with discrete tokens extracted by a 3D VAE within the VQVAE framework [32] . A codebook is learned jointly with the VAE for quantization. SVD [4] , AnimateDiff [15] , VideoCrafter [6] , etc., are video models trained with continuous latent extracted by a 2D VAE without quantization, rather than a 3D VAE. SD image VAE is the commonly used 2D VAE. One reason is that video models are difficult to train from scratch and they are always initialized with the weights of a pretrained T2I model such as Stable Diffusion UNet [28] . Hence, the corresponding image VAE is used to extract latents from a video. Since the image VAE can only perform spatial compression, the temporal compression is realized by uniform frame sampling. This strategy ignores the motion between key frames.There lacks a video VAE that is compatible with the pretrained T2I or video models. Though it is not difficult to train a video VAE (3D VAE) independently with high reconstruction accuracy; it will result in a latent space gap between the learned video VAE and existing pre-trained image and video models that are always used as initialization. The Open-Sora-Plan project [1] offers a video VAE; however, it is not compatible with existing image or video models. Large computational resources and a long training time are required to bridge the gap. In this work, we propose a latent space regularization method to train a video VAE whose latent space is compatible with pretrained models. We propose a latent space regularization method for training a video VAE that is compatible with pretrained image and video models. We examine multiple strategies for implementing the regularization, focusing on either the encoder or the decoder of the image VAE. Additionally, we explore four types of mapping functions to develop the regularization loss. To enhance the efficiency of the video VAE, we introduce an architecture that employs different inflation strategies in distinct blocks, instead of incorporating 3D convolutions in all blocks. 
latent space regularization: 
 We inflate a 2D VAE into a 3D VAE, initializing it with the 2D VAE's weights. The 3D VAE is designed to be capable of encoding both image and video (see details in Sec.3.2). The key of building a compatible video VAE is the latent space alignment between the video VAE and the image VAE.Notations. Let x ∈ R H×W ×3 denote an image in RGB space and X ∈ R (T +1)×H×W ×3 denote a video with T + 1 frames. When T = 0, X degrades into an image and the video VAE will process it with temporal padding. z ∈ R h×w×c denotes the latent tokens extracted by either an image VAE or a video VAE. Z ∈ R (t+1)×h×w×c is latent tokens extracted by the video VAE. ρ s = H/h = W/w and ρ t = T /t are the spatial and temporal compression rates. Let E i and D i denote the encoder and decoder of the image VAE, respectively. While E v and D v are for the video VAE. Then, we havez = E i (x), Z = E v (X), and z = E v (x). x = D i (z) = D i (E i (x)), X = D v (Z) = D v (E v (X)), and x = D v (z) = D v (E v (x)) are the reconstructed image and video from the latent tokens.Regularization. We assume the latent of the image VAE follow a distribution, i.e, z ∼ p i (z).The joint distribution of t + 1 independent frames isp i (Z) = t+1 k p i (z k ).The latent distribution of the video VAE can be denoted as Z ∼ p v (Z). To achieve the alignment between the latent spaces of the image and video VAEs, we have to build mappings between p i (Z) and p v (Z). Since both distributions have no analytic formulation, distance metric for measuring differences between distributions is not applicable.Here, we build the cooperation between the image VAE and the video one to construct reconstruction loss for space alignment. When exploiting the encoder of the image VAE for alignment, the latent extracted from the image encoder should be corrected decoded by the decoder of the video VAE, i.e., Xv i = D v (E i (ψ(X))). The illustration is shown in Fig. 3 (a). For a given input video X ∈ R (T +1)×H×W ×3 , we use a mapping function ψ to sample ψ(X) ∈ R (T /ρt+1)×H×W ×3 . Thus the reconstructed video Xv i is the same as the shape of X. Then, the reconstruction loss of using the image encoder can be defined asEQUATION)When exploiting the decoder of the image VAE, the latent extracted by the video encoder can be decoded by the decoder of the image VAE, i.e., Xi v = D i (E v (X)). The illustration is shown in Fig. 3(b) . For a given input video X ∈ R (T +1)×H×W ×3 , the reconstructed video is Xv i ∈ R (T /ρt+1)×H×W ×3 . Then, the reconstruction loss of using the image decoder can be defined asEQUATION)Mapping Functions. To bridge the dimension gap between Xi v or Xv i and X, we investigate four types of mapping functions ψ as follows. 1) First frame. We compare only the first frame of the input video and the reconstructed one. The regularization loss degenerates to measure the difference between the input and reconstruction of the image. 2) Slice. ψ samples one frame every ρ t frames to form a shorter video. It starts from the second frame and the first one is reserved. 3) Average. ψ computes the average of every ρ t frames, starting from the second frame. 4) Random. ψ randomly samples one frame from every ρ t frames, starting from the second frame.Training Objective. Following the training of the 2D VAE in LDM [28] , our basic objective is a combination of a reconstruction loss [43] , an adversarial loss [12] , and a KL regularization [19] , i.e.,L AE = min Ev,Dv max Dv L rec (X, D v (E v (X)) -L adv (D v (E v (X))) + log D v (X) + L KL (X; E v , D v ),where the first term is the reconstruction loss, the second and third are the adversarial loss, and the last is the KL regularization. D v is the discriminator that differentiates original videos from reconstructed ones. It is inflated from the image discriminator in LDM by injecting 3D convolutions. Then, for latent space alignment, our full training objective is:EQUATIONwhere λ 1 and λ 2 are trade-off parameters. We explore different settings of λ 1 and λ 2 and find that using the decoder only achieves the best performance. The framework of CV-VAE is shown in Fig. 3 (c) and evaluations between different regularization methods can be found in Tab. 6.We design the architecture of the video VAE according to the image VAE in LDM [28] . The detailed architecture is presented in the Appendix A.1. We explain the key modifications as follows.Model Inflation. Considering the latent space compatibility and the convergence speed of the video VAE, we make full use of the pretrained weights of the image VAE for initialization, instead of training from scratch. We inflate the image VAE into the video VAE by replacing 2D convolutions with 3D convolutions. 3D convolutions are used to model the temporal dynamics among frames.To initialize the 3D convolutions, we copy the weights of the 2D Conv kernel to the corresponding positions in the 3D Conv kernel and set the remaining parameters to zero. We set the stride to achieve temporal downsampling and increase the number of 3D kernels by a factor of s to achieve s× temporal upsampling. To enable the video VAE to handle both image and video, given T + 1 frames as input, we use reflection padding in the temporal dimension for the first frame. By initializing the video VAE using the above operations, we can reconstruct images without training, significantly accelerating the training convergence speed on video datasets.Efficient 3D Architecture. Expanding 2D Convs to 3D Convs (e.g., k × k → k × k × k)results in k× parameters and computational complexity. To improve the computational efficiency of the model, we adopt a 2D+3D network structure. Specifically, we retain half of the convolutions in the ResBlock as 2D Convs and set the other half as 3D Convs. We find that, compared to setting all Convs to 3D, the number of parameters and the computational complexity are reduced by roughly 30%, while the reconstruction performance remains nearly the same. See Sec. 4.2 for experimental comparisons.Temporal Tiling for Arbitrary Video Length Existing image VAEs employ spatial tiling on large spatial resolution images to achieve memory-friendly processing, which cannot handle long videos.As a result, we introduce temporal tiling processing. During encoding, the video X is divided into block. X i and X i+1 have a one-frame overlap in the temporal dimension. After encoding each X i to obtain Z i , we discard the first frame of Z i when i ̸ = 0 and concatenate all Z i in the temporal dimension to obtain Z. The decoding process is handled similarly to the encoding process. By combining our method with 2D tiling, we can encode videos with arbitrary resolution and length.[X 1 , X 2 , ...X n ], where X i ∈ R (1+f •ρt)×H×W ×34 Experiments 
experimental setups: 
 Datasets and Metrics. We evaluate our CV-VAE on the COCO2017 [21] validation dataset and the Webvid [3] validation dataset which includes 1024 videos. Both images and videos are resized and cropped to a resolution of 256 × 256. Each video is sampled with 33 frames and a frame stride of 3.We evaluate the reconstruction performance of CV-VAE on images and videos using metrics such as PSNR, SSIM [36] , and LPIPS scores [43] . We employ 3D tiled processing to encode and decode videos with arbitrary resolution and length within a limited memory footprint. During inference, we allow a single video block size of 17 × 576 × 576. We evaluate the video generation quality of our model using 2048 randomly sampled videos from UCF101 [30] and MSR-VTT [39] . Videos are resized and cropped to a resolution of 576 × 1024 to fit the SVD [4] . We use Frechet Video Distance (FVD) [31] , Kernel Video Distance (KVD) [31] , and Perceptual Input Conformity (PIC) [38] metrics to evaluate video generation quality. For evaluating image generation quality, we use 2048 samples from the COCO2017 validation dataset and employ FID [16] , CLIP score [26] , and PIC score metrics.Training Details. We train our CV-VAE model using image datasets including LAION-COCO [9] and Unsplash [23] , as well as the video dataset Webvid-10M [3] . For image datasets, we employ two resolutions, i.e., 256 × 256 and 512 × 512. In the case of video datasets, we use two settings of frames and resolutions: 9 × 256 × 256 and 17 × 192 × 192. The batch sizes for these four settings are 8, 2, 1, and 1, with sampling ratios of 40%, 10%, 25%, and 25%, respectively. We employed the AdamW optimizer [22] with a learning rate of 1e-4 and cosine learning rate decay. To avoid numerical overflow, we trained CV-VAE using float32 precision, and the training was carried out on 16 A100 GPUs for 200K steps. To fine-tune the SVD on CV-VAE, we utilize in-house data with a frame rate and resolution of 97 × 576 × 1024. We employ deepspeed stage 2 [27] , gradient checkpointing [8] techniques, and train with bfloat16 precision. We used a constant learning rate of 1e-5 with the AdamW [22] optimizer, and only optimized the last layer of U-Net. The training was carried out on 16 A100 GPUs for 5K steps.We evaluated the reconstruction quality of various VAE models on image and video test sets. The comparison group includes: (1) VAE-SD2.1 [28] which is widely used in the community for image and video generation models. (2) VQGAN [12] which encoding pixels into discrete latents. We use the f8-8192 version for comparision. (3) TATS [14] Text-to-Image Models We tested the compatibility of our CV-VAE by integrating it into the pretrained SD2.1 [28] , replacing the original 2D VAE without any finetuning. We evaluated it on We also visualized the text-to-image generation results of both models in Fig. 5 . In each pair, the left side depicts the results of SD2.1, while the right side shows the results generated by our CV-VAE, which replaced the original VAE, using the same random seed and prompt. The results show that both models generate images with almost identical content and texture, with only slight differences in color. This further validates the feasibility of building a compatible VAE via latent constraint.Image-to-Video Models The primary objective of CV-VAE is to train a model that can compress both time and space, while also being compatible with the existing 2D VAE. In this section, we validate the compatibility of CV-VAE with existing video generation models. We integrate CV-VAE into SVD [4] , replacing the original VAE, and decoded the generated video latents. CV-VAE offers the flexibility to decode either in image mode (CV-VAE-I) or video mode (CV-VAE-V); the former decodes n frames of latent into n frames of video, while the latter decodes n frames of latent into 1 + (n -1) × 4 frames of video. We tested the video generation quality of both models. Furthermore, we fine-tuning the SVD for better alignment.As shown in Tab. 3, incorporating 'CV-VAE-I' into a frozen SVD immediately yields video generation quality comparable to the original VAE. Using CV-VAE in video mode can also decode videos generated by SVD, and further improvements in video decoding quality can be achieved by finetuning only the output layer (approximately 12k parameters). One of the reasons for the noticeable gap in test metrics between 'SVD+CV-VAE-I' and 'SVD+CV-VAE-V' is that they use different numbers of frames, making a direct comparison challenging.In Fig. 6 , we also display the comparison results with SVD [4] . The top row shows the generated results by SVD, and the bottom row shows the generated results after inserting CV-VAE into SVD and fine-tuning the output layer. We use the first frame as a condition and generate with the same random seed. The U-Net generates 25 frames of latent, which are decoded by CV-VAE into a 97-frame video.As can be seen, compared to the original SVD, our results exhibit smoother motion. It is worth noting that both models have the same computational complexity during the diffusion process, which means that our model is more scalable. By fine-tuning a small number of parameters, the image-to-video model can generate smoother and longer videos through CV-VAE, effectively serving as a frame interpolation method. Therefore, we compared CV-VAE with existing video interpolation models [18] , conducting experiments on MSR-VTT [39] . We first used SVD to generate a video of size 25 × 576 × 1024, and then applied the interpolation model to expand the video from 25 frames to 97 frames. For our results, we directly generated a video of size 97 × 576 × 1024 using CV-VAE and the fine-tuned SVD. The comparison results are shown in Tab. 4.3, where CV-VAE outperformed the FIRE [18] in two out of three metric, validating the potential of CV-VAE as an interpolation model.Text-to-Video Models In this section, we integrate CV-VAE into the existing text-to-video model to futher validate the effectiveness. 'VC2' refers to decoding the results generated by VideoCrafter2 [7] using the original 2D VAE, 'VC2+CV-VAE-I' indicates decoding the results using CV-VAE in image mode, and 'VC2+CV-VAE-V' denotes decoding the results using CV-VAE in video mode, which generates videos of 4× frames. We only fine-tuned a small number of parameters in U-Net, including the first and last layers. We use captions from the validation set of MSR-VTT [39] for evaluation, with the resolution of 320×512. Following the approach taken by previous studies [37] , we used the CLIP [26] metric to evaluate the generation quality of text-to-video models, including Frame Consistency (F.C.) and Textual Alignment (T.A.). The experimental results are shown in Tab. 4.3, where the 'VC2+CV-VAE-V' setting achieved the best generation performance through fine-tuning VideoCrafter2. Check Appendx A.5 for quantitative comparison.Influence of Regularization Type We evaluated the impact of three types of latent regularization, which are: (1) 2D Enc. , i.e., λ 1 = 0 and λ 2 = 1 in Eq. 3; (2) 2D Dec. , i.e., λ 1 = 1 and λ 2 = 0 in Eq. 3; (3) 2D Enc. + Dec. , i.e., λ 1 = 1 and λ 2 = 1 in Eq. 3. The 2D decoder decodes n frames of latents into n frames of video, while the 3D decoder decodes the same n frames of latents into 1 + (n -1) × 4 frames of video. Therefore, we need to mapping the input video to n frames to calculate the regularization loss in Eq. 2. We evaluated four mapping functions mentioned in Sec. 3.1.As shown in Tab. 7, the four methods have similar effects on image reconstruction, with the main differences being in video reconstruction. The '1st Frame' approach yields the worst video reconstruction results due to the lack of regularization and guidance for subsequent frames. The 'Slice' method results in poor reconstruction quality for the three unsampled middle frames. The 'Average' method is inferior to 'Random' in video reconstruction, primarily because calculating the mean for multiple consecutive frames leads to motion blur in the target. 
conclusion and limitations: 
 We propose a novel method to train a video VAE that is compatible with existing image and video models trained with SD image VAE. The video VAE provides a truly spatio-temporally compressed latent space for latent generative video models, as opposed to uniform frame sampling. Due to the latent space compatibility, a new video model can be trained efficiently with the pretrained image or video models as initialization. Besides, existing video models such as SVD can generate smoother videos with four times more frame using our video VAE by slightly fine-tuning a few parameters.Extensive experiments are performed to demonstrate the effectiveness of the proposed VAE.Limitations. The performance of the proposed video VAE relies on the channel dimension of the latent space. A higher dimension may yield better reconstruction accuracy. Since we pursue the latent space compatibility with existing image and video models trained with SD image VAE, the channel dimension of our video VAE is limited to be the same as the image VAE. This can be improved if an image VAE with a higher channel dimension becomes available, e.g., the VAE of SD3 [13] . Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Code and checkpoints will be released upon the acceptance of this paper. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Details are in the experiment settings.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We use the commonly used protocol and metrics for evaluation.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the corresponding papers of the used assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: As illustrated in Fig. 7 , we introduce the structure of the CV-VAE. The architecture of CV-VAE is primarily derived from the VAE in Stable Diffusion [28] , with several notable differences: (1) Some or all 2D convolutions within the network are transformed into 3D convolutions, while retaining their weights. (2) Temporal downsampling is executed in the encoder through the use of strides. (3) Temporal upsampling is accomplished by increasing the output channel number of 3D convolutions by a specific factor. (4) A discriminator, comprising 3D convolutions, is utilized. The main differences are marked in red text in Fig. 7 . More latent channels generally lead to better reconstruction performance in VAEs [13, 10] , which is crucial for image and video editing tasks. In this subsection, I conducted experiments using a VAE with more latent channels (z=16). We used the 2D VAE from SD3 [13] as the baseline and trained the CVVAE-SD3 based on latent regularization. Testing was conducted under the same settings as in Tab. 1. The comparison results are shown in Tab. 8. CVVAE-SD3 outperforms VAE-SD3 in image reconstruction but is inferior to VAE-SD3 in video reconstruction. The main reason is that CVVAE-SD3 has a higher compression ratio in video compression, resulting in the loss of more information, though its reconstruction quality is significantly higher than that of VAE with z=4.In Fig. 8 , we showcase additional image reconstruction results using CV-VAE. we use the version of '2D + 3D'. These images are sourced from the COCO2017 [21] dataset with a resolution of 512 × 512.The reconstructed image precisely shares the same colors and textures as the original, demonstrating the high fidelity of our CV-VAE in encoding and reconstructing images. Interestingly, in Fig. 5 , slight color differences can be observed between the images decoded by the Image VAE and CV-VAE, given the same latent generated by the Image Diffusion Model. This suggests that there is still a minor discrepancy between the latent spaces of the video VAE trained with latent regularization and the Diffusion Model. This gap can be bridged with minimal additional training.As shown in Fig. 9 , we present the reconstruction results of 4 consecutive frames from a video clip (33 × 576 × 1024) using CA-VAE. The reconstructed video frames maintain consistency in color, structure, and motion with the ground truth. According to CA-VAE, these continuous frames are condensed into a single latent frame, signifying that even a single latent frame encapsulates motion information.We tested the compatibility of CV-VAE with existing text-to-video diffusion models, such as Videocrafter2 [7] , which also employs a 2D VAE from the SD as its first-stage model. We adopted a strategy similar to the training of 'SVD + CV-VAE' in Sec. 4.3, by fine-tuning the last layer of the U-Net in VC2 to adapt it to CV-VAE. We finetuned the model using in-house data at a resolution of 61 × 320 × 512, which is equivalent to a latent size of 16 × 40 × 64.As shown in Fig. 10 , compared to the original VC2, the 'VC2 + CV-VAE generates videos approximately four times longer, resulting in smoother motion. This further validates the feasibility of obtaining a compatible video VAE through latent regularization, thereby avoiding the massive computational power required to train a video diffusion model from scratch.The CV-VAE can be seamlessly integrated into existing diffusion models, replacing the original 2D VAE for image or video generation, which may result in potential societal implications. While it proves beneficial in fields such as entertainment and advertising, by providing more realistic and immersive content, it also raises ethical and safety concerns. The ease of generating high-quality synthetic images and videos could lead to a surge in the production of harmful or misleading content, such as deepfakes, potentially exacerbating issues of misinformation and privacy invasion. We condemn the misuse of generative AI that harms individuals or spreads misinformation. • The answer NA means that the paper does not include experiments.Real Reconstructed• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The paper is reviewed by four expert reviewers and got overall postiive ratings of accept, two weak accepts and one borderline accepts. Reviewers find that the paper tackles an important problem of desiging 3D VAE for compact video representation and well executed. Some reviewers concerns regarding the comparison to more recent works, lack of technical novelty, have been addressed during rebuttal. AC agrees that the paper provides a good insight on building compact video representation and recommend for acceptance.

Reviews: review: paper_summary: paper_summary: This paper focuses on a compact video VAE suitable for both image and video generation tasks. The motivation stems from the absence of spatial-temporal compressed continuous latent 3D VAE models for video generation. To effectively utilize 2D VAE and seamlessly integrate image generation models, the authors propose a latent space regularization and initialization method to maximize the use of 2D VAE. A 3D CNN-based architecture is also proposed to compress both temporal and spatial dimensions efficiently. The evaluation of the proposed method is primarily based on reconstruction tasks and image/video generation tasks.
strength_weakness: strength_weakness: 1. Improvements over the baseline are presented in Table 1, where the authors compare the main reconstruction results with different open-sourced VAE methods. However, I don’t think it is a fair comparison to claim superior performance. The training data is a key factor in this table. The authors use in-domain data, such as Webvid, for both training and evaluation, whereas other methods do not. Therefore, it is unclear if the proposed method can truly outperform other methods when trained on the same data.

2. Many generation models have been released based on SD VAE. Since compactness does not require further fine-tuning, it is crucial to evaluate more models to strengthen the claim. Evaluating only one model in Table 2/3 is not convincing.
questions: questions: The term "mapping function" seems to require clarification, as it appears to be more of a simple sampling process from video frames. In Table 7, a higher CLIP score is better. Why does random sampling work best? Providing more exploration and explanation can help readers understand this better.
limitations: limitations: Yes, the author discussed the limitations and potential social impact.
paper_summary: paper_summary: This paper proposes a new video VAE, starting with pretrained image VAE. It includes several techniques that are still capable of handling images and do not suffer from significant computational overhead. First, they use pretrained weights of 2D VAE (from Stable diffusion) by using their weights as initialization with model inflation. They also used efficient 3D architecture to only half of 2D convolutions as 3D convolutions to minimize the computation increase and also applied temporal tiling to handle long videos. They also use a pretrained frozen image decoder to regularize the latent space from the 3D encoder/encoder to ensure the latent space becomes similar to the stable diffusion latent space. The paper shows the proposed video encoder can have a compact latent space with a temporal compression factor (of 4) but is very similar to pretrained image latent space, which can be used for longer and smoother video generation by finetuning existing latent video diffusion models in this latent space.
strength_weakness: strength_weakness: - The paper lacks a comparison with recent video autoencoders. Specifically, the baselines that the paper provided are quite outdated (e.g., TATS and VQGAN), considering that there are many recent attempts to design a better video autoencoder, to name a few [1, 2, 3]. The authors should discuss what's the pros and cons of this approach compared with these approaches, and if possible, the authors should compare the performance as well. In particular, [2] provides very similar intuition to this paper because they also try to construct a latent space capable of jointly handling images and videos. 
- Lack of novelty: The proposed method is mainly composed of components that have already been widely used to extend image models to videos.
- The paper does not mention the total training time (even though it includes the number of GPUs and the total number of training iterations).

[1] Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.  
[2] Language Model Beats Diffusion: Tokenizer is key to visual generation, ICLR 2024.  
[3] Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition, ICLR 2024.
questions: questions: - The paper only provides video generation results on image-to-video generation tasks; I wonder if one naturally achieves more smooth video generation model by fine-tuning existing text-to-video generation models in this latent space (such as modelscope). 
- Are you planning to release the code and the model parameters?
limitations: limitations: The paper addresses the limitation appropriately.
paper_summary: paper_summary: This paper designs a 3D VAE consistent with the 2D VAE, whose output mode can losslessly switch between 2D and 3D while retaining the characteristics of both the 2D VAE and 3D VAE. This allows for obtaining performance similar to the original 2D VAE while increasing the frame count of the 3D output.
strength_weakness: strength_weakness: 1. Figure 10 attempts to demonstrate the effect of the 3D VAE on frame count enhancement. I understand that the conclusion of this figure is actually the most critical value proof of this paper: retaining the 2D VAE latent consistency while having the embedding capability of 3D features, such as extending the frame count. This is quite challenging, as the 3D VAE might tend to completely collapse to perform similarly to the original 2D VAE, simply replicating the 2D results. The loss between replicated 2D frames is very small, making optimization difficult. While the distance of the horse's legs in Figure 10 might prove it's not direct replication, the other two cases are hard to distinguish differences. Can the authors provide more evidence to explain that the model is not just learning simple replication?

2. In Table 1, when comparing, which pretrained 2D VAE is aligned with "Ours"?
questions: questions: 1. Would training the VAE with bf16 cause a performance collapse?

2. In Lines 218-219, how are the images of 256 × 256 and 512 × 512, as well as the videos of 9 × 256 × 256 and 17 × 192 × 192 organized during the training cycle? Do you train on images first, then videos, or start with low resolution and then move to high resolution?

3. What impact does training the VAE at low resolution or low frame rate have when performing SVD on high resolution and high frame rate?
limitations: limitations: Author added a limitation section in paper and I think there is no obvious potential negative societal impact.
paper_summary: paper_summary: The paper introduces CV-VAE, a 3D VAE that is trained through latent space regularization using an existing two-dimensional VAE decoder. This design facilitates seamless integration with 2D VAE-based diffusion models. The concept, while simple and straightforward, is notably efficacious and offers practical utility. Moreover, relative to conventional 2D VAEs, the proposed framework significantly compresses the latent space dimensionality by integrating temporal downsampling within the 3D VAE architecture.
strength_weakness: strength_weakness: 1. As presented in Figure 6, CV-VAE suffers from more severe flickering problems than the original SVD, which may possibly be due to the gap between the latent distribution of CV-VAE and SD-2.1. In the paper, the author also pointed out that text-to-image generation may experience differences in color. Authors may convince readers that such flaws can be overcome through further fine-tuning, e.g., fine-tuning full parameters of SVD, or fine-tuning for longer iterations.

2. Missing details about training: The authors didn’t present details about training batch size, augmentations, video fps, etc. Also, whether the default CV-VAE displayed in the experiment section is trained on video and image data separately or jointly (as vae of Opensora) remains unknown.
questions: questions: 1. Please refer to weakness.

2. In Table 3 and figure 6, authors may also present the metrics of SVD on 97 frames straightly or by using the frame interpolation model of SVD.

3. Authors may specify whether the decoder-based regularization is still employed when training with images as inputs for CV-VAE.

4. Authors may also present metrics comparison between SVD and SVD + CV-VAE on text-to-video generation.

5. Can the method be supported by theoretical proof? For example, can we minimize the distribution distance between video latents of CV-VAE and one of corresponding image latents encoded by 2d VAE?
limitations: limitations: N/A

Comments: title: Response to Rebuttal
Response
Thanks for the rebuttal
Response
Response
Thanks for rebuttal.
comment: Thank you for your rebuttal and my concerns have been addressed. Accordingly, I have raised my rating to 6.
Thanks for the further clarification. I increased my rating from 4 to 5 accordingly.
The additional results addressed my concerns.
The comparison results of FID and FVD metrics on MSR-VTT are as follows. we use the first frame of MSR-VTT to calculate FID.


| Methods      | Comp.      | FID  | FVD   |
|:--------------:|:------------:|:------:|:-------:|
| VAE-SD2.1    | -          | 1.31 | **7.42**  |
| VQGAN        | $\times$   | 7.56 | 23.59 |
| TATS         | $\times$   | 8.27 | 19.36 |
| VAE-OSP      | $\times$   | 1.28 | 9.81  |
| PVDM         |$\times$    | 7.74 | 22.38 |
| Open-MAGVIT2 | $\times$   | 2.38 | 12.13 |
| Ours         |$\checkmark$| **1.26** | 8.55  |

Due to quantization error, discrete VAEs usually lose more information than continuous VAEs. For example, MAGVIT2 encodes an image with resolution of $256\times 256$ into **integers** ranging from 0 to 262144 with a size of **$32\times 32$**, while continuous VAEs (SD-VAE2.1, VAE-OSP, CV-VAE) encode an image with resolution of $256\times 256$ into **floating-point vectors** with a size of **$32\times32\times z$** where $z$ represents channels of latent and are equal to 4 for the continuous VAEs in Table 1. Therefore, it is not surprising that continuous VAEs can achieve better reconstruction results than discrete VAEs. 

Moreover, reconstruction quality is not our primary goal in designing CV-VAE, since the reconstruction quality can be easily improved by increasing the size of $z$ without changing the model structure and size [1][2]. For example, in Stable Diffusion 3[2], the PSNR of VAE with $z = 4$ is 25.12dB, while the PSNR of VAE with $z=16$ is 28.62dB. Our primary goal is to design a compatible Video VAE based on the existing 2D VAE, so we must keep the size of $z$ the same as the 2D VAE ($z=4$ for VAE-SD2.1).


[1] Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack, arXiv 2023.

[2] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, ICML 2024.
Thanks for the further clarification - can authors can provide rFID and rFVD (reconstruction FID and FVD) of the proposed methods and other baselines in MSR-VTT? It seems MAGVIT-2 results show too high LPIPS and low SSIM scores considering that Open-MAGVIT2 shows quite a good performance on ImageNet and the original MAGVIT2 paper shows surprising low LPIPS on UCF-101.
Thank you for your comments. We clarify that CV-VAE and MAGVIT are completely different works: (1) MAGVIT2 represents images or videos as **discrete tokens**, which are used in **autoregression models** to generate images or videos; Our CV-VAE represents images or videos as **continuous latents**, which are used in **diffusion models** to generate images or videos. (2) Our main goal and contribution is to design a Video VAE that is compatible with 2D VAEs, while MAGVIT2 is not compatible with any other VAEs. here are the responses:

**R1:** The term "Comp." has the same meaning as in Table 1 of the main paper, indicating compatibility. This also represents the core contribution of our work, which is to design a Video VAE that is compatible with other 2D VAEs, Image Diffusion Models, and Video Diffusion Models.

**R2:** We use the 8x downsampling version from MAGVIT2 for comparison, which is also the same as the downsampling factor in our CV-VAE.
Thanks for the clarifications. Many of my concerns are now addressed. Please add this discussion, particularly for a comparision with MAGVIT2 because it shares lots of similarities to this work. I have a few more questions: 
 
1. What is a meaning of "Comp" in Table 1 in the rebuttal PDF file?
2. I checked the MAGVIT2 repository and it seems there are two versions available - the first one uses 16x downsampling and the second uses 8x downsampling factor for training and inference. Which model is used for the comparison?
I have no further questions, and I stand by my rating. Personally, I believe the quality and contribution of this paper generally meet the bar for NeurIPS.