CV-VAE: A Compatible Video VAE for Latent Generative Video Models
Authors: Sijie Zhao
Yong Zhang
Xiaodong Cun
Shaoshu Yang
Muyao Niu
Xiaoyu Li
Wenbo Hu
Ying Shan
Venue: NeurIPS 2024 poster (2024)
Keywords: Variational Autoencoder
Video Generation
Image Generation
Primary area: generative_models
TL;DR: A video VAE for latent generative video models, which is compatible with pretrained image and video models, e.g., SD 2.1 and SVD

Abstract: Spatio-temporal compression of videos, utilizing networks such as Variational Autoencoders (VAE), plays a crucial role in OpenAI's SORA and numerous other video generative models. For instance, many LLM-like video models learn the distribution of discrete tokens derived from 3D VAEs within the VQVAE framework, while most diffusion-based video models capture the distribution of continuous latent extracted by 2D VAEs without quantization. The temporal compression is simply realized by uniform frame sampling which results in unsmooth motion between consecutive frames. Currently, there lacks of a commonly used continuous video (3D) VAE for latent diffusion-based video models in the research community. Moreover, since current diffusion-based approaches are often implemented using pre-trained text-to-image (T2I) models, directly training a video VAE without considering the compatibility with existing T2I models will result in a latent space gap between them, which will take huge computational resources for training to bridge the gap even with the T2I models as initialization. To address this issue, we propose a method for training a video VAE of latent video models, namely CV-VAE, whose latent space is compatible with that of a given image VAE, e.g., image VAE of Stable Diffusion (SD). The compatibility is achieved by the proposed novel latent space regularization, which involves formulating a regularization loss using the image VAE. Benefiting from the latent space compatibility, video models can be trained seamlessly from pre-trained T2I or video models in a truly spatio-temporally compressed latent space, rather than simply sampling video frames at equal intervals. To improve the training efficiency, we also design a novel architecture for the video VAE. With our CV-VAE, existing video models can generate four times more frames with minimal finetuning. Extensive experiments are conducted to demonstrate the effectiveness of the proposed video VAE.

CV-VAE: A Compatible Video VAE for Latent Generative Video Models
Sijie Zhao
Yong Zhang
Xiaodong Cun
Shaoshu Yang
Muyao Niu
Xiaoyu Li
Wenbo Hu
Ying Shan
Tencent Al Lab
Abstract
opatio-tempont compression of videos, utilizing networks such as Variational Autoencoders (VAE), plays a crucial role in OpenAl's SORA and numerous other video generative models. For instance, many LLM-like video models learn the dis- tribution of discrete tokens derived from 3D VAFs within the VQVAE framework, while most diffusion-based video models capture the distribution of continuous latent extracted by 2D VAEs without quantization. The temporal compression is simply realized by uniform frame sampling which results in unsmooth motion between consecutive frames. Currently, there lacks of a commonly used continuous video (3D) VAE for latent diffusion based video models in the research community. Moreover, since current diffusion-basee based approaches are often implemented using pre- pre-trained text-to- to-image (T2I) models, directly training video VAE without considering the compatibility with existing T2I models will result in a latent space gap between them, which will take huge computational resources for training bridge the gap even with the T2I models as initialization. To address this issue, we propose method for training video VAE of latent video models, namely CV-VAE, whose latent space is compatible with that of given image VAE, e.g., image VAE of Stable Diffusson (SD). The compatibility is achieved by the pro- posed novel latent space regularization, which involves formulating a regularization Joss using the image VAE. Benefiting from the latent space compatibility, video models can be trained seamlessly from pre-trained T2I or video models in truly opatio-temporaale compressed latent space, rather than simply sampling video frames at equal intervals. To improve the training efficiency, we also design a novel architecture for the video VAE. With our CV-VAE, existing video models can generate four times more frames with minimal finetuning. Extensive experiments are conducted to demonstrate the effectiveness of the proposed video VAE.
Introduction
Video generation has gained significant public attention, especially after the announcement of OpenAI SORA [2] Current popular video models can divided into two categories based on the modeling space, i.e., pixel and latent space. Imagen Video [17], Make-a-video [29]. and Show-1 1 [2] are representative video diffusion models that directly learn the distribution of pixels. On the other hand, Phenaki (33] MAGVIT [1I. VideoCrafter (6), AnimateDiff [15). VideoPeot [20]. and SORA, etc, are representative latent generative video models that are trained the latent space formed using variational autoencoders (VAEs), The latter category is more prevalent due to its training efficiency.
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Time
(a) Uniform frame sampling
(b)
3D comv
(c)
(b) 3D convolution for temporal compression
Figure 1: Temporal compres- sion difference between an image VAE and our video one.
Figure 2: SVD inference (a) The pretrained SVD with the inde- pendently trained Video VAE. (b) Finetuning SVD based on (a). (c) The pretrained SVD with video VAE.
Furthermore, latent video generative models can be classified into two groups according to the type of VAE they utilize: LLM-like and diffusion- baaee video models. LLM-like models train a transformer on discrete tokens extracted by 3D VAE with a quantizer within the VQ-VAF framework [32]. For example, VideoGPT [40] initially trains 3D-VQU VOVEE and subsequently an autoregressive transformer the latent space. The 3D- VQVAE is inflated from the 2D- VQVAE 32 used in image generation. TATS [4] and MAGVIT 41 use 3D- for better visual quality by employing discriminators, utilizes transformer- based encoder and decoder, namely CViVIT.
while Phenaki 33
However, recent latent diffusion-barre video models typically exploit 2D VAEs, rather than 3D VAEs, generate continuous latents to train UNet or DiT (25] The commonly used 2D VAE is the image VAE (28) from Stable Diffusion, as training a video model from scratch can be quite challenging. Almost all high-performing latent video models are trained with the SD image model [28 as
initialization for the inflated UNet or DIT. Examples include Align-your-he latent 5. VideoCrafter! (
AnimateDiff [55], SVD 4.. Modelscope 34, LiVie (35), Magic Video 44) Latte 24 etc. Temporal
compression is simply achieved by uniform frame sampling while ignoring the motion information between frames (see Fig. [T.. Consequently, the trained video models may not fully understand smooth motion, even when FPS is set as condition. When projecting sampled latent sequence to a video using the decoder of the 2D VAE, the generated video exhibits a low FPS and lacks visual smoothness.
Currently, the research community lacks commonly used 3D video VAE for generating continuous latent variables with spatio-temporal compression for latent video models. Training a high-quality video VAE without considering the compatibility with existing pretrained image and video models might not be too difficult. However, even though the trained video VAE exhibits low reconstruction errors, gap exists between its learned latent space and the one used by pretrained models, such as the video VAE of Open-Sora-Plan [1] This means that bridging the gap requires significant computa- tional resources and extensive training time, even when using pre-trained models as initialization.
One example is shown in Fig. 2 When training video VAE independently without considering compatibility, the sampled latent of SVD 4 cannot be projected into the pixel space correctly due to the latent space gap, as shown in Fig. 2 a). After finetuning the SVD model in the new latent space 16 100 for 58K iterations, the quality of the generated video still poor (see Fig. b)). In contrast, video VAE achieves promising results in the pretrained SVD even without finetuning the UNet as shown in Fig. c).
In this work, we propose a novel method to train video VAE to extract continuous latents for generative video models, which is compatible with existing pretrained image and video models, e.g. Stable Diffusion (28) and SVD 4 We also inflate the SD image VAE to form video VAE by adding 3D convolutions to both encoder and decoder of the 2D VAE, which allows us train video models efficiently with the pretrained models initialization in truly spatio-temporally compressed

---

latent space, instead of uniform frame sampling for temporal compression (see Fig. D. Consequently, the generated videos will be smoother and have a higher FPS than those produced using a 2D VAE.
To ensure latent space compatibility between 2D and 3D VAEs, we propose a latent space regulariza- tion to avoid distribution shifts. We examine the effectiveness of using either the encoder or decoder of the 2D VAE to form constraints and explore four types of mapping functions to design regularization. Moreover, improve video VAE efficiency, we investigate its architecture and partially integrate 3D convolutions instead of exploiting 3D convolution in all blocks. The proposed video VAE can be used not only for training new video models with pretrained ones as initialization but also as frame interpolator for existing video models with slight finetuming.
Our main contributions summarized follows: (1) We propose video VAE that provides truly spatio-temporally compressed continuous space for training latent generative video models, which is compatible with existing image and video models, greatly reducing the expense of training or finetuning video models. (2) We propose a latent space regularization to avoid distribution shifts and design an efficient architecture for the video VAE. (3) Extensive experiments are conducted to demonstrate the effectiveness of the proposed video VAE.
2 Related Work
Variational Autoencoder. Variational Autoencoders (VAEs), introduced by [19], have been widely used two-stage generative models. The first stage involves compressing the pixels into a lower- dimensional latent representation, followed by second stage that generates pixels from this latent space. VAEs can be divided into two groups according to the token, i.e., discrete and continuous latent. The difference between the two types of VAEs is the quantization. Continuous VAEs have no quantization, while discrete VAEs learn a codebook for quantization and use to convert the continuous latent features to discrete indices, called VQVAE (32). When training discrete VAEs, some methods exploit a discriminator to improve the visual image quality, called VQGAN [2].
In video generation, 2D VAEs are typically inflated into 3D ones by injecting 3D Conv or temporal attention. 3D Convs are for based VAEs, e.g., 3D-VQVAE [40), 3D-VAQGAN [4 41.
Attentions are for transformer-based VAEs, e.g., CViViT [33. Although there are several discrete 3D VAES for video generation, there are no commonly used continuous 3D VAEs.
Video Generative models. Video generation has achieved remarkable progress in recent years. The announcement of Imagen Video [17] and Make-A-Video (29] made researchers see the hope of purely Al-generated videos. Then, the launch of OpenAl SORA (2 brought the enthusiasm
of researchers in academia and industry to a climax. Many video generation models [17 29
directly learn the distribution of pixels while some others (5 6 44
34
35|
24 14 33
15
learn the distribution of tokens in a latent space. The tokens are always extracted by a variational autoencoder [.]. Latent video generation models can be categorized into two groups according to whether the token is discrete or continuous. TATS 14). MAGVIT 4Л). VideoGPT [40], and Phenaki (33 are representative models trained with discrete tokens extracted by a 3D VAE within the VQVAE framework (32). A codebook is learned jointly with the VAE for quantization. SVD [ Difm [15]. VideoCrafter [6), etc., are video models trained with continuous latent extracted by 2D VAE without quantization, rather than 3D VAE. SD image VAE is the commonly used 2D VAE. One reason is that video models are difficult to train from scratch and they are always initialized with the weights of pretrained T21 model such as Stable Diffusion UNet (28). Hence, the corresponding image VAE is used to extract latents from a video. Since the image VAE can only perform spatial compression, the temporal compression is realized by uniform frame sampling. This strategy ignores the motion between key frames.
There lacks a video VAE that is compatible with the pretrained T2I or video models. Though it is not difficult to train a video VAE (3D VAE) independently with high reconstruction accuracy; will result in a latent space gap between the learned video VAE and existing pre-trained image and video models that are always used as initialization. The Open-Sora- Plan project [I] offers a video VAE; however, it is not compatible with existing image or video models. Large computational resources and a long training time are required to bridge the gap. In this work, we propose a latent space regularization method to train a video VAE whose latent space is compatible with pretrained models.
3

---

Method
A+
Ensader
Decoder
SD Ericader
Z
Dleander
x(X)
20 Encodar
Prifant
Latert
liper
Recordtrsded
Prateire y0 becotee
3D Warriinntte
Mapsing
Dccodor
Resoratructio Loss
CAN LOM
Pubtaine 20 Decader
Regubbiizaton
Regularizaion
Figure 3: (a-b): Two different regularization methods; (c) The framework of CV-VAE with the regularization the pretrained 2D decoder.
We propose latent space regularization method for training a video VAF that is compatible with pre- trained image and video models. We examine multiple strategies for implementing the regularization, focusing on either the encoder or the decoder of the image VAE. Additionally, we explore four types of mapping functions to develop the regularization loss. To enhance the efficiency of the video VAE, we introduce an architecture that employs different inflation strategies in distinct blocks, instead of incorporating 3D convolutions in all blocks.
3.1 Latent Space Regularization
We inflate 2D VAE into a 3D VAE, initializing it with the 2D VAE's weights. The 3D VAE is designed to be capable of encoding both image and video (see details in Sec The key of building a compatible video VAE is the latent space alignment between the video VAE and the image VAE.
Notations. Let I HW denote an image in RGB space and E (T x3 denote a video with T' + 1 frames. When T = 0, X degrades into image and the video VAE will process with temporal padding. 2 E mbxwxe denotes the latent tokens extracted by either an image VAE or a video VAE. Z E (+++))xxxxxe latent tokens extracted by the video VAE. P. = H/h W/w and Pr: T/l are the spatial and temporal compression rates. Let & and D; denote the encoder and decoder of the image VAE, respectively. While &, and D. are for the video VAE. Then, we have z (x), = (X), and = (x). ã = Dz(z) = (x)), X (2) = (X)).. and (=) (E, are the reconstructed image and video from the latent tokens.
Regularization. We assume the latent of the image VAE follow distribution, i.e, 2 ~ p²(2). The joint distribution of + independent frames is p2( ITk +1 p'(z)). ). The latent distribution of the video VAE can be denoted as (Z). To achieve the alignment between the latent spaces of the image and video VAEs, we have to build mappings between (Z) and p" (Z). Since both distributions have no analytic formulation, distance metric for measuring differences between distributions is not applicable.
Here, build the cooperation between the image VAE and the video to construct reconstruction loss for space alignment. When exploiting the encoder of the image VAE for alignment, the latent extracted from the image encoder should be corrected decoded by the decoder of the video VAE, i.e., X (E)(())) The illustration is shown in Fig. a). For a given input video X R(T+1)x xxx3
mapping function v to sample (xX E xxx Thus the reconstructed video X the same the shape X. Then, the reconstruction loss of using the image encoder can be defined as
L
Ligg
X X1112.
(I)

---

When exploiting the decoder of the image VAE, the latent extracted by the video encoder can be decoded by the decoder of the image VAE, i.e., X = (X)). The illustration is shown the reconstructed video is X; E R PT(TPP xx Then, the reconstruction loss of using the image decoder can be defined as
in Fig. b). For given input video X E (R(++)xx**
dh (X) X:112
Mapping Functions. To bridge the dimension gap between X or X and X, we investigate four types of mapping functions as follows. 1) First frame. We compare only the first frame of the input video and the reconstructed one. The regularization loss degenerates to measure the difference between the input and reconstruction of the image. 2) Slice. v samples one frame every Pt frames to form shorter video. It starts from the second frame and the first one is reserved. 3) Average. ei computes the average of every P: frames, starting from the second frame. 4) Random. zú randomly samples one frame from every frames, starting from the second frame.
Training Objective. Following the training of the 2D VAE in LDM 28), our basic objective is a combination of reconstruction loss [43, an adversarial loss 12 and a KL regularization [19] i.e.,
LAE = min max (X,, (e(E. Ludv (X))) + log D. + LKL (X; Ex; Do), D. D.
where the first term is the reconstruction loss, the second and third are the adversarial loss, and the last the KI. regularization. D2 is the discriminator that differentiates original videos from reconstructed ones. is inflated from the image discriminator in LDM by injecting 3D convolutions. Then, for latent space alignment, our full training objective is:
L aligo LAE A I.dee A-
where ê and X2 are trade-off parameters. We explore different settings of a and X2 and find that using the decoder only achieves the best performance. The framework of CV-VAE is shown in Fig. 3( and evaluations between different regularization methods can be found in Tab. 6
3.2
Architecture Design of Video VAE
We design the architecture of the video VAE according to the image VAE LDM 28 The detailed architecture is presented in the Appendix A.I We explain the key modifications as follows.
Model Inflation. Considering the latent space compatibility and the convergence speed of the video VAE, we make full use of the pretrained weights of the image VAE for initialization, instead of training from scratch. We inflate the image VAE into the video VAE by replacing 2D convolutions with 3D convolutions. 3D convolutions are used to model the temporal dynamics among frames. To initialize the 3D convolutions, we copy the weights of the 2D Conv kernel to the corresponding positions in the 3D Conv kernel and set the remaining parameters to zero. We set the stride to achieve temporal downsampling and increase the number of 3D kernels by a factor of to achieve sx temporal upsampling. To enable the video VAE to handle both image and video, given T + 1 frames as input, we use reflection padding in the temporal dimension for the first frame. By initializing the video VAE using the above operations, we can reconstruct images without training, significantly accelerating the training convergence speed on vidco datasets.
Efficient 3D Architecture. Expanding 2D Convs to 3D Convs (e.g., k k k x k) results in k parameters and computational complexity. To improve the computational efficiency of the model, we adopt 2D+3D network structure. Specifically, we retain half of the convolutions in the ResBlock as 2D Convs and set the other half as 3D Convs. We find that, compared to setting all Convs to 3D, the number of parameters and the computational complexity reconstruction performance remains nearly the same. See Sec. 4.2 for experimental comparisons.
reduced by roughly 30%, while the
Temporal Tiling for Arbitrary Video Length Existing image VAEs employ spatial tiling large spatial resolution images to achieve eemory-friendll processing, which cannot handle long vidcos. As result, we introduce temporal tiling processing, During encoding, the video X is divided into [X Xn],, where X; e xx xx33 and is a parameter controlling the size of each
5

---

COCO-Val
Webvid- Val
Method
Params
FCR Comp.
PNSR(()
SSIM(T) LPIPS(C) PNSR(1)
SSIM(() LPIPS(
VAE-SD2.1 1
28
34M + 49M
1x
26.6
0.773
0.127
28.9
0.810
0.145
VQGAN 12
26M 38M
1x
22.7
0.678
0.186
24.6
0.718
0.179
TATS 14
7M 16M
4x
23.4
0.741
0.287
24.1
0.729
0.310
VAE-OSP !
94M +135M
4x
27.0
0.791
0.142
26.7
0.781
0.166
Ours(2D+3D)
68M + 114M
4x
27.6
0.805
0.136
28.5
0.817
0.143
Ours(3D)
100M + 156M
4x
27.7
0.805
0.135
28.6
0.819
0,145
Table 1: Quantitative evaluation image and video reconstruction. FCR represents the frame
compression rate, and Comp. indicates compatibility with existing generative models.
block. X; and X have a one-frame overlap in the temporal dimension. After encoding each X; to obtain Z, we discard the first frame of Zi when i of 0 and concatenate all Z; in the temporal dimension obtain Z. The decoding process is handled similarly to the encoding process. By combining our method with 2D tiling, we can encode videos with arbitrary resolution and length.
4 Experiments
4.1 Experimental Setups
Datasets and Metrics. We evaluate our CV-VAE on the COC02017 21 validation dataset and the Webvid [3] validation dataset which includes 1024 videos. Both images and videos are resized and cropped to resolution of 256 x 256. Each video is sampled with 33 frames and frame stride of 3. We evaluate the reconstruction performance of CV-VAE on images and videos using metrics such as PSNR, SSIM [36]. and LPIPS scores 43). We employ 3D tiled processing to encode and decode videos with arbitrary resolution and length within limited memory footprint. During inference, we allow a single video block size of 17 x 576 x 576. We evaluate the video generation quality of our model using 2048 randomly sampled videos from UCF101 (30 and MSR- (39] Videos are resized and cropped to resolution of 576 x 1024 to fit the SVD [4]. We use Frechet Video Distance
(FVD) 31, Kernel Video Distance (KVD) 31 and Perceptual Input Conformity (PIC) 38 metrics to evaluate video generation quality. For evaluating image generation quality, we use 2048 samples from the COC02017 validation dataset and employ FID [16]. CLIP score (26] and PIC score metrics.
Training Details. We train our CV-VAE model using image datasets including LAION-COCO [9] and Unsplash 233, as well as the video dataset Webvid- 10M (3. For image datasets, we employ two resolutions, i.e., 256 x 256 and 512 x 512. In the case of video datasets, we use two settings of frames and resolutions: x 256 x 256 and 17 x 192 x 192. The batch sizes for these four settings are 8, 2, 1, and 1, with sampling ratios of 40%. 10%, 25%, and 25%, respectively. We employed the AdamW optimizer 22 with a learning rate of le-4 and cosine learning rate decay. To avoid numerical overflow, we trained CV-VAE using float32 precision, and the training was carried out on A100 GPUs for 200K steps. To fine-tune the SVD on CV-VAE, we utilize in-house data with frame rate and resolution of 97 576 x 1024. We employ deepspeed stage 2 [27), gradient checkpointing [8] techniques, and train with bíloat precision. We used a constant learning rate of le-5 with the AdamW [22) optimizer, and only optimized the last layer of U-Net. The training was carried out on 16 A10 GPUs for 5K steps.
4.2 Image and Video Reconstruction
We evaluated the reconstruction quality of various VAE models on image and video test sets. The comparison group includes: (1) VAE-SD2. (28) which is widely used in the community for image
and video generation models. (2) VQGAN 12 which encoding pixels into discrete latents. We use the f8-8192 version for comparision. (3) TATS [4: 3D VQGAN designed for video generation. (4) VAE-OSP M: a 3D VAE from Open-Sora- Plan which is initialized from VAE-SD2. and trained with video data. (5) Our CV-VAE (2D+3D): retains half of the 2D convolutions to reduce computational overhead. (6) Our CV-VAE (3D): utilizes only 3D convolutions.
As illustrated in Tab. I we present the parameter count (Params), Frame Compression Ratio (FCR), and compatibility with existing diffusion models (Comp.) for various VAE models. Thanks to the latent constraint, our model is compatible with current diffusion models, compresses videos by 4x in the temporal dimension, and achieves top-tier image and video reconstruction quality. This enables the generation of longer vidcos under roughly the same computational resources. Reconstruction

---

eihh Tyse
SELISS
Groundtruth
VQGAN
VAB-SD2.1
CV-VAB(Ours)
Groundrruth
VAE-OSP
CV-VAE(OUrs)
Figure 4: Qualitative comparison of image and video reconstruction. Top: Reconstruction with differ- ent Image VAE models (i.e., VQGAN [2] and VAE-SD2.1 (28) on images; Bottom: Reconstruction with different Video VAE models (t.e., TATS [4 and VAE-OSP T video frames.
liot, coloctul, low-polly and orange poly poly mesh, biccy.. composition
freshly made hot floral nea glass kettle table, angled shot, midday Nikon 105mm, up
Chitered house in the woods, snime, painting. cortagecone, ghibli inqiredd,
portrait pritly blonde wonton, crown, cartbyy makeup, sunsct or salure scenc, green and gold color scheme
latlio design, hird, minimalistic, black while drawing, detailed, Sk.
cat under the snow with blue eyes, covered by snow, cinconatic style, mediuu shot, professionll photo, zaimal
Figure 5: Text-to-image generation comparison. In each pair, the left is generated by the SD2. with the image VAE while the right is generated by the SD2.1 with our video VAE.
quality improves as the number of latent channels increases. For comparison results with 16 latent channels, please refer Appendx A.2
We also conducted a qualitative comparison the reconstruction results for different VAE models, shown in Fig. 4 In the top row, we reconstructed images with resolution of 512 512 and compared them with Image VAE models. All three models compressed the images to latent size of 64 64. Our results were close those of VAE-SD2.1, while VQGAN had the worst performance. In the bottom row, we reconstructed videos with a resolution of 33 x 512 x 512 and compared them with Video VAE models. All three models compressed the videos latent size of 9 64 x 64. Comparing the decoded videos at the same frames, our model achieved the best results. Check Appendx and A.4 for more reconstruction results.
and
4.3 Compatibility with Existing Models
Text-to-Image Models We tested the compatibility of our CV-VAE by integrating it into the pretrained SD2.1 (28), replacing the original 2D VAE without any finetuning. We evaluated it

---

COCO2017-Val
Trainable
FID(J)
CLIP(L)
PIC(T)
SD2.1 [28]
X
57.3
0.312
0.354
SD2. ICCV
X
57.6
0.311
0.360
Table 2:
Quantitative results of text-to-image generation.
Method
Trainable
FCR
Frames
UCF-101
MSR-VTT
FVD(J)
KVD(I)
PIC(T)
FVD(J)
KVD())
PIC(T)
SVD [4|
X
1x
25
402
8.20
0.791
310
1.30
0.588
SVD+CV-VAEJ
x
1x
25
419
6.73
0.763
262
1.67
0.609
SVD
4x
97
762
I5.7
0.791
"39
3.31
0.696
VVD-CV-VAEVV Output layer
4x
681
13.1
0.858
295
2.26
0.734
Table 3: Evaluation results of image-to- generation. FCR denotes the frame compression rate.
the COCO- Val [2]] dataset and compared the results with the SD2.1 I model using PID, CLIP score, and PIC metrics. The data (see Tab. 2 suggest that both models perform similarly in text-to-image generation.
We also visualized the text-to-image generation results of both models in Fig. In each pair, the left side depicts the results of SD2.1, while the right side shows the results generated by our CV-VAE, which replaced the original VAE, using the same random seed and prompt. The results show that both models generate images with almost identical content and texture, with only slight differences in color. This further validates the feasibility of building compatible VAE via latent constraint.
Image-to- Video Models The primary objective of CV-VAE is to train a model that can compress both time and space, while also being compatible with the existing 2D VAE. In this section, we validate the compatibility of CV-VAE with existing video generation models. We integrate CV-VAE into SVD [4], replacing the original VAE, and decoded the generated video latents. CV-VAE offers the flexibility to decode either in image mode (CV-VAE-I) or video mode (CV- the former decodes n frames of latent into n frames of video, while the latter decodes n frames of latent into 1 1 (n 1) x frames of video. We tested the video generation quality of both models. Furthermore, we fine-tuning the SVD for better alignment.
As shown in Tab. incorporating 'CV-VAE-I' into frozen SVD immediately yields video generation quality comparable to the original VAE. Using CV-VAE in video mode can also decode videos generated by SVD, and further improvements in video decoding quality can be achieved by fine- tuning only the output layer (approximately 12k parameters). One of the reasons for the noticeable gap test metries between WUD+CV-VVE-J and VDD--VVVVA-VV is that they use different numbers of frames, making a direct comparison challenging.
In Fig. d we also display the comparison results with SVD [4] The top row shows the generated results by SVD, and the bottom row shows the generated results after inserting CV-VAE into SVD and fine-tuning the output layer. We use the first frame as a condition and generate with the same random seed. The U-Net generates 25 frames of latent, which are decoded by CV-VAE into a 97-frame video. As can be seen, compared to the original SVD, our results exhibit smoother motion. It is worth noting that both models have the same computational complexity during the diffusion process, which means that our model is more scalable.
By fine- tne-tunnng a small number of parameters, the image-to-vides model can generate smoother and longer videos through CV-VAE, effectively serving as frame interpolation method. Therefore, we compared CV-VAE with existing video interpolation models [8]. conducting experiments on MSR-VTT [39] We first used SVD to generate a video of size 25 x 576 x 1024, and then applied the interpolation model to expand the video from 25 frames to 97 frames. For our results, we directly generated video of size 97 576 x 1024 using CV-VAE and the fine-tuned SVD. The comparison results are shown in Tab. 4.3 where CV-VAE outperformed the FIRE [8] in two out of three metric, validating the potential of VAE as an interpolation model.
Text-to- Video Models In this section, we integrate CV-VAE into the existing text-to-video model to futher validate the effectiveness. 'VC2' refers to decoding the results generated by VideoCrafter2 using the original 2D VAE, VVC2+CV-VAE-P indicates decoding the results using CV- in image mode, and VVC++VVVAABV denotes decoding the results using CV-VAE in video mode, which generates vidcos of 4x frames. We only fine-tuned small number of parameters in U-Net, including
8

---

SVD
SVD + CV-VAE
SVD
SVD + CV-VAE
Figure 6:
Comparison between the image VAE and our video VAE -mage-to-video generation of SVD []. 'SVD' means using the image VAE. 'SVD + CV-VAE' means using our video VAE and tuning the output layer of SVD. Click to play the video clips with Adobe Foxit PDF Reader.
Method SVD+RIFE 18
FVD())
KVD(L)
PIC(t)
253
3.12
0.721
SVD+CV-VAE
295
2.26
0.734
Table 4: Comparison between CV- and frame interpolation model.
the first and last layers. We use captions from the validation set of MSR-VT -TT 39) for evaluation, with the resolution of 320x512. Following the approach taken by previous studies [37], used the CLIP [26] metric to evaluate the generation quality of text-to-vidco models, including Frame Consistency (F.C.) and Textual Alignment (T.A.). The experimental results are shown in Tab. 4.3
where the VC2+CV-VAE-V setting achieved the best generation performance through fine-tuning VideoCrafter2. Check Appendx A.5 for quantitative comparison.
4.4 Ablation Study
Influence of Regularization Type which are: (1) 2D Enc. i.e., X and A2 = in Eq 3 (2) 2D Dec. i.e., A1 and X in
We evaluated the impact of three types of latent regularization,
Eq. (3) 2D Enc. + Dec. i.e., A and X = 1 in Eq
CLIP Metric
Methods
FCR
Trainable
F.C.
T.A.
VC2 7
1x
X
0.292
0.960
VC2+CV-VAE.I
1x
X
0.290
0.964
VC2+CV-VAE-V
4x
X
0.285
0.953
VC2+CV-VAE-V
4x
V
0.301
0.987
Table 5; Evaluation results of text-to-video generation.

---

Constraint
COCO-Val
Webvid-Va Val
PNSR(1)
SSIM(T)
LPIPS(C)
PNSR(1)
SSIM(1)
LPIPS(L)
2D Enc.
26.0
0.759
0.205
26.0
0.748
0.222
2D Dec.
27.5
0.801
0.151
28.0
0,803
0.158
2D Dnc. Dec.
27.9
0.808
0.150
27.6
0.795
0.176
Table 6: Comparison of different regularization types.
Mapping
COCO-WA Val
Webvid Val
Function
PNSR(1)
SSIM(1)
LPIPS(()
PNSR(1)
SSIM(t)
LPIPS(()
1st Frame
27.3
0.797
0.156
26.6
0.771
0.191
Average
27.5
0.801
0.151
27.9
0.801
0.172
Slice
27.5
0,802
0.152
27.7
0.799
0.168
Random
27.6
0.803
0.138
28.4
0.811
0.153
Table Comparison of different mapping functions.
Tab. 6Jshows the impact of various latent regularization methods. Using the 21) decoder for latent regularization results in better reconstruction for both image and video test sets compared to the 2D encoder. This likely because the gradient backpropagation through the 2D decoder provides better guidance for the 3D VAE's learning, while the frozen 2D encoder doesn't propagate gradients. The '2D Enc. + Dec.' method performs slightly better on image test sets but worse on video datasets compared to '2D Enc.' Since our main goal is video reconstruction and for simplicity, we use the 2D decoder for regularization.
Influence of Mapping Functions The 2D decoder decodes frames of latents into n frames of video, while the 3D decoder decodes the same n frames of latents into 1 + (n 1) x 4 frames of video. Therefore, we need to mapping the input video n frames calculate the regularization loss in Eq. We evaluated four mapping functions mentioned in Sec. 3.1
As shown in Tab. 7 the four methods have similar effects image reconstruction, with the main differences being in video reconstruction. The 'Ist Frame' approach yields the worst video reconstruction results due the lack of regularization and guidance for subsequent frames. The 'Slice' method results in poor reconstruction quality for the three unsampled middle frames. The 'Average' method is inferior 'Random' in video reconstruction, primarily because calculating the mean for multiple consecutive frames leads to motion blur in the target.
Conclusion and Limitations
We propose novel method to train video VAE that is compatible with existing image and video models trained with SD image VAE. The video VAE provides truly tpaporltemporrll compressed latent space for latent generative video models, as opposed to uniform frame sampling. Due to the latent space compatibility, new video model can trained efficiently with the pretrained image or video models as initialization. Besides, existing video models such as SVD can generate smoother videos with four times more frame using our video VAE by slightly fine-tuning few parameters. Extensive experiments are performed to demonstrate the effectiveness of the proposed VAE.
Limitations. The performance of the proposed video VAE relies on the channel dimension of the latent space. A higher dimension may yield better reconstruction accuracy. Since we pursue the latent space compatibility with existing image and video models trained with SD image VAE, the channel dimension of our video VAB is limited to be the same as the image VAE. This can be improved if an image VAE with a higher channel dimension becomes available, egg., the VAE of SD3 [3]
10

---

References
[1]
Open-sora-plan. Accessed May 15, 2024 [Online].
URL
PKU- T
[2] Openai sora. Accessed May 13, 2024 [Online]. URL https://openai.com/
video -etereeiinnnnnneeeeeeee -wwwwrrldd soorld-simme simulators/
[3] M. Bain, Nagrani, G. Varol, and Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
[4] A. Blattmann, T. Dockhom, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231..15127 2023.
[5] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563-22575, 2023.
[6] H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, Xing, Y. Liu, Q. Chen, X. Wang, al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023.
[7] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang. C. Weng, and Y. Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047, 2024.
[8] T. Chen, B. Xu, Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604.660747444 2016.
[9] S. Christoph, K. Andreas, V. Richard, C. Theo, and B. Romain. Laion-coco: 600m synthetic captions from laion2b-en, 2022. httpps//s// /ts/blloo/laaooncccooo/
[10] X. Dai, J. Hou, C.-Y. Ma, S. Tsai, J. Wang, R. Wang, P. Zhang, S. Vandenhende, X. Wang, A. Dubey, et al. Emu: Enhancing image generation models using photogenic needles a haystack, arXiv preprint 23X 2023.
[11] C. Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05900 2016.
[12] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[13] Esser, S. Kulal, Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.088 032006, 2024.
[14] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision, pages 102-118. Springer, 2022.
[15] Y. Guo, C. Yang, A. Rao, Y. Wang. Y. Qiao, D. Lin, and B. Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
[16] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[17] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint ar 22Xiv:2210.023000 2022.
11

---

[18] Z. Huang, T. Zhang. W. Heng, B. Shi, and S. Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision, pages 624 642. Springer, 2022.
[19] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.
[20] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, R. Hornung, H. Adam, H. Akbari, Y. Alon, V. Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14122, 2023.
[21] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[22] I. Loshchilov and F. Hutter. arXiv:1711.05101, 2017.
Decoupled weight decay regularization. arXiv preprint
[23] A. Z. Luke Chesser Timothy Carbone. Unsplash. http://giinbb.cmmuussllllaaaaat
2023.
[24] X. Ma, Y. Wang, G. Jia, X. Chen, Z. Liu, Y.-F Li, C. Chen, and Y. Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint urXiv:2401.0304 2024.
[25] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
[26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PMLR, 2021.
[27] S. Rajbhandari,, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing. Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
(28) R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[29] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
[30] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint
2012.
[31] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly. To- wards accurate generative models of video; A new metric & challenges. arXiv preprint 1rXiv:1812.011717, 2018
[32] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[33] R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
[34] J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang. Modelscope text-to-video technical report. arXiv preprint 230.:2308..0571, 2023.
[35] Y. Wang. X. Chen, X. Ma, S. Zhou, Z. Huang, Y. Wang, C. Yang, Y. He, J. Yu, P. Yang, et al. Lavie: High-quality video generation with eascaded latent diffusion models. arXiv preprint arXiv:2309.15/00 2023.
12

---

[36] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600-6) 612, 2004.
[37] Z. Wu, Y. Ge, Wang. S. W. Lei, Y. Gu, Y. Shi, Hsu, Y. Shan, X. Qie, and M. Z. Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023.
[38] J. Xing, M. Xia, Y. Zhang, H. Chen, X. Wang, T.-T. Wong, and Y. Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprini urXiv:2310.12190, 2023.
[39] Xu, T. Mei, Yao, and Y. Rui. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288 5296, 2016.
[40] W. Yan, Y. Zhang, Abbeel, and Srinivas. Videogpt: Video generation using vq-vae and transformers, arXiv preprint arXiv:2104.1011 2021.
[41] L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann. M.-H. Yang, Hao, I. Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEF/CVF Conference
Computer Vision and Pattern Recognition, pages 10459 10469, 2023.
[42] D. Zhang, J. Z. Wu, J.-W. Liu, R. Zhao, L. Ran, Y. Gu, D. Gao, and M. Z. Shou. Show- 1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818, 2023.
[43] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
[44] D. Zhou, W. Wang, H. Yan, W. Lv, Zhu, and Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 2022.
13

---

CV-VAE
Discriminator
Encoder
Decoder
Conv 3x3x3, 3->64, stride 2x2x2
Conv 3x3x3, 3->128
Conv 3x3x3, 128->3
ResBlock
64->128
2x
ResBlock 128
2x
ResBlock 128
Downsample 3x3x3,
ResBlock 256->128
ResBlock
128->128, 2x2x2
128->256
Upsample 3x3x3,
ResBlock 128->256
256->512, stride 1x1x1
ResBlock
256->512
ResBlock 256
2x
ResBlock 256
ResBlock 512->512
Downsample 3x3x3,
ResBlock 512->256
256->256, stride 1x2x2
ResBlock 256->512
UPsample 3x3x3, 512->512, stride 1x1x1
Conv 3x3x3, 512->1
ResBlock
3x
ResBlock 512
Downsample 3x3x3,
Upsample 3x3x3, 512->1024, stride 1x1x1
512->512, stride 2x2x2
Cony 3xdod, X-aV
Conv 1xlx1
X->Y
Conv Bx3x3
3X
ResBlock 512
ResBlock 512
ResBlock X->
Attention
Attention
ResBlock 512
ResBlock 512
Conv 3x3xx
3x3x3, 512->8
3x3x3,
Conv 3x3x3
3x3
Sample
ResBlock
Figure 7: Architecture of CV-VAE.
A Appendix
A.1 CV-VAE Model Architecture
As illustrated in Fig. 7 we introduce the structure of the CV-VAE. The architecture of CV-VAE is primarily derived from the VAE in Stable Diffusion (28), with several notable differences: (1) Some or all 2D convolutions within the network are transformed into 3D convolutions, while retaining their weights. (2) Temporal downsampling executed the encoder through the use strides. (3) Temporal upsampling is accomplished by increasing the output channel number of 3D convolutions by a specific factor. (4) A discriminator, comprising 3D convolutions, is utilized. The main differences
marked in red text in Fig.
14

---

COCO-Val
Webvid-Val Val
Method
Params
FCR Comp. PNSR(() SSIM(T) (PIPS( PNSR(1)
SSIM(T) LPIPS(
VAE-SD3 [HE
34M + 49M
1x
32.6
0.899
0.064
33.4
0.910
0.064
CVVAE-SD3
68M 114M
4x
33.6
0.916
0.072
32.0
0.899
0.087
Table 8: Quantitative evaluation image and video reconstruction between. FCR represents the frame compression rate, and Comp. indicates compatibility with existing generative models.
A.2 CV-VAE with more latent channels
More latent channels generally lead to better reconstruction performance in VAEs 10 which is crucial for image and video editing tasks. In this subsection, I conducted experiments using VAE with more latent channels (z=16). We used the 2D VAE from SD3 [33] as the baseline and trained the CVVAE-SD: based on latent regularization. Testing was conducted under the same settings as in Tab. | The comparison results are shown in Tab. 8 CVVAE-SD3 outperforms VAE-SD3 in image reconstruction but is inferior to VAE-SD3 in video reconstruction. The main reason is that CVVAE-SD3 has a higher compression ratio in video compression, resulting the loss more information, though its reconstruction quality is significantly higher than that of VAE with z=4.
A.3 Qualitative Examples of Image Reconstruction
In Fig. 8 we showcase additional image reconstruction results using CV-VAE. use the version of '2D + 3D These images are sourced from the COCO2017 (2] dataset with a resolution of 512 x 512. The reconstructed image precisely shares the same colors and textures as the original, demonstrating the high fidelity of our CV-VAE in encoding and reconstructing images. Interestingly, in Fig. 5 slight color differences can be observed between the images decoded by the Image VAE and CV- VAE, given the same latent generated by the Image Diffusion Model. This suggests that there is still minor discrepancy between the latent spaces the video VAE trained with latent regularization and the Diffusion Model. This gap can be bridged with minimal additional training.
A.4 Qualitative Examples of Video Reconstruction
As shown in Fig. 9 present the reconstruction results of 4 consecutive frames from a video clip (33 x 576 x 1024) using CA-VAE. The reconstructed video frames maintain consistency in color, structure, and motion with the ground truth. According CA-VAE, these continuous frames condensed into single latent frame, signifying that even a single latent frame encapsulates motion information.
A.5 Compatibility with Existing Text-to-video Model
We tested the compatibility of CV-VAE with existing text-to-video diffusion models, such as Videocrafter2 77 which also employs 2D VAE from the SD as its first-stage model. We adopted strategy similar to the training of 'SVD CV-VAE in Sec. [.3 by fine-tuning the last layer of the U-Net in VC2 to adapt to CV-VAE. We finetuned the model using in-house data at resolution of 61 x 320 x 512, which is cquivalent to latent size of 16 40 x 64.
As shown in Fig. 10 compared to the original VC2, the *VC2 + CV-VAE generates videos ap- proximately four times longer, resulting in smoother motion. This further validates the feasibility of obtaining a compatible video VAE through latent regularization, thereby avoiding the massive computational power required to train a video diffusion model from scratch.
B
Society Impacts
The CV-VAE can be seamlessly integrated into existing diffusion models, replacing the original 2D VAE for image or video generation, which may result in potential societal implications. While proves beneficial in fields such entertainment and advertising, by providing more realistic and immersive content, also raises ethical and safety concerns. The ease of generating high-quality synthetic images and videos could lead to surge in the production of harmful or misleading content, such as deepfakes, potentially exacerbating issues of misinformation and privacy invasion. We condemn the misuse of generative AI that harms individuals or spreads misinformation.
15

---

Real
Reconstructed
Real
Reconstructed
STOP
Figure 8: Our CV-VAE is capable of encoding and reconstructing images with high fidelity.
16

---

1st Frame
2nd Frame
3rd Frame
4th Frame
Real
Reconstructed
Real
Reconstructed
Real
Reconstructed
Real
Reconstructed
Figure 9: Reconstruction results consecutive frames using CV-VAE.
17

---

1st Frame
2nd Frame
3rd Frame
4th Frame
5th Frame
VC2
VC2+CV-VAE
Prompt: pianist playing somber music, abstract style, non-repoeee non-representational colors shapes, expression of feelings, highly detailed
VC2
VC2-CV-VAE
Prompt: a girl astronaut exploring the cosmos, floating among plancts stars, high quality detail
VC2
VC2 -V--CVVVE
Prompt: A professional photograph of astronaut riding horse
VC2
VC2 CV-VVE
Prompt: black dress and sunglasses with white hair, in the style of becky cloonan, ross tran, dark white and cyan, close up, tsubasa nakai, light black and black, bold outlines, flat colors
Figure 10: Comparison between the image VAE and our video VAE on text-to-video generation of VC2 [7]. We fine-tuned the last layer of U-Net in VC2 to adapt it CV-VAE. VC2 generates vidcos with resolution x 320 512, while the 'VC2 + CV-VAE' produces videos of x 320 x 512 resolution under the same computation. The missing frames in the VC2 results marked in gray.
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose a video VAE for latent generative video models. The Method sec- tion shows the technical contribution of the design and training of the VAE. The Experiment part provides comprehensive experiments to demonstrate the claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
We use paragraph to discuss the limitations in the manuscript.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The method part and experiment settings show the design of the model as well the parameters for training.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Code and checkpoints will be released upon the acceptance of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
stteeeeuuumssiioooooooooc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/tpps.cc/public//uue
CodeeSumiiii wwabbbmmsssiinni ssionpolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
21

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes] Justification: Details are in the experiment settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We use the commonly used protocol and metrics for evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are in the experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
We follow the instruction.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are in the supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the corresponding papers of the used assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
24

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "z45wWFLOrz", "reviewer": "Reviewer_xmxT", "summary": "This paper focuses on a compact video VAE suitable for both image and video generation tasks. The motivation stems from the absence of spatial-temporal compressed continuous latent 3D VAE models for video generation. To effectively utilize 2D VAE and seamlessly integrate image generation models, the authors propose a latent space regularization and initialization method to maximize the use of 2D VAE. A 3D CNN-based architecture is also proposed to compress both temporal and spatial dimensions efficiently. The evaluation of the proposed method is primarily based on reconstruction tasks and image/video generation tasks.", "questions": "The term \"mapping function\" seems to require clarification, as it appears to be more of a simple sampling process from video frames. In Table 7, a higher CLIP score is better. Why does random sampling work best? Providing more exploration and explanation can help readers understand this better.", "limitations": "Yes, the author discussed the limitations and potential social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation is clear: the current video generation method may suffer from a lack of high-quality and high-efficiency spatial-temporal VAE if it can be used with the existing T2I model, which also makes it more general and effective. \nThe decoder loss seems a simple and novel design, which helps the model learn spatial latent efficiently and align with the original 2D VAE. Initialization of 3D VAE  with 2D ones seems to provide some insights in practice. \nThe qualitative results are good.", "weaknesses": "1. Improvements over the baseline are presented in Table 1, where the authors compare the main reconstruction results with different open-sourced VAE methods. However, I don’t think it is a fair comparison to claim superior performance. The training data is a key factor in this table. The authors use in-domain data, such as Webvid, for both training and evaluation, whereas other methods do not. Therefore, it is unclear if the proposed method can truly outperform other methods when trained on the same data.\n\n2. Many generation models have been released based on SD VAE. Since compactness does not require further fine-tuning, it is crucial to evaluate more models to strengthen the claim. Evaluating only one model in Table 2/3 is not convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:  Fairness of training and evaluation data**\n\nThank you for your comments on the fairness. **(1)** Unified training data. Since the VAE models in Table 1 are trained on different datasets and some of them did not release their training and dataset details, such as VAE-OSP, it is difficult to train all models on the same dataset. **(2)** Unified testing data. We appreciate you pointing out that evaluating on the in-domain validation set is unfair. Therefore, we conduct additional evaluations of all VAE models on the publicly available MSR-VTT validation set that is not mentioned as the training set of those models. The evaluation results are presented in the attached file **(Table 1)**. Our method performs better than other video VAE models. \n\n**W2: Evaluation on different diffusion models**\n\n**(1)** We would like to first clarify that we have performed compatibility validation on multiple diffusion models, including quantitative evaluations on SD2.1 (text-to-image), SVD (image-to-video) (Table 1, 2), and visual evaluations on SD2.1, SVD, and Videocrafter2 (text-to-video) (Figure 5, 6, 10). **(2)** To further validate our compatibility with text-to-video models, we have additionally conducted quantitative evaluations on Videocrafter2+CV-VAE and also provided visual results in the attached file **(Table 4, Figure 2)**.\n\n**Q1:  Clarification of \"mapping function\"**\n\nThank you for your comments. We will revise \"mapping function\" to \"sampling function\" to clarify the expression. We observed that the \"1st Frame\" results in poorer reconstruction of subsequent frames due to the lack of constraints on other frames, \"Average\" leads to more motion blur in the reconstructed frames, and \"Slice\" causes frames without 2D Decoder constraints to be more prone to artifacts. \"Random\" is proposed based on \"Slice\" to cover the content in the whole sequence and uses the randomly sampled frames instead of the average frame in a slice to avoid motion blur.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JvqA11gASq", "reviewer": "Reviewer_tAbR", "summary": "This paper proposes a new video VAE, starting with pretrained image VAE. It includes several techniques that are still capable of handling images and do not suffer from significant computational overhead. First, they use pretrained weights of 2D VAE (from Stable diffusion) by using their weights as initialization with model inflation. They also used efficient 3D architecture to only half of 2D convolutions as 3D convolutions to minimize the computation increase and also applied temporal tiling to handle long videos. They also use a pretrained frozen image decoder to regularize the latent space from the 3D encoder/encoder to ensure the latent space becomes similar to the stable diffusion latent space. The paper shows the proposed video encoder can have a compact latent space with a temporal compression factor (of 4) but is very similar to pretrained image latent space, which can be used for longer and smoother video generation by finetuning existing latent video diffusion models in this latent space.", "questions": "- The paper only provides video generation results on image-to-video generation tasks; I wonder if one naturally achieves more smooth video generation model by fine-tuning existing text-to-video generation models in this latent space (such as modelscope). \n- Are you planning to release the code and the model parameters?", "limitations": "The paper addresses the limitation appropriately.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is generally well-written and well-motivated. In particular, the paper tackles an important problem of constructing a compact latent space of videos that is similar to image space.\n- The performance drop from fine-tuning does not seem that large and shows a good potential for training a latent video generation model.\n- The paper shows a real use-case of this VAE by finetuning stable video diffusion.", "weaknesses": "- The paper lacks a comparison with recent video autoencoders. Specifically, the baselines that the paper provided are quite outdated (e.g., TATS and VQGAN), considering that there are many recent attempts to design a better video autoencoder, to name a few [1, 2, 3]. The authors should discuss what's the pros and cons of this approach compared with these approaches, and if possible, the authors should compare the performance as well. In particular, [2] provides very similar intuition to this paper because they also try to construct a latent space capable of jointly handling images and videos. \n- Lack of novelty: The proposed method is mainly composed of components that have already been widely used to extend image models to videos.\n- The paper does not mention the total training time (even though it includes the number of GPUs and the total number of training iterations).\n\n[1] Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.  \n[2] Language Model Beats Diffusion: Tokenizer is key to visual generation, ICLR 2024.  \n[3] Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition, ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Comparison between recent video autoencoders**\n\n**(1)** There might be a missunderstanding. We have compared with the latest Video VAE in Table 1: VAE-OSP, which was released in May 2024. (2)  MAGVIT2 [2] and CMD [3] are SOTA methods, but they have not released their weights. Following the suggestion of the reviewer to compare with more recent Video VAEs, we add new quantitative and qualitative comparisons of Open-MAGVIT2 [4] (released in July 2024) and PVDM[1] in the attached file **(Table 2, Figure 1)**. Open-MAGVIT2 is an opensource project to replicate MAGVIT2. \n\n\n[1] Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023.\n\n[2] Language Model Beats Diffusion: Tokenizer is key to visual generation, ICLR 2024.\n\n[3] Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition, ICLR 2024.\n\n[4] Open-MAGVIT2: Democratizing Autoregressive Visual Generation, 2024.\n\n\n**W2: Lack of novelty**\n\nWe would like to claim that the major contribution of this work is the framework (latent regularization with 2D decoder) to obtain a video VAE from a pretrained image VAE, which is compatible with existing image and video diffusion models, rather than a novel network component or a loss.  The compatibility greatly saves additional efforts of training diffusion models to adapt the VAE and being compatible with a wide range of community models. For example, Open-Sora-Plan, based on the incompatible Video VAE, spent an additional ~7138 GPU hours to obtain the image diffusion model. \n\n\n**W3: Training Time**\n\n Training the CV-VAE took approximately 800 A100 GPU hours. We will clarify this in the final version.\n \n \n **Q1: Results of Text-to-video diffusion model**\n \nThanks for your concern. **(1)** We have provided the visual results of CV-VAE + VideoCrafter2 (text-to-video) in Figure 10 and the supplementary material. **(2)** We also provide the qualitative results and more visualizations in the attached file **(Table 4, Figure 2)**.\n\n\n**Q2： Release of code and weights**\n\nWe will make the code and weights publicly available to promote community development. Following the rules of response, we have submitted a preview version of the open-source code to the Area Chair.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwTzjEOzTj", "reviewer": "Reviewer_U7Gm", "summary": "This paper designs a 3D VAE consistent with the 2D VAE, whose output mode can losslessly switch between 2D and 3D while retaining the characteristics of both the 2D VAE and 3D VAE. This allows for obtaining performance similar to the original 2D VAE while increasing the frame count of the 3D output.", "questions": "1. Would training the VAE with bf16 cause a performance collapse?\n\n2. In Lines 218-219, how are the images of 256 × 256 and 512 × 512, as well as the videos of 9 × 256 × 256 and 17 × 192 × 192 organized during the training cycle? Do you train on images first, then videos, or start with low resolution and then move to high resolution?\n\n3. What impact does training the VAE at low resolution or low frame rate have when performing SVD on high resolution and high frame rate?", "limitations": "Author added a limitation section in paper and I think there is no obvious potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Designing a 3D VAE with 2D latent consistent characteristics is very meaningful, and this paper successfully achieves this.\n\n2. The bidirectional VAE alignment method designed in this paper is very clever. A similar design is found in [1], but the scenario used in this paper is completely different.\n\n3. Achieving effective performance in both 2D and 3D simultaneously is of high practical value to the community, for example, supporting pre-training and frame interpolation in video Diffusion models.\n\n4. The writing is fluent, and the experimental design is excellent.\n\n[1] Christensen, Anders, et al. \"Image-free classifier injection for zero-shot classification.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "weaknesses": "1. Figure 10 attempts to demonstrate the effect of the 3D VAE on frame count enhancement. I understand that the conclusion of this figure is actually the most critical value proof of this paper: retaining the 2D VAE latent consistency while having the embedding capability of 3D features, such as extending the frame count. This is quite challenging, as the 3D VAE might tend to completely collapse to perform similarly to the original 2D VAE, simply replicating the 2D results. The loss between replicated 2D frames is very small, making optimization difficult. While the distance of the horse's legs in Figure 10 might prove it's not direct replication, the other two cases are hard to distinguish differences. Can the authors provide more evidence to explain that the model is not just learning simple replication?\n\n2. In Table 1, when comparing, which pretrained 2D VAE is aligned with \"Ours\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1： More examples to explain the model is not just learning simple replication**\n\nThank you for your suggestion. **(1)** Due to the size limitation of the paper (50MB), we did not include the text-to-video results as videos in the pdf. The visual results were provied in the supplementary material. **(2)** We alslo provide more visual results of VideoCrafter2 + CV-VAE and quantitative metrics in the attached file **(Table 4, Figure 2)**. It can be observed that the motion between frames is smooth, and it is not simply a duplication.\n\n**W2:  Which pretrained 2D VAE is aligned with \"Ours\" in Table 1?**\n\nThank you for your comments. In Table 1, both of our models (2D+3D, 3D) are aligned and compatible with VAE-SD2.1. Our models are also aligned with a lot of open-source diffusion models in the community that use VAE-SD2.1 as the auto-encoder.\n\n**Q1: Would training the VAE with bf16 cause a performance collapse?**\n\nYes, training the model with either bf16 or fp16 would result in numerical collapse (NaN), which might be due to the instability caused by GAN loss. Therefore, CV-VAE is trained with fp32.\n\n\n**Q2:  Jointing training between images and videos**\n\nWe use four settings for joint training: 256 × 256 images, 512x512 images, 9 × 256 × 256 video clips, and 17 × 192 × 192 video clips. In each iteration, a batch from one of the four settings is generated with different batch sizes (8, 2, 1, 1), allowing samples of different resolutions to be trained simultaneously.\n\n\n**Q3： Performing SVD on high resolution and high frame rate**\n\nSince CV-VAE is composed of convolutional networks and we train it on different resolutions and frame rates, CV-VAE can adapt to higher resolutions and different frame rates with a negligible performance degradation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VekWXaRYqW", "reviewer": "Reviewer_LvuQ", "summary": "The paper introduces CV-VAE, a 3D VAE that is trained through latent space regularization using an existing two-dimensional VAE decoder. This design facilitates seamless integration with 2D VAE-based diffusion models. The concept, while simple and straightforward, is notably efficacious and offers practical utility. Moreover, relative to conventional 2D VAEs, the proposed framework significantly compresses the latent space dimensionality by integrating temporal downsampling within the 3D VAE architecture.", "questions": "1. Please refer to weakness.\n\n2. In Table 3 and figure 6, authors may also present the metrics of SVD on 97 frames straightly or by using the frame interpolation model of SVD.\n\n3. Authors may specify whether the decoder-based regularization is still employed when training with images as inputs for CV-VAE.\n\n4. Authors may also present metrics comparison between SVD and SVD + CV-VAE on text-to-video generation.\n\n5. Can the method be supported by theoretical proof? For example, can we minimize the distribution distance between video latents of CV-VAE and one of corresponding image latents encoded by 2d VAE?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is novel and delicate, which effectively compresses video latents and maintains latent distribution.  \n\n2. Experiments and visualizations demonstrate that CV-VAE performs equally to SD VAE on image-based/frame-based encoding and falls marginally behind on video encoding with little cost of finetuning SD models.", "weaknesses": "1. As presented in Figure 6, CV-VAE suffers from more severe flickering problems than the original SVD, which may possibly be due to the gap between the latent distribution of CV-VAE and SD-2.1. In the paper, the author also pointed out that text-to-image generation may experience differences in color. Authors may convince readers that such flaws can be overcome through further fine-tuning, e.g., fine-tuning full parameters of SVD, or fine-tuning for longer iterations.\n\n2. Missing details about training: The authors didn’t present details about training batch size, augmentations, video fps, etc. Also, whether the default CV-VAE displayed in the experiment section is trained on video and image data separately or jointly (as vae of Opensora) remains unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Flickering problems of CV-VAE**\n\nOur optimization loss is a trade-off between compatibility and better reconstruction quality. Therefore, there is still a domain gap between CV-VAE and diffusion models, which leads to color shifts or flickering problems. This offset can be alleviated by further fine-tuning.\n\n**W2: Missing details about training**\n\nThank you for your comments. Actually, we provided the training details in the manuscript. (1) In the \"Training Details\" paragraph of Section 4.1, we introduced the training datasets, resolution, batch size, learning rate, and other augmentations. We also mentioned joint training with two image datasets of different resolutions (256x256, 512x512) and two video datasets of different resolutions (9x256x256, 17x192x192), with batch sizes of 8, 4, 1, and 1 for the four settings, respectively. (2) We will add some details about the video fps. During the training process, we randomly used a value between 1 and 4 as the frame stride for sampling.\n\n**Q2: Comparison between CV-VAE and interpolation model**\n\nThank you for your suggestion. As SVD only released versions for 14 and 25 frames, direct inference of 97 frames would cause the video to collapse. Therefore, we provided comparative results for the frame interpolation model in attached file **(Table 3)**. We use RIFE [4] as the frame interpolation model for comparison, as it is popular and has 4.2K stars on GitHub. Our model outperforms RIFE in 2 out of 3 metrics, further validating the effectiveness of our method.\n\n**Q3: Decoder-based regularization during training with images**\n\nDuring the training process, we use both images and videos for joint training, so decoder-based regularization is also employed when taking images as inputs.\n\n**Q4: Comparison between SVD and SVD + CV-VAE on text-to-video generation**\n\nThank you for your suggestion. Regrettably, SVD only released the image-to-video version. As an alternative, we compared the performance of VideoCrafter2 and VideoCrafter2+CV-VAE on text-to-video in Figure 10. Additionally, we provide the quantitative metrics and extra visual results for text-to-video in the attached file **(Table 4, Figure 2)**, which further validates the compatibility of CV-VAE with various video generation models.\n\n**Q5: Can the method be supported by theoretical proof?**\n\nCurrently, the effectiveness of this method is proven through empirical evidence, and latent regularization can also be seen as a form of knowledge distillation [1], which is applied in different fields [2][3]. Providing a detailed theoretical proof is quite challenging and can be a direction for future exploration.\n\n[1] Distilling the knowledge in a neural network, arXiv 2015.\n\n[2] Adversarial diffusion distillation, arXiv 2023.\n\n[3] MiniLLM: Knowledge distillation of large language models, ICLR 2024.\n\n[4] Real-Time Intermediate Flow Estimation for Video Frame Interpolation, ECCV 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful analysis and feedback, we are glad the reviewers find that

* Ours proposed question are valuable
    * "Designing a 3D VAE with 2D latent consistent characteristics is very meaningful, and this paper successfully achieves this " - Reviewer U7Gm.
    * "Achieving effective performance in both 2D and 3D simultaneously is of high practical value to the community" - Reviewer U7Gm.
    * "the paper tackles an important problem of constructing a compact latent space of videos that is similar to image space." - Reviewer tAbR.

* Our solutions are novel and effective and our experiments are well-conducted
    * "The bidirectional VAE alignment method designed in this paper is very clever." - Reviewer U7Gm.
    * "The method is novel and delicate, which effectively compresses video latents and maintains latent distribution." - Reviewer LvuQ.
    * "The decoder loss seems a simple and novel design, The qualitative results are good." - Reviewer xmxT.

* Our paper is well-written
    * "The writing is fluent" - Reviewer U7Gm.
    * "The paper is generally well-written" - Reviewer tAbR.

Attached you can find a file containing new experiments suggested by the reviewers.

*  Reconstruction comparison results of different VAEs on a public out-domain dataset.
*  Quantitative and visual comparison results of CV-VAE with some recent methods.
*  Quantitative and visual results of CV-VAE in text-to-video generation models.
*  Comparison results between CV-VAE and interpolation model.

Meta-review: The paper is reviewed by four expert reviewers and got overall postiive ratings of accept, two weak accepts and one borderline accepts. Reviewers find that the paper tackles an important problem of desiging 3D VAE for compact video representation and well executed. Some reviewers concerns regarding the comparison to more recent works, lack of technical novelty, have been addressed during rebuttal. AC agrees that the paper provides a good insight on building compact video representation and recommend for acceptance.
Decision: Accept (poster)