High-Quality Joint Image and Video Tokenization with Causal VAE
Authors: Dawit Mureja Argaw
Xian Liu
Qinsheng Zhang
Joon Son Chung
Ming-Yu Liu
Fitsum Reda
Venue: ICLR 2025 Poster (2025)
Keywords: Autoencoding
Generative Modelling
Causal Video VAE
FILM
Video Tokenization
Primary area: generative models
TL;DR: A causal video VAE for joint image and video tokenization

Abstract: Generative modeling has seen significant advancements in image and video synthesis. However, the curse of dimensionality remains a significant obstacle, especially for video generation, given its inherently complex and high-dimensional nature. Many existing works rely on low-dimensional latent spaces from pretrained image autoencoders. However, this approach overlooks temporal redundancy in videos and often leads to temporally incoherent decoding. To address this issue, we propose a video compression network that reduces the dimensionality of visual data both spatially and temporally. Our model, based on a variational autoencoder, employs causal 3D convolution to handle images and videos jointly. The key contributions of our work include a scale-agnostic encoder for preserving video fidelity, a novel spatio-temporal down/upsampling block for robust long-sequence modeling, and a flow regularization loss for accurate motion decoding. 
Our approach outperforms competitors in video quality and compression rates across various datasets. Experimental analyses also highlight its potential as a robust autoencoder for video generation training.

Published as a conference paper at ICLR 2025
HIGH-QUALITY JOINT IMAGE AND VIDEO TOKENIZA- TION WITH CAUSAL VAE
Dawit Mureja Argaw 1+ Xian Liu
Qinsheng Zhang
Joon Son Chung
Ming-Yit Yu Liu 2
Fitsum Reda 2t
Korea Advanced Institute of Science and Technology (KAIST)
NVIDIA
ABSTRACT
Generative modeling has seen significant advancements in image and video synthe- sis. However, the curse of dimensionality remains a significant obstacle, especially for video generation, given its inherently complex and high-dimensional nature. Many existing works rely on low-dimensional latent spaces from pretrained image autoencoders. However, this approach overlooks temporal redundancy in videos and often leads to temporally incoherent decoding. To address this issue, we propose a video compression network that reduces the dimensionality of visual data both spatially and temporally. Our model, based on variational autoencoder, employs causal 3D convolution to handle images and videos jointly. The key contributions of our work include scale-agnostic encoder for preserving video fidelity, a novel spatio-temporal down/upsampling block for robust long-sequence modeling. and flow regularization loss for accurate motion decoding. Our ap- proach outperforms competitors in video quality and compression rates across various datasets. Experimental analyses also highlight its potential as a robust autoencoder for video generation training. Code and models can be found here.
INTRODUCTION
Recently, generative modeling has taken the world by storm, showcasing remarkable advancements in various domains such as sext-to-image (Rombach et al., 2022; Chen et al., 2024; Pernias et al., 2024; Kang et al., 2023; Nguyen et al., 2023), text-to-speech (Le et al., 2023; Wang et al., 2023; Shen et al., 2024) and most recently, text-to-video generation (Blattmann et al., 2023c; Brooks et al., 2024; Blattmann et al., 2023a). Following from the seminal work (Rombach al., 2022), many generative models (Blattmann et al., 2023c; Brooks et al., 2024; Blattmann et al., 2023a; Yu et al., 2023c; Hu et al., 2023a) now rely on a low-dimensional latent space to achieve high-resolution synthesis. This strategic choice stems from the need to combat the curse of dimensionality in high- resolution synthesis. In video generation, the importance of acquiring robust low-dimensional latent representation is further emphasized by the complex and high-dimensiooal nature of video data.
Most works on video compression used a generation context typically fall into two main categories based on the nature of their latent space: discrete or continuous. While discrete latent spaces have been an active area of investigation, with recent works like (Yu et al., 2024; Gupta et al., 2023) enhancing compression and reconstruction quality, there has been comparatively less exploration of continuous latent spaces. For example, recent video diffusion models (Blattmann et al., 2023c;a; Hu et al., 2023a) still heavily rely pretrained image autoencoders with added 3D convolutions for temporal alignment, resulting in lingering temporal artifacts such as flickering in decoded videos. This reliance image autoencoders also means that temporal compression is often overlooked. Consequently, the well-documented temporal redundancy of videos (Lu et al., 2019; Lin et al., 2020; Hu et al.. 2023b) is not fully exploited in continuous video autoencoders. This, in turn, has significantly limited the ability of state-of-the-at models (Blattmann et al., 2023c;a; Hu et al., 2023a) to effectively encode and generate longer-duration videos, until the recently introduced Sora (Brooks et al., 2024) model. In light of this, our work introduces video compression network in continuous
"Work done during a research internship at NVIDIA.
1 "ooresponniig author.

---

Published as a conference paper at ICLR 2025
time-space that reduces the dimensionality of visual data into learned latent and maps the generated latent back to pixel space with high fidelity.
Our video compression network is based on variational autoencoder (VAE) (Kingma & Welling, 2013), where the encoder compresses the input video both spatially and temporally into a latent representation and the decoder reconstructs the input video from this latent representation. We explored both Transformer-bane
( (Vaswani et al., 2017) and 3D convolution- based architectures for a video VAE. Our experiments have shown that fully Transformer-based video VAE tends to introduce blocking artifacts and struggles to generalize effectively across various temporal and spatial resolutions compared to 3D convolutions, consistent with findings in Yu al. (2024). Therefore, we opt for 3D convolutions interleaved with self-attention layers to design our video VAE. To seamlessly integrate joint image and video compression within a single model, we use temporally causal 31) convolutions and self-attention layers. In addition to introducing a continuous video VAE for high- quality spatio-temporal compression, our main contributions lie in identifying and addressing three key issues in video VAEs.
First, we observe that a standard video VAE with symmetric encoder-decoder architecture struggles to maintain the fidelity of a video when it contains small and fast-moving objects, especially at higher compression rates. This is mainly because small, fast-moving objects tend to disappear at the deeper levels of the encoder feature pyramid. Moreover, at these levels, the significantly smaller feature dimension compared to the input dimension makes it difficult to preserve large motion information. To mitigate this challenge, we draw inspiration from FILM (Reda et al., 2022) and propose weight- shared encoder that learns to aggregate features across different scales of the input video (refer to Fig. 2(a)). The intuition here is that large motion at finer scales (higher resolution) should be equivalent to small motion at coarser scales (lower resolution). Thus, sharing encoder weights and aggregating features from different depths of the feature pyramid allows us to increase the number of pixels available to effectively encode large motion. Our experiments show that incorporating FILM encoder into a video VAE significantly improved the decoding of large motion.
Second, we carefully examine spatio-temporal downsampling and upsampling in video VAEs. A commonly used approach in previous works (Rombach et al., 2022; Yu et al., 2023a) to employ non-learnable kernels for down/upsampling followed by convolutional layer i.e. average pooling for downsampling and nearest interpolation for upsampling. However, this approach suffers from information, non non-learable kernels treat all
the potential loss of high- high-frequency f features within the pooling (interpolation) window equally. To address the issue, MAGVIT-v2 (Yu al., 2024) recently proposed using learnable down/upsampling with convolutional layers. While video VAE trained with learnable spatio-temporal down/upsampling generally performs well, it tends to overfit to the temporal sequence length it has been trained on, with performance notably dropping when inference is done with different sequence lengths, thus limiting the model's adaptability for arbitrarily long videos. Our work introduces robust ppatio-temport down/upsampling module that addresses the aforementioned limitations. The module is designed as a dual-path network, effectively leveraging both learnable and non-learnable kernels. We empirically show that the proposed video VAE is temporally more adaptable and can encode and decode arbitrarily long videos at varying lengths without significant performance degradation.
Third, to ensure that the encoded latent representation faithfully preserves the motion dynamics of the input video, we propose a flow regularization loss for video VAE training. The loss is incorporated by optimizing the mean-squared error between the optical flows of the input video frames and their corresponding optical flows in the decoded video frames. We use a state-of-the-art model (Teed & Deng, 2020) compute the optical flows. Our experiments reveal that imposing flow regularization loss not only results in the decoding of temporally smoother motion but also helps to achieve a temporal compression rate of up to 16x without a significant loss in decoded video quality.
Building on these findings, we train a causal video VAE at various spatio-temporal compression rates: 4 x 8 8 (256),& 8 x 8 x 8 (512), 16 8 x 8 (1024) and 16 16 x 16 (4096). We compare our method with several state-of-the-arr approaches (Rombach et al., 2022; Blattmann et al., 2023c; Yan al., 2021b; Lab & etc., 2024; Zheng et al., 2024; Wang al., 2024a) across multiple video and image benchmarks (Pont-Tused et al., 2017; Niklaus & Liu, 2020; Su et al., 2017: Russakovsky et al., 2015), using a comprehensive suite of metrics. Our experimental results demonstrate that the proposed autoencoder consistently outperforms the competing baselines, both qualitatively and quantitatively. To showease the effectiveness of our video VAE in generative modeling, we integrate
2

---

Published as a conference paper at ICLR 2025
Input video
3D
Cousa/DRssSlecc
STAlltffock
STAitrdlock
Middle Block
Encoder
STCownBrock
Blokk
Decoder
STupBlock
Cousal BBccclocckkk
-X
STAttnälock
Causal BDakk
STubBllock
STCoowndtock
STAIL
Cousa BDR:sBokk
Cousal BDRc
Cursal 3D Conv
Output video
Figure I: Architecture Overview: Our causal video VAE comprises three main components: an encoder, a middle block, and a decoder. The encoder compresses input visual data spatially and temporally, the middle block samples from the learned distribution of encoded features to generate latent representation, and the decoder maps this latent representation back pixel space. N denotes the number of down/upsampling steps.
our pretrained autoencoder into open-source video generation frameworks (Wang et al., 2024b; Zheng et al., 2024) and train them for unconditional video generation tasks on the SkyTimelapse (Zhang et al., 2020) and UCF-101 (Soomro, 2012) datasets.
We also perform extensive ablation studies and experimental analyses further confirm the benefits of the proposed autoencoder. To evaluate the robustness of the video VAE decoder against erroneous latent predictions from generative models, we perturb the encoded latents using various corruption schemes, such as Gaussian noise and quantization, and analyze the model's sensitivity in Sec. B.1.1. Additionally, we assess the video VAE's effectiveness in reliably encoding and decoding videos of varying lengths sampled at different intervals in Sec. 1.1.2. Our results strongly affirm the model's potential as robust autoencoder for training video and image generation models.
METHOD
We propose a video (image) compression network reduce the dimensionality of visual data into learned latent space. Our network is based on a variational autoencoder (VAE) (Kingma & Welling, 2013) architecture consisting of an encoder and decoder. Given a video clip V of size (1 + 7) x H x W 3, where T denotes the number of frames in the video, the encoder compresses the input video both temporally and spatially into a latent representation v of dimension (1+t) x h x X c, with a spatio-temporal compression rate of x x H W. The decoder takes the generated latent u as input and maps back video V in pixel space. We use temporally causal 3D convolutional and self-attention layers in network design to scamlessly integrate joint image and video compression within a single model. The overview of our method is depicted in Fig. I.
2.1 ENCODER
The encoder network in the video VAE mainly consists of three components. These are causal 3D residual block ( spatio-temporal attention block (STArmBlock). As shown in Fig. 1, the encoder is constructed by cascading stack of these blocks in a top-down fashion.
spatio-temporal downsampling block (STDownBlock) and
Causal 3D Residual Block The DResal3DResBlocki is residual network composed of two tempo- rally causal 3D convolution layers, along with group normalization (GN) and Swish activation layers (see Fig. 2(b)). For kernel size (ke, Ki. kww) the padding scheme in the temporal axis of vanilla 3D convolutional layer is to add [--l] frames before and ? after the input frames, respectively. In comparison, a causal 3D convolution layer pads with be frames before the input frames and nothing after, ensuring that the output for each frame solely relies the preceding frames (Yu et al., 2024; Gupta et al., 2023). As result, the first frame remains independent of the subsequent frames, enabling our model compress single images well. We test three different temporal padding schemes: zero, constant, and replication. Our experiments reveal that employing the replication padding results in a model decoding temporally smoother videos (refer to Sec. B.3).
Spatio-Temporal Downsampling Given feature volume of size b (1 t) h where b denotes batch size, the STDownBlock reduces the spatial and temporal dimensions by half, Le.. x + { x z x c. A commonly employed approach in previous works (Rombach
3

---

Published as a conference paper at ICLR 2025
B+ x
Cauasl3OReeBllck
STDownBlock
Sarall 5J Dare
Carad Cary
1D Avaruge
(Scale- saalle-agnsstccl Encoder (Weight th shared)
Cayaa
STAttnBlock
STUpBllck
Altention
Causal trroposed labilee 21
interpaation
Attane on
Couasi Corn
Cencat
(al
Figure 2: (a) FILM Encoder: Our weight-shared encoder aggregates features from multiple scales input video, enhancing its ability to handle large motion decoding. (b) Building Blocks: Design specifications for the causal 3D residual block (Ceusal)DeesBoocl)) spatio-temporal upsampling block (STUpBlock), and spatio-temporal attention block (STAmbBlock).
downsampling block (Sookk)
et al., 2022; Yu et al., 2023a) is to do average pooling followed by convolution layer. However, such an approach is not optimal, particularly for higher compression rates, because average pooling treats all features within the pooling window equally, potentially resulting in the loss of high- high-frequency spatial or temporal information. Recently, MAGVIT-v2 (Yu et al., 2024) opted for using strided convolutions instead of average pooling to leverage learned kernels. While video VAE trained with learnable downsampling kernels generally performs well, it often fails to be temporally agnostic, Le.. it overfits the specific sequence lengths it is trained with and performance notably drops when inference is done by sampling the input video at different sequence lengths (see Table 4). This particularly limits the adaptability of the model for Long-sequencr encoding and the robustness of its latents to noise corruption.
To address these limitations, we introduce temporally more adaptable downsampling module for a video VAE. We design the STDownBlock as dual-path module, leveraging both learnable and non-learnable kernels, where the input feature passed through strided causal 3D convolutional layer and 3D average pooling layer concurrently, and the resulting outputs are combined via summation as shown in Fig. 2(b). We experimentally observe that the proposed STDownBlock effectively addresses the aforementioned limitations and leads to notably better performance. This intuitive as downsampling via average pooling mitigates the risk of the temporal receptive field overfitting specific sequence length while learnable kernels facilitate downsampling by selectively emphasizing or suppressing certain features based on their relevance.
Spatio-Temporal Attention The STAAtBlock captures spatial and temporal dependencies within an input video using self-attention layers (Vaswani et al., 2017). Given feature of size b x () x h x w X c, the spatial attention is done by reshaping the feature to b(1 t) x fw x c and passing to a single self-attention layer. This is followed by causal attention in the temporal di- mension by reshaping the input feature to bhw (1 + t) x as depicted in Fig. 2(b). To manage the increasing computational complexity caused by merging spatial dimensions during spatial attention, we apply the STAunBlock only at the lower resolutions of the feature pyramid in the encoder.
2.2 FILM ENCODER
Reducing the dimensionality of video while maintaining its fidelity could quite challenging particularly when it involves small and fast-moving objects. This is mainly because small objects with large motion disappear at the deeper levels of the encoder feature pyramid. Furthermore, there are significantly fewer pixels at the deeper levels of the pyramid preserve large motion information. To overcome these challenges, we draw inspiration from FILM (Reda et al., 2022) and design our video VAE with shared encoder weights across different scales shown in Fig. 2(a). Given a video Vo of size (1 + T') x И x W x 3, we first construct an input pyramid { (Vo, V1, Vi,..., V by successively resizing the video, where Vg has a shape of (1 + T) x Te x 7E x 3. Each clip in the input pyramid is then fed into weight-shared (FILM) encoder create set of feature pyramids (Fq.. (1)):
4

---

Published as a conference paper at ICLR 2025
(o (F) (F)) = FILM (Vo, Vi.....,VV)).
(I)
F Concat (Fo 12 ,..., F
where d denotes the maximum depth in the feature pyramid. A scale-agnostic feature F is constructed by channel-wise concatenating features from different depths but with the same spatial dimensions (Eq. (2)). We use temporal average pooling to align the dimensions of the features {FT--1 ..... F%) with F3 before concatenation, as depicted in Fig. 2(a). By sharing the encoder weights across the input pyramid, the key intuition here is that large motion observed at the deeper depths of Vo should be the same as small motion at the shallower depths of Vk, thus, aggregating features from different depths of the feature pyramid allows to boost the number of pixels available to effectively encode large motion (Reda et al., 2022).
,...,FF.
2.3 MIDDLE BLOCK
The output of the FILM encoder, as defined in Eq. (2), is fed into the middle block, where the scale-agnostic feature is projected into latent representation with reduced channel size. This block is tasked with sampling from the learned distribution of the encoded latent space and generating latent representation v with reduced spatial and temporal dimensionality. As shown in Fig. 1, the middle block consists of a cascade of Causal3 STAitnBlock, and Gaussian sampling layer. Following previous works (Kingma Welling, 2013; Rombach et al., 2022), we adopt an isotropic Gaussian distribution parameterize the encoded latent variables, from which samples are drawn by the sampling layer.
2.4 DECODER
The decoder of our video VAE maps the generated latent back to the input visual data. The network architecture of the decoder resembles that of the base encoder, constructed in bottom-up manner as shown in Fig. 1. It includes a causal 3D residual block (Causal3IResBlock) spatio-temporal upsampling block (STUpBlock), and spatio-temporal attention block (STAttBlooc).
Spatio-Tempore Upsampling Given feature volume E of size b x (1 + x the STUpBlock increases the spatial and temporal dimensions of x by a factor of two, Lee. b x (1 + 21) 2h 2w c. Similar to the downsampling block, the STUpBlock is designed as a dual-path module, incorporating both learnable and non-learnable kernels, which are combined through summation (see Fig. 2(b)). We use causal transposed 3D convolutional layer for learnable upsampling. To map 1 + t frames to 1 + 2t frames, thus enabling the joint usage of our model for both images and videos, we discard the first frame after the upsampling process. A nearest-neighbor interpolation in both spatial and temporal dimension is used for non-learnable upsampling.
2.5 NETWORK TRAINING
Following previous works (Rombach et al., 2022; Kingma & Welling, 2013), we train our video VAE using the standard set of loss functions. First, we compute the reconstruction loss LR which calculated as the L2 loss between the input video V and the decoded video V (Eq. (3)). We also optimize the perceptual similarity Lp between each input video frame and the corresponding reconstructed frame using frame-wise LPIPS (Zhang et al., 2018) loss.
1+T
LR
Σ 1=1
V4
V.
V.J
(3)
where, Vú denotes the ,th frame index in the input video. To mitigate arbitrary high-variance in the encoded latent spaces, we apply KL regularization loss LKI. by guiding the learned latent distribution towards a standard normal (Kingma & Welling, 2013; Rombach et al., 2022).
Flow Regularization To ensure that the decoded video accurately preserves the motion dynamics of the input video, we propose a flow regularization loss Lnow for video VAE training. We define LRow as the mean-squared error between the optical flows of the input video frames and their corresponding optical flows in the decoded video frames. To compute the optical flows, we use pretrained RAFT (Teed & Deng, 2020) model on-the-fly. We employ a bidirectional scheme, as shown in Eq. (4), to ensure robust motion supervision. Our experiments reveal that a model trained with flow regularization decodes temporally smoother motion compared to a model trained without (refer to Sec. 4). Li also facilitates better capturing of motion at higher temporal compression rates.
5

---

Published as a conference paper at ICLR 2025
flow Σ
[|5411-1 fi11-
(4)
where fi-s++1 represents the optical flow between Vi and Vi+1 frames in the decoded video. The total loss for training our video VAE is formulated as follows:
Lvee = CR + Lp + O'toow Cnow + aKLLKL
(5)
where Ofloww and d'L denote the weights for the flow and KL regularization losses, respectively.
GAN Training In addition to standard network training, we incorporate adversarial training, fol- lowing prior works (Esser et al., 2021; Rombach et al., 2022; Blattmann et al., 2023c), to enhance the quality of the decoded video. We optimize a 3D convolution- et al., 2017) to distinguish between real videos and those generated by our video VAE. Our discrimi- nator architecture builds Pix2Pix (Isola et al., 2017), adapting it for video data by replacing 2D convolutional and batch normalization layers with their 3D counterparts. For further details, please refer to the official Pix2Pix implementation here.
PatchGAN discriminator (Isola
EXPERIMENT
3.1
VIDEO/IMAGE AUTOENCODING
Implementation Details We use the WebVid-2M (Bain et al., 2021) dataset for model training. For each step, we randomly sample T + 1 consecutive frames from a video, where T E {8, 16), and crop them to size of 128 128. The resulting clip with dimensions (1 + 7) x 128 x 128 x 3 is fed into the video VAF. The FILM encoder uses an input pyramid with k = 3. GAN training begins after the initial 100K iterations with Lvae. The flow and KL regularization loss weights are set to O'flow le- 3 and aKK = le 6, respectively. We use the Adam (Kingma & Ba, 2014) optimizer with learning rate of 4.5c- 5. Training is conducted for 250K iterations with batch size of 48 on 48 NVIDIA A100 (40GB) GPUs.
Baseline Methods We benchmark our approach against state-of-the-an methods for which open- source code or models are available. These include autoencoders from VideoGPT (Yan et al., 2021b),
LDM (Rombach et al., 2022), Video LDM (Blattmann et al., 3023c), SVD (Blattmann et al., 2023b),
Open-Sora (Zheng et al., 2024), Open-Sora-PPan (Lab & etc., 2024), CV-VAE (Zhao al., 2024), HVDM (Kim et al., 2024a), and OmniTokenizer (Wang et al., 2024a), Cog VideoX (Yang et al., 2024).
All models, including ours, encode latents with channel size of 4 in continuous space, except for DoniiTokenierr (channel size of 8), Cog VideoX (channel size of 16), and VideoGPT, which encodes latents in discrete space. For fair comparison, we also include baselines of model with latent channel sizes of 8 and 16, as shown in Table
Evaluation Datasets and Metrics We evaluate our model and competing approaches on the video autoencoding task using two representative datasets that feature medium to large degrees of mo- tion: Xiph-2K 2K (Niklaus & Liu, 2020) and DAVIS (Pont-Tuse et al., 2017), both at 480p resolution. Additionally, benchmark image autoencoding performance using the ImageNet validation set (Rus- sakovsky et al., 2015) at 256 x 256 resolution. Decoded video/image quality is assessed using PSNR, SSIM, and LPIPS (Zhang al., 2018), while temporal coherence between frames is assessed using an optical flow- temporal smoothness (TS) metric (Shen et al., 2020). We also use two video-based metrics: reconstruction FVD (Unterthiner et al., 2019) (rFVD) and reconstruction STREAM (Kim et al., 2024b) (rSTRavg). which focus on the spatio-temporal perceptual quality and fidelity of the decoded videos. For STREAM, we report the average of the fidelity (STREAM-F) and temporal flow (STREAM-T) scores. Please refer to Kim et al. (2024b) for further details.
3.1.1 VIDEO COMPRESSION
In Table 1, we comprehensively evaluate our approach and state-of-the-an methods on the video autoencoding task. We train variants of our model with different spatio-temporal compression rates and latent channel sizes. As expected, methods that focus solely spatial compression, such as Video LDM (Blattmann et al., 2023c), demonstrate strong performance, particularly on videos with large motion. However, shown in Table our model gives a highly competitive, if not superior, performance compared to competing approaches with significantly higher dimension reduction. For instance, VideoGPT (Yan et al., 2021a) attains an average PSNR of 30.58 dB across the two

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison on spatio-temporal video compression
Method
Comp. Kute
Size
Xiph-2K
DAVIS
Ex b
[8]
PSNR
I.PIPS
TS
(FVD). |
15T
PSNRT
1.PIPS
TS
(FVD)
18TR 1
30.36
0.788
Video LDM
1x 8
4
30.32
0.120
-1.76
30.46
0.924
32.30
0.131
-0.81
40.10
0.889
SVD
1x888
4
30:40
0.122
-1.58
30.96
0.880
31.74
0.138
-0.77
42.24
01818
0.777
Ours
1x8x8
34.08
0.068
-2.04
17.26
0.982
32:49
0.118
-0.98
36.12
0.916
VideoGPT
4x4x4
31.04
0.225
-1.09
57.11
0.762
30.12
0.306
0.14
93.67
0,760
32.25
Open-Sora
8
4
28.20
0.162
-1.20
42.39
0.789
26.79
0.262
0.27
80.20
0.747
Opra-Sora-Plan
4X8X8
29.33
0.138
-1.07
35.02
0.757
27.17
0.257
0.38
78.67
0.707
CV-VAB
8
0.164
Ours
1X888
32.12
0.107
-1.53
27.16
0.868
30.31
0.1.9
-0.24
48.67
0,806
Ornii Ornn/Takenizer
Takenizer
8
8
28.61
0.153
-1.05
38.83
0.753
26.86
0.260
0.41
79.599
0.712
Cog VideoX
4
0.852
Ours
4
8
33.39
0.084
-1.70
21.32
0.999
31.65
0.128
-0.54
39.18
0.855
Ours
8.888.888
16
34,51
0.062
-1.88
15.74
0.952
32.86
0.104
-0.86
31,84
0.892
09.04
0.786
Ours
x6 x.
4 29.74
0.162
-0.96
41.12
0.731
27.51
0.270
0.39
82.65
0.679
Ours
16 16 1.6.x.15
4
26.91
0.250
-0.76
63.45
0.683
24.60
0.356
0.47
108.98
0.644
Input (1
VLDM
VideoGPT (4x1x x
Open-Sora-Plan (4 x x
Ours
1)
(1 x x
(8 8 x
Figure 3: Qualitative comparison of decoded video frames between our model and competing approaches
datasets at a compression rate of 4 x 4 x 4. In comparison, our model achieves 30.16 dB at compression rate of 8 x x 8. This can be attributed to the proposed spatio-temporal downsampling and upsampling modules, which effectively leverage both learnable and non-learnable kernels to facilitate the encoding and decoding of information at higher compression rates. Additionally, the use of the FILM encoder mitigates the challenges of encoding large motion, contributing the strong performance of our video VAE on targe-motion datasets (Niklaus & Liu, 2020; Pont- Tuset et al., 2017). It can also be observed that best competing method at a 4 x 8 x 8 compression rate, Open-Sora-Plan (Lab & etc., 2024). The proposed flow regularization loss enables faithful reconstruction motion between decoded frames, resulting in robust autoencoding performance at higher temporal compression, as evident in Table 1.
16
8 model achieves performance comparable the
In Fig. 3. we qualitatively compare our model with the best-performing baselines. As shown in the figure, our 8 x 8 x 8 model successfully reconstructs a small, fast-moving object (highlighted in the green box), whereas Open-Sora-Plle Plan fails at 4 x 8 compression rate. It is also evident that our approach reconstructs human faces with great fidelity, while other methods with much lower compression rates, such as VideoGPT, struggle to preserve details (see the nose and mouth in the blue box). As illustrated in Fig. 3, our model generally reconstructs sharper frames while maintaining the structural details of objects that are far from the camera, compared to other methods (see the details in the red box). Please refer to the supplemental video and the appendix for further analyses.
3.1.2 IMAGE COMPRESSION
We evaluate our approach and other baselines the image autocncoding task, as shown in Table 2. As can be inferred from the table, our causal video VAE demonstrates competitive image compression performance across various metrics. Notably, our model, trained exclusively video clips without explicit optimization for images, performs comparably to a of-e--of-the-rr image VAE (Rombach al., 2022). These results highlight the advantages of the causal formulation in our video VAE for joint image and video compression model.
3.2 VIDEO GENERATION
Our work has practical applications in tasks such as image/video generation, where the encoder compresses visual data into latent space for training generative models, and during inference, the
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparison on image compression
Table 3: Comparison on video generation
Mathod
ImageNet Val
Method
SkyTimelapse
UCF-101
Cx b [+]
PSNK 1
SSIM 1
LPIPS
FVD16
FVD
FVDic 4 FVD L
LDM
1x8x 8
29.06
0.684
0.137
VideoGPPT
222.7
2880.6
Open-Sora
Open Sora
CV-VAE
OnmiTokenizer
4x8x8
8x8x8
8xxxx
26.67
27.69
27.57
27.05
0.602
0,652
0.65]
0.150
0.141
StyleGAN-
DIGAN
PVDM
79.52
71:46
197.0
159.9
1431.0
457.4
2243.7
1773.4
902.2
Dues 4x888
29.02
0.897
0.088
LDM Open- Sora
53.38
127.5
266.8
657.6
VogVidooX
4
16
29.66
0.712
0,075
Open-Sora
55.12
130.9
662.9
Dars
4x888
0.744
Ocus + Open- Sora
49.50
119.8
248.7
572.1
decoder reconstructs the data from the generated latents (Brooks et al., 2024). To demonstrate this, we plug in our pretrained video VAE into the diffusion-based open-source generative framework, Open-Sora (Zheng et al., 2024), and train it for unconditional video generation.
Implementation Details We use the train-split of commonly used video synthesis benchmarks, Sky Timelapse (Zhang et al.. 2020) and UCF-101 (Soomro, 2012), for model training. Consistent with previous works (Yu et al., 2023b; Skorokhodov et al., 2022), we resize each frame in the datasets to a resolution of 256 x 256 and sample video clips of length + T, where T E {16, 128). Our video generation experiments are conducted on 16 NVIDIA A100 (80GB) GPUs adhering the training configuration in Zheng et al. (2024).
Baseline Methods We compare our video VAE (4 x 8) with the autoencoders from LDM (1 x 8 x 8) and Open-Sora (4 x 8 x 8) on the video generation task by integrating them into a video generation model (Zheng et al., 2024). All autoencoders, including ours and the baselines, are kept frozen during the generation experiments. Additionally, we report the results of state-of-the- art video generation methods, including VideoGPT (Yan et al., 2021a), DIGAN (Yu et al., 2022), StyleGAN-V (Skorokhodov al., 2022), and PVDM (Yu et al., 2023b).
Evaluation We use the Fréchet Video Distance (FVD) (Unterthiner al., 2018) to quantitatively assess the quality of the generated clips. Following the evaluation protocol in previous works (Yu et al., 2023b; Skorokhodov et al., 2022), we measure the FVD score on video clip lengths of 16 and 128 frames. For both FVD and FVV 128, we evaluate using 2,048 real and generated video clips.
3.2.1 RESULTS
In Table 3, we present a quantitative comparison of Open-Sora (Zheng et al., 2024) (which utilizes latents encoded by LDM (Rombach al., 2022), Open-Soral 's video VAE, and our video VAE) and previous video generation methods. The significant performance improvement of Open-Sora-bare generation over prior GAN-based (Yu al., 2022: Skorokhodov et al., 2022) and autoregressive- based (Yan et al., 2021 approaches, as evident from the table, can be attributed to the ofate-of-the-art liffusion-based architecture in Zheng et al. (2024). It can also inferred that using our video VAE (Ours + Open-Sora) delivers the best performance in both short and long video generation, notably outperforming Open-Sora's 's own video VAE, Additionally, our video VAE achieves better generation results than LDM, as shown in Table 3, while being 4x more efficient in temporal compression. These results highlight the effectiveness of our autoencoder in generative modeling.
In Fig. 4. visualize sequence of video frames generated using our video VAE. The first three rows illustrate generated results based on the SkyTimelapse dataset, while the last three rows show generated results from the 101 dataset. As shown in the figure, Open-Sora, powered by video VAE, generates realistic sky time-lapse videos with high fidelity. It can also be seen that our method, trained solely on the challenging UCF-101 dataset, synthesizes videos of human faces with satisfactory details and videos of human actions exhibiting spatio-temporal consistency, as depicted in Fig. 4.
4 ABLATION STUDIES
We conduct extensive ablation studies analyze the contributions of the different components in the proposed video VAE, including the FILM encoder, the spatio-temppral down/upsampling blocks, the spatio-temporal attention block, and the flow regularization loss. All experiments are conducted on a video VAE with a spatio-temporal compression rate of 8 x 8 x 8. The results on the Xiph-2K (Niklaus Liu, 2020) and DAVIS (Pont-Tusee et al., 2017) datasets are summarized in Table 4.
8

---

Published as a conference paper at ICLR 2025
UCK-10 SkyToellppe Sky Trmeeoope Sky MoyTeellpee
UCR-101
Figure 4: Qualitative analysis of videos generated by Open-Sora, enabled by video VAE (Ours Open-Sora). The top three rows are from a model trained on SkyTimelapse, and the bottom three on UCF-101.
FILM Encoder We explore the advantages of employing a weight-share shared encoder across the input video pyramid to address the challenges associated with decoding small and fast-moving objects, as outlined in Sec. 2.2. To accomplish this, we train our video VAE using a single (base) encoder that takes the original resolution of the input video and compare its performance with a model that uses FILM encoder. As can be inferred from Table 4(a), a model with a FILM encoder consistently outperforms a single encoder model across all metrics. Specifically, on the DAVIS dataset, which predominantly contains large motion, using FILM encoder improves performance by 1.81 dB on the PSNR metric. The qualitative results in Fig. 5 further show that video VAE with a FILM encoder reconstructs sharper frames compared to one without it. These findings underscore the advantage of learning scale-agnostic features through shared encoder weights, which helps mitigate the challenges of large-motion decoding (Reda et al., 2022).
Down/Upsampling Here, we study how different network designs for the down/upsampling modules impact our video VAE. First, we train model using non-learnable kernels (followed by a convolution layer), employing average pooling for downsampling and nearest interpolation for upsampling, in line with previous works (Rombach et al., 2022; Lab & etc., 2024). Additionally, we train a separate model using learnable kernels (Yu et al., 2024), employing strided convolution for downsampling and strided transposed convolution for upsampling. We compare these baselines against our proposed approach, which integrates both learnable and nea-laaeea kernels, described in Sec. 2. The results for input videos sampled different sequence lengths presented in Table 4(b). As can be inferred from the table, a model based on only learnable kernels outperforms model based
non-learnable kernels for sequence length seen during training (T = 8). However, when tested at a different sequence length (T" = 32), the performance of the model based on learnable kernels significantly drops compared to the model based on non-learnable kernels. We observe that while ppatio-temporal down/upsampling with 3D convolutions generally performs well, the model tends to overfit to the training sequence length. This results in a notably worse performance at different sequence lengths during inference. In comparison, non-learnable spatio-temporal down/upsampling robust across different sequence lengths, seen in Table 4(b). Our proposed method which combines the best of both worlds as depicted Fig. 2(b). not only outperforms both approaches but is also temporally more adaptable to different sequence lengths.
video VAE results in relatively lower performance but is more
To examine the benefit of imposing motion constraints in video VAE training, train model with and without the flow regularization loss (Eq. (4)). The results are presented in Table 4(c). As evident from the table, a model trained with flow regularization loss Cnow consistently outperforms model trained without it. In particular, there is significant improvement in the
Flow Regularization

---

Published as a conference paper at ICLR 2025
Input
w/o FILM
w/ FILM
Input
w/o FILM
w/ FILM
Figure 5: Qualitative analysis of large-motion decoding with (w/) and without (w/) the FILM encoder
Table 4: Ablation experiments on different components of our causal video VAE
Method
Xipl--K
DAVIS
PSNR
SSIM
LPIPS
TS
PSNR 1
SSIM
LPIPS 1
TS
FILM eneoder w/ FILM encoder
30.44
0.775
0.139
27.23
0.587
0.251
0.29
31.28
0.797
0.121
-1.37
29.04
0.624
0.226
0.16
(8)
non- -onmmaaaablle kemnel (7 80
29.98
0.728
0.156
-1.13
27.89
0.560
0.257
0.21
non-lcamable kennel
52)
60 0.38)
0.710
0.175
0.42)
0.276
0.26
learnable kernel (2 8)
30.60
0.743
0.153
-1.18
28.03
0.555
0.257
0.18
Jearnable kernel (2" (2)
28.98 1.62)
0.713
0.182
-1.01
26.95 (1 1.08)
0.489
0.305
0.35
ours (? 8)
31.28
0.797
0.121
-1.37
29.04
0.624
0.226
0.16
ours (?' 32)
30.85 0:43)
0.785
0.130
-1.25
28.82 () 0.22)
0.619
0,234
0.17
(c) wio Bow regularization (Crow)
30.65
0.784
0.130
-0.68
28.26
0.607
0.236
0.58
w/ flow regularization (Crow)
31.28
0.797
0.121
-1.37
29.04
0.624
0.226
0.16
(4) wlo STAtmBloch
30.52
0.773
0.139
-0.85
27.36
0.603
0,240
0.36
w/ STAmmBlack
31.28
0.797
0.121
-1.37
29.04
0.624
0.226
0.16
temporal smoothness of the decoded frames, as noted from the TS metric in Table 4(c). This implies that incorporating flow regularization during training is advantageous for decoding temporally consistent videos. Please refer to the supplemental video for qualitative examples.
Spatio-Temporal Attention Here, we analyze the importance of incorporating self-attention layers (Vaswani et al., 2017) into our video VAE to explicitly capture spatio-temporal dependencies. To achieve this, we train our model after removing all the attention blocks from the encoder, middle block, and decoder parts of our network (see Fig. 1). As be observed from Table 4(d), model trained without STAunBlock already gives a competitive performance, as causal 3D convolutional layers are strong capturing spatio-temporal correspondences in videos (Yu al., 2024). However, adding spatio-temporal attention blocks in the video VAE results in notable performance boost.
5 CONCLUSION
This work presents a causal video VAE for high-quality video compression. We make three key contributions. First, we propose a weight-shared encoder that efficiently captures multi-scale features from the input video. This mitigates the challenges associated with reconstructing videos with large motions. Second, we introduce robust spatio-temporal down/upsampling blocks, overcoming the limitations of prior methods. Third, to preserve motion dynamics during high compression, we introduce a flow regularization loss for video VAE training. We demonstrate the effectiveness of our model through comprehensive experiments. We also demonstrate the potential of our proposed model as a robust autocncoder for video generation training.
ACKNOWLEDGMENT
This work was partially supported by IITP grant funded by the Korea government (MSIT) (RS-2024- 00457882, National AI Research Lab Project).
REFERENCES
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Andreas Blattmann, Tim Dockhom, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023a.
10

---

Published as a conference paper at ICLR 2025
Andreas Blattmann, Tim Dockhom, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15117 2023b.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 22563-22575, 2023c.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL ttttps://openee com/ research/ Hileo--eeeeran -oo-oodenllloeoooo --as---asssse assworld- simulators.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- -aaart-Saalpaa:: Fast training of diffusion transformer for photorealistic text-to-image synthesis. The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp///pp///eeeeeeee/eeeeeeeeeee net. forum? Fow.mmmd=eeAnneeee.....
Abdelaziz Djelouah, Joaquim Campos, Simone Schaub-Meyer, and Christopher Schroers. Neural inter-frame compression for video coding. Computer Vision (ICCV), pp. 6420-6420 2019. doi: 10.1 201100//CCV.22119000
2019 IEEE/CVF International Conference on
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high righ-resolution image synthesis. Proceedings the pp. 12873-12883, 2021.
the IEEF/CVE conference on computer vision and paltern recognition,
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprini arXiv:2312.0666 2023.
Amirhossein Habibian, Ties Van Rozendaal, Jakub Tomezak, and Taco Cohen. Video compression with rate-distortion autoencoders. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, October 2019, doi: 10. 10.11099icc..201900013... URL http:/ /xxp doi. org/10. 1109 ICCV 2019. 2019.00713.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation, 2023a.
Zhihao Hu, Dong Xu, Guo Lu, Wei Jiang, Wei Wang, and Shan Liu. Fve: An end-to-end framework towards deep video compression in feature space. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(4):45694458) 2023b. doi: 10.1 10.109TTAMM2220.3110622
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. taage-to-imagee translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-11344 2017.
Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusieee 2024.
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Kihong Kim, Hancol Lee, Jihye Park, Seycon Kim, Kwanghee Lee, Seungryong Kim, and Jacjun Yoo. Hybrid video diffusion models with 2d triplane and 3d wavelet representation. arXiv preprint arXiv:2402. 13729, 2024a.
Pum Jun Kim, Scojun Kim, and Jaejun Yoo. Stream: Spatio-temporal evaluation and analysis metric for video generative models. arXiv preprint 0066099903 2024b.

---

Published as a conference paper at ICLR 2025
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprini
PKU-Yuan Lab and Tuzhan Al etc. etc. Open-sora-plan, April 2024, URL htt.httts///.iorrgge
5281/zenodo. 10948109.
Matthew Le, Apoory Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicehox: Text-guided multilin- gual universal speech generation at scale. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: http///pppeeeeeeeeeeeeeeeeeeeeee www.eeww.eeeeeeeeeeeeeeeeeeeeeee forum? iid gzCS2 52hco.
Jianping Lin, Dong Liu, Houqiang Li, and Feng Wu. M-Ive: Multiple frames prediction for learned video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework, 2019.
Fabian Mentzer, George Toderici, David Minnen, Sergi Caelles, Sung Jin Hwang, Mario Lucic, and Eirikur Agustsson. VCT: A video compression transformer. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https::t//.ooo. /pwnreveeeeeeeeeeee net. Forum? me SMM.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. /5505, 2023.
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco. Sewoong Oh, and Ludwig Schmidt. Improving multimodal datasets with image captioning. In l'hirty-evennth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: openreview.
Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp:////////eeeeeeee net/ fortm? din
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Korkine-Hornung and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 1e04..0040077. 2017.
Zhiwu Qing, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yujie Wei, Yingya Zhang, Changxin Gao, and Nong Sang. Hierarchical spatio-temporal decoupling for text-to-video generation, 2023.
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. European Conference on Computer Vision, pp. 250-266. Springer, 2022.
Oren Rippel, Alexander G. Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir Bourdev. Elf-ve: Efficient learned flexible-rate video coding. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14459- 14459-14468, 2021. 01421.
doi: 10. 10.1109//CCV48922..009
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confee ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2025
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV). 115 1-252, 2015. doi: 100/0/ 1263-015-0816-y.
Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, sheng zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, 2024. URL net/ / forum?id-Re) Rdw.VVL......
Wang Shen, Wenbo Bao, Guangtao Zhai, Li Chen, Xiongkuo Min, and Zhiyong Gao. Blurry video frame interpolation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5114-5123, 2020.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 3626-3636, 2022.
K Soomro. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1rXiv:1212.04022 2012.
Shuochen Su, Mauricio Delbracio, Jue Wang. Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. Deep video deblurring for hand-held cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1279-1288, 2017.
Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand, Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on Circuits and Systems for Video Technology, 22(12):1649-166 2012. doi: 10. 1000TTCST..00..22211110
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECC ECCV 2020: 16th European Conference, Glasgow. UK, August 23-28. 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Iliia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang. Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers, 2023.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng. Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.09399 2024a.
Xiang Wang, Shiwei Zhang, Hangjie Yuan, Zhiwu Qing, Biao Gong, Yingya Zhang, Yujun Shen, Changxin Gao, and Nong Sang. A recipe for scaling up text-to-video generation with text-free videos. In CVPR, 2024b.
T. Wiegand, G.J. Sullivan, G. Bjontegaard, and A. Luthra. Overview of the h.264/ave video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7):500-576, 2003. doi: 10.11099/CSST..003..111
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation, 2023.
13

---

Published as a conference paper at ICLR 2025
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021a.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.0015 2021b.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 2024.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. MAGVIT: Masked generative video transformer. In CVPR, 2023a.
Lijun Yu, José Lezama. Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffussin-ttkeniier is key to visual generation. 2024.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awan implicit generative adversarial networks. arXiv preprint arXiv:2202.10571, 2022.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space, 2023b.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18456-18466, 2023c.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation, 2023.
Jiangning Zhang. Chao Xu, Liang Liu, Mengmeng Wang, Xia Wu, Yong Liu, and Yunliang Jiang. Dtvnet: Dynamic time-lapse video generation via single still image. In European Conference on Computer Vision, pp. 300-315. Springer, 2020.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Sijie Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cv-vae: A compatible video vae for latent generative video models. arXiv preprint arXiv:2405.20279, 2024.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URL https: /tt//ttttuu/ github..h...eegggoggggg com
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
B RELATED WORKS
Video Compression Traditionally, video compression has relied on handcrafted algorithms such as H.264 (Wiegand et al., 2003) and H.265 (Sullivan al., 2012). These algorithms achieve excellent compression rates but require video coding experts to meticulously design and optimize the various components of the compression pipeline. In contrast, neural networks that can automatically learn important features during training have emerged as strong contenders (Mentzer et al.. 2022; Djelouah et al.. 2019; Rippel et al., 2021; Lin al.. 2020). Pioneering works such as DVC (Lu et al., 2019) mimic traditional compression methods, replacing certain components within the framework with neural networks. Following works (Agustsson al., 2020; Lin al., 2020; Djelouah et al., 2019; Rippel et al., 2021) build upon this framework, introducing enhancements to various modules. Branching away from the complex residual coding- based framework which consists of many modules, some works (Habibian et al., 2019: Mentzer et al., 2022; Hu et al., 2023b) opt for simpler autoencoder- based approach. For instance, Habibian et al. (2019) uses a 3D autoencoder and an auto-regressive prior coding model, Hu al. (2023b) combines previous modules for compression in feature space using an autoencoder-style network, and Mentzer et al. (2022) utilizes image autoencoders to obtain frame- -wie representations of videos, which are then quantized through trained Transformer (Vaswani et al., 2017). Except for few works (Hu et al., 2023b; Mentzer et al., 2022; Habibian et al., 2019) that perform dimension reduction, although not significantly, the primary focus of neural compression has been on reducing bitrate rather than the dimension.
Video Compression for Generation In recent years, generative modelling (Esser et al., 2021; Yu et al., 2024; Rombach et al., 2022; Yu et al., 2023a; Chang et al., 2022) has ushered a new era of neural video compression approaches, with a primary focus achieving low-dimensional latent rep- resentations. The emphasis on obtaining low-dimensional latents because highly compressed latents allow generative models to ignore imperceptible pixel variations, providing efficient generation en- vironment and reducing computational costs. These approaches can broadly be divided into those that learn discrete latent representations and those that learn contimuous latent representations. Discrete neural video compression approaches mainly follow a pioneering work (van den Oord et al., 201 8), which uses a learned codebook through the k-means clustering algorithm for image compression. For example, Yan et al. (2021b) follows a similar approach but uses 3D convolutions to properly model spatio-temporal dynamics for effective video compression. Recently, follow-up works (Yu al., 2024; Mentzer et al., 2023) have focused on non- non-learnable and look-up free quantization schemes instead of the k-means clustering to learn a bigger codebook for more faithful reconstructions.
On the other hand, continuous neural video compression approaches follow after the VAE (Kingma & Welling, 2013) framework, incorporating perceptual losses and GAN training to enhance perceptual quality, as introduced in Rombach et al. (2022) for image compression. Notably, techniques such as Zeng et al. (2023); Qing et al. (2023); Xing et al. (2023); Jain et al. (2024); Hu et al. (2023a) utilize pretrained image autoencoder for independent spatial compression of video frames. However, this approach may induce flickering artifacts (Blattmann et al., 2023c). To address this issue, Blattmann et al. (2023c;a) fine-tune pretrained image autoencoder with added 3D convolutions to align latents for temporal reasoning, employing a training framework adapted for videos. Another noteworthy approach is Yu et al. (2023c), which performs spatial compression by learning triplane low- dimensional representation of the video. However, these approaches neglect to exploit the well- documented temporal redundancies in videos (Lu et al., 2019; Lin et al., 2020; Hu et al., 2023b), resulting in larger latent dimensions. To address these limitations, our work leverages temporal redundancies and proposes a continuous video autoencoder that performs compression in both the spatial and temporal domains. We also empirically demonstrate the potential of our proposed model as a robust autoencoder for video generation training.
B.I EXPERIMENTAL ANALYSES
B. LATENT SENSITIVITY
In this analysis, investigate the robustness of the video VAE decoder to erroneous latent predictions from a generative model. To do so, we intentionally corrupt the encoded latents using various corruption methods and assess the quality of the video reconstructed by the decoder. First, we use
15

---

Published as a conference paper at ICLR 2025
Table 5: Experimental analyses on latent sensitivity and arbitrary sequence length on Adohe240 dataset
Spatio- Temporal Compression
Method
Quantization
Sequence Length
Ux h
0
0.4
0.8
Ip32
k/16
mint8
T 8
T= 61
T 128
LDM Viden LDM
1 8
28.53
27.86
26.91
28.53
28.41
15.56
28.33
28.53
28.53
X 8
29.51
28.30
27.02
29.51
29.49
16.50
29.51
29.16
04.04
Open-Sone
28.43
Ours
8 8
31.50
30.96
29.03
31,50
31.42
17.49
31.50
31.22
30.83
Ours (T 64)
8
31.22
30.89
28.92
31,22
31.20
17.16
Input
0. fp33
0.4
0.8
bfl6
nin8
Buggk
ahan 2 cT-bo
nann FRoz dr
Figure 6: Qualitative analysis on the effect of noise corruption and quantization on encoded latents
noise corruption scheme where we add different magnitudes of Gaussian noise (of the same size as the encoded latent) the encoded latent i.e. + a NO 1), where denotes the encoded latent. We experiment with 3 scales of noise: a 0 (clean), a = 0.4 (medium noise) and a = 0.8 (large noise). As anticipated, the decoded video quality decreases in proportion to the scale of the added Gaussian noise as shown in Table 5. However, our video decoder still performs reasonably well even when the latents are corrupted with significant noise. similar conclusion can be drawn from the visualization in Fig. 6, where the overall structure of the video is well-maintained for = 0.8 despite the noisy output. better robustness to noise for longer input sequence lengths. For example, there is 0.33 dB drop for a = 0.4 when = 64 compared to 0.54 dB drop when = 8. This result further underscores the importance of temporally agnostic video VAEs. Please refer the supplemental video for qualitative examples.
also noteworthy from Table 5 that our video VAE exhibits relatively
We also investigate the sensitivity of video VAE latent quantization. We achieve this by storing the encoded latents in various data type formats: 32-bit floating point (/p32), 16-bit brain floating point (bf16), and unsigned 8-bit integer (uin18). As demonstrated in Table 5, by16 quantization performs comparably to fp32 despite the reduced precision, while uint8 quantization leads to poor decoding performance across all methods. This observation is further supported by Fig. where our model with 6 decodes video frames indistinguishable from those generated with fp32, whereas uint& only preserves the edges of the frame. This failure occurs because uint8 significantly shifts the distribution of the encoded latent, which is optimized to resemble a standard normal distribution via KL regularization (refer to Sec. 2.5).
B.1.2 SEQUENCE LENGTH
The applicability of a video VAE hinges on its ability to robustly encode and decode arbitrarily long videos sampled at varying lengths. In Table 5, we assess different approaches this regard using three different sequence lengths: T = 8, T = 64, and T 128. As expected, methods that only perform spatial compression (Blattmann et al., 2023c) tend to perform consistently across different sequence lengths compared to those that perform both spatial and temporal compression (Lab & etc., 2024). For example, Video LDM (Blattmann et al., 2023c) (1x in the temporal dimension) shows a 0.47 dB performance drop when the sampling rate changes from T 8 to T 128, while Open-Sora-Plan (Lab & etc., 2024) (4x) exhibits larger drop of 1. 1.09 dB under the same conditions. In comparison, our method (8x) experiences 0.67 dB performance drop. These results show that our model, even with a higher compression rate, generalizes reasonably well to varying sequence lengths.
16

---

Published as a conference paper at ICLR 2025
Adobe240
Adobo240
0.0 D00GFFT
Ours
Ours
Ours
Open-Sore--P
32
VPdeoGPT
Curs
PSVR (083)
UBN
Durs
Open Stre-
Curs
27
0.1
26
10
122
0.0
10r
10
Compression rate (log-stale]
Compression rate (logrscale)
Xiph-2K
Xiph-2K
34
1.0
0.9
Ours
0.3
JTM
Open-50ra
0.7
PSNR (0B)
-2
Ours
0.6
Ours
Weddeo LDM
LOW
Open Sare--SSe-PPeee Plan
28-
Curs
10
Compression (log-scale)
Compression rate (logrscele)
DAVIS
DAVIS
3indeo LDM
FEEEAAM
Durs
Ours
Open Sera Pan
PSWR (dB)
Ours
Ours
SS/M
Curs
26-
Open
0.4
0.5
26
0.2
25
0.1
74
103
0.
107
107
Campression rata (log-scalo)
Compression rate (lng (rg-)rale)
Figure 7: PSNR and SSIM analysis across spatio-temporal compression rates
Table 6: Effect of padding in causal 3D convolution
Method
PSNR
SSIM
T.PIPS
zero padding
0.232
constant pudding
28 28.86
0.620
0,231
0.22
roplication padding
0.226
B.2 VIDEO COMPRESSION COMPARISON
Here, present visualizations of the results summarized in Table 1 of our paper. In Fig. 7, illustrate the PSNR and SSIM (on the y-axis) against the spatio-temporal compression rate of our model and competing approaches the Adobe240 (Su al., 2017), DAVIS (Pont-Tuset et al., 2017). and Xiph-2K (Niklaus & Liu, 2020) datasets. As evident from the plots, our model achieves highly competitive, if not superior, performance compared competing approaches while accomplishing significantly higher dimension reduction.
B.3 PADDING IN CAUSAL 3D CONVOLUTION
Here, we explore the impact of various padding schemes in the causal 3D convolutional layers within our video VAE. We conduct experiments using three different temporal padding schemes: zero, constant, and replication. Training our 8 8 x model with each padding scheme, we evaluate their performance on the DAVIS dataset (Pont-Tuset et al., 2017). The results, outlined Table 6, indicate that replication padding yields the best performance. Moreover, we observe that replication padding leads to temporally smoother videos compared to zero or constant padding.
17


Reviews:
[{"review_id": "qn4fn7vctv", "reviewer": "Reviewer_76n7", "summary": "The paper presents a novel approach to joint image and video compression via a causal variational autoencoder (VAE) that leverages 3D convolutional and self-attention mechanisms. Designed to tackle the challenge of temporal and spatial redundancy in video data, the proposed model incorporates a dual-path architecture for downsampling and upsampling, as well as a flow regularization loss to maintain temporal coherence. Key contributions include a scale-agnostic encoder, a dual-path network for robust spatio-temporal feature extraction, and a loss function to enhance motion preservation in compressed sequences. The experimental results show the model’s superiority in video fidelity and compression rates over existing methods, particularly in handling large motions and maintaining temporal consistency.", "questions": "Please see Weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Hybrid Approach: The paper successfully integrates causal 3D convolutions with self-attention layers to handle temporal and spatial redundancy in video compression, creating a versatile model capable of high-quality compression for both images and videos. This hybrid approach is particularly relevant as it extends beyond existing methods that often separate these tasks or overlook temporal redundancy.\nRobust Encoder-Decoder Design: By leveraging a dual-path downsampling and upsampling mechanism, the proposed model achieves a high compression rate without significant quality loss, evidenced by its superior PSNR and SSIM scores in Table 1. Additionally, the FILM encoder’s ability to handle large motion in compressed videos aligns well with contemporary needs in video compression.\nComprehensive Evaluation: The paper rigorously evaluates its model across multiple datasets and compression settings, supporting its claims with ablation studies and comparisons to several state-of-the-art methods. The qualitative results further emphasize its performance in preserving motion fidelity and detail, especially for fast-moving objects.", "weaknesses": "* The paper should cite the recent ECCV 2024 work, Hybrid Video Diffusion Models with 2D Triplane and 3D Wavelet Representation, as this paper also explores autoencoder backbones and addresses related challenges in video compression. A comparative analysis, including quantitative and qualitative evaluations against this baseline, would strengthen the evidence for the proposed model’s advantage.\n* The claim that the dual-path network is \"temporally agnostic and can encode and decode arbitrarily long videos at varying lengths\" may be overstated. The performance improvements in Table 4’s ablation study do support better temporal adaptability; however, the reliance on learnable kernels suggests some degree of generalization error remains, as evidenced by performance drops with different sequence lengths. It would be more accurate to moderate this claim, acknowledging the limitations indicated by remaining performance variability.\n* Given the paper’s focus on maintaining high spatio-temporal quality in compressed videos, metrics specifically targeting spatio-temporal fidelity must be also compared, such as STREAM (ICLR 2024 STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models). These metrics could provide a more nuanced understanding of the model’s strengths and areas needing improvement in terms of temporal coherence and spatial detail.\n* The model’s large size and complexity might raise concerns about overfitting, particularly when evaluated on datasets like UCF101 and SkyTimelapse. To address this, a thorough analysis for potential overfitting or memorization is recommended, especially in cases where generated samples closely resemble training data. Expanding the qualitative analysis in Figure 4 to include more diverse sample comparisons would help clarify this aspect.\n* Table 1 demonstrates the model’s robustness at more severe compression rates (e.g., 4\\times8\\times8 and 8\\times8\\times8), but the performance at commonly used settings such as 1\\times8\\times8 and 4\\times4\\times4 is not discussed. Adding a direct comparison at these standard settings would provide a more balanced view, allowing for an \"apple-to-apple\" comparison with other models. Such an addition could reveal how the model’s advantages scale across different compression intensities, offering deeper insights into the model’s comparative performance under varying constraints.\n* To provide readers with an estimation of computational resources and convergence speed, including additional details on training time until convergence would be beneficial. This information would help contextualize the model’s practical application feasibility.\n* Please elaborate more on 3D Model Extension in GAN Training.\n* Although the introduction mentions experiments on noise corruption and varying sequence lengths, these are not included in the main text. Readers may find it disorienting without a clear reference directing them to the Appendix for these analyses. Please note this clearly. \n* Is normalization applied during the concatenation of FILM’s outputs? Wouldn't it be problematic if they are not normalized (due to different output statistics from different scales)?\n* Any specific reason for using 1+T over T? \n* In the conclusion, \"3 key contributions\" -> \"three key contributions\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "adDMo1yYGy", "reviewer": "Reviewer_LuUv", "summary": "The paper introduces a novel architecture for 3D Causal VAE aimed at image and video compression, addressing three key issues found in existing video VAE architectures: \n\n1. Handling Small and Fast-Moving Objects at Higher Compression Rates: Standard encoders struggle with these objects. The authors tackle this by proposing a FILM-based encoder.\n\n2. Downsampling and Upsampling Challenges: Non-learnable downsampling loses high-frequency spatiotemporal (ST) information, while learnable downsampling tends to overfit. To address this, the authors propose a novel Down and Upsampling layer design that employs a dual-path network with parallel branches utilizing both learnable and non-learnable kernels.\n\n3. Motion Reconstruction Quality and Smoothness: To enhance these aspects, the authors introduce a novel Flow Regularization loss that facilitates in Video VAE training.\n\nThe authors trained their 3D VAEs at various compression rates (T × H × W) such as 4 × 8 × 8 (256), 8 × 8 × 8 (512), 16 × 8 × 8 (1024), and 16 × 16 × 16 (4096). Their experiments showed strong results when compared to other baselines.\n\nOverall, this is a good paper that introduces novel architectural decisions for Video-VAE and shows strong experimental results. However, it would benefit from including additional experimental results and details (See Weaknesses and Questions).", "questions": "My main concerns and comments regarding the paper are related to the experiment section:\n\n1. Tables 1 and 2 would benefit from including results for CogVideoX VAE [1] and CV-VAE [2], adding rFVD metric,  providing information about evaluation resolution and inference time of the models. Please refer to weaknesses 1, 2, 4, and 6 for more details.\n\n2. The paper would benefit from adding limitation analysis of the proposes method. Especially interesting to see how the method generalizes to different video resolutions (128x128, 256x256, 512x512, 480p, and 720p) in comparison with other methods. For more details see weaknesses 5 and 7.\n\n3. Table 3 provides video generation results for relatively simple datasets and tasks. It is recommended to test the model on text-conditional video generation on the WebVid dataset if possible. Refer to weaknesses 3 for more details.\n\n[1] Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., ... & Tang, J. (2024). CogVideoX: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072.\n\n[2] Zhao, S., Zhang, Y., Cun, X., Yang, S., Niu, M., Li, X., ... & Shan, Y. (2024). CV-VAE: A Compatible Video VAE for Latent Generative Video Models. arXiv preprint arXiv:2405.20279.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Importance: This paper addresses a critical issue of training a high-quality VAE for image and video compression.\n\n2. Novelty: The paper introduces a unique and previously unexplored approach to tackle the aforementioned problem.\n\n3. Results Quality: Experimental results demonstrate strong performance compared to existing baselines. Notably, the paper provides results for very high video compression rates (16 × 8 × 8 and 16 × 16 × 16).\n\n4. Clarity: The main text of the paper is well written and easy to follow.\n\n5. Ablations: Ablation studies clearly illustrate the positive impact of the proposed architectural decisions.", "weaknesses": "1. Missed comparisons: Tables 1 and 2 do not include comparisons with CogVideoX VAE [1] and CV-VAE [2], which are significant baselines in this field.\n\n2. Video quality metrics: Table 1 reports only frame-wise video quality metrics and neglects any temporal consistency metrics. It is recommended to measure the Fréchet Video Distance (FVD) between original and reconstructed videos (rFVD) to provide a more comprehensive evaluation.\n\n3. Video generation results: The video generation results in Table 3 were conducted on relatively simple video datasets such as SkyTimelapse and UCF-101. It is recommended to include experiments on more complex datasets and tasks, such as text-conditional video generation on the WebVid dataset, to better assess the proposed method's capabilities.\n\n4. Evaluation resolution: The authors have not specified the evaluation resolution for videos in Tables 1 and images in Table 2, which is crucial for assessing the results accurately. \n\n5. Analysis on different resolutions: The paper lacks an analysis of how the proposed VAE performs across different resolutions compared to other baselines. The performance on high-resolution videos (e.g., 720p) is particularly relevant and should be explored.\n\n6. Inference time: Information about the inference time of the models in Tables 1 and 2 is missing. Including this data would provide a better understanding of the practical utility of the proposed method.\n\n7. Method’s limitations: The paper does not provide an analysis of the limitations of the proposed method, which is necessary for a comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ia5jKJcLMi", "reviewer": "Reviewer_tD7c", "summary": "This paper introduces a video VAE for video compression and generation tasks. It features 4 major components: a causal 3D residual block, spatio-temporal downsampling module, spatio-temporal attention module, and FILM encoder. The proposed VAE is tested with the autoencoding and generation tasks.", "questions": "(1) The title “joint image and video compression” is misleading. I believe the task is focused primarily on video generation, not video transmission or compression (i.e. the autoencoding task). If the task is video compression/autoencoding, the resulting rate-distortion performance of the proposed method should be compared with that of the prior works on end-to-end learned video compression. However, this was not done when the authors reported results on the video/image autoencoding task. Just because the resulting latents have smaller dimensionality does not suggest that they would have smaller bit rates.  \n\n(2) In Table 1, Video LDM (1 x 8 x 8) actually performs quite well particularly on DAVIS with large motion, as opposed to your better performing variant (4 x 8 x 8). Notably, Video LDM has relatively better TS performance. This leads to an impression that the proposed method cannot model well fast-motion sequences.\n\n(3) The model size and MAC/pixel are currently missing in Table 1. Also, except TS, all the quality metrics are for images, not video. Why not use VMAF?\n\n(4) I wonder if the competing methods adopt the GAN loss for training. This needs to be clarified in Table I. In addition, it is unclear how much the GAN loss contributes to the performance of the proposed method.\n\n(5) For the image autoencoding task, how is the proposed method used?\n\n(6) For the video generation task, it is unclear whether the proposed VAE is trained or fine-tuned together with the diffusion model. It appears to me that the proposed VAE is pre-trained to generate latents that follow a simple Gaussian distribution. \n\n(7) In addition, the proposed VAE can be a stand-alone approach to video generation. I wonder how it performs as compared to the diffusion-based modeling of the video latents.  \n\n(8) In Figure 4, I wonder whether the proposed method can generate well fast-motion sequences. \n\n(9) FILM encoder is shown to be effective in the ablation study. But, the proposed VAE does not appear to work well on the autoencoding task in Table I, as compared to Video LDM.  This is a bit confusing.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method is evaluated with both the video/image autoencoding task and the video generation task. Moreover, extensive ablation studies were conducted to validate the effectiveness of the proposed method.", "weaknesses": "(1) The title “joint image and video compression” is misleading. I believe the task is focused primarily on video generation, not video transmission or compression (i.e. the autoencoding task). If the task is video compression/autoencoding, the resulting rate-distortion performance of the proposed method should be compared with that of the prior works on end-to-end learned video compression. However, this was not done when the authors reported results on the video/image autoencoding task. Just because the resulting latents have smaller dimensionality does not suggest that they would have smaller bit rates.\n\n(2) In terms of the newly proposed components, I have the impression that their novelty is not very high.\n\n(3) The necessity of causality is unclear in the context of video generation. If the aim is to enable both image and video generation, the image generation task is not explored in the current writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TuQBrxhN7y", "reviewer": "Reviewer_MHdT", "summary": "This paper presents several improvements to the Video VAE architecture. First, it introduces 3D causal convolutions to encode images and videos simultaneously. Second, it employs a 2 + 1D attention layer for separate interactions of spatiotemporal information, referred to as the STAttnBlock in the paper. Third, it utilizes a two-pathway approach in the upsampling module, with one pathway being learnable and the other non-learnable. Lastly, an interesting contribution is the introduction of the FILM encoder, which shares encoder weights to handle videos resized to different scales. The experiments claim that the proposed method achieves significant results. However, I think that important parameters are not presented in the experiments, and these parameters could greatly affect the experimental metrics.", "questions": "1. Has the author explored the number of latent channels? If so, I would like to see those findings and understand why they were not presented in the paper.\n2. How can the authors explain the superior performance of the 8x8x8 compression rate model over the 4x4x4 compression rate model, given the 64 times difference in compression ratio? Similarly, the 16x8x8 model outperforms the 4x4x4 model in LPIPS metrics, despite a 128 times difference in compression ratio. I believe this discrepancy cannot be solely attributed to training tricks or model design.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed FILM Encoder is designed well: its pyramid structure enhances the model's ability to reconstruct targets of varying sizes, improving detail retention and overall reconstruction quality. The use of shared modules also bolsters the model's robustness. Additionally, the ablation experiments presented in the paper demonstrate that the FILM Encoder indeed improves reconstruction performance.\n2. The Flow Regularization introduced in this paper will contribute positively to motion continuity in video reconstruction.\n3. The experiments conducted on video generation effectively demonstrate that this VAE structure can be utilized for downstream diffusion training.", "weaknesses": "Although the model is well designed, I am skeptical about the correctness of the experiment. In previous work (such as the SD3 report and the CogVideo X report), the number of latent channels has been emphasized as having a significant impact on reconstruction performance and metrics. The CogVideo report demonstrates that a model with a configuration of 4x8x8 and 8 latent channels achieves about 3 points higher PSNR compared to a model with 32 latent channels. However, in the experiments of this paper, the 4x8x8 model shows nearly 4 points higher PSNR than the compression rate model, while the 8x8x8 compression rate model surpasses the PSNR of the 4x4x4 token compression rate VideoGPT by 0.23 points, despite having a 64 times higher compression rate. Even in the 16x8x8 model, the LPIPS metric remains superior to VideoGPT, which is quite surprising. The paper does not specify the number of latent channels in the model, raising concerns about the comparability with similar models (e.g., Open-Sora, which has only 4 latent channels). I believe that the overall compression rate (considering latent channels) is a fundamental factor limiting VAE reconstruction performance. If the total compression rates were the same, the differences in metrics would likely not be so pronounced. Therefore, I am concerned about the rigor of the experiments presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KnSp8IllIH", "reviewer": "Reviewer_ea5M", "summary": "This paper presents valuable improvements in the video VAE domain. The main innovations include a multiple-scales encoder, dual-path spatio-temporal sampling module, and flow regularization loss. The experimental design is comprehensive, particularly in downstream task validation.", "questions": "1. Clarification of the Ablation Study\n   How can the ablation study section be enhanced to clearly demonstrate the impact of adding or removing individual metrics? Additionally, would it be beneficial to include comparisons for configurations such as baseline + a, baseline + b, baseline + c, baseline + d, and the combined baseline + a + b + c + d?\n\n2. Inclusion of Visualization Analysis in Experiments\n   While the experiments are thorough and encompass downstream tasks like image and video generation with metric comparisons, the analysis lacks essential visualization components. Could the authors incorporate visualization analyses to complement the metric-based evaluations?\n\n3. Comparison with SVD-VAE in Table 1 \n   I recommend adding a comparison with SVD-VAE in Table 1 to provide a more comprehensive evaluation of the proposed method against existing models.\n\nThese refined questions and suggestions aim to provide constructive feedback that can help clarify ambiguities, address potential limitations, and enhance the overall quality of the work during the rebuttal and discussion phases.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper presents valuable improvements in the video VAE domain. The main innovations include a multiple-scales encoder, dual-path spatio-temporal sampling module, and flow regularization loss. The experimental design is comprehensive, particularly in downstream task validation.", "weaknesses": "If the following areas can be optimized, the quality of the paper will improve. Key areas for enhancement include:\n\n1.Enhanced Visualization in Experiments\nCurrent Observation: While the experiments are comprehensive, covering downstream tasks such as image and video generation, this section primarily relies on metric comparisons and lacks crucial visualization analysis.\nRecommendations:\nFigure 4 Enhancement: Add a comparison of videos generated by OpenSora and the combination of your method with OpenSora. This will provide a clearer visual demonstration of the improvements introduced by your approach.\nAblation Study Visualization: In Figure 5, include visual results showcasing the effects of individual components. For example, display results with and without Spatio-Temporal Down/Upsampling, with and without flow regularization, and with and without Spatio-Temporal Attention. This will help illustrate the contribution of each component to the overall performance.\n\n2.Comparison with SVD-VAE\nCurrent Observation: The paper lacks a comparison with SVD's Variational Autoencoder (SVD-VAE), which is relevant to the presented work.\nRecommendation: Incorporate a comparison with SVD-VAE in Table 2. This will provide a more comprehensive evaluation of your method against existing approaches and highlight its relative strengths and weaknesses.\n\n3.Innovation of Improvement Methods\nCurrent Observation: The three proposed improvement methods in the paper are generally intuitive but lack a certain degree of innovation.\nRecommendation: To enhance the paper's contribution, consider elaborating on the novelty of each improvement method. Provide deeper insights into how these methods advance the current state of the art beyond intuitive enhancements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)