VideoPanda: Video Panoramic Diffusion With Multi-view Attention

Authors: Kevin Xie, Amirmojtaba Sabour, Jiahui Huang, Despoina Paschalidou, Umar Iqbal, Sanja Fidler, Xiaohui Zeng
Primary area: generative models
Keywords: video generation, diffusion model, panorama

Abstract:
High resolution panoramic video content is paramount for immersive experiences in Virtual Reality, but is non-trivial to collect as it requires specialized equipment and intricate camera setups. 
In this work, we introduce \ourmodel, a novel approach for synthesizing $360^\circ$ videos conditioned on text or single-view video data. \ourmodel leverages multi-view attention layers to augment a video diffusion model, enabling it to generate consistent multi-view videos that can be combined into immersive panoramic content. \ourmodel is trained jointly using two conditions: text-only and single-view video, and supports autoregressive generation of long-videos. 
To overcome the computational burden of multi-view video generation, we randomly subsample the duration and camera views used during training and show that the model is able to gracefully generalize  to generating more frames during inference.
Extensive evaluations on both real-world and synthetic video datasets demonstrate that \ourmodel generates more realistic and coherent $360^\circ$ panoramas across all input conditions compared to existing methods.
Visit the project website at https://mvpanovideo.github.io/VideoPanda/ for results.

Decision: reject
Reviewer scores: 3, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730118009857

thread_type:
official_review

<thread_object>
creator: reviewer_u9nk

<content>
summary:
This paper proposes a framework for generating panoramic videos, based on a pretrained video generation model. The authors introduce multi-view generation capabilities by embedding several Multi-view Blocks. Additionally, they propose a random matrix training strategy, using videos of random frames and views to train the model, which increases the model's generalization to longer/more viewed videos while conserving computational resources. The authors trained the model using WEB360 and tested their method on 100 in-distribution and 100 out-of-distribution videos.

strengths:
1. The authors successfully adapted a pretrained video generation model to the multi-view/panoramic video generation task, achieving relatively good visual results.
2. The proposed method supports using text or video as conditions to generate panoramic videos.

weaknesses:
1. In terms of model design, the authors introduced a Multi-view Block on top of SVD (Stable Video Diffusion) to enable multi-view generation, a concept similar to MVDiffusion. However, while MVDiffusion focuses on panoramic image generation, this paper is designed for panoramic video generation, adding a temporal block (see Figure 2). Essentially, this paper can be seen as an application combining SVD and MVDiffusion, with the spatial and temporal blocks derived from SVD and the multi-view block from MVDiffusion. From this perspective, the novelty of the proposed method may be somewhat limited, so it would be beneficial for the authors to explain how their approach differs from these methods and whether they have made specific design choices to address challenges unique to this application.
2. Regarding the training strategy, the proposed random matrix strategy is essentially a compromise due to limited computational resources; theoretically, using more views or frames in training would yield better results. From the experimental results in Table 3, it can be seen that improvements in FID and FVD scores are achieved at the cost of PSNR (rows 2 and 3).
As for the multi-task strategy proposed by the authors—randomly dropping some conditions (such as text, the first frame, or single-view video), or conditioning on only part of the information or a subset of modalities—is a common trick in training diffusion models. For example, dropping text is often used in text-to-image/video tasks [GLIDE, SVD, Imagen, etc.], or conditioning on text and image in image editing [InstructPix2Pix, Tim Brooks et al. 2022]. Therefore, it would be helpful for the authors to clarify how their approach differs from these methods to demonstrate the novelty of their method.

questions:
1. In Table 3, after applying the multi-task training strategy, several metrics, such as FVD and PSNR, show a decline. Could the authors provide an appropriate explanation for this?
2. In section 4.6, the authors tested autoregressive long video generation, producing videos up to 61 frames. Have the authors attempted to generate even longer videos? As video length increases, does generation quality continuously degrade? If so, what feasible strategies might help mitigate this issue?
3. In Table 1, the authors compare their method with 360DVD and report performance improvements. However, 360DVD uses the text-to-image model SD1.5, while the proposed method uses the SVD video generation model, which inherently offers an advantage in video smoothness. Did the authors conduct a relatively fair comparison, such as applying SVD to 360DVD or using SD1.5 with the proposed method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533058660

thread_type:
official_review

<thread_object>
creator: reviewer_eysz

<content>
summary:
This work present a method for generating videos from text or single-view video data, which employs multi-view attention to enhance a video diffusion model and produce consistent multi-view content suitable for immersive panoramas. The work demonstrates performance improvements and provides code.

strengths:
1. The ideas are very easy to understand, reliable and well written.

2. The boost in quantitative metrics looks good.

3. Open source code for easy reproduction by readers

weaknesses:
1.Poor qualitative results. I feel that the overfitting is evident in the effect, that is, the watermarks are being generated, and on closer inspection you can see the airpano.

2.Some of the diagrams in the paper don't have good consistency when zoomed in.

3.The technology is low-innovative, and multi-view concerns are common in the 3D AIGC field[1].

[1] Liu J, Huang X, Huang T, et al. A comprehensive survey on 3D content generation[J]. arXiv preprint arXiv:2402.01166, 2024.

questions:
See weakness.
I will check the author's response and revise the score after combining other review comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730600256147

thread_type:
official_review

<thread_object>
creator: reviewer_czzr

<content>
summary:
This paper presents a novel method for long panoramic video generation from a text prompt or a perspective video. Different from existing 360-DVD that generates equi-rectangular panorama video directly, it builds on existing video diffusion models by adding multi-view attention layers to generate consistent multi-view outputs. It is expect to help maintain the capability of the pre-trained model without domain gap hindering. Extensive experimental results tells the superiority of the proposed method, especially the quantitative evaluation and user study. Anyhow, it still suffers from obvious visual artifacts, as shown in the demonstrated video results.

strengths:
- This paper is well written and easy to follow. Every design is well-motivated and clear clarified.

- The key idea to formulate panorama video generation as multi-view video generation makes sense and novel. Quantiative results (i.e. Table 1) evidence the superiority over existing baseline method 360DVD.

- The experimental evaluation is well-conducted and sufficient. Multiple metrics are adopted and user study is performed.

weaknesses:
-The major weakness is that most of the generated panorama videos are not good as expected, which renders the importance of the key technical innovation not well supported.
First, most of the results present ambiguous semanic structure or broken scene, such as the [autoregressive generation] showcase "anime girl standing on a boat", and the [video-conditioned with different prompts] showcase "A view of the seashore with the sea coming up against a cliff.“ This is a unignorable weakness about the performance.
Besides, almost all results present obvious seamline across views.
It seems the newly introduced multi-view attention does not work as expected. A possible attempt is jointly finetuning the base model with LORA, which may help the model better adapt to the panorama video distribution.

-As stated above, some of results suffers from broken content and semantic ambiguity. In contrast, although showing over-smooth textures, the semantics of 360DVD are more natural and the scene of content are more identifiable than the proposed method. This impedes the justification of the proposed method is better than existing 360DVD.

-The comparative examples with 360DVD are almost static scene, which makes the evaluation less convincing. It is highly required to evaluate on cases with moving objects (such as "moving car on the street", "astronaut riding a horse on grass", etc.), because the consistency of dynamic objects is one of the major focus in video generation task.

questions:
- How many samples are used for the evaluation in Table 2? How do you collected the prompts? Do they cover good diversity?

- It is hard to understand why multi-task training is better than single task training. This is contrast with some common feeling, for example all the T2V or I2V modes are processed with fixed frames, because varied number of tokens in the attention computation may hinder the performance on a specific frames and resolution. I would like to see more elaboration over this.

- The artifacts of full matrix shown in Figure. 7, actually also happen in the result of the proposed method (i.e. random matrix), such as in the [video-conditioned generation] showcase "A view of a panda's face peeking from behind a tree branch amidst lush green foliage“.
So it seems both setting suffers from the over blurry and structure mixture artifacts.

- According to the paper, the model is finetuned with the base model layers frozen. In my understanding, the distribution of each view of the panorama still deviate with the original real-world image. Would it be helpful to the generation quality (such as the ambiguous/broken scene semantics) if the base model is tuned with LORA?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691834950

thread_type:
official_review

<thread_object>
creator: reviewer_txrj

<content>
summary:
This paper introduces a method named VideoPanda, designed to synthesize 360-degree videos from text prompts or single-view videos. VideoPanda builds on existing video diffusion models by adding multi-view attention layers to produce consistent multi-view outputs. Both quantitative and qualitative results are presented.

strengths:
The concept of expanding single-view videos into panoramic videos is interesting. Some of the generated results appear visually good.

weaknesses:
1. The model appears to be overfitted to the WEB360 dataset. The test videos are directly sourced from the WEB360 dataset or selected from similar scenes on the airpano channel (e.g., the pandas in "100329" from the WEB360 dataset, and the ice and mountains in "100666"). The results generated by the method often contain the airpano watermark, whereas the few results without watermarks on the webpage exhibit strange artifacts in other regions. This indicates a lack of generalization to real-world scenarios.
2. The authors state, "Since the out-of-distribution condition inputs do not originate from 360 videos, we cannot compute metrics that require ground truth images, such as pairwise FVD." However, to the best of my knowledge, FID and FVD do not necessarily require paired prediction-ground truth data. FID and FVD are distribution-matching metrics and do not require one-to-one correspondence between generated and real data.
3. There are no ablation studies on the multi-view attention mechanism. The paper does not clearly explain the differences between the proposed multi-view attention and existing methods, such as MVDream.
4. The paper lacks experiments on conditions with varying fields of view (FOV) and view directions. The results only demonstrate conditioning using a 0-degree latitude image. In practical scenarios, adapting to different FOVs and viewing angles is a common requirement.

questions:
1. Can VideoPanda generate videos conditioned on different FOVs and view directions?
2. What is the specific implementation of the multi-view attention, and how does it differ from existing methods?
3. Can the authors provide more quantitative and qualitative results without watermarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732217698740

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary Response to All (we will also individually reply to each reviewer shortly)

comment:
We thank all the reviewers for their time and great feedback! To address some of the common concerns raised by the reviewers, we made the following additions. 


**1 Expanding and Improving OOD Evaluations**

We greatly expanded and improved our OOD evaluations for both the text-conditional and video-conditional settings.

Our text-conditional OOD evaluations are now using the full VBench suite of prompts (1700+ prompts) with great diversity and coverage of different dimensions and categories.

Our video-conditional OOD evaluations are using the 50 videos we curate from publicly available AI generated videos and they can feature cases with non-standard FOV and elevation.

We show many additional qualitative examples now on our website (along with comparison to 360DVD). Please see the qualitative results at links below:

OOD Text-cond VBench prompts: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html 

OOD Col-cond AI generated videos: https://mvpanovideo.github.io/VideoPanda/release_col_cond_comp/release_col_cond_comp/index.html 

Please take a look!

Update: We have computed quantitative metrics for our OOD evaluation comparing VideoPanda to 360DVD for the text-conditional setting on using all 946 VBench-all_dimension prompts including Clip score, FVD to HDVila3k and FID to HDVila3k first frames. Unpaired FVD and FID was based on suggestion from Reviewer TXRj. We thank them very much for the suggestion. Our original VideoPanda model on VideoLDM (presented in the paper) performs better than 360DVD across all the metrics on this benchmark. Please see the quantitative metrics in the comment **(Point 7 OOD Evaluation: Quantitative Metrics)** above.


**2 Results on new base model (CogVideoX-2B)**

Many reviewers were concerned about the OOD generalization ability of our VideoPanda model. 
To clarify, in the main paper we used the VideoLDM model (not SVD) which was presented in the work Align your Latents [Blattmann et al., 2023a] as cited in our paper. This model is a text-to-video model based on SD1.5 and is quite similar in design to AnimateDiff.


It shows quite good generalization in the video-conditional setting but some OOD prompts can be challenging in the text-conditional setting. Please see the extended qualitative OOD video and text-conditional results on the updated websites linked above (in point 1) for a better understanding of the model’s capabilities.
We found that this weakness on OOD text-conditional setting primarily stems from the weakness of the base model itself not understanding the prompts.


So we implemented the VideoPanda method on top of a more powerful model, CogVideoX-2B which is a medium-sized transformer DiT.

Our method readily applies without any major alterations. Per-frame multi-view attention layers are added to the model in between the 3D video attention that now operates on each of the views separately. The weights of the multiview-attention is initialized from the 3D video attention layer and zero-residual initialized.
We finetune the full model end-to-end using the same dataset (WEB360). Additional training details will be included in the supplementary materials.


With no major alterations to our framework, the new model VideoPanda (CogVideoX-2B) demonstrates large improvements in OOD generalization and video quality in both the text-conditional and video-conditional settings.


We have also addressed each reviewer individually with our responses.
Once again we want to thank the reviewers for their great feedback and suggestions.

[Yang et al, 2024] Yang, Zhuoyi et al. “CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer.” ArXiv abs/2408.06072 (2024): n. pag.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732554790441

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Point 7 OOD Evaluation: Quantitative Metrics

comment:
**7 OOD Evaluation: Quantitative Metrics**

As mentioned earlier, we have improved and expanded our OOD evaluations. For the text-conditional setting, we now use the popular VBench evaluation prompts. These prompts offer good diversity and include fantastical examples, such as "pandas sitting in a cafe," to stress test OOD generalization. Qualitative visual results are shown here: https://mvpanovideo.github.io/VideoPanda/release_text_cond_comp/release_text_cond_comp/index.html
To evaluate our performance on OOD cases, we also compute and report some quantitative metrics. These are done for the text-conditional setting, in which we compare 360DVD against VideoPanda using VideoLDM (from the paper) as well as VideoPanda using CogVideoX-2B which we explained further in point 2 above.

We used the gpt enhanced prompts released by VBench here: https://github.com/Vchitect/VBench/tree/master/prompts/gpt_enhanced_prompts 

**Clip Score:** We compute a clip score between our generated videos and the input prompt to quantify video-text alignment.

**FID/FVD**: Following the suggestion by reviewer TXRj, we evaluated non-paired FID and FVD for the OOD text-conditional case (meaning that the true set of videos and our generated videos do not share the same prompts as we do not have a reference ground truth set to correspond with the prompts).
For this we decided to use the popular video dataset HDVila for the reference set. In particular, we use 3,000 random videos from HDVila for FVD computations and use the first frames from the same set for FID computations. As this dataset contains perspective view data, we also extract perspective views from our generated panorama videos in order to perform a fair comparison.

**Perspective View Extraction** In addition to the 8 horizontal (zero degree elevation) extracted views we used in the paper, we also included views with non-zero elevation. In particular, we create an additional setting where we extract 8 views in total with 4 views at negative 60 degree elevation looking downwards and another 4 views at positive 60 degree elevation looking upwards. The FOV is kept at 90 degrees. We refer to this setting as “Elevation=+/-60degree Views” in the table below.
These views better capture a complete picture of the panorama while still remaining within the distribution of natural camera angles.

We present the results in the table below:
| Text conditional  Vbench All Dimensions (946 prompts x 3 seeds) | Elevation=+/-60degree 8 Views |                         |             | Horizontal 8 Views (elevation=0) |                         |             |
|-----------------------------------------------------------------|:-----------------------------:|:-----------------------:|:-----------:|:--------------------------------:|:-----------------------:|:-----------:|
|                                                                 | FID (to HDVila3k frames)      | FVD (to HDVila3k video) | Clip Score  | FID (to HDVila3k frames)         | FVD (to HDVila3k video) | Clip Score  |
| 360DVD                                                          | 149.7                         | 901.1                   | 23.39       | 127.1                            | 801.9                   | 27.63       |
| Ours (VideoLDM + web360)                                        | **130.6**                     | **826.6**               | **24.12**   | **112.2**                        | **677.8**               | **27.78**   |
| Ours (Cogvideo + web360)                                        | **_109.9_**                   | **_675.9_**             | **_25.99_** | **_97.2_**                       | **_624.7_**             | **_29.33_** |

The original model we presented in our paper (VideoPanda using VideoVLDM base model) outperforms 360DVD across all of these metrics. 
VideoPanda significantly outperforms 360DVD in the 60 degree elevation views, highlighting its superior ability to generate the ground and sky views, which are distorted in the equirectangular representation used by 360DVD.
Additionally, the superior performance gained by using the CogVideoX-2B base model with VideoPanda is clearly seen by the quantitative evaluation above, with massive improvements observed across all metrics.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper received overall negative scores. The reviewers liked the concept of extending single view videos into panoramic videos. At the same time, they listed a bunch of weaknesses, such as weaker overall results, semantic issues, not convincing evaluation, lack of novel ideas. The only reviewer (EySz) who gave a somewhat positive score, provided a superficial and very short review. Hence the recommendation is to reject the manuscript.