MVTokenFlow: High-quality 4D Content Generation using Multiview Token Flow

Authors: Hanzhuo Huang, Yuan Liu, Ge Zheng, Jiepeng Wang, Zhiyang Dou, Sibei Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 4D Generation, Dynamic 3D Gaussian Splatting, Dynamic Reconstruction, Diffusion Models

Abstract:
In this paper, we present MVTokenFlow for high-quality 4D content creation from monocular videos. Recent advancements in generative models such as video diffusion models and multiview diffusion models enable us to create videos or 3D models. However, extending these generative models for dynamic 4D content creation is still a challenging task that requires the generated content to be consistent spatially and temporally. To address this challenge, MVTokenFlow utilizes the multiview diffusion model to generate multiview images on different timesteps, which attains spatial consistency across different viewpoints and allows us to reconstruct a reasonable coarse 4D field. Then, MVTokenFlow further regenerates all the multiview images using the rendered 2D flows as guidance. The 2D flows effectively associate pixels from different timesteps and improve the temporal consistency by reusing tokens in the regeneration process. Finally, the regenerated images are spatiotemporally consistent and utilized to refine the coarse 4D field to get a high-quality 4D field. Experiments demonstrate the effectiveness of our design and show significantly improved quality than baseline methods.  Project page: https://soolab.github.io/MVTokenFlow.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730472208614

thread_type:
official_review

<thread_object>
creator: reviewer_f8rm

<content>
summary:
This paper proposed the enlarged self attention on the temporal dimension and incorporate the flow to warp the feature as the additional constraint to improve the temporal consistency. Moreover, the regeneration and refinement have been used to further improve the final performance. The experiment has been on Consistent4D dataset to demonstrate the performance.

strengths:
1. The enlarged self attention on the temporal dimension to maintain the temporal consistency.
2. Using 2D flow to warp the feature as the additional constraint to improve the temporal consistency.
3. The regeneration and refinement have been used to further improve the final performance.
4. The experiment has been on Consistent4D dataset to demonstrate the performance.

weaknesses:
1. The author does not present clearly about the motivation and main contributions of this work in the introduction part. 
2. In the related work, the author firstly claimed the diffusion model applied in 4D generation will be discussed, while in the section, 4D Scene Representation, only NeRF and 3DGS based method have been discussed. The related works need to be reorganized to ensure the logical coherence between each parts.
3. Applying the self attention on the temporal dimension has been well studied and proved to be effective way to maintain the temporal consistency. Incorporating this strategy may not have enough novelty.
4. No need to further training not have enough novelty since building the original temporal multi view diffusion model requires huge training while no need for retraining for enlarging the self attention is straight forward.

questions:
Please refer to the Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730535031828

thread_type:
official_review

<thread_object>
creator: reviewer_4rtf

<content>
summary:
The paper addresses the problem of multi-view and temporal consistency in video-to-4D content generation. The proposed method, MVTokenFlow, adopts Era3D for multi view video generation to create coarse 3DGS, and TokenFlow to enhance the temporal consistency of the coarse 3DGS. With this method, the authors are able to generate 4D content while preserving the detail of the content over the timeline. Experiments demonstrate the effectiveness of the proposed method in 4D generation, both quantitatively and qualitatively.

strengths:
(1) The task of video-to-4D content generation, while preserving the details (spatial and temporal information) of an input object, is challenging and well-motivated.

(2) They appropriately chose the multi-view generation method Era3D for spatial consistency and the video generation method TokenFlow for temporal consistency, using the scene representation 3DGS, to construct their pipeline.

(3) The proposed method achieves SOTA performance on video-to-4D generation compared to existing baselines.

(4) The paper is easy to follow.

weaknesses:
(1) Era3D uses Row-wise Multiview Attention (RMA) to achieve multi-view consistency. In L232, the authors say that they use enlarged self-attention to improve the temporal consistency of multi-view video frames. Using $K=6$ viewpoints and a keyframe interval of 8 frames of video, the enlarged self-attention across all different timesteps may be quite memory-heavy. I think this process is a distinguishing feature compared to the previous work Era3D, but the authors don't provide an ablation study on how this component improves temporal consistency in the coarse stage.

(2) As I understand it correctly, the coarse stage produces less temporal-consistent 3DGS, and the fine stage renders 2D flows to guide the re-generation of the multi-view images and create the final temporally consistent 3DGS. When describing this, Fig. 2 does not intuitively illustrate the process. It would be better if the two stages were completely separated.

(3)  The ablation studies on token propagation and flow loss for multi-view video generation show only qualitative results. Quantitative results (using the same metrics as in Tab. 1) are needed to show the generality of these modules.

(4) Similar to point (3), the authors use the normal loss, but this feature is not ablated, either qualitatively or quantitatively.

questions:
(1) How long does it take for the coarse stage and the fine stage, and how much GPU memory is required for each stage? Also, how about the other baselines?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538124051

thread_type:
official_review

<thread_object>
creator: reviewer_wwg8

<content>
summary:
This work focuses on dynamic object generation from a monocular video. The authors propose a two-stage approach, enabling a 3D diffusion model to generate temporally coherent multi-view videos based on reference view video. Specifically, they expand self-attention to all images of different timestamps and generate pseudo multi-view videos in the first stage. The pseudo videos are utilized to reconstruct coarse dynamic gaussians with motion trajectories. In the second stage, they propose a token propagation technique based on 2D optical flow rendered by dynamic gaussians, which helps the 3D diffusion model generate more temporally coherent videos. Experiments on public datasets validate the effectiveness of the proposed framework.

strengths:
1) It is natural to incorporate knowledge of video generation and editing methods into 4D generation. TokenFlow is a reasonable attempt.
2) The authors conduct comparisons and ablation studies to support their claims.

weaknesses:
1. Token Propagation: As illustrated in Section 3.1, token propagation is only conducted between key frames and non-key frames, which means the temporal consistency among key frames is not guaranteed. Although the authors utilize enlarged self-attention for key frames, I think this cannot ensure full temporal consistency, especially when the objects have large movements.

2. Experiments:

a) Dataset: The authors are encouraged to conduct qualitative experiments on the official Consistent4D dataset instead of its demo videos. The official dataset contains 7 synthetic objects with multi-view videos; please refer to the SV4D paper for the performance of other video-to-4D methods on that dataset.

b) Comparison: The authors are encouraged to conduct novel-view synthesis comparisons with other methods if possible.

c) Ablations: The authors are encouraged to conduct quantitative ablations and provide more qualitative analyses.

d) Visualizations: The supplementary videos have only two views of the object, one of which is the input view, which does not provide readers with a full understanding of the performance of the proposed method. The authors are encouraged to provide videos with more views. Additionally, in Figure 3, the comparison with state-of-the-art methods is conducted in the input view, which may not be appropriate.

questions:
The authors claim they compare with all open-sourced methods, but as far as I know, there are other open-sourced video-to-4d works which are not included for comparison, for example, DG4D and Diffusion^2. I'm not asking the authors to comapre with all open-sourced works, instead, I suggest to modify the inappropriate expression.

Besides, I'm willing to raise my score if the authors address my concerns during rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730741798140

thread_type:
official_review

<thread_object>
creator: reviewer_fpxc

<content>
summary:
This paper presents a method by which multiview images generated by Era3D independently per frame in a video can be adjusted to be temporally consistent.  A coarse dynamic 3DGS reconstruction is made from the initial multiview videos and 2D flows are computed from these videos.  These flow fields are used to ensure tokens associated by flow between frames are similar.  The multiview images are regenerated from these modified tokens and then a final dynamic 3DGS reconstruction is built.

Results are presented on Consistent4D and a self-collected dataset.

strengths:
* method seems relatively clean without unneeded bells and whistles
* qualitative results look good
* quantitative metrics show improvement
* ablation study is presented

weaknesses:
* Minor point, but I'd recommend the authors add a clear list of contributions at the end of the intro.

* Presented videos in the supplemental are pretty limited.  The columns also aren't labeled.  I'm guessing the left most image and flow field correspond to the input and the right most is some arbitrary second view?  It would have been nice to see some sort of orbit around the object as the video plays.

questions:
* Will the custom dataset be made available?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a two-stage method for generating 4D content from monocular videos by enabling 3D diffusion models to produce temporally coherent multi-view videos. In the first stage, multiview diffusion models generate multi-view images at different timesteps. In the second stage, a token propagation technique based on 2D optical flow helps the 3D diffusion model generate more temporally coherent videos. The proposed method effectively addresses a challenging task and demonstrates state-of-the-art performance both qualitatively and quantitatively. Concerns regarding the experiments and other issues raised during the review process were effectively addressed in the rebuttal.