PuppetMaster: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics

Authors: Ruining Li, Chuanxia Zheng, Christian Rupprecht, Andrea Vedaldi
Primary area: generative models
Keywords: Video generation, motion, diffusion models

Abstract:
We present PuppetMaster, a video generator that understands part-level object dynamics. Given an image of an object and a number of drags defining the desired trajectory of selected points of the object, PuppetMaster synthesizes a video where the object moves according to the specified drags in a physically plausible manner. PuppetMaster is obtained by fine-tuning an off-the-shelf video diffusion model, extended with a new component that encodes the input drags. PuppetMaster also introduces all-to-first attention, a replacement for the common spatial attention module, which removes artifacts that arise from fine-tuning a video generator out-of-domain and significantly improves the quality of the synthesized videos. PuppetMaster is learned from Objaverse-Animation-HQ, a new dataset of curated part-level motion clips obtained by rendering synthetic 3D animations. We propose strategies to automatically filter out sub-optimal animations and augment the synthetic renderings with meaningful drags. By using this data, PuppetMaster learns to generate part-level motions, unlike other motion-conditioned video generators that mostly move the object as a whole. PuppetMaster generalizes well to real images, outperforming existing methods in real-world benchmarks in a zero-shot manner. We refer the reader to the supplementary material for video visualizations.

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730583606190

thread_type:
official_review

<thread_object>
creator: reviewer_h6mh

<content>
summary:
This paper addresses the problem of generating videos controlled by drag. The key idea is to take the pre-trained stable video diffusion and fine-tune it by conditioning it on drag control. They used all-to-first attention to improve the consistency between the reference images and video frames. A dataset with paired drag input and output videos is collected while building upon the Objaverse animation dataset.

strengths:
First, the collected dataset could be helpful for further study in this direction. 
Second, compared with previous works such as DragAPart, which used an image generation model, it makes more sense to use pre-trained video diffusion to generate temporal consistent videos.

weaknesses:
First, some descriptions might be confusing to understand, making it difficult to follow. For example, in line 113, "The resulting datasets, Objaverse- Animation and Objaverse-Animation-HQ, contain progressively fewer animations of higher quality." what does it mean by progressively fewer animations?  
From Lines 197-205, they mentioned there are two main challenges. However, it seems that both challenges are about injecting drag control into the pre-trained video diffusion. 

Second, to my understanding, the problem itself is not quite properly defined. Given the drag input in the image space only, the input control itself is ambiguous. For example, whether the drag is supposed to zoom-in the object or manipulate part of the object is ambiguous. From my understanding, it also depends on the training set. In this case, the training is severely overfitted to the training set.

Third, the technical contributions are not clear here. Except for collecting a dataset, the technical novelty is limited. For example, the drag encoder seems to be very similar to that proposed in DragAPart. The all-to-first attention(replacing K, V in the attention) is also not new in the context of video generation or image generation, such as Consistent123.

questions:
1) What is the difference between the drag encoder and that in the DragAPart paper?
2) Why not demonstrate the generated videos of DragAPart? 
3) What are the failure cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645802056

thread_type:
official_review

<thread_object>
creator: reviewer_tfym

<content>
summary:
The paper presents PuppetMaster, a conditional 2D video generation model that takes an reference object image and some conditional input drags then synthesizes a video with part-level object dynamics. The authors finetunes an off-the-shelf video diffusion model and extend it to encode input conditional drags. Additionally, they propose all-to-first attention method to remove the artifacts when generating out-of-domain video and improve the quality. They also made their own dataset, Objaverse-Animation-HQ, with data curation from Objaverse dataset. The authors argued that the proposed method generalizes well on real images and outperforms in real-world benchmarks in a zero-shot manner.

strengths:
1. The motivation of the paper to generate object animation given a reference object image and user-controllable drag condition seems practical. Compared to the existing methods that generate a static image of the final-state of objects, the proposed method can produce fine-grained object motion as a video.

2. The numerical experiments are conducted in rigorous validation by retraining the comparable method (DragAPart) with an identical data setting as the proposed method. To validate the accuracy of part-level dynamics, the authors introduce the 'flow error' metric by computing errors between predicted and ground-truth point trajectories.

weaknesses:
1. Insufficient generation quality: Although the authors leverage and finetune the video foundational model (Stable Video Diffusion), the generated video quality is not sufficient to determine that the learned model well-captures object dynamics. In the video results provided by the authors in supplementary materials, the generated results do not preserve the ground-truth object's appearance details, e.g., the generated object colors are changed in every frame (flickering effect). The reviewer wonders if the authors consider these low temporal smoothness problems and make any solution in the proposed method. 

2. Limited qualitative result: The reviewer thinks that the authors should provide more qualitative results compared with DragAPart. Although DragAPart is a method that generates a static image, the video results can be obtained by dense sampling on the trajectory and defining fine-grained drag conditions. The reviewer found two samples in the supplementary document, but the reviewer thinks that more diverse object samples should be reported in the main paper.

questions:
1. In Sec. 5.1, the authors argue that the proposed model obtains smaller errors when averaging all foreground points (L412). However, the reviewer thinks that the flow error is smaller with the averaged conditioning points. It would be helpful if the authors could provide more explanation and analysis of the results.

2. In Table 2, the authors report the ablation study results. In models A and B, the reviewer wonders why the authors do experiments without an end location of the drags. Is it possible to encode input drags without an end point? It would be great if the authors give more explanation of baseline models in ablation studies. 

3. Impact of drag modulation with scale: The authors argue that utilizing the adaptive normalization layer helps to improve the generation quality. The reviewer needs more explanations about the authors' design choices.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713170786

thread_type:
official_review

<thread_object>
creator: reviewer_w1d9

<content>
summary:
The authors present a framework called PuppetMaster for generating videos with part level object motion from a single image. The approach uses "drag" inputs to control fine grained parts of the generated video. This approach is trained on the Objaverse-Animation-HQ dataset that contains videos with part level motion annotation. State of the art quantitative performance is demonstrated on the videos generated for various classes (animals, humans, objects).

strengths:
1. **Clarity**: The paper is well written with adequate attention to detail. All the design components are explained in detail to aid in reproducibility. 
2. **Result Quality**: The generated videos show improved quality over baselines with similar drag controls. 
3. **Comparison**: Both quantitative and qualitative performance is provided against a number of baselines. Additionally, out-of-domain performance on the Human3.6 datasets shows the efficacy of the approach.
4. **Ablations**: The provided ablations highlight the need for the all-to-first attention and the right parameterization needed to represent the drag constraints.

weaknesses:
1. **Claims of Novelty**: The authors claim the "all-to-first" attention. Although not previously demonstrated in its exact form. The mechanism is essentially a variant of cross attention looking at KV pair from just the first frame. To that the claim that authors present a "novel all-to-first attention can be softened. For instance, one can highlight the finding of all-to-first attention while emphasizing that the variant of cross-attention that always looks at first frames is better than no attention at generating videos with fine grained part control.

questions:
1. What is maximum number of frames that can be generated ? Can the all-to-first attention be replaced with a full 3D attention for better performance?
2. How is the performance of the framework on non articulated objects? In particular, providing a video example of a non articulated object along with arbitrary drags is helpful to demonstrate the framework's default behavior. 
3. What happens when the drag condition input to the framework is zero? Does the network generate a video with arbitrary motion or just a static image? Showing an example of this setting would be helpful to understand "unconditional" synthesis  from the network.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731765818055

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal: Responses to Shared Concerns & Questions

comment:
We would like to thank all three fellow reviewers for your time devoted to providing valuable feedback to our work. We are encouraged by the positive comments: “**well written** (W1d9)”, “**improved quality** (W1d9)”, **rigorous validation and thorough comparison** (W1d9, tFyM) and the **recognition of our dataset contribution** (H6mH). We address individual questions in separate threads, but first highlight some shared concerns.

## Comparison with DragAPart (tFyM, H6mH)
We provided two examples in the supplementary material (Sup. Mat.). We agree that providing more examples, especially as videos, could better visualize our model’s improved temporal consistency. **We provide 8 examples in `video-results/comparison-DragAPart.gif` in the revised Sup. Mat.**

## Novelty of the All-to-First Attention (W1d9, H6mH)
The idea of using cross attention in image/video generation is not new. As pointed out by H6mH, the closest to our proposed All-to-First Attention is Consistent123 [1], where they attend each noised novel view to the key and value obtained from the conditioning view for the novel view synthesis (NVS) problem. Our work is highly inspired by [1] (we stated this explicitly in L307-308). **To the best of our knowledge, we are the first to introduce this cross-frame spatial attention mechanism in the domain of video generation.**

We would also like to emphasize that **All-to-First Attention is not our only contribution**. Generating intrinsic, part-level object motion prompted by drags is non-trivial, complicated further by the lack of high-quality data. In addition to the introduction of All-to-First Attention, (1) we **curate a large-scale high-quality 4D dataset, Objaverse-Animation-HQ**. Enormous efforts were dedicated to curating this dataset, from classifier training and GPT-4V prompting to large-scale rendering. We believe this dataset can significantly benefit the community. (2) We **introduce drag tokens to make the cross attention modules in SVD spatial-aware**. We believe this design can inspire future research in conditional image/video generation.

Thank you again for your inputs! We are more than willing to discuss further if you have follow-up questions regarding our work.

References:
[1] Consistent123: Improve Consistency for One Image to 3D Object Synthesis, Weng et al.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an approach to control articulated objects by "dragging" parts of the objects. The reviews place the paper slightly below the borderline. While the reviewers acknowledge the clarify of the manuscript, the practicality of the problem, and extensiveness of numerical experiments, they also list several issues and questions. The reviewers suggest that the quality of generations is expected to be better. The AC agrees with this statement. AC checked the supplement and found the quality insufficient. It was not clear whether it's the gif encoding that introduces color artifacts or the backbone video model. One of the reviewers mentioned that too. In general AC believes that choosing gif format in the supplement doesn't do a good service to the manuscript. Besides, the supplement just lacks enough comparisons. Reviewers mentioned that too, and additional comparisons were added during the discussion, however, AC believes that 8 comparisons is not enough. 

In summary, the paper didn't get enough support from reviewers. That's due to quality, certain concerns with novelty and technical contribution, and the amount of visual results provided.