AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation

Authors: Yukang Cao, Liang Pan, Kai Han, Kwan-Yee K. Wong, Ziwei Liu
Primary area: generative models
Keywords: 4D human avatar generation, compositional generation, human-object interaction

Abstract:
Recent advancements in diffusion models have led to significant improvements in the generation and animation of 4D full-body human-object interactions (HOI). Nevertheless, existing methods primarily focus on SMPL-based motion generation, which is limited by the scarcity of realistic large-scale interaction data. This constraint affects their ability to create everyday HOI scenes. This paper addresses this challenge using a zero-shot approach with a pre-trained diffusion model. Despite this potential, achieving our goals is difficult due to the diffusion model's lack of understanding of ''where'' and ''how'' objects interact with the human body. To tackle these issues, we introduce **AvatarGO**, a novel framework designed to generate animatable 4D HOI scenes directly from textual inputs. Specifically, **1)** for the ''where'' challenge, we propose **LLM-guided contact retargeting**, which employs Lang-SAM to identify the contact body part from text prompts, ensuring precise representation of human-object spatial relations. **2)** For the ''how'' challenge, we introduce **correspondence-aware motion optimization** that constructs motion fields for both human and object models using the linear blend skinning function from SMPL-X.  Our framework not only generates coherent compositional motions, but also exhibits greater robustness in handling penetration issues. Extensive experiments with existing methods validate AvatarGO's superior generation and animation capabilities on a variety of human-object pairs and diverse poses. As the first attempt to synthesize 4D avatars with object interactions, we hope AvatarGO could open new doors for human-centric 4D content creation.

Decision: accept (poster)
Reviewer scores: 6, 3, 8, 5

Reviews and discussion:
<thread>
creation date:
1730426424338

thread_type:
official_review

<thread_object>
creator: reviewer_bzxt

<content>
summary:
This paper introduces AvatarGO, a novel framework for generating realistic and animatable 4D human-object interaction (HOI) scenes directly from textual descriptions. Addressing the challenges of determining "where" and "how" objects interact with human bodies, the authors propose two key innovations:

1. **LLM-Guided Contact Retargeting**: Utilizing Lang-SAM to identify contact body parts from text prompts, ensuring precise human-object spatial relations.
2. **Correspondence-Aware Motion Optimization**: Constructing motion fields for both human and object models using the linear blend skinning (LBS) function from SMPL-X to maintain correspondence during animation, reducing penetration issues.

The proposed method operates in a zero-shot manner using pre-trained diffusion models, overcoming the scarcity of realistic large-scale interaction data. Extensive experiments demonstrate that AvatarGO outperforms existing methods in generating coherent compositional motions and handling various human-object pairs and poses.

strengths:
- The paper tackles the challenging and less-explored problem of generating visually realistic and animated 4D human-object interactions directly from textual inputs, which is crucial for applications in AR/VR and gaming.
- To the best of my knowledge, using LLM-Guided Contact Retargeting and Correspondence-Aware Motion Optimization to improve HOI in 4D Gaussian Splatting generation is novel. The proposed methods clearly solve the “where” and “how” problems raised in the introduction.
- The proposed method operates in a zero-shot manner with pre-trained diffusion models. It avoids the need for scarce large-scale interaction datasets.

weaknesses:
A significant problem of the paper is the lack of a substantial body of related work in HOI generation, particularly in 4D HOI generation methods. Important works such as Programmable Motion Generation, Chain-of-Contacts, CG-HOI, HSI, among others, are not discussed. These methods have contributed significantly to the field by generating SMPL and SMPL-X poses for HOI scenarios. 

While the paper focuses on generating realistic appearances using Gaussian splatting, which differs from the approaches of these methods, it is essential for the authors to analyze and position their work in the context of existing studies in 4D HOI generation. For instance, the use of large language models (LLMs) to generate contact-based constraints was first proposed in Chain-of-Contacts. By not acknowledging these relevant works, the paper misses the opportunity to highlight how AvatarGO builds upon or differs from established methods, potentially overlooking important insights and contributions from the existing literature.

Another weakness of the proposed method, as mentioned by the authors in the limitation section, is that it assumes a consistent geometric relationship between the interacting body part and the object. This prohibits it from generating complex manipulations such as tossing an object or passing an object between hands. 

### References

- Liu, Hanchao, et al. "Programmable Motion Generation for Open-Set Motion Control Tasks." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 2024.
- Xiao, Zeqi, et al. "Unified Human-Scene Interaction via Prompted Chain-of-Contacts." *The Twelfth International Conference on Learning Representations (ICLR)*.
- Diller, Christian, and Angela Dai. "Cg-hoi: Contact-guided 3d human-object interaction generation." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.
- Jiang, Nan, et al. "Scaling up dynamic human-scene interaction modeling." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

questions:
- How does AvatarGo compare to SMPL-based 4D HOI generation methods?
- Aside from the apparent appearance advantage, does AvatarGo achieve a similar interaction quality to the SMPL-based methods?
- What advantages does direct 4D Gaussian splatting optimization offer over a two-stage approach (SMPL generation followed by GS fitting)?
- Why do the results trend toward anime/cartoon styles? What technical modifications would enable more photorealistic outputs?
- How does runtime scale with sequence length and scene complexity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618911774

thread_type:
official_review

<thread_object>
creator: reviewer_e4gh

<content>
summary:
This paper introduces AvatarGO, a novel framework that utilizes diffusion models and language guidance to enable zero-shot generation of realistic, animatable 4D human-object interactions from text prompts.

strengths:
1. The paper is well-written, with a clear motivation and an easy-to-follow presentation of the core idea. Related work is thoroughly covered, incorporating up-to-date research in the field.
2. Extensive experiments and ablation studies convincingly demonstrate the effectiveness of AvatarGO, showcasing appropriate human-object interactions and superior robustness against penetration issues.

weaknesses:
I believe this paper is ready for publication, as I found no major weaknesses. My only question concerns the diversity of objects included in the text prompts used in the experiments. Additionally, how does your method perform across different types of objects?

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654722810

thread_type:
official_review

<thread_object>
creator: reviewer_1bkd

<content>
summary:
The paper proposes AvatarGO, a framework that generates 4D human-object interaction (HOI) to address the challenges of the lack of realistic large-scale interaction datasets. The authors generate 4D HOI samples in a zero-shot manner by leveraging a pre-trained diffusion model (Stable Diffusion). To mitigate the diffusion model's lack of understanding of "where" and "how" objects interact with humans, the authors propose 1) LLM-guided contact retargeting, which identifies the contact body part from the text prompt, and 2) correspondence-aware motion optimization, which models human and object deformation fields using linear blend skinning function. The authors argue that their extensive experiments validate the proposed method's superior generation capabilities.

strengths:
1. The qualitative results of the proposed method seem more plausible than the compared methods in Figure 3. Moreover, the authors conduct user studies to compare the proposed method with comparable methods in diverse criteria, e.g., penetration, motion quality, and overall performance.

2. The paper is well-structured, with clear sections from introduction to methodology and experiments. The authors provide detailed explanations about their task with various citations, which helps the reviewer to understand the problem the authors focused on.

weaknesses:
1. The reviewer wonders if the human motion is given as input or is generated by the proposed method. In Figure 2 caption (L185), the authors explain that correspondence-aware motion optimization jointly optimizes human and object animation, but in L345, it is written that the motion sequence is given. The reviewer requests clear explanations of what are the inputs and outputs of the proposed pipeline.

2. The reviewer thinks that the proposed method's novelty is limited. The objective function of the proposed generative model is spatial-aware SDS (L276), that is introduced by ComboVerse (Chen et al., 2024b), not newly proposed by the authors. Moreover, the authors propose LLM-guided contact retargeting, which identifies 3D Gaussians by unprojecting 2D segmentation masks. This process operates with a heuristically pre-defined threshold $\alpha$, and the reviewer wonders if this heuristic threshold $\alpha$ applies to various contact body parts, e.g., fingers, feet, head, and legs.

3. Since the proposed method only handles the composition of generated 3D humans and objects, the overall quality is prone to be bounded by the 3D asset generation module. Then, if the 3D objects and humans are generated in poor quality, how can the authors mitigate the error propagation in the proposed method?

4. The qualitative results in Figure 3 only show the interaction with human hands. The reviewer requests more visualization results of diverse objects and body parts, such as the head, feet, and legs.

questions:
1. In L310, the reviewer wonders how the authors define the threshold $\alpha$.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680021522

thread_type:
official_review

<thread_object>
creator: reviewer_d2nt

<content>
summary:
The paper introduces AvatarGO, a framework for generating and animating 4D human-object interactions (HOI) in a zero-shot manner using pre-trained diffusion models. The main contributions of the paper include: (1) LLM-guided ****contact ****retargeting, which addresses the challenge of determining where objects should interact with the human body by using large language models (LLMs) to identify appropriate contact areas from text prompts, and (2) correspondence-aware motion optimization, which ensures coherent motion between humans and objects while preventing issues like object penetration during animation. AvatarGO generates realistic 4D HOI scenes directly from textual inputs, overcoming limitations in existing methods that rely on SMPL-based models. Qualitative and quantitative results show that the AvatarGO outperforms existing methods in terms of generating plausible human-object interaction animation.

strengths:
- The paper tackles the new task of zero-shot generation of 4d human-object interaction (HOI).
- The proposed LLM-guided contact retargeting helps initialize the object in reasonable 3D locations to help generate plausible HOI.
- The correspondence-aware motion optimization helps prevent penetration and maintain plausible interactions throughout the animation.
- Visually, the method outperforms previous methods in the quality of generated HOI. The higher CLIP score and user preference also support this.

weaknesses:
- The overall appearance quality of generated humans and objects is still limited. The results are often blurry and lack realism. There is also still human-object penetration (see “Naruto in Naruto Series stepping on a football under his foot” on the webpage).
- The notations in object animation and correspondence-aware motion optimization are poorly explained. What is G_c and G_o? How are they computed? The paper mentioned they are derived from x_ci and x_oi, but it’s unclear.
- It seems the method requires training for each human-object pair, which can be quite inefficient for large-scale generation. Is there anyway for the method to train the same character interacting with multiple objects?

questions:
- Is each HOI trained with one motion only? Can it generalize to multiple motions?
- Fig. 4 only shows one frame of animation, which is not animation but reposing. Multiple frames need to be shown.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733052602194

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their time and effort in reviewing our paper!

Our method presents the first method for zero-shot text-guided 4D human-object interaction (HOI) generation featuring realistic geometry and texture. To realize this objective, our method makes two primary contributions:

**LLM-guided contact retargeting:** By utilizing Lang-SAM, we pinpoint the contact body part based on the text prompts, enhancing the precision of human-object spatial relationships.

**Correspondence-aware motion optimization:** Leveraging the linear blend skinning function from SMPL-X, we construct two distinct motion fields for humans and objects, improving the overall motion quality.

Being the first endeavor to synthesize 4D avatars engaged in object interactions, our method attains state-of-the-art performance in both composite 3D HOI generation and dynamic 4D HOI animation, demonstrating robustness against penetration issues.

**Below, we summarize the changes made according to the reviews:**

1. We discuss the factors contributing to the quality of our results, in terms of the geometry and appearance. Additionally, we present the potential of the generated quality (`#D2nt`).

2. We explain the meaning of the components in our correspondence-aware motion fields (Eq. (16)) and show the process for obtaining them (`#D2nt`). 

3. We explain the setting of our method to generate the dynamic scene for each pair of humans and objects. we explore the potential for extending our method to enable a single human to interact with a range of objects and animate them with diverse motions without the need for re-training (`#D2nt`).

4. We will improve the Fig.4 to showcase multiple frames in the revised edition (`#D2nt`).

5. We discuss the inputs and outputs of our method (`#1bKd`).

6. We explain how we employ spatial-aware score distillation sampling while emphasizing the importance of our proposed LLM-guided contact retargeting (`#1bKd`).

7. We describe how we use the threshold alpha in our experiments and provide an explanation of how it is defined (`#1bKd`).

8. We showcase how the quality of the generated human and object impacts our work, such as when the quality is poor, and we explain how our method mitigates this influence. (`#1bKd`).

9. We showcase our method to generate diverse interaction between human and various body parts. Additionally, we explain the common parts where interactions between humans and objects typically occur (`#1bKd`).

10. We discuss the different objects in the text prompts we used for our experiments and how well our method works with various types of objects (`#E4gh`).

11. We further discuss our work with existing SMPL-based 4D HOI generation techniques and agree that the SMPL-based 4D HOI generation techniques lay the basis of our method (`#bzXT`). 

12. We discuss the limitation in our current approach, specifically in addressing only continuous contact between humans and objects (`#bzXT`).

13. We present comparisons in terms of interaction quality with SMPL-based methods (`#bzXT`).

14. We discuss the potential and challenges of achieving our objectives in a two-stage process (`#bzXT`).

15. We explain why our method tends to produce a cartoon-like style and explore methods for attaining photorealistic results (`#bzXT`).

16. Lastly, we discuss how the runtime of our method scales with the length of the motion sequence and the complexity of the scene (`#bzXT`).


We sincerely thank all reviewers and the AC(s) again for their valuable suggestions, which have greatly helped strengthen our paper.

If you have any further questions, we would be happy to discuss them!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method for diffusion-guided synthesis of 4D human and object interactions. The key innovations that the work introduces to address this problem are: (a) LLM-guided contact retargeting: a language-guided feedback module that segments the Gaussians pertaining to the part of the body that interacts with the object and places the object at the start of the optimization process to learn a joint composition of the object and person, close to it; and (b) Correspondence-aware motion optimization: a correspondence-aware model for learning the joint animation of the human body and the object to explicitly avoid penetrations between them. This work is compared to several existing baselines to demonstrate its efficacy both quantitatively and qualitatively, along with several ablation studies with visual results only to showcase the contribution of each of the method's individual components.