AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation
Authors: Yukang Cao
Liang Pan
Kai Han
Kwan-Yee K. Wong
Ziwei Liu
Venue: ICLR 2025 Poster (2025)
Keywords: 4D human avatar generation
compositional generation
human-object interaction
Primary area: generative models
TL;DR: an interactable animatable pipeline for generating and animating 4D avatars with object interactions from text prompts.

Abstract: Recent advancements in diffusion models have led to significant improvements in the generation and animation of 4D full-body human-object interactions (HOI). Nevertheless, existing methods primarily focus on SMPL-based motion generation, which is limited by the scarcity of realistic large-scale interaction data. This constraint affects their ability to create everyday HOI scenes. This paper addresses this challenge using a zero-shot approach with a pre-trained diffusion model. Despite this potential, achieving our goals is difficult due to the diffusion model's lack of understanding of ''where'' and ''how'' objects interact with the human body. To tackle these issues, we introduce **AvatarGO**, a novel framework designed to generate animatable 4D HOI scenes directly from textual inputs. Specifically, **1)** for the ''where'' challenge, we propose **LLM-guided contact retargeting**, which employs Lang-SAM to identify the contact body part from text prompts, ensuring precise representation of human-object spatial relations. **2)** For the ''how'' challenge, we introduce **correspondence-aware motion optimization** that constructs motion fields for both human and object models using the linear blend skinning function from SMPL-X.  Our framework not only generates coherent compositional motions, but also exhibits greater robustness in handling penetration issues. Extensive experiments with existing methods validate AvatarGO's superior generation and animation capabilities on a variety of human-object pairs and diverse poses. As the first attempt to synthesize 4D avatars with object interactions, we hope AvatarGO could open new doors for human-centric 4D content creation.

Published as a conference paper at ICLR 2025
AVATARGO: ZERO-SHOT 4D HUMAN-OBJECT INTER- ACTION GENERATION AND ANIMATION
Yukang Cao Liang Pan2ti Kai Han Kwan-Yee K. Wong Ziwei Liu
SS-Lab, Nanyang Technological University, Shanghai Al Laboratory, The University of Hong Kong https: /ttpss///uaaggaaooo gi rithub. io/ AvatarGo/
ABSTRACT
Recent advancements in diffusion models have led to significant improvements in the generation and animation of 4D full-body human-object interactions (HOI). Nevertheless, existing methods primarily focus on SMPL.-based motion generation, which is limited by the scarcity of realistic large-scale interaction data. This con- straint affects their ability to create everyday HOI scenes. This paper addresses this challenge using zero-shot approach with a pre-trained diffusion model. Despite this potential, achieving our goals is difficult due to the diffusion model's lack of understanding of "where" and "how" objects interact with the human body. To tackle these issues, we introduce AvatarGO, a novel framework designed to gener- ate animatable 4D HOI scenes directly from textual inputs. Specifically, for the "where" challenge, we propose LLM-guided contact retargeting, which employs Lang-SAM to identify the contact body part from text prompts, ensuring precise representation of numan-objec spatial relations. 2) For the "how" " challenge, we introduce correspondence-wwarr motion optimization that constructs motion fields for both human and object models using the linear blend skinning function from SMPL-X. Our framework not only generates coherent compositional mo- tions, but also exhibits greater robustness in handling penetration issues. Extensive experiments with existing methods validate AvatarGO's ' superior generation and animation capabilities on variety of human-object pairs and diverse poses. As the first attempt to synthesize 4D avatars with object interactions, we hope AvatarGO could open new doors for human-centric 4D content creation.
INTRODUCTION
The creation of 4D human-object interaction (HOI) holds immense significance across a wide range of industries, including augmented/virtual reality (AR/VR) and game development, as it forms the foundation of the 4D virtual world. Traditionally, developing such models has required extensive human effort and specialized engineering expertise. Fortunately, thanks to the collections of HOI datasets (Li et al., 2023b; Bhatnagar et al., 2022; Jiang et al., 2023a) and the recent advancements in diffusion models (Saharia et al., 2022; Ramesh et al., 2022; Balaji et al., 2022; Stability.AI. 2022; 2023), existing HOI generative techniques (Zhang et al., 2022; 2023; 2024; Shafir et al., 2023; Kapon et al., 2024; Chen et al., 2024a) have exhibited promising capabilities by generating 4D human motions with object interactions from textual inputs. Nonetheless, these methods primarily focus on SMPL-based (Loper et al., 2015; Pavlakos et al., 2019) motion generation, which struggles to capture the realistic appearance of subjects encountered in everyday life. Although InterDreamer (Xu et al., 2024b) has recently proposed to generate text-aligned 4D HOI sequences in a zero-shot manner, their output is still largely constrained by the SMPL model. This highlights pressing need for more realistic and generalizable methods tailored specifically to model 4D human-object interactive content. We take the initiative and showcase the potential of addressing this challenge by leveraging the 3D generative methods in a zero-shot manner.
In recent times, 3D generative methods (Poole et al., 2022: Tang et al., 2023; Liu al., 2023e; Lin et al., 2023; Wang et al., 2023e; Cao et al., 2023b; Liao et al., 2023) and Large Language
* Part of the work has been done when interning at Shanghai AI Laboratory. Project lead.
Corresponding authors

---

Published as a conference paper at ICLR 2025
Body buiggeuieerr der carrying dumbbel in his shand-
Dragon Ball grasping <hana's
fron Man. kalding of Thor his shand-
Goku Dragon
Narito
Einstain
Harry bear head
AK-47*
microphone
Searheed
Jobs iPhone"
football
Figure 1: Examples of 4D animation results obtained via AvatarGO. AvatarGO effectively
produces diverse human-object compositions with correct spatial correlations and contact areas. achieves joint animation of humans and objects while avoiding penetration issues.
Models (LLMs) (Wu et al., 2023a) have garnered increasing interest. These progressives have compositional generation techniques that are capable of
led the development of text-guided comprehending intricate relations and creating complex 3D scenes incorporating multiple subjects. Notably, GraphDreamer (Gao et al., 2023) utilizes LLMs to construct graph where nodes represent objects and edges denote their relations. ComboVerse (Chen et al., 2024b) proposes spatial-aware score distillation sampling to amplify the spatial correlation. Subsequent studies (Epstein et al., 2024; Zhou al., 2024) further explore the potential of jointly optimizing layouts to composite different components.
Despite the promising performance demonstrated by existing methods, they encounter major challenges in generating 4D) HOI scenes: /) Incorrect contact area: While LLMs excel at capturing the relationships, optimization with diffusion models faces difficulties in accurately defining the contact area between various objects, particularly those with complex articulated structures like human bodies. Although efforts like InterFusion (Dai al., 2024) have constructed 2D human-object interaction datasets to retrieve human poses from text prompts, they still encounter challenges in defining the optimal contact body parts for cases outside the training distribution. 2) Limitations in 4D compositional animation: While existing techniques like DreamGaussian
(Ren et al., 2023) and TC4D (Bahmani et al., 2024) employ video diffusion models (Blattmann et al., 2023; Guo et al., subject during optimization, leading to unrealistic animation results. Despite initiatives like Comp4D (Xu et al., 2024a), which utilize trajectories to animate 3D objects individually. modeling contact between various subjects remains a challenge.
2023) to animate 3D static scenes, they often treat the entire scene
In this paper, propose AvatarGO, novel framework for compositional 4D avatar generation with object interactions. By taking the text prompts as inputs, we assume that the 3D human and object models as well as the human motion sequences can be individually generated by adopting existing
2

---

Published as a conference paper at ICLR 2025
generative techniques (Tang et al., 2023; Liu et al., 2023d; Zhang et al., 2023; 2024). Specifically, we adopt DreamGaussianDD (Ren et al., 2023) as our baseline considering its superior training efficiency and focus addressing the challenges associated with human-object interactions. To achieve this objective, AvatarGiO integrates two key innovations to learn "where" and "how" the object should interact with the human body: 1) LLM-guidee contact retargeting. Given the limited availability of human-object interaction images in the 2D dataset used for diffusion model training, it's difficult to identify the most appropriate contact area between humans and objects. To tackle this issue, we propose leveraging Lang- SAM (lan, 2023) to identify the contact body part from text prompts, which serves as the initialization for the optimization procedure. 2) Correspondence-wware motion optimization. Building upon the observation that penetration is absent in static composited models, we introduce correspondence-awant motion optimization that leverages SMPL-X as an intermediary to maintain the correspondence between humans and objects when they are animated to new pose, thus demonstrating greater robustness in handling penetration issues.
We thoroughly assess AvatarGO by compositing diverse pairs of 3D humans and objects and animating them across various motion sequences (see Fig. 1). Our experimental results show that our method excels at identifying optimal contact areas and exhibits greater robustness in handling penetration issues during animation, significantly outperforming existing techniques. We will make our code publicly available.
2 RELATED WORK
3D Content Generation, Leveraging advances in diffusion-based text-to-2D image generation (Sa- haria et al., 2022; Ramesh et al., 2022; Balaji et al., 2022; Stability.AI, 2022; 2023). DreamFusion introduced Score Distillation Sampling (SDS) to generate 3D content via pre-trained models, uti- lizing technologies like NeRF (Mildenhall et al., 2020), DMTFT (Shen et al., 2021), )), 3D Gaussian Splatting (Kerbl ct al., 2023)). Subsequent research has focused on enhancing output quality (Lin et al., 2023; Chen et al., 2023b; Wang et al., 2023c), controlling generation processes (Metzer et al., 2022: Seo et al., 2023), improving training efficiency (Wang et al., 2023a; Wu et al., 2024; Tang et al., 2023), and extending capabilities on 3D texturing (Richardson et al., 2023; Cao et al., 2023a; Chen et al., 2023a; Tang et al., 2024b). Addressing 3D human body complexity, recent studies (Cao et al., 2023b;e; Han al., 2023; Liao et al., 2023; Jiang al., 2023b; Huang al., 2023b; Kolotouros et al., 2023; Zeng et al., 2023; Huang et al., 2023a; Cao et al., 2024) have been proposed for creating controllable 3D human avatars, although these still require significant input-specific training time. The prolifcration of large 3D datasets (Deitke et al., 2023; 2024; Wu et al., 2023b) has propelled 3D generation techniques forward. Notably, 1-to- (Liu et al., 2023c), Zero123++ (Shi et al., 2023a), and MVDream (Shi et al., 2023b) use 2D diffusion models to generate consistent multi-view images, serving as inputs for efficient 3D model generation tools like SyncDreamer (Liu al., 2023e), Wonder3D (Long et al., 2023), One-2-3-45 (Liu et al., 2023b;a), UniDream (Liu et al., 2023(), MVDiffusion++ (Tang et al., 2024c), and Hake-Your-3D (Liu et al., 2024a). Additionally, building on transformer (Vaswani et al., 2017) and image processor advancements (e.g., DINO (Caron et al., 2021; Oquab et al., 2023)), Large Reconstruction Models (Hong et al., 2023; Wang et al., 2023b; Xu al., 2023; Li et al., 2023a) implement transformer-hase from image features. 3DTopia (Hong et al., 2024) uses hybrid diffusion priors to produce high-fidelity 3D objects. Meanwhile, methods like LGM (Tang et al., 2024a), CRM (Wang et al., 2024), and GRM (Yinghao et al., 2024) explore various 3D representations for improved performance, such as 3D Gaussian Splatting (Kerbl et al., 2023) and FlexiCube (Shen et al., 2023). Despite these advances, challenges remain in generating complex compositional 3D scenes.
architectures derive 3D tri- plane tokens
3D Compositional Generation. To address the compositional nature of 3D content, a few efforts have been made recently. Epstein et al (Epstein et al., 2024) and GALA3D (Zhou et al., 2024) propose optimizing component layouts for integrated object scenes. Combo (Chen al., 2024b) introduces apat score distillation sampling (SSDS) to effectively learn object spatial relations. GraphDreamer (Gao et al., 2023) uses large language models to form graph structures where nodes and edges represent objects and their relationships, respectively, showing promising results. Challenges remain in modeling interactions between humans and objects. InterFusion (Dai et al., 2024) develops a 2D dataset for human-object interactions, enabling text-guided pose retrieval and scene generation. However, this approach lacks precise control over interaction areas and is not readily adaptable 4D scenarios.
3

---

Published as a conference paper at ICLR 2025
Text-driven- humaan object comuposition
4D Avatar Generation with Object Interactions
Trajnable parameters
CLMugguded comtact resargeing
Spntial
Stable Difluioo
Compesitinnl
Cios
Scale
SMPL-X parametors
Bodybuilder holding dumbbel shatnat
(it) Cerrespondenne--ve awart (CA) mation field
Homaa Rxinattio
Stable Diftowoon
Axns
inear Blennd skinning
Non-rigid motion
Compasitonnl rendering
Corvesoondanet awart
motiow sptimizziioo
LCA
Okjestt grnnmation.
Average
Rotation
Esns
Transtation
Terture-Strccurr Join diflusion
Trainable parameters
Figure 2: Overview of AvatarGO. AvatarGO takes the prompts as input to generate 4D avatars with object interactions. At the core of our network are: 1) Text-driven 3D human and object composition that employs large language models retarget the contact areas from texts and spatial- aware SDS to composite the 3D models. 2) Correspondence-ave ewware optimizes the animation for humans and objects. It effectively maintains the spatial correspondence during animation, addressing the penetration issues.
aware motion optimization which jointly
4D Content Generation. Recent advances in video diffusion models and score distillation sampling have spurred variety of 4D scene generation techniques. Make-A-Video:33 (MAV3D) (Singer al., 2023) utilizes HexPlane features for 4D representations. 4D-fy (Bahmani et al., 2023) and DreamGaussian4D) (Ren et al., 2023) employ multi-stage optimization pipelines to transform static 3D into dynamic 4D scenes. Dream-in-4D (Zheng et al., 2023) allows for personalized 4D generation using image guidance, while Consistent4D (Jiang et al., 2023c) uses video inputs with RIFE (Huang et al., 2022) and a super-resolution module for scene creation. 4DGen (Yin et al., 2023) and AnimatableDreamee (Wang al., 2023c) focus controllable motion generation via driving videos. More recently, Comp4D (Xu et al., 2024a) and TC4D (Bahmani et al., 2024) introduced trajectory- based approaches for creating 4D compositional scenes. While these technologies show promise, they often struggle to produce 4D avatars that effectively interact with objects. Although GAvatar (Yuan al., 2023) excels 4D human animation, its object interaction capabilities are limited. On the other hand, SMPL-based 4D HOI generation remains an open challenge for 4D content generation. Methods like Chain-of-
Xiao et al. (2023), CG-HOI Diller & Dai (2024), and others Liu et al. (2024b); Jiang et al. (2024); Wu et al. (2022); Jiang et al. (2023a); Yang et al. (2024); Wang et al. (2023d) have significantly advanced the field. However, these techniques primarily focus on generating motion sequences for SMPL models, which lack intricate clothing details. Additionally, these methods rely on training datasets, limiting their adaptability in practical scenarios.
3 METHODOLOGY
Given generated 3D avatar and specific 3D object, AvatarGO generates compositional 4D avatars with object interactions based on text instructions. In the subsequent sections, we first introduce the preliminaries (in Sec. 3.1,, including static 3D content generation and parametric human model SMPL-X. Next, we will describe the key components of AvatarGO, including (1) text-driven 3D) human and object composition (in Sec. 3.2), and (2) correspondence-wware motion optimization for achieving synchronized human and object animation (in Sec. 3.3). The overview of AvatarGO is shown in Fig. 2.
3.1 PRELIMINARIES
3D Model Generation. Recently, DreamGaussian (Tang et al., 2023) showcases promising results with largely improved training efficiency by incorporating major components:
4

---

Published as a conference paper at ICLR 2025
(I) 3D Gaussian Splatting (3DGS). 3DGS (Kerbl et al., 2023) directly defines the 3D space through a set of Gaussians parameterized by their 3D position μe, opacity a, anisotropic covariance Σ. and spherical harmonic coefficients sh. The sh term is used to capture the view-dependent appearance of the scene and E can be decomposed to:
Σ RSS R
where is the rotation matrix expressed by quaternion q E SO(3), and S is the scaling matrix, represented by a 3D vector s. Essentially, each Gaussian centered at point (mean) j is defined as:
G(x, 1)) = - ((x-n)"x=" (x-N),
(2)
where x is the 3D query point.
For rendering the 3D Gaussians onto the 2D image space, 3DGS incorporates tile-based rasterizer and point-based a-blend rendering. Specifically, the color C(u) of pixel u can be calculated as:
i-1
C(u)= = E SHshs, SEN
T: = G(x,
II (1 - (xx,a, j=1
where T represents the transmittance, SH denotes the spherical harmonic function, and e indicates the viewing direction. By optimizing the Gaussian attributes {G: q. cJ and dynamically adjusting the density of 3D Gaussians (i.e., densifying and pruning), DreamGaussian achieves high-quality generations from either textual or visual inputs.
(2) Score Distillation Sampling (SDS). Starting with the latent feature z extracted from 3DGS rendering æ, SDS introduces random noise to z. yielding a noisy latent variable x,. This variable is then processed by a pre-trained denoising function cø (zy; y. t) to estimate the added noise. The SDS loss then calculates the difference between predicted and added noise, with its gradient calculated by:
Oz 8x dx d0
V VoLsDs (e.sss((.g(0) = (E,,,~N(0.1)) w(t) (Eg (z -)) E)
(4)
where y denotes the text embedding, w²() weights the loss from noise level t. We do not apply the mesh extraction and texture optimization proposed in DreamGaussian to obtain the 3D models.
SMPL-- (Loper et al., 2015; Pavlakos et al., 2019). With pose parameter 0, shape parameter B, and expression parameter é as inputs, SMPL-X maps the canonical model to the observation space:
M(3 0, 6) LBS (3, 0, o), 0, W),
(5a)
T 0, の) = T B, (3) + (0) + B,(0),
(5b)
where M denotes the function defining the mesh model of human body, and T represents the transformed vertices. W stands for blend weights, B,, Be, and B, are functions respectively for shape, expression, and pose blend shapes. LBS(-) indicates the linear blend skinning function that poses each body vertex of SMPL-X according to:
Vc. 9 K
V. = -V.
(0,
where V. and v. represent SMPL-X vertices under the canonical pose and observation space, respectively. wa is the skinning weight, Gu jx) is the affine deformation that maps the k-th joint ju from the canonical space observation space, and K denotes the number of neighboring joints.
3.2 TEXT-DRIVEN 3D HUMAN AND OBJECT COMPOSITION
With the help of DreamGaussian (Tang et al., 2023), we efficiently generate the 3D avatar G₁, and the 3D object G. individually based on 3DGS and SDS (discussed Sec. 3.). We noticed that even with manual adjustments, such as rescaling and rotating the 3D objects, it's difficult to directly rig the generated 3D human and object models accurately (see Appendix). Therefore, we strive to seamlessly composite GA and G, based on the text prompt in this stage. Specifically, the Gaussian attributes of Gn, and G, would be optimized, as well as three trainable global parameters of G. including rotation R E R4, scaling factor S R. and the translation matrix T E 33:
XG. == S (XG. R + T):
5

---

Published as a conference paper at ICLR 2025
where XG. is the set of static Gaussian points.
However, solely utilizing SDS for optimization could frequently lead to disproportionate relationships and erroneous contact areas (see Fig. 3). This issue can be attributed to two potential factors: (1) the absence of emphasis on words describing human-object interaction, which decreases the model's ability to comprehend the relationships between humans and objects; (2) the complexity inherent in human subjects, posing challenges for the diffusion model to identify the most suitable contact areas (see Sec. 4.3).
Spatial-aware SDS (SSDS). Following Corbo (Chen et al., 2024b), we employ SSDS to facilitate the compositional 3D generation between the human and the object. Specifically, SSDS augments the SDS with a spatial relationship between the human and the object by scaling the attention maps of the designated tokens <token"> with a constant factor (where e > 1):
c. ATTstokan>,
<token> = <token">,
ATT: =
(8)
(ATT<token>>
otherwise.
Here, <token"> corresponds to the tokens encoding the human-object interaction term, such as < 'holding's >, which can be identified through Large Language Models (LLLMs) specified by the user. Consequently, the spatial-awar SDS loss can be written as:
dæ VøLssDs(Ø" (0)) = ((,00) w(()) (co- (2 - dx 00
(9)
where o² denotes the pre-trained denoising function with the adjusted attention maps.
LLM-guided Contact Retargeting, While spatial-aware SDS could benefit in understanding spatial correlations, still faces difficulties in identifying the most appropriate contact area (See Fig. 3), which serves as key component for human-object interaction. According to our studies (see Appendix for visualization), the diffusion model struggles accurately estimate contacts, even in the 2D images generated for human-object interaction. To tackle this issue, we propose leveraging Lang-SAM (lan, 2023) to identify the contact area from text prompts. Specifically, starting from the 3D human model G₁, render it from frontal viewpoint to produce the image I. This image, alongside textual inputs, undergoes Lang-SAN model to derive 2D segmentation masks M:
LangSAM(1, <body
M,
(10)
where <body-part>> part>> represents the text describing the human body part, such as 'hand's Subsequently, we back-project the 2D segmentation labels onto the 3D Gaussians via inverse render- ing (Chen et al., 2023c). Specifically, for each pixel the segmentation maps, update the mask value (0 or 1) back to the Gaussians via:
W; Σ C((u)) (T) M(u),, IEN
(11)
where W; represents the weight of the i-th Gaussian, A is the collection of Gaussians that can be projected onto the pixel u. o(-), T(.), and M(.) respectively denote the opacity, transmittance, and segmentation mask value. Following the weight updates, we assess whether Gaussian corresponds to the segmented region of the human body part by comparing its weight against pre-defined threshold We then initialize the translation parameter T according
T (w- #1) Σ w,
(12)
where w = {w1, ... wx lw, 0/1} E Nx1 1xxx,,1 p== (UI,..,,NN E NN33 and N is the number of Gaussain points within the human model Gj.
3.3 CORRESPONDENCE AWARE MOTION FIELD
Following the compositional integration of 3D humans and objects, animating them synchronously presents an additional challenge owing to potential penetration issues. This problem stems from the absence of well-defined motion field for the object. To this end, establish the motion fields for both human and object models using the linear blend skinning function from SMPL-X (as in Eq. 6), and propose correspondence-awrr motion optimization aimed optimizing the trainable global

---

Published as a conference paper at ICLR 2025
Joker holding microphone in his hand
Goku in Dragon Ball Series holding a torch in his hand
Kratos God of War holding an ase of Thor kis hand Ours (Var-A)
Human Gaussian
GraphDreamer
Ours
Figure 3:
Comparisons on 3D compositional generations.
parameters of the object model, i.e. rotation (R) and translation (7), to improve robustness against penetration issues between humans and objects.
Human Animation. Given the motion sequence, we first construct deformation field, which consists of two components: (1) articulated deformation utilizing the SMPL-X linear blend skinning function LBS(.), and (2) non-rigid motion learning the offset based on HexPlane features (Cao Johnson, 2023), to deform the point x from the canonical space x, in the observation space:
= G
MLP(F(Xx.: t)),
(13)
where F-) denotes the HexPlane-band based feature extraction network, and t indicates the timestamp. We derive G from the closet canonical SMPL-X vertex to x, via Eq. 6.
Object Animation. Similar to the human animation, we calculate the deformation matrix G. for each Gaussian point within the object model Go based its closest canonical SMPL-X vertex. Given our experimental definition of 3D objects as rigid bodies, we then compute their average to establish the intermediate motion field for the object:
X. = G. Xc, G. = EiELLM -iELi Go.
(14)
M
where X, = (Xx, ...., Xos }. X. (Xor ..., XeM ), and M is the total number of Gaussian points within Go. Although animating the object directly using SMPL-X lincar blend skinning may seem like simple solution, it can result in penetration issues between the human and the object (see Fig, 6). This challenge arises primarily from the absence proper constraints to maintain the correspondence between these two models.
Correspundence-wware Motion Optimization. Drawing insight from the fact that our method robust in handling penetration issues in static composited models across various scenarios, propose a correspondece-aware motion optimization to preserve the correspondence between human and object, thereby addressing the penetration problem. Specifically, extend the above motion field (Eq. 14) to include two additional trainable parameters R and T:
X == X, TT
(15)
where X,, is obtained in Eq. 14. Rather than naïvely optimizing the parameters via SDS, we propose novel correspondence-awaat training objective that leverages SMPL-X an intermediary to maintain the correspondence between human and object when they are animated to new pose:
LCs MSE(Ge, Go); Ge (grg ... Geseh Go (g -.. Goss
(16)
7

---

Published as a conference paper at ICLR 2025
Bodybuilder holding dumbbel in his hand
6oku in Dragon Ball Series holding a torch in kis hand
Steven Paul Jabs holding iPhone in kis hand TC4D
DreamGaussian4D)
HumanGaussian*
Ours (Var-B)
Figure 4: Comparisons on 4D avatar animation with object interactions. We present HumanGaus- sian in three poses, whereas other methods are shown in three poses and three different views. 's' indicates that the SMPL LBS function is directly employed on HumanGaussian directly for animation, without additional training as in other methods.
where Ge, derived via Eq. 6 by finding the nearest vertex in the canonical SMPL model to Xe. while Go similarly obtained by locating the closest vertex in the observed SMPL model Xo;
In addition to our correspondence-avare loss, we also incorporate the spatial-aware SDS as in Eq. and the texture-structure joint SDS from HumanGaussian (Liu al., 2023d) to enhance the overall quality:
Oz Ox
TeLsDs(ó,g(0) = d1 [Et,e~ wot) (Ea (Zari8, 6) -xx)
dx 00
aza 0d dd 00
w(t) (co y. ea) + A2 ((,..1(0.11)
(17)
where A1 and A2 are hyper-parameters to balance the impact of structural and textural losses, while denotes the depth renderings.
The overall loss function to optimize the 4D animative scene is then given by:
C λCA LCA + ASDS LSDS + AssDS LSsDS,
(18)
where ACa, ASDS, and Assps represents weights to balance the respective losses.
EXPERIMENTS
We now validate the effectiveness and capability of our proposed framework to animate various 3D avatar-object pairs with different poses and provide comparisons with existing 3D and 4D compositional generation methods.
Implementation Details. We follow DreamGaussian4D (Ren et al., 2023) to implement the 3D Gaussian Splatting (Kerbl et al., 2023) and the HexPlane (Cao & Johnson, 2023) our method. We utilize the pre-trained Pexture-Structure joint diffusion model from HumanGaussian (Liu et al., 2023d) and version of Stable Diffusion (Stability.AI, 2022) to respectively calculate the SDS and spatial-aware SDS in implementation. Typically, for each 3D avatar-object pair, train the 3D stage with a batch size of 16 for 400 epochs, and the 4D stage with batch size epochs. The training takes around 10 minutes for the 3D stage and 20 minutes for the 4D stage single NVIDIA 1100 GPU. We use Adam (Kingma & Ba, 2015) optimizer for back poocpprrpagttion. Additional implementation details can be found in the Appendix.
for 400
Comparison Methods for 3D Statie Generation. We first compare the 3D static generation results with HumanGaussian (Liu al., 2023d) and Graphrreamer (Gao al., 2023). Since
8

---

Published as a conference paper at ICLR 2025
Combo Verse (Chen et al., 2024b) lacks official code release and relies image inputs, compare static AvatarGO with an alternative variant, Le. "Ours (Var-A)", by only using the spatial- aware score distillation sampling (SSDS) in ComboVerse to composite 3D humans and avatars. We cannot compare with GALA3D as their source code is not publicly accessible.
Comparison Methods for 4D Animation. Since there are specific methods tailored for 4D avatar animation with object interactions, we access AvatarGO's 's efficacy against three recent 4D generation techniques (i.e., DreamGaussian4D (Ren et al., 2023), HumanGaussiall (Liu al., 2023d), and TC4D (Bahmani et al., 2024)), as well as one alternative variant "Ours (Var-B)". To implement Var-B, we utilize human hand motion sequences as trajectories to guide the transformation of 3D objects and follow Comp4D to integrate the video diffusion model to compute SDS, Because InterDreamer (Xu et al., 2024b) and InterFusion (Dai al., 2024) have not released their code, we could not include their results for comparison. See more motivation for designing "Ours (Var-A)" and "Ours (Var-B)" in Appendix.
4.1 QUALITATIVE EVALUATIONS
4D Avatar Generation with Object Interaction. In Fig. 1, we present diverse collection of avatar-object pairs that are animated to different poses. These renderings consistently showcase high-fidelity results from various viewpoints. Thanks to our proposed L.I. M-gguddedd contact retargeting and correspondence-awant motion optimization, our method can deliver appropriate human-object interactions and demonstrate superior robustness to the penetration issues.
Comparison on 3D Generation. We provide qualitative comparisons with existing methods on 3D generation Fig. 3. We can observe: 1) without the aid of LLMs, HumanGaussian struggles to determine the spatial correlations between humans and objects; 2) Despite using graphs to establish relationships, GraphDreamer is confused by the meaningful contact, resulting in unsatisfactory outcomes. 3) Optimizing R, S, and T with only SSDS is inadequate to move the object to the correct area. Conversely, AvatarGO consistently outperforms with precise human-object interactions.
Comparisons on 4D Animation. In Fig. 4, compare our 4D animation results with SOTA methods. We take the rendering from our 3D compositions stage as the input for DreamGaussian4D. The following observations can be made: Even with human-object interaction images, Dream- Gaussian4D, which employs video diffusion models, struggles with animating the composited scene. 2) Direct animation via SMPL LBS function, as in HumanGaussian, tends to yield unsmooth results, especially for the arms. 3) TC4D faces similar issues as the DreamGaussian44. Mcanwhile, it treats the entire scene as a single entity, lacking both local and large-scale motions for individual objects. 4) One may think applying trajectory to objects seems like a simple solution (as in Comp4D). However, as seen in "Ours (Var-B)", can disrupt spatial correlations between humans and objects. These points further validate the necessity of AvatarGO. Our method can consistently deliver superior results with correct relationships and better robustness to penetration issues See the Appendix for more comparisons.
4.2 QUANTITATIVE EVALUUTIOOS
CLIP-based Metrics. We use CLIP-based met- ries (CLIP-Score (CLIP-S), CLIP image similar- ity (CLIP-Image), and CLIP Directional Simi- larity (CLIP-DS) (Brooks et al., 2023; Gal et al., 2022)) with CLIP- ViT-L/4 model. Among them. CLIP-S measures the similarity between texts and their corresponding models, Imip-Image denotes the similarity between compositional models and human models, and CLIP-DS represents the alignment between changes in text captions (e.g., "Iron Man" to "Iron Man holding an axe of Thor in his hand") and corresponding changes in images. Through Tab. 1, our method maintain the human identity in the composited scenes (see ILIIeImmae). Note that "Ours (Var- A)" and GraphDreamer slightly better for this metric as they struggle do the composition (see Fig. 3). Meanwhile, "Ours" and "Ours (static)" consistently achieve better results than HumanGaussian and other variants, further affirming the objective superiority of AvatarGO.
Table 1: Quantitative Evaluation.
GrophDDweeer TCM Dimuniisssin
diexs (Waz-a)
(Var-B)
Oues
(static)
Chilehiegge CLIPS CLINNIS
96.51
01.83
90.11
$1.65
19.81
69
31.00
1.71
15.35
4.71
091
25.90
35,00
20:0
User Studies We further conduct user studies to compare with DreamGaussian4D, HumaaGaussian,, TC4D, and "Ours (Var-A)". 24 Volunteers rated these methods independently based seven criteria
9

---

Published as a conference paper at ICLR 2025
w/ SDF distance loss w/ SDF label loss
Ours
w/o R. AcA
w/o Lea
Ours
fron Man kalding an axe his hand
Figure 6: Analysis of correspondence-wware motion field.
from (worst) to (best): (1) Level of penetration; (2) Accuracy of the relative scale between humans and objects; (3) Accuracy of contact; (4) Motion quality; (5) Motion amount; (6) Text alignment; (7) Overall performance. Detailed results have been presented in Tab. 5. Key observations include: 1) Both DreamGausiann4D and HumanGaussian have difficulty providing satisfactory outcomes for human-object interaction (HOI) scenes. 2) Although TC4D performs well with HOI generations, it only produces global motions, leading to less optimal motion quality and quantity compared to our method. Our final design consistently delivers superior results for all seven criteria, outperforming the other methods across the board.
4.3 ABLATION STUDIES
Analysis of LLM-guided Contact Retargeting, We first conduct evaluations to validate the efficacy of employing Lang-SAM for retargeting the accurate contact area. See Fig. 3. By comparing "Ours (Var-A)" and Ours, we can conclude that without Lang-SAM, the model struggles to produce correct human-object interaction in the 3D compositional generation.
Figure 5: User studies.
Motion Field. Fig. 6, we first compare our proposed training objectives LC with two alternative strategies: 1) "SDF distance loss" which mini- mizes the change of signed distance field (SDF) between objects and humans when they are an- imated new pose, and 2) "SDF label loss"
Analysis of Correspondence-awant
Level penetration Accurmc of relutive scale
1.257
1,064
1,551
4,872
1.183
1,002
4.305
3,347
4.785
Accurscy of contact
1.654
1.137
4.412
2.137
4,502
Motion quality
1.331
2.196
1.947
1.0%%
4,592
Motion anount
2.118
3.781
1.517
4.159
4,934
Test alienment"
2.047
1.915
4.515
2,402
4.767
Performance
that supervise the label of SDF instead. These comparisons demonstrate the effectiveness of our proposed method for maintaining spatial correlations during the animations. Additionally, we validate our model's i design by further comparing it with two variants: 1) "w/o R. 'T, LCA" which disables the trainable parameters R. T. Eq. 15, and our proposed loss LC This setting represents the scenario where the object is moved directly with the contact point. and 2) "w/o LC which trains the animation network solely with SDS loss (C LSDS). These comparisons underscore the necessity of these components in achieving 4D animation with better robustness to the penetration issues.
Analysis of Spatial-aware SDS. We finally assess the effectiveness of spatial-aware SDS (SSDS) and present the results in Fig. Notably, observe that SSDS plays a crucial role in preventing the optimization of R, S, T from vanishing during 3D compositional generation. Addition- ally, there is drop in the quality of the animated avatars when disabling SSDS.
Captain halding
Woodyy Story halding microphant wio Eas (dysamic)
hand
wio Liss (static)
Durs (static)
Ours
Figure 7: Analysis of Spatial-aware SDS.
CONCLUSIONS
In this paper, we have introduced AvatarGO, the first attempt for text-guided 4D avatar generation with object interactions. Within AvatarGO, we proposed to employ large language model for comprehending the most suitable contact area between humans and objects. We also presented novel correspondence-awar motion optimization that utilizes SMPL-X as an intermediary to enhance the model's resilience to penetration issues when animating 3D humans and objects into new poses. Extensive evaluations demonstrated that our method has achieved high-fidelit 4D animations across diverse 3D objeet pairs and poses, surpassing current state-of sate--ofthhe-ar by a large margin.
10

---

Published as a conference paper at ICLR 2025
ACKOWLEDGEMENT
This study is supported by the National Key R&D Program of China No. 20.222220011010.. This study is also supported by Shanghai AI Laboratory, the Ministry of Education, Singapore, under its MOE AcRF Tier 2 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). This study is also supported by the Hong Kong Research Grants Council General Research Fund (Grant No.: 17211024).
MOE-T2PP20211-0022 and under the RIE2020
REFERENCES
Language segment anything. https://githhi com/ wwtp.///p..lu.e.eerrraaaaaaaaaaa hut 2023. 3,6 6
sam giit,
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311.17944 2023. 4
Sherwin Bahmani, Xian Liu, Yifan Wang. Ivan Skorokhodov. Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. To4d: Trajectory-conditione conditioned text-to- 4d generation. arXiv preprini arXiv:2403. 114440337722 17920, 2024. 2, 4,
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint 2rXiv:2211.01322 2022. 1,3
Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2022.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15117, 2023. 2
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.7
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023. 7,
Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, and Kangxue Yin. Texfusion: Synthesizing 3d textures with text-guided image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4169-4181, 2023a. 3
Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan- Yee K Wong. Dreamavatar: Text-and- shape guided 3d human avatar generation via diffusion models. arXiv preprint arXiv:2304.00916 2023b. 1,3
Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yer K Wong. Guide3d: Create 3d avatars from text and image guidance. arXiv preprint arXiv:2308.00 09705, 2023c. 3
Yukang Cao, Masoud Hadi, Liang Pan, and Ziwei Liu, Gs-vton: Controllable 3d virtual try-on with gaussian splatting. arXiv preprint arXiv:2410.05255 2024. 3
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021. 3
Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Yine Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303. 11396. 2023a. 3
11

---

Published as a conference paper at ICLR 2025
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. arXiv preprint arXiv:2303. 2023b.
Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming diffusion probabilisttic models for character control. In SIGGRAPH, 2024a.
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang. Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. arXiv preprint arXiv:2311.1452,, 2023c.
Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Comboverse: Compositional 3d assets creation using spatially-aware diffusion guidance. arXiv preprint
arXiv:2403.12409, 2024b. 2, 6, 6,
Sisi Dai, Wenhao Li. Haowen Sun, Haibin Huang, Chongyang Ma, Hui Huang, Kai Xu, and Ruizhen Hu. Interfusion: Text-driven generation of 3d human-object interaction. arXiv preprint
arXiv:2403. 15612, 2024. 2, 2,3,
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024. 3
Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 19888-199 19901, 2024. 4
Dave Epstein, Ben Poole, Ben Mildenhall, Alexei A Efros, and Aleksander Holynski. Disentangled 3d scene generation with layout learning. arXiv preprint arXiv:2402.169936 2024. 2,
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 2022.
Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. Graphdreamer: Compositional 3d scene synthesis from scene graphs. arXiv preprint arXiv:2312.000993 2023. 2, 3,
Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Chongyang Ma, Weiming Hu, Zhengjun Zha, Haibin Huang, Pengfei Wan, et al. 12v-adapter: A general image-to-videe adapter for video diffusion models. arXiv preprint wxXv:2312.11669) 2023. 2
Xiao Han, Yukang Cao, Kai Han, Xiatian Zhu, Jiankang Deng. Yi-Zhe Song, Tao Xiang, and Kwan- Yee K Wong. Headsculpt: Crafting 3d head avatars with text. arXiv preprint 206.038306.03038. 2023. 3
Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang. Liang Pan, Dahua Lin, and Ziwei Liu. 3dtopia: Large text-to-3d generation model with hybrid diffusion priors. arXiv preprint arXiv:2405 022344 2024.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:231 1..44000 2023. 3
Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, and Qing Wang. Humannorm: Learning normal diffusion model for high-quality and realistic 3d human generation. arXiv preprint 2300v:23100014000 2023a.
12

---

Published as a conference paper at ICLR 2025
Yukun Huang, Jianan Wang. Ailing Zeng, He Cao, Xianbiao Qi, Yukai Shi, Zheng-Jun Zha, and Lei Zhang. Dreamwaltz: Make a scene with complex 3d animatable avatars. arXiv preprint DrXiv:2305.12529, 2023b.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision, pp. 624-642. Springer, 2022.
Nan Jiang, Tengyu Liu, Zhexuan Cao, Jieming Cui, Zhiyuan Zhang, Yixin Chen, He Wang, Yixin Zhu, and Siyuan Huang. Full-body articulated human-object interaction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9365-9376 2023a. 1,
Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin
Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1737-1747, 2024.
Ruixiang Jiang, Can Wang, Jingbo Zhang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control. arXiv preprint arXiv:2303. /7606, 2023b. 3
Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 { ('deg] dynamic object generation from monocular video. arXiv preprini urXiv:23/1.02844 2023c. 4
Roy Kapon, Guy Tevet, Daniel Cohen-Or, and Amit H Bermano. Mas: Multi-view ancestral sampling for 3d motion generation using 2d diffuusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1965-1974, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4))1-14, 2023. 3, 5,
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. 8
Nikos Kolotouros, Thiemo Alldieck, Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Fieraru, and Cristian Sminchisescu. Dreamhuman: Animatable 3d avatars from text. arXiv preprint arXiv:2306.09329, 2023. 3
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.062/ 2023a.
Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Trans. Graph., 42(6), 2023b.
Tingting Liao, Hongwei Yi, Yuliang Xiu, Jiaxaing Tang, Yangyi Huang, Justus Thies, and Michael J Black. Tada! text to animatable digital avatars. arXiv preprint arXiv:2308. 10899, 2023. 1,3
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition, 2023. 1,
Fangfu Liu, Hanyang Wang, Weiliang Chen, Haowen Sun, and Yueqi Duan. Make-your-3d: Fast and consistent subject-driven 3d content generation. arXiv preprint arXiv:2403.09625, 2024a. 3
Hanchao Liu, Xiaohang Zhan, Shaoli Huang, Tai-Jiang Mu, and Ying Shan. Programmable motion generation for open-set motion control tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1399-1408, 2024b. 4
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang, Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng. Jiayuan Gu, and Hao Su. One-2-3-45+4: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint XXix:23/1.07883 2023a.
13

---

Published as a conference paper at ICLR 2025
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. arXiv preprint arXiv:2306.1692 16928, 2023b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9298-9309, 2023c. 1,3
Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. arXiv preprini arXiv:2311. 2023d. 3, 8,
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. images from a single-view image. arXiv preprint
Syncdreamer: Generating
arXiv:2309.03453, 2023e.
Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for relightable text-to-3d generation. arXiv preprint arXiv:2312 287544, 2023f.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image 3d using cross-domain diffusion. arXiv preprint arXiv:23 15008, 2023. 3
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics, Asia, 2015. 1,5
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guipe-guieddd generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600, 2022. 3
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In IEEE Conference on Computer Vision and Pattern Recognition, 2019. 1,5
Ben Poole. Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference Learning Representations, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06//25 2022. 1,
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dream- gaussian4d: Generative 4d gaussian splatting. arXiv preprint 12Xiv:231..17142, 2023. 2,3, 3,3,,, 4, 8,
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. arXiv preprint arXiv:2302.01721, 2023. 3
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic -mnagee diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487, 2022.
Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin- Kim, Jiyoung Lee, and Seungryong Kim. Let diffusion model know 3d-consistency for robust text-to-3d generation, arXiv preprint arXivv2303.079.77 2023.
14

---

Published as a conference paper at ICLR 2025
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion generative prior. arXiv preprint arXiv:2303.014100 2023.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. In Advances in Neural Information Processing Systems, 2021."
Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient- mesh optimization. ACM Transactions on Graphics (TOG), 42(4):1-16, 2023. 3
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol single image consistent multi-view diffusion base model. arXiv preprint #rXiv:2310.15110 2023a. 3
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv:2308. 16512, 2023b. 3
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprini arXiv:2301. 12880, 2023. 4
Stability.AI.
Stable diffusion.
https://stabilityy.e www.lllyyyyyy.iae.. ai/lo//
stable-di do--iiffuut ffuu www.ssnnuuuulee-----o relaeee lease, 2022. 3,
Stability.AI. Stability AI releases DeepFloyd IF, a powerful text-to-image model pttts::/sttablityye
that smartly integrate text into images.
Seepfloyd-oo ffffetttttoooo tooxx-tto--o imagemmodeee 2023. 1,3
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309. /6653, 2023. 1, 3,4, 4,
Jaaxian Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprins arX1v:2402.05054, 2024a.
Jiaxiang Tang, Ruijie Lu, Xiaokang Chen, Xiang Wen, Gang Zeng, and Ziwei Liu. Intex: Interactive text-to-texture synthesis via unified depth- aware inpainting. arXiv preprint arXiv:2403.11875 2024b.
Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mydiffusiont+: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. arXiv preprint 2222:22402.127122 2024c.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 3
Peihao Wang, Zhiwen Fan, Dejia Xu, Dilin Wang, Sreyas Mohan, Forrest landola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, al. Steindreamer: Variance reduction for text-to-3d score distillation via stein identity. arXiv preprint arXiv:2401.0060 0000044 2023a.
Peng Wang, Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lmm: Pose-free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:2311. 12024, 2023b.
Xinzhou Wang, Yikai Wang. Junliang Ye, Zhengyi Wang. Fuchum Sun. Pengkun Liu, Ling Wang, Kai Sun, Xintong Wang, and Bin He. Animatabledreamer: Text-guided non-rigid 3d model generation and reconstruction with canonical score distillation. arXiv preprint urXiv:2312.03799 2023c. 4
15

---

Published as a conference paper at ICLR 2025
Yinhuai Wang, Jing Lin, Ailing Zeng. Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics- based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393, 2023d. 4
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, arXiv preprint arXiv:2305. 162 13, 2023e. 1,3
Zhengyi Wang. Yikai Wang. Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image 3d textured mesh with convolutional reconstruction model. arXiv preprint 2433:2403300034, 2024. 3
Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1 122-11221111 136, 2023a.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, Liang Pan, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, et al. Omniobject3d: Large-vocabular 3d object dataset for realistic perception, reconstruction and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 803-814, 2023b.
Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, and Siyu Tang. Saga: Stochastic shole-body grasping with contact. In European Conference on Computer Vision, pp. 257-274 Springer, 2022.
Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, and Hanwang Zhang. Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior. arXiv preprint arXiv:2401.09050, 2024. 3
Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiang- miao Pang. Unified human-scene interaction via prompted chain-of-contace arXiv preprint arXiv:2309.07918. 2023. 4
Dejia Xu, Hanwen Liang Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: Llm-guided compositional 4d scene generation. arXiv preprint
2xXi:22403.16993 2024a. 2.4
Sirui Xu, Ziyin Wang, Yu-Xiong Wang, and Liang- Yan Gui. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. arXiv preprint arXiv:2403.19552 2024b. 1.9
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.09211 2023.
Jie Yang, Xuesong Niu, Nan Jiang, Ruimao Zhang, and Siyuan Huang. F-hoi: Toward fine-grained semantic-aligned 3d human-object interactions. In European Conference on Computer Vision, pp. 91-110. Springer, 2024. 4
Yuyang Yin, Dejia Xu, Zhangyang Wang. Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arX2::2312.17225, 2023.
Xu Yinghao, Shi Zifan, Yifan Wang, Chen Hansheng, Yang Ceyuan, Peng Sida, Shen Yujun, and Wetzstein Gordon. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation, 2024. 3
Ye Yuan, Xueting Li, Yangyi Huang, Shalini De Mello, Koki Nagano, Jan Kautz, and Umar lqbal. Gavatar: Animatable 3d gaussian avatars with implicit mesh learning. arXiv preprint arXiv:2312.11461, 2023. 4
Yifei Zeng, Yuanxun Lu, Xinya Ji, Yao Yao, Hao Zhu, and Xum Cao. Avatarbooth: High-quality and customizable 3d human avatar generation. arXiv preprint arXiv:2306.098k. 2023.
16

---

Published as a conference paper at ICLR 2025
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022. 1
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: astrieval-augmenie motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 364-373, 2023. 3,3
Mingyuan Zhang. Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang. Chongzhi Zhang. Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified multi-modal motion generation. arXiv preprint urXiv:240..01284 2024. 1,3
Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. unified approach for text-and image-guided 4d scene generation. arXiv preprint 1xX14 2023.
Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deqing Sun, and Ming-Hssun Yang. Gala3d: Towards text-to-3d complex scene generation via layout- guided generative gaussian splatting, arXiv preprint arXiv:2402.07207 2024. 2, 3
17


Reviews:
[{"review_id": "zOBQPvYa1Q", "reviewer": "Reviewer_D2nt", "summary": "The paper introduces AvatarGO, a framework for generating and animating 4D human-object interactions (HOI) in a zero-shot manner using pre-trained diffusion models. The main contributions of the paper include: (1) LLM-guided ****contact ****retargeting, which addresses the challenge of determining where objects should interact with the human body by using large language models (LLMs) to identify appropriate contact areas from text prompts, and (2) correspondence-aware motion optimization, which ensures coherent motion between humans and objects while preventing issues like object penetration during animation. AvatarGO generates realistic 4D HOI scenes directly from textual inputs, overcoming limitations in existing methods that rely on SMPL-based models. Qualitative and quantitative results show that the AvatarGO outperforms existing methods in terms of generating plausible human-object interaction animation.", "questions": "- Is each HOI trained with one motion only? Can it generalize to multiple motions?\n- Fig. 4 only shows one frame of animation, which is not animation but reposing. Multiple frames need to be shown.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles the new task of zero-shot generation of 4d human-object interaction (HOI).\n- The proposed LLM-guided contact retargeting helps initialize the object in reasonable 3D locations to help generate plausible HOI.\n- The correspondence-aware motion optimization helps prevent penetration and maintain plausible interactions throughout the animation.\n- Visually, the method outperforms previous methods in the quality of generated HOI. The higher CLIP score and user preference also support this.", "weaknesses": "- The overall appearance quality of generated humans and objects is still limited. The results are often blurry and lack realism. There is also still human-object penetration (see “Naruto in Naruto Series stepping on a football under his foot” on the webpage).\n- The notations in object animation and correspondence-aware motion optimization are poorly explained. What is G_c and G_o? How are they computed? The paper mentioned they are derived from x_ci and x_oi, but it’s unclear.\n- It seems the method requires training for each human-object pair, which can be quite inefficient for large-scale generation. Is there anyway for the method to train the same character interacting with multiple objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5TjY8FxRw", "reviewer": "Reviewer_1bKd", "summary": "The paper proposes AvatarGO, a framework that generates 4D human-object interaction (HOI) to address the challenges of the lack of realistic large-scale interaction datasets. The authors generate 4D HOI samples in a zero-shot manner by leveraging a pre-trained diffusion model (Stable Diffusion). To mitigate the diffusion model's lack of understanding of \"where\" and \"how\" objects interact with humans, the authors propose 1) LLM-guided contact retargeting, which identifies the contact body part from the text prompt, and 2) correspondence-aware motion optimization, which models human and object deformation fields using linear blend skinning function. The authors argue that their extensive experiments validate the proposed method's superior generation capabilities.", "questions": "1. In L310, the reviewer wonders how the authors define the threshold $\\alpha$.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The qualitative results of the proposed method seem more plausible than the compared methods in Figure 3. Moreover, the authors conduct user studies to compare the proposed method with comparable methods in diverse criteria, e.g., penetration, motion quality, and overall performance.\n\n2. The paper is well-structured, with clear sections from introduction to methodology and experiments. The authors provide detailed explanations about their task with various citations, which helps the reviewer to understand the problem the authors focused on.", "weaknesses": "1. The reviewer wonders if the human motion is given as input or is generated by the proposed method. In Figure 2 caption (L185), the authors explain that correspondence-aware motion optimization jointly optimizes human and object animation, but in L345, it is written that the motion sequence is given. The reviewer requests clear explanations of what are the inputs and outputs of the proposed pipeline.\n\n2. The reviewer thinks that the proposed method's novelty is limited. The objective function of the proposed generative model is spatial-aware SDS (L276), that is introduced by ComboVerse (Chen et al., 2024b), not newly proposed by the authors. Moreover, the authors propose LLM-guided contact retargeting, which identifies 3D Gaussians by unprojecting 2D segmentation masks. This process operates with a heuristically pre-defined threshold $\\alpha$, and the reviewer wonders if this heuristic threshold $\\alpha$ applies to various contact body parts, e.g., fingers, feet, head, and legs.\n\n3. Since the proposed method only handles the composition of generated 3D humans and objects, the overall quality is prone to be bounded by the 3D asset generation module. Then, if the 3D objects and humans are generated in poor quality, how can the authors mitigate the error propagation in the proposed method?\n\n4. The qualitative results in Figure 3 only show the interaction with human hands. The reviewer requests more visualization results of diverse objects and body parts, such as the head, feet, and legs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6UM2z6Jice", "reviewer": "Reviewer_E4gh", "summary": "This paper introduces AvatarGO, a novel framework that utilizes diffusion models and language guidance to enable zero-shot generation of realistic, animatable 4D human-object interactions from text prompts.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with a clear motivation and an easy-to-follow presentation of the core idea. Related work is thoroughly covered, incorporating up-to-date research in the field.\n2. Extensive experiments and ablation studies convincingly demonstrate the effectiveness of AvatarGO, showcasing appropriate human-object interactions and superior robustness against penetration issues.", "weaknesses": "I believe this paper is ready for publication, as I found no major weaknesses. My only question concerns the diversity of objects included in the text prompts used in the experiments. Additionally, how does your method perform across different types of objects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fn0ofnufwO", "reviewer": "Reviewer_bzXT", "summary": "This paper introduces AvatarGO, a novel framework for generating realistic and animatable 4D human-object interaction (HOI) scenes directly from textual descriptions. Addressing the challenges of determining \"where\" and \"how\" objects interact with human bodies, the authors propose two key innovations:\n\n1. **LLM-Guided Contact Retargeting**: Utilizing Lang-SAM to identify contact body parts from text prompts, ensuring precise human-object spatial relations.\n2. **Correspondence-Aware Motion Optimization**: Constructing motion fields for both human and object models using the linear blend skinning (LBS) function from SMPL-X to maintain correspondence during animation, reducing penetration issues.\n\nThe proposed method operates in a zero-shot manner using pre-trained diffusion models, overcoming the scarcity of realistic large-scale interaction data. Extensive experiments demonstrate that AvatarGO outperforms existing methods in generating coherent compositional motions and handling various human-object pairs and poses.", "questions": "- How does AvatarGo compare to SMPL-based 4D HOI generation methods?\n- Aside from the apparent appearance advantage, does AvatarGo achieve a similar interaction quality to the SMPL-based methods?\n- What advantages does direct 4D Gaussian splatting optimization offer over a two-stage approach (SMPL generation followed by GS fitting)?\n- Why do the results trend toward anime/cartoon styles? What technical modifications would enable more photorealistic outputs?\n- How does runtime scale with sequence length and scene complexity?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the challenging and less-explored problem of generating visually realistic and animated 4D human-object interactions directly from textual inputs, which is crucial for applications in AR/VR and gaming.\n- To the best of my knowledge, using LLM-Guided Contact Retargeting and Correspondence-Aware Motion Optimization to improve HOI in 4D Gaussian Splatting generation is novel. The proposed methods clearly solve the “where” and “how” problems raised in the introduction.\n- The proposed method operates in a zero-shot manner with pre-trained diffusion models. It avoids the need for scarce large-scale interaction datasets.", "weaknesses": "A significant problem of the paper is the lack of a substantial body of related work in HOI generation, particularly in 4D HOI generation methods. Important works such as Programmable Motion Generation, Chain-of-Contacts, CG-HOI, HSI, among others, are not discussed. These methods have contributed significantly to the field by generating SMPL and SMPL-X poses for HOI scenarios. \n\nWhile the paper focuses on generating realistic appearances using Gaussian splatting, which differs from the approaches of these methods, it is essential for the authors to analyze and position their work in the context of existing studies in 4D HOI generation. For instance, the use of large language models (LLMs) to generate contact-based constraints was first proposed in Chain-of-Contacts. By not acknowledging these relevant works, the paper misses the opportunity to highlight how AvatarGO builds upon or differs from established methods, potentially overlooking important insights and contributions from the existing literature.\n\nAnother weakness of the proposed method, as mentioned by the authors in the limitation section, is that it assumes a consistent geometric relationship between the interacting body part and the object. This prohibits it from generating complex manipulations such as tossing an object or passing an object between hands. \n\n### References\n\n- Liu, Hanchao, et al. \"Programmable Motion Generation for Open-Set Motion Control Tasks.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 2024.\n- Xiao, Zeqi, et al. \"Unified Human-Scene Interaction via Prompted Chain-of-Contacts.\" *The Twelfth International Conference on Learning Representations (ICLR)*.\n- Diller, Christian, and Angela Dai. \"Cg-hoi: Contact-guided 3d human-object interaction generation.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.\n- Jiang, Nan, et al. \"Scaling up dynamic human-scene interaction modeling.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)