Title: InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction
Venue: NeurIPS 2024
Accepted: yes
Keywords: human object interaction

human motion generation
Abstract: Text-conditioned human motion generation has experienced significant advancements with diffusion models trained on extensive motion capture data and corresponding textual annotations. However, extending such success to 3D dynamic human-object interaction (HOI) generation faces notable challenges, primarily due to the lack of large-scale interaction data and comprehensive descriptions that align with these interactions. This paper takes the initiative and showcases the potential of generating human-object interactions without direct training on text-interaction pair data. Our key insight in achieving this is that interaction semantics and dynamics can be decoupled. Being unable to learn interaction semantics through supervised training, we instead leverage pre-trained large models, synergizing knowledge from a large language model and a text-to-motion model. While such knowledge offers high-level control over interaction semantics, it cannot grasp the intricacies of low-level interaction dynamics. To overcome this issue, we introduce a world model designed to comprehend simple physics, modeling how human actions influence object motion. By integrating these components, our novel framework, InterDreamer, is able to generate text-aligned 3D HOI sequences without relying on paired text-interaction data. We apply InterDreamer to the BEHAVE, OMOMO, and CHAIRS datasets, and our comprehensive experimental analysis demonstrates its capability to generate realistic and coherent interaction sequences that seamlessly align with the text directives.

InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction

Abstract

Text-conditioned human motion generation has experienced significant advancements with diffusion models trained on extensive motion capture data and corresponding textual annotations. However, extending such success to 3D dynamic human-object interaction (HOI) generation faces notable challenges, primarily due to the lack of large-scale interaction data and comprehensive descriptions that align with these interactions. This paper takes the initiative and showcases the potential of generating human-object interactions without direct training on text-interaction pair data . Our key insight in achieving this is that interaction semantics and dynamics can be decoupled. Being unable to learn interaction semantics through supervised training, we instead leverage pre-trained large models, synergizing knowledge from a large language model and a text-to-motion model. While such knowledge offers high-level control over interaction semantics, it cannot grasp the intricacies of low-level interaction dynamics. To overcome this issue, we introduce a world model designed to comprehend simple physics, modeling how human actions influence object motion. By integrating these components, our novel framework, InterDreamer, is able to generate text-aligned 3D HOI sequences without relying on paired text-interaction data. We apply InterDreamer to the BEHAVE, OMOMO, and CHAIRS datasets, and our comprehensive experimental analysis demonstrates its capability to generate realistic and coherent interaction sequences that seamlessly align with the text directives.

1 Introduction

Text-guided human motion generation has made unprecedented progress through advancements in diffusion models [41, 105, 106, 131], leading to synthesis outcomes that are realistic, diverse, and controllable. This progress has ignited an increased interest in exploring expanded tasks related to textguided human interaction generation, such as social interaction [69] and human-scene interaction [44]. However, many of these explorations are limited in that the dynamics of objects is not involved or text-guided. Aiming to bridge such a gap, this paper tackles a more challenging task generating versatile 3D human-object interactions (HOIs) through language guidance , as illustrated in Figure 1.

Although a direct solution, as suggested by the concurrent work [28, 65, 91, 107, 136, 140], would be replicating the success observed in human motion generation and adopting a similar supervised approach for learning text-driven HOIs, it is not scalable. As can be observed, generating social or scene interactions is heavily dependent on extensive collections of text-interaction pair data [34, 69, 83, 130], and scaling these methods to address more complex HOIs outlined in our study could require datasets of comparable magnitude. Achieving this goal appears unattainable by merely annotating existing 3D HOI datasets [7, 30, 45, 47, 53, 66, 169, 170, 177, 180], which are relatively limited in

Semantics

Dynamics

How do humans

carry backpacks?

A person picks up a

backpack, puts it on, and

then walks forward

clockwise.

K²=()(6)

size. Although recent studies [28, 65, 91, 155] have annotated some of these datasets, the volume of text-interaction pairs still lags behind that available for existing text-driven motion generation efforts.

An intriguing question naturally arises: given the limited annotations of the text, what is the potential of learning for text-conditioned HOI generation without text supervision , which is the main focus of this paper. However, formulating the task in such a setting presents significant challenges, primarily due to the inability to directly learn the alignment between text and HOI dynamics. Our key observation is that interaction semantics and dynamics can be decoupled . That is, the high-level semantics of an interaction, aligned with its textual description, can be informed by human motion and the initial object pose . Meanwhile, the low-level dynamics of the interaction - specifically, the subsequent behavior of the object - is governed by the forces exerted by the human, within the constraints of physical laws. Motivated by these insights, we introduce InterDreamer - a novel framework that synergizes knowledge of interaction semantics and dynamics (Figure 1), both of which do not necessarily require learning from text-interaction pairs, if they are decoupled.

The semantics of interaction, although not available through direct supervised training, can be harnessed from prior knowledge without text-interaction pair datasets. Specifically, to acquire semantically aligned interaction, we first consult a large language model (LLM), such as GPT-4 [88] and Llama 2 [120], to provide understanding including how humans typically use specific body parts in interactions with particular objects, by exploiting its in-context learning capability with few-shot prompting [10] and chain-of-thought prompting [134]. The intermediate thoughts and the final thought are then used to ( i ) generate semantically aligned human motion with a pre-trained textto-motion model; and ( ii ) identify an initial object pose that is harmonious with the generated human pose and text description, following a philosophy similar to retrieval-augmented generation [62].

While these large models can offer high-level motion semantic modeling, they lack crucial low-level dynamics knowledge. Nevertheless, by decoupling interaction dynamics from semantics, a key advantage emerges in our InterDreamer framework: interaction dynamics can be learned from motion capture data without the necessity of text annotations . We instantiate this idea by developing a world model , which predicts the subsequent state of an object affected by the interaction. The key here is to reach generalizable representations in different motion and objects. To do so, we exert control over the object through the motion of vertices on the human body. These vertices are solely sampled in regions where contact occurs, agnostic to the overall object shape and whole-body motion. Such abstraction empowers the model to learn the simple dynamics from a publicly available 3D HOI dataset BEHAVE [7], and generalize naturally to other datasets [47, 66]. The plausibility of the generated interaction is further enhanced by a subsequent optimization procedure on the synthesized human and object motion.

To summarize, our contributions are: ( i ) We address the task of synthesizing whole-body interactions with dynamic objects guided by textual commands, achieving this notably without the need for paired text-interaction data, a novel paradigm to the best of our knowledge. ( ii ) We introduce a framework that decomposes semantics and dynamics, and they can be easily integrated. Specifically, it harnesses knowledge from a large language model (LLM) and a text-to-motion model as external resources, alongside our proposed world model. Remarkably, the only component that requires additional training is the world model, which highlights the ease of use of our framework. ( iii ) Experimental results demonstrate that our framework, InterDreamer, is capable of producing semantically aligned and realistic human-object interactions, and generalizes beyond existing HOI datasets .

2 Related Work

Text-Conditioned Human Motion Generation. Significant progress has been witnessed in human motion synthesis tasks, given different kinds of external conditions, including action categories [2, 36, 61, 93], past motion [5, 17, 86, 110, 149, 150, 163], trajectories [31, 50, 51, 100, 122, 145], scene context [12, 29, 39, 44, 113, 114, 125-127, 130, 153, 175, 182, 183], and without condition [96]. Recently, human motion synthesis guided by textual descriptions [1, 6, 18, 24, 26, 34, 35, 49, 54, 57, 64, 71, 77, 81, 94, 95, 97, 104, 116, 133, 160, 162, 168, 172, 174, 178, 179, 181, 185, 187] is popular and extended to various applications, including the text-conditioned generation of multipleperson [33, 43, 63, 75, 132] and human-scene interaction [14, 21, 44, 48]. Our goal is to model human and object dynamics concurrently guided by text.

Human-Object Interaction Generation. Synthesizing hand-object interactions [11, 15, 20, 68, 73, 79, 80, 119, 137, 161, 165, 167, 184, 186] and single-frame human-object interactions [25, 42, 56, 92, 128, 143, 152, 154, 166] are popular topics and extended to zero-shot settings [52, 67, 156, 157]. Recently, researchers explore whole-body dynamic interaction generation, in kinematic-based approaches [22, 27, 32, 38, 58-60, 66, 84, 85, 99, 107-109, 111, 123, 136, 139, 140, 148, 151, 176] and physics-based approaches [4, 8, 16, 23, 40, 72, 74, 78, 87, 89, 115, 117, 124, 129, 142, 146, 147, 158]. Current methods in HOI synthesis are often restricted by a narrow scope of actions, the use of non-dynamic objects, and a lack of comprehensive whole-body motion. Our work aims to generate diverse whole-body interactions with various objects, and enables control through language input. Recent datasets [7, 30, 45, 47, 53, 66, 112, 138, 144, 155, 164, 169, 170, 180] provide the groundwork for research in this area, and concurrent efforts [28, 65, 91] demonstrate the feasibility of applying supervised learning methods via annotating datasets. However, the amount of data currently available fall short when compared to more extensive text-motion datasets [34, 70, 83]. This discrepancy in data volume limits the capability of supervised methods to capture the complexity of human-object interactions, motivating us to investigate the potential of zero-shot generation.

External Knowledge from LLMs. Large language models (LLMs) are being used for advanced visual tasks, such as editing images based on instructions [9]. In digital humans, they are used to reconstruct 3D human-object interactions [128] and generate human motion [3, 46, 159, 178] as well as human-scene interactions [141]. Our approach is inspired by [128], which uses LLMs to infer contact body parts with a given object for reconstructing 3D human-object interactions - a task different from ours. Our approach utilizes GPT-4 [88] or Llama 2 [120], to not only understand contact body parts but also narrow the distribution gap between different tasks, and provide knowledge for interaction retrieval. This is accomplished by utilizing the in-context learning capabilities of LLMs [22] and their support for retrieval-augmented generation [62].

3 Methodology

Problem Formulation. Our goal is to synthesize a sequence of 3D human-object interactions x that satisfies a descriptive text p . This sequence is a series of tuples [( h 1 , o 1 ) , ( h 2 , o 2 ) , . . . , ( h M , o M )] , where h i represents the human pose parameters defined in the SMPL model [76], while the shape of the human is unified the same as [34]. o i defines the pose of the rigid object in terms of its 3D spatial position and orientation. The sequence length M is variable and is dynamically determined by our text-to-motion model based on the input text p . We do not require text supervision for training.

Overview. Our framework, illustrated in Figure 2, can be conceptualized as a Markov decision process (MDP). We begin by dividing the motion sequence into T segments, each with m frames,

Text-to-Motion

Interaction Retrieval

Dynamics Modeling

Low-Level Control

World Model

s

1

s

t

,

Abstracting

states (object) and actions (human)

Actions as

spatial conditioning

controls

to the dynamics model

Past state

abstraction

Future state

abstraction

Action

abstraction

ℱ

k

Cross

attention

a

1

, …,

a

t

a

t

+1

a

t

,

a

1

Optimization

a

t

+1

s

t

+1

You will be given a sentence that describes the interaction between a person

and an object. You will need to extract information from the description

and answer three questions based on the following rules and examples.

1. Extract the category of the object with which the person interacts.

Answers should be selected from the following objects: [trashbin, monitor, boxlarge,

stool, boxsmall, backpack, boxlong, plasticcontainer, tablesquare, yogaball, yogamat, toolbox,

chairwood, chairblack, boxmedium, boxtiny, suitcase, tablesmall].

[start of rules]

•

Pick the most similar object from the list if the description is not in the list.

[end of rules]

2. Infer the body part that contacts with the object at the beginning of the

interaction.

Answers should be selected from the following body parts: [right hand, left hand,

arm, back of the hip, upper leg, leg, upper back, left foot, right foot, front body, back, shoulder,

no contact

].

[start of rules]

•

If the description does not specify which hand or foot, randomly choose from

'left' or 'right'.

…

[end of rules]

3. Simplify and modify the description.

[start of rules]

•

The subject of the sentence should be "a person".

…

[end of rules]

[start of examples]

A person uses their feet to apply force to the medium-sized box. Answer:

boxmedium|left foot|A person kicks the box with left foot.

…

[end of examples]

The description of human-object interaction:

Someone is physically pulling the chair on the floor to adjust its position.

Please output the answer following the format of

"Answer: Answer 1|Answer 2|Answer 3"

High-Level Planning

/u1D4A2

k

.

M

！

where M = T × m . Object motion { o i } i M =1 can be seen as a sequence of environmental states { s t } T t =1 , and human motion { h i } i M =1 is described as a sequence of actions { a t } T t =1 that interact with the environment. Under such an MDP setup, our framework starts with high-level planning L , which deciphers textual interaction description p by g = L ( p ) (Sec. 3.1). Then, a text-tomotion model π translates context g into human actions, modeled as a t +1 ∼ π ( a t +1 | s t , { a i } t i =1 , g ) (Sec. 3.2). The interaction retrieval R proposes an initial object state s 1 ∼ R ( s 1 | a 1 , g ) , based on the initial action a 1 and context g (Sec. 3.2). After that, a world model P is trained to predict future states s t +1 ∼ P ( s t +1 | a t , s t , a t +1 ) from the current action and state (Sec. 3.3). Our world model incorporates an optimization process, for both state and action refinement (Sec. 3.4). Notably, the text-to-motion and world models are executed iteratively until text-to-motion generates an end frame.

3.1 High-Level Planning

Leveraging LLMs' strong reasoning capabilities and inherent common sense, our high-level planning L yields interaction details g = L ( p ) that cannot be naïvely extracted in textual descriptions p . The process undertaken by L encompasses three steps: ( i ) Determining the object : the LLM is employed to translate described objects into corresponding categories from a predefined list. ( ii ) Determining initial human-object contact : the LLM infers the body parts involved in the interaction, drawing from a list defined in the SMPL model [76]. And most importantly, ( iii ) reducing the distribution gap : the LLM bridges the distribution gap between the free-form textual input and the language used within the training data of the text-to-motion model [34]. This involves standardizing syntax and content according to designed guidelines. In Figure 2, we demonstrate the prompt we used with the few-shot prompting [10]. We define intermediate thoughts and the final thought, i.e. , answers to three questions, as detailed information g = L ( p ) , which guides the subsequent procedure, structuring the entire framework with a philosophy similar to retrieval-augmented generation [62]. Our high-level planning operates indirectly in the generation of interactions. Nonetheless, it narrows the vast range of possible interactions in the real world into a more manageable distribution within the capabilities of our framework. We incorporate GPT-4 [88] and Llama-2 [120] for evaluation.

3.2 Low-Level Control

With the information g derived from the description p , the low-level control aims to create a sequence of human actions { a t } T t =1 by a text-to-motion model, and an initial state s 1 by interaction retrieval, such that they correspond to the objectives outlined by g .

Text-to-Motion. We utilize a text-to-motion model π to develop actions to be executed in the world model. At each timestep t , π receives the sequence of previous actions { a i } t i =1 and the text tokens encoded from the rewritten description in g = L ( p ) , and produces a next action a t +1 , which later in Sec. 3.4 will be adjusted through an optimization process that intertwines actions with the object state s t . Thus, the overall process can be formally defined as a t +1 ∼ π ( a t +1 | s t , { a i } t i =1 , g ) , while the initial action a 1 ∼ π ( a 1 | g ) is influenced merely by context g without prior actions or states, which will be used in interaction retrieval. π builds upon existing text-to-motion models, where we evaluate MDM[118], MotionDiffuse [172], ReMoDiffuse [173], and MotionGPT [46].

Interaction Retrieval. The interaction retrieval component R establishes the initial state s 1 ∼ R ( s 1 | a 1 , g ) , based on the initial action a 1 generated by the text-to-motion model. We propose a user-friendly pipeline for this purpose built on handcrafted rules. First, we create databases by extracting HOI frames from the training sets of each target datasets - BEHA VE [7], OMOMO [66], and CHAIRS [47]. The indexing key for retrieval is a tuple consisting of the body part in contact and the category of the involved object. Each retrieval value is a per-frame contact map, represented by a list of K vertex pairs { ( d i h , d i o ) } K i =1 . Here, d i h refers to the contact vertex on the human surface, while d i o refers to the corresponding contact vertex on the object surface. This contact map is linked to its corresponding key, creating a searchable record of interactions. During the inference stage, using the body part and object information provided by the high-level planning (Sec. 3.1), we retrieve all relevant contact maps from the database. We then sample one map { ( d i h , d i o ) } K i =1 and use it to establish the object state s 1 ∼ R ( s 1 | a 1 , g ) , thus initializing the interaction. Further details including how we ensure consistency between the sampled state and human action are provided in Sec. B.1 of the Appendix. We also discuss an alternative learning-based approach in Sec. B.1.

3.3 World Model

Our world model combines a dynamics model and the optimization process, dedicated to simulating state transitions affected by applied actions. While drawing inspiration from similar concepts utilized in robotics [103, 135] and autonomous driving systems [55], we use it here to generate HOI trajectories. This model, trained on the training set of a 3D HOI dataset such as BEHA VE [7], serves a similar role as a simulator but is much simpler - it takes the preceding object state s t along with a pair of consecutive actions a t and a t +1 , and predicts the subsequent object state s t +1 . The interplay between the low-level control and the world model ultimately produces a coherent interaction rollout.

In designing the dynamics model, a naïve method would be directly taking raw actions, states, and object geometry as input. However, this suffers from a severe generalization problem during inference: the dynamics model is likely to encounter human actions and object geometry that do not exist in the training set, since our text-to-motion model is not trained with object interaction data. To overcome this limitation, we instead focus on encoding interactions through the contact vertices on the object, which capture both the action and object geometry, as shown in Figure 2. This locality ensures that the dynamics model remains focused on interactions in the contact region, without being distracted by the motion of body parts and geometry details that are irrelevant to the interaction.

̸

Input Representation. Specifically, at each timestep t , we abstract the past actions as H historical vertex trajectories {{ v j i } N j =1 } H i =1 , and the future actions as F = m future vertex trajectories {{ v j i } N j =1 } H + F i = H +1 , where non-fixed variable N is the number of sampled contact vertices, and m is the length of segments as mentioned in the overview of Sec 3. Note that we train our dynamics model to forecast over a longer duration than the past motion ( F > H ), only the foremost future action will be used for autoregressive generation during the inference, as suggested in [19]. To determine these N vertices, we start with object's signed distance fields { sdf i } H i =1 over the past H frames, derived from the past state s t . We then sample vertices that meet the following criteria: | sdf i ( v j i ) | ≤ δ 1 , sdf i ( v k i ) | ≤ δ 1 , ∀ i = 1 , . . . , H, ∀ j , and ∥ v j i -v k i ∥ ≥ δ 2 , ∀ j = k , where δ 1 and δ 2 are two hyperparameters. The objective is to sparsely sample contact vertices while ensuring that they are sufficient to encompass the interaction. We characterize each vertex trajectory { v j i } H + F i =1

with a feature f j to provide (i) human vertex coordinates at T-pose, providing information about the position of the human vertex on the body surface; (ii) the vertex-to-object surface vector, indicating vertex's impact on the object as well as inherently including information related to the object's shape; and (iii) the vertex's velocity relative to its nearest object vertex. Thus, the model needs to learn how the features of human action f j affect the evolution of the state of the object.

Architecture. As demonstrated in Figure 2, the network comprises two components: G that operates without contact vertex conditions, applicable in scenarios where no contact occurs, and F , which incorporates contact vertex conditions into the object trajectory when contact is present. The kth layer of G can be initiated as G k ( x k , Θ ) , mapping the input feature map x k at the k -th layer to another feature map, with Θ denoting the MLP's parameters. To incorporate human vertex controls, we introduce a second network F k ( y j k , Θ v ) operating on N vertex features { y j k } N j =1 , where Θ v is its parameters. With a cross-attention layer Attn , a dynamics block is formulated as: x k +1 , { y j k +1 } N j =1 = Attn( G k ( x k , Θ ) , {F k ( y j k , Θ v ) } N j =1 ) . We stack multiple dynamics blocks to form the model. The initial input, x 0 , corresponds to the previous state s t , while each y j 0 represents the feature of the vertex trajectory, containing both the trajectory { v j i } H + F i =1 and its associated feature vector f j . The output of this model is preliminary and subject to further optimization as introduced in Sec. 3.4, which will yield the final future state. We utilize the Mean Squared Error loss to train the dynamics model. For more explanations, please refer to Sec. B.2 of the Appendix.

3.4 Optimization

Optimization plays a role in introducing prior knowledge and avoiding the accumulation of errors. During inference, we input the action a t +1 and state s t +1 and refine them. This refinement is achieved through gradient descent on the human and object pose parameters. Our optimization includes several loss terms: a fitting loss to align optimized results with their preliminary one, a velocity loss for temporal smoothness, a contact loss to promote occurring contact, and a collision loss to reduce penetration. We provide detailed formulations in Sec. B.3 of the Appendix.

4 Experiments

Extensive comparisons evaluate the performance of our InterDreamer across two motion-relevant tasks. Details of the evaluation settings are provided in Sec. 4.1. We present both quantitative (Sec. 4.2) and qualitative (Sec. 4.3) results for our approach. Additionally, we perform ablation studies to verify the efficacy of each component within our framework. These studies also cover the

Methods

Planning

(

Ours

)

R-Precision

↑

FID

↓

MMDist

↓

Multimodality

↑

Diversity

→

Top 1

Top 2

Top 3

Ground Truth

-

0.044

±

0

.

004

0.095

±

0

.

008

0.151

±

0

.

009

0.000

±

0

.

000

6.858

±

0

.

006

-

5.660

±

0

.

110

MDM[118]

×

0.056

±

0

.

005

0.096

±

0

.

007

0.135

±

0

.

006

16.638

±

0

.

631

7.110

±

0

.

063

2.446

±

0

.

456

5.862

±

0

.

520

✓

0.062

±

0

.

006

0.109

±

0

.

004

0.155

±

0

.

008

15.735

±

0

.

285

6.889

±

0

.

082

2.663

±

0

.

520

6.461

±

0

.

841

MotionDiffuse [172]

×

0.048

±

0

.

006

0.094

±

0

.

008

0.143

±

0

.

013

15.442

±

0

.

231

5.799

±

0

.

054

1.658

±

0

.

209

5.981

±

0

.

516

✓

0.075

±

0

.

005

0.141

±

0

.

015

0.189

±

0

.

009

10.815

±

0

.

093

5.916

±

0

.

094

1.677

±

0

.

264

5.718

±

0

.

522

ReMoDiffuse [173]

×

0.062

±

0

.

003

0.111

±

0

.

005

0.160

±

0

.

012

15.479

±

0

.

209

5.690

±

0

.

049

1.179

±

0

.

145

6.032

±

0

.

540

✓

0.067

±

0

.

004

0.127

±

0

.

006

0.174

±

0

.

006

14.560

±

0

.

080

5.678

±

0

.

033

1.193

±

0

.

202

5.368

±

0

.

417

MotionGPT [46]

×

0.061

±

0

.

005

0.114

±

0

.

006

0.152

±

0

.

006

18.472

±

0

.

528

6.358

±

0

.

076

4.553

±

0

.

244

6.726

±

0

.

156

✓

0.064

±

0

.

007

0.121

±

0

.

007

0.164

±

0

.

009

17.512

±

0

.

498

6.287

±

0

.

041

4.470

±

0

.

191

7.048

±

0

.

169

A person holds a plastic container with their

right hand and moves it in different directions.

A person places a large box on the right side, then moves it to the left side.

A person moves a long box in their left hand

downward, to the right and then to the left.

A person transfers a plastic container from their left hand

to their right hand.

A person stands up from a chair.

A person lifts a chair with their left hand.

interaction prediction task [148] to evaluate our dynamics model. Additional details and results are presented in Sec. C and Sec. D of the Appendix. Please refer to our website for video results.

4.1 Experimental Setup

Datasets. Weuse BEHAVE [7], CHAIRS [47], and OMOMO [66] datasets for quantitative evaluation. The BEHAVE dataset includes recordings of 8 individuals interacting with 20 everyday objects, and our analysis focuses on 18 objects for which interaction sequences are available at 30 Hz. The human pose is modeled using SMPL-H [102], with hand poses set to an average pose due to the absence of detailed hand pose in the dataset . We manually segment the long interaction sequences in the test set, and annotate them with descriptions as well as their starting and ending indices, leading to 532

A person is seated on the small box with legs crossed.

A person turns to their right and steps on and over a wooden chair and steps down.

Someone can be seen on chair 96 and adjusts their sitting position.

A seated person on chair 63 adjusts the sitting position repeatedly.

1.yogamat;

2.shoulder;

1.yogaball;2.righthand;

Language Planning

3.apersoniscarrying anobjectontheshoulder.

Language Planning

3.aperson throws anobjectwithrighthand towards theground.

Text-to-Motion

InteractionRetrieval

Text-to-Motion

InteractionRetrieval

subsequences for evaluation. For qualitative evaluation, we go beyond using annotations specifically created and employ free - form text to demonstrate our model's capability on out-of-distribution inputs. To assess our model's performance with novel objects, we expand our retrieval database to include objects from the OMOMO [66] and CHAIRS [47] datasets, while we do not fine-tune the dynamics model on them-a direct qualitative evaluation without additional adaptation.

Metrics. The evaluation metrics are divided into three categories: ( i ) Human motion quality : The Fréchet Inception Distance ( FID ) measures the distance between the generated motion and ground truth. The MultiModality ( Multimodality ) and Diversity metrics assess the variance in generated human motion. R-Precision evaluates the consistency between the text and the generated human motion within the latent space. MultiModal distance ( MMDist ) is the distance between the motion feature and the text feature. We follow [34] to generate motion and text features. ( ii ) Interaction quality : We propose CMD to measure the distance between contact maps of real interactions and those generated. The per-sequence contact map is defined by the percentage of time that each body part is actively in contact. The detailed formulation is provided in Sec. C of the Appendix. We also measure the collision ( Pene. [148]), which calculates the average percentage of object vertices that have non-negative values in the human signed distance fields [90]. ( iii ) Object motion accuracy : The dynamics model's performance in the interaction prediction task [148] is evaluated by the accuracy of predicted object motion, including Trans. Err. , the average distance between predicted and ground truth, and Rot. Err. , the average distance between the predicted and ground truth. A person lifts an object with force with left hand Applying great forces, the man lifts the chair with one hand.

Someone can be seen sitting on a yogaball.

A person is seated on an object.

Someone can be seen sitting on a yogaball.

w/oplanning

w/ planning

Positioned in front of the bag,

Someone can be seen sitting on a yogaball.

A person is holding an object with both hands. the individual is holding onto it. (a) Low-level control w/o planning v.s. w/ planning

(b) Text feature w/ planning v.s. w/o planning

(a) human motion as action

(b) contact vertices as action

Baselines. Most recent research on text-to-HOI synthesis follows a supervised learning approach [28, 91], making direct quantitative comparisons unfair. Therefore, we primarily focus on qualitative comparisons with these methods. To enable quantitative evaluation, we develop a range of baselines to assess both the overall performance of our framework and the effectiveness of its individual components. In the context of high-level planning, we utilize GPT-4 [88] and Llama-2 [120], illustrating the effectiveness of our prompts across different language models. For low-level motion generation control, our baselines include MDM [118], MotionDiffuse [172], ReMoDiffuse [173], and MotionGPT [46], which span a range of text-to-motion approaches trained on HumanML3D [34] and show the generalizability of our framework. To evaluate the dynamics model, we include different design choices: ( i ) unconditional dynamics model which operates object dynamics independently of human motion; ( ii ) using human marker features as actions to the dynamics model, similar to [148]; ( iii ) using unprocessed human motion and object pointcloud motion as input to the dynamics model; ( iv ) our proposed vertex-based actions where only the contacting vertices are used for control.

4.2 Quantitative Results

In Table 1, comparing our framework to baselines with unconditional dynamics model, InterDreamer achieves better interaction quality in terms of CMD and penetration scores, showing the importance of human influence on object motion. Against methods that utilize direct raw human motion or markers for action features, our method demonstrates enhanced performance by offering more fine-grained guidance and extracting generalizable features for dynamics modeling. Tables 2 and 3 present a comparative analysis of our approach of combining high-level planning with low-level control, where we adopt various text-to-motion models against their counterparts without high-level planning on the BEHAVE and OMOMO datasets. Our approach consistently outperforms baselines. Specifically, InterDreamer exhibits superior motion quality, reflected by a lower FID, higher R-Precision, and better diversity, highlighting the benefits of incorporating our planning to reduce the distribution gap for the motion generator to generalize in the HOI synthesis task.

4.3 Qualitative Results

Figure 3 displays several results guided by the free-form text. Our method exhibits proficiency in interpreting the textual input and synthesizing dynamic, realistic interactions, despite the absence of training with text-interaction paired data. More importantly, as illustrated in Figure 4, we selectively use more complex sequences of interactive descriptions that are beyond the scope of the existing HOI dataset . Figure 5 further exemplifies our method that is able to generalize effectively to the CHAIRS dataset, despite our dynamics model not being trained on it. Figure 6 depicts the retrieval procedure, resulting in a diverse set of interactions that are both high-quality and semantically aligned. More experimental results and the user study are presented in Sec. D of the Appendix.

4.4 Ablation Study

Adaptability of High-Level Planning. Is our framework adaptable across different large language models (LLMs)? As illustrated in Table 4, our analysis contains two types of language models: GPT-4 [88], which is accessible through APIs and operates as a black box model; and Llama-2 [120], an open-source model. We demonstrate that language models with large parameters exhibit very high accuracy in responding to questions tailored to our prompts, validating the framework's adaptability.

Effectiveness of High-Level Planning with Low-Level Control. In consistency with Table 2, Figure 7 offers a qualitative comparison of text-to-motion results, contrasting results with and without LLM-revised text descriptions. The comparison shows that motion generated without LLM-enhanced descriptions often fails to correspond to the intended text, if the text is too challenging, e.g., not in the distribution of HumanML3D [34], which is used to train text-to-motion models. This underscores the LLM's critical role in bridging the distribution gap. In Figure 7(b), we visualize the CLIP [98] features of descriptions from HumanML3D, our raw annotations, and those processed by high-level planning. Quantitative evidence is provided in Table 5. Text processed through high-level planning demonstrates greater similarity to the HumanML3D dataset. Additionally, we test on more challenging out-ofdistribution text, selecting examples with an average cosine similarity to HumanML3D text of less than 0 . 85 . High-level planning successfully rephrases these texts, significantly increasing their similarity. For example, in Figure 7(a), the text 'Someone can be seen sitting on a yoga ball' has a cosine similarity of 0 . 874 to the closest in-distribution text, while the rephrased text after planning, 'A person is seated on an object,' achieves a similarity of 0 . 958 .

Effectiveness of World Model. In the quantitative evaluation, we show that the performance of our framework is enhanced by the tailored design of our world model. Table 1 provides additional evidence of the effectiveness by integrating the proposed world model, as interaction correction within the InterDiff framework [148] in the interaction prediction task. This implementation demonstrates enhanced conditionality in the object dynamics modeling across two tasks, attributed to the vertexlevel condition as actions. Doing so effectively removes the whole-body complexity, most of which tends to be irrelevant to the interaction. Figure 8 further indicates that our vertex-based condition can establish consistent interactions over time, while the condition by raw motion is not robust.

5 Conclusion

We tackle the task of text-guided 3D human-object interaction generation, aiming to accomplish this without relying on paired text-interaction data. To this end, we present InterDreamer that decouples interaction dynamics from semantics, formulating the task as retrieval-augmented generation and Markov decision process, where high-level planning and low-level control are introduced to generate semantically aligned human motion and initial object pose, while a world model is responsible for the object dynamics guided by the interaction. Our approach demonstrates effectiveness in this novel task, suggesting its potential for various real-world applications.

Limitations. The current utilization of dynamics modeling could be enhanced. A prospective improvement involves incorporating model-based learning techniques, which empower the agent to more effectively interact and learn a broader range of skills. The results may not be physically plausible and lead to artifacts in some cases, for example, foot skating. Hand poses are rough because they are missing from the dataset, but could be improved by integrating a physics simulator.

Acknowledgments

We thank Qiusi Zhan for supporting the implementation and evaluation of high-level planning, Jiangwei Yu and Xiyan Xu for their efforts in dataset annotation, and Xiang Li for the discussions. This work was supported in part by NSF Grant 2106825, NIFA Award 2020-67021-32799, the IBM-Illinois Discovery Accelerator Institute, the Toyota Research Institute, and the Jump ARCHES endowment through the Health Care Engineering Systems Center at Illinois and the OSF Foundation. This work used computational resources on NCSA Delta and PTI Jetstream2 through allocations CIS220014, CIS230012, CIS230013, and CIS240311 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, and on TACC Frontera through the National Artificial Intelligence Research Resource (NAIRR) Pilot.

References

Ahuja, C., Morency, L.P.: Language2pose: Natural language grounded pose forecasting. In: 3DV (2019) 3

Athanasiou, N., Petrovich, M., Black, M.J., Varol, G.: Teach: Temporal action composition for 3d humans. In: 3DV (2022) 3

Athanasiou, N., Petrovich, M., Black, M.J., Varol, G.: SINC: Spatial composition of 3d human motions for simultaneous action generation. In: ICCV (2023) 3

Bae, J., Won, J., Lim, D., Min, C.H., Kim, Y.M.: Pmp: Learning to physically interact with environments using part-wise motion priors. In: SIGGRAPH (2023) 3

Barquero, G., Escalera, S., Palmero, C.: BeLFusion: Latent diffusion for behavior-driven human motion prediction. In: ICCV (2023) 3

Barquero, G., Escalera, S., Palmero, C.: Seamless human motion composition with blended positional encodings. In: CVPR (2024) 3

Bhatnagar, B.L., Xie, X., Petrov, I., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: BEHA VE: Dataset and method for tracking human object interactions. In: CVPR (2022) 1, 2, 3, 5, 6, 7, 8, 22, 25

Braun, J., Christen, S., Kocabas, M., Aksan, E., Hilliges, O.: Physically plausible full-body hand-object interaction synthesis. arXiv preprint arXiv:2309.07907 (2023) 3

Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR (2023) 3

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. In: NeurIPS (2020) 2, 4

Cao, J., Liu, J., Kitani, K., Zhou, Y .: Multi-modal diffusion for hand-object grasp generation. arXiv preprint arXiv:2409.04560 (2024) 3

Cao, Z., Gao, H., Mangalam, K., Cai, Q.Z., Vo, M., Malik, J.: Long-term human motion prediction with scene context. In: ECCV (2020) 3

Casas, D., Comino-Trinidad, M.: SMPLitex: A generative model and dataset for 3d human texture estimation from single image. In: BMVC (2023) 7

Cen, Z., Pi, H., Peng, S., Shen, Z., Yang, M., Zhu, S., Bao, H., Zhou, X.: Generating human motion in 3d scenes from text descriptions. In: CVPR (2024) 3

Cha, J., Kim, J., Yoon, J.S., Baek, S.: Text2HOI: Text-guided 3d motion generation for hand-object interaction. In: CVPR (2024) 3

Chao, Y.W., Yang, J., Chen, W., Deng, J.: Learning to sit: Synthesizing human-chair interactions via hierarchical control. In: AAAI (2021) 3

Chen, L.H., Zhang, J., Li, Y., Pang, Y., Xia, X., Liu, T.: HumanMAC: Masked motion completion for human motion prediction. In: ICCV (2023) 3

Chen, X., Jiang, B., Liu, W., Huang, Z., Fu, B., Chen, T., Yu, G.: Executing your commands via motion diffusion in latent space. In: CVPR (2023) 3

Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., Song, S.: Diffusion policy: Visuomotor policy learning via action diffusion. In: RSS (2023) 5

Christen, S., Hampali, S., Sener, F., Remelli, E., Hodan, T., Sauser, E., Ma, S., Tekin, B.: Diffh2o: Diffusion-based synthesis of hand-object interactions from textual descriptions. arXiv preprint arXiv:2403.17827 (2024) 3

Cong, P., Dou, Z.W., Ren, Y., Yin, W., Cheng, K., Sun, Y., Long, X., Zhu, X., Ma, Y.: LaserHuman: Language-guided scene-aware human motion generation in free environment. arXiv preprint arXiv:2403.13307 (2024) 3

Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F.: Context-aware human motion prediction. In: CVPR (2020) 3

Cui, J., Liu, T., Liu, N., Yang, Y ., Zhu, Y ., Huang, S.: AnySkill: Learning open-vocabulary physical skill for interactive agents. In: CVPR (2024) 3

Dabral, R., Mughal, M.H., Golyanik, V., Theobalt, C.: MoFusion: A framework for denoisingdiffusion-based motion synthesis. In: CVPR. pp. 9760-9770 (2023) 3

Dai, S., Li, W., Sun, H., Huang, H., Ma, C., Huang, H., Xu, K., Hu, R.: InterFusion: Textdriven generation of 3d human-object interaction. In: ECCV (2024) 3

Dai, W., Chen, L.H., Wang, J., Liu, J., Dai, B., Tang, Y.: Motionlcm: Real-time controllable motion generation via latent consistency model. arXiv preprint arXiv:2404.19759 (2024) 3

Daiya, D., Conover, D., Bera, A.: COLLAGE: Collaborative human-agent interaction generation using hierarchical latent diffusion and language models. arXiv preprint arXiv:2409.20502 (2024) 3

Diller, C., Dai, A.: CG-HOI: Contact-guided 3d human-object interaction generation. In: CVPR (2024) 1, 2, 3, 9, 21

Diller, C., Funkhouser, T., Dai, A.: FutureHuman3D: Forecasting complex long-term 3d human behavior from video observations. In: CVPR (2024) 3

Fan, Z., Taheri, O., Tzionas, D., Kocabas, M., Kaufmann, M., Black, M.J., Hilliges, O.: ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In: CVPR (2023) 1, 3

Feng, H., Ma, W., Gao, Q., Zheng, X., Xue, N., Xu, H.: Stratified avatar generation from sparse observations. In: CVPR (2024) 3

Ghosh, A., Dabral, R., Golyanik, V., Theobalt, C., Slusallek, P.: IMoS: Intent-driven full-body motion synthesis for human-object interactions. arXiv preprint arXiv:2212.07555 (2022) 3

Ghosh, A., Dabral, R., Golyanik, V., Theobalt, C., Slusallek, P.: ReMoS: Reactive 3d motion synthesis for two-person interactions. arXiv preprint arXiv:2311.17057 (2023) 3

Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating diverse and natural 3d human motions from text. In: CVPR (2022) 1, 3, 4, 8, 9, 10, 25

Guo, C., Zuo, X., Wang, S., Cheng, L.: Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In: ECCV (2022) 3

Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., Cheng, L.: Action2motion: Conditioned generation of 3d human motions. In: ACMMM (2020) 3

Han, S., Joo, H.: CHORUS: Learning canonicalized 3d human-object spatial relations from unbounded synthesized images. In: ICCV (2023) 22

Hassan, M., Ceylan, D., Villegas, R., Saito, J., Yang, J., Zhou, Y., Black, M.J.: Stochastic scene-aware motion prediction. In: ICCV (2021) 3

Hassan, M., Ghosh, P., Tesch, J., Tzionas, D., Black, M.J.: Populating 3d scenes by learning human-scene interaction. In: CVPR (2021) 3

Hassan, M., Guo, Y., Wang, T., Black, M., Fidler, S., Peng, X.B.: Synthesizing physical character-scene interactions. In: SIGGRAPH (2023) 3

Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020) 1

Hou, Z., Yu, B., Tao, D.: Compositional 3d human-object neural animation. arXiv preprint arXiv:2304.14070 (2023) 3

Huang, B., Li, C., Xu, C., Pan, L., Wang, Y., Lee, G.H.: Closely interactive human reconstruction with proxemics and physics-guided adaption. In: CVPR (2024) 3

Huang, S., Wang, Z., Li, P., Jia, B., Liu, T., Zhu, Y ., Liang, W., Zhu, S.C.: Diffusion-based generation, optimization, and planning in 3d scenes. In: CVPR (2023) 1, 3

Huang, Y., Taheri, O., Black, M.J., Tzionas, D.: InterCap: Joint markerless 3D tracking of humans and objects in interaction. In: GCPR (2022) 1, 3

Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., Chen, T.: MotionGPT: Human motion as a foreign language. In: NeurIPS (2023) 3, 5, 6, 7, 8, 9, 26

Jiang, N., Liu, T., Cao, Z., Cui, J., Chen, Y ., Wang, H., Zhu, Y ., Huang, S.: CHAIRS: Towards full-body articulated human-object interaction. In: ICCV (2023) 1, 2, 3, 5, 7, 8, 22

Jiang, N., Zhang, Z., Li, H., Ma, X., Wang, Z., Chen, Y., Liu, T., Zhu, Y., Huang, S.: Scaling up dynamic human-scene interaction modeling. In: CVPR (2024) 3

Jin, P., Li, H., Cheng, Z., Li, K., Yu, R., Liu, C., Ji, X., Yuan, L., Chen, J.: Local action-guided motion diffusion model for text-to-motion generation. arXiv preprint arXiv:2407.10528 (2024) 3

Karunratanakul, K., Preechakul, K., Suwajanakorn, S., Tang, S.: GMD: Controllable human motion synthesis via guided diffusion models. In: ICCV (2023) 3

Kaufmann, M., Aksan, E., Song, J., Pece, F., Ziegler, R., Hilliges, O.: Convolutional autoencoders for human motion infilling. In: 3DV (2020) 3

Kim, H., Han, S., Kwon, P., Joo, H.: Zero-shot learning for the primitives of 3d affordance in general objects. arXiv preprint arXiv:2401.12978 (2024) 3

Kim, J., Kim, J., Na, J., Joo, H.: ParaHome: Parameterizing everyday home activities towards 3d generative modeling of human-object interactions. arXiv preprint arXiv:2401.10232 (2024) 1, 3

Kim, J., Kim, J., Choi, S.: Flame: Free-form language-based motion synthesis & editing. In: AAAI (2023) 3

Kim, S.W., Zhou, Y., Philion, J., Torralba, A., Fidler, S.: Learning to simulate dynamic environments with gamegan. In: CVPR (2020) 5

Kim, T., Saito, S., Joo, H.: NCHO: Unsupervised learning for neural 3d composition of humans and objects. In: ICCV (2023) 3

Kong, H., Gong, K., Lian, D., Mi, M.B., Wang, X.: Priority-centric human motion generation in discrete latent space. In: ICCV (2023) 3

Krebs, F., Meixner, A., Patzer, I., Asfour, T.: The kit bimanual manipulation dataset. In: Humanoids (2021) 3

Kulkarni, N., Rempe, D., Genova, K., Kundu, A., Johnson, J., Fouhey, D., Guibas, L.: NIFTY: Neural object interaction fields for guided human motion synthesis. arXiv preprint arXiv:2307.07511 (2023)

Lee, J., Joo, H.: Locomotion-Action-Manipulation: Synthesizing human-scene interactions in complex 3d environments. In: ICCV (2023) 3

Lee, T., Moon, G., Lee, K.M.: Multiact: Long-term 3d human motion generation from multiple action labels. In: AAAI (2023) 3

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. In: NeurIPS (2020) 2, 3, 4

Li, B., Ho, E.S., Shum, H.P., Wang, H.: Two-person interaction augmentation with skeleton priors. In: CVPR (2024) 3

Li, C., Chibane, J., He, Y., Pearl, N., Geiger, A., Pons-Moll, G.: Unimotion: Unifying 3d human motion synthesis and understanding. arXiv preprint arXiv:2409.15904 (2024) 3

Li, J., Clegg, A., Mottaghi, R., Wu, J., Puig, X., Liu, C.K.: Controllable human-object interaction synthesis. arXiv preprint arXiv:2312.03913 (2023) 1, 2, 3

Li, J., Wu, J., Liu, C.K.: Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) 42 (6), 1-11 (2023) 1, 2, 3, 5, 7, 8, 22, 25, 26

Li, L., Dai, A.: GenZI: Zero-shot 3d human-scene interaction generation. In: CVPR (2024) 3

Li, Q., Wang, J., Loy, C.C., Dai, B.: Task-oriented human-object interactions generation with implicit neural representations. arXiv preprint arXiv:2303.13129 (2023) 3

Liang, H., Zhang, W., Li, W., Yu, J., Xu, L.: InterGen: Diffusion-based multi-human motion generation under complex interactions. arXiv preprint arXiv:2304.05684 (2023) 1

Lin, J., Zeng, A., Lu, S., Cai, Y ., Zhang, R., Wang, H., Zhang, L.: Motion-X: A large-scale 3d expressive whole-body human motion dataset. In: NeurIPS (2023) 3

Liu, H., Zhan, X., Huang, S., Mu, T.J., Shan, Y.: Programmable motion generation for open-set motion control tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1399-1408 (2024) 3

Liu, L., Hodgins, J.: Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Transactions on Graphics (TOG) 37 (4), 1-14 (2018) 3

Liu, S., Zhou, Y., Yang, J., Gupta, S., Wang, S.: Contactgen: Generative contact modeling for grasp generation. In: ICCV (2023) 3

Liu, Y., Chen, C., Ding, C., Yi, L.: PhysReaction: Physically plausible real-time humanoid reaction synthesis via forward dynamics guided 4d imitation. arXiv preprint arXiv:2404.01081 (2024) 3

Liu, Y., Chen, C., Yi, L.: Interactive humanoid: Online full-body motion reaction synthesis with social affordance canonicalization and forecasting. arXiv preprint arXiv:2312.08983 (2023) 3

Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: A skinned multiperson linear model. ACM transactions on graphics (2015) 3, 4, 25, 26

Lu, S., Chen, L.H., Zeng, A., Lin, J., Zhang, R., Zhang, L., Shum, H.Y.: HumanTOMATO: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978 (2023) 3

Luo, Z., Wang, J., Liu, K., Zhang, H., Tessler, C., Wang, J., Yuan, Y., Cao, J., Lin, Z., Wang, F., et al.: SMPLOlympics: Sports environments for physically simulated humanoids. arXiv preprint arXiv:2407.00187 (2024) 3

Ma, J., Chen, X., Bao, W., Xu, J., Wang, H.: MADiff: Motion-aware mamba diffusion models for hand trajectory prediction on egocentric videos. arXiv preprint arXiv:2409.02638 (2024) 3

Ma, J., Xu, J., Chen, X., Wang, H.: Diff-IP2D: Diffusion-based hand-object interaction prediction on egocentric videos. arXiv preprint arXiv:2405.04370 (2024) 3

Ma, S., Cao, Q., Zhang, J., Tao, D.: Contact-aware human motion generation from textual descriptions. arXiv preprint arXiv:2403.15709 (2024) 3

Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9 (11) (2008) 8

Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.: AMASS: Archive of motion capture as surface shapes. In: ICCV (2019) 1, 3

Mandery, C., Terlemez, O., Do, M., Vahrenkamp, N., Asfour, T.: The kit whole-body human motion database. In: ICAR (2015) 3

Mandery, C., Terlemez, O., Do, M., Vahrenkamp, N., Asfour, T.: Unifying representations and large-scale whole-body motion databases for studying human motion. IEEE Transactions on Robotics 32 (4), 796-809 (2016) 3

Mao, W., Liu, M., Salzmann, M.: Generating smooth pose sequences for diverse human motion prediction. In: CVPR (2021) 3

Merel, J., Tunyasuvunakool, S., Ahuja, A., Tassa, Y., Hasenclever, L., Pham, V., Erez, T., Wayne, G., Heess, N.: Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG) 39 (4), 39-1 (2020) 3

OpenAI: ChatGPT. https://chat.openai.com/ (2023) 2, 3, 4, 7, 8, 9, 10, 23, 26

Pan, L., Wang, J., Huang, B., Zhang, J., Wang, H., Tang, X., Wang, Y.: Synthesizing physically plausible human motions in 3d scenes. arXiv preprint arXiv:2308.09036 (2023) 3

Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: Learning continuous signed distance functions for shape representation. In: CVPR (2019) 8

Peng, X., Xie, Y., Wu, Z., Jampani, V., Sun, D., Jiang, H.: HOI-Diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553 (2023) 1, 2, 3, 9, 21

Petrov, I.A., Marin, R., Chibane, J., Pons-Moll, G.: Object pop-up: Can we infer 3d objects and their poses from human interactions alone? In: CVPR (2023) 3

Petrovich, M., Black, M.J., Varol, G.: Action-conditioned 3d human motion synthesis with transformer vae. In: ICCV (2021) 3

Petrovich, M., Black, M.J., Varol, G.: TEMOS: Generating diverse human motions from textual descriptions. In: ECCV (2022) 3

Petrovich, M., Black, M.J., Varol, G.: TMR: Text-to-motion retrieval using contrastive 3d human motion synthesis. In: ICCV (2023) 3

Raab, S., Leibovitch, I., Li, P., Aberman, K., Sorkine-Hornung, O., Cohen-Or, D.: MoDi: Unconditional motion synthesis from diverse data. In: CVPR (2023) 3

Raab, S., Leibovitch, I., Tevet, G., Arar, M., Bermano, A.H., Cohen-Or, D.: Single motion diffusion. arXiv preprint arXiv:2302.05905 (2023) 3

Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021) 8, 10

Razali, H., Demiris, Y.: Action-conditioned generation of bimanual object manipulation sequences. In: AAAI (2023) 3

Rempe, D., Luo, Z., Bin Peng, X., Yuan, Y., Kitani, K., Kreis, K., Fidler, S., Litany, O.: Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In: CVPR (2023) 3

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022) 22

Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics 36 (6) (2017) 7

Seo, Y., Hafner, D., Liu, H., Liu, F., James, S., Lee, K., Abbeel, P.: Masked world models for visual control. In: CoRL (2023) 5

Shafir, Y ., Tevet, G., Kapon, R., Bermano, A.H.: Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418 (2023) 3

Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML (2015) 1

Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020) 1

Song, W., Zhang, X., Li, S., Gao, Y., Hao, A., Hou, X., Chen, C., Li, N., Qin, H.: HOIAnimator: Generating text-prompt human-object animations using novel perceptive diffusion models. In: CVPR (2024) 1, 3

Starke, S., Zhang, H., Komura, T., Saito, J.: Neural state machine for character-scene interactions. ACM Trans. Graph. 38 (6), 209-1 (2019)

Starke, S., Zhao, Y., Komura, T., Zaman, K.: Local motion phases for learning multi-contact character movements. ACM Transactions on Graphics (TOG) 39 (4), 54-1 (2020) 3

Sun, J., Chowdhary, G.: Towards globally consistent stochastic human motion prediction via motion diffusion. arXiv preprint arXiv:2305.12554 (2023) 3

Taheri, O., Choutas, V., Black, M.J., Tzionas, D.: GOAL: Generating 4d whole-body motion for hand-object grasping. In: CVPR (2022) 3

Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: A dataset of whole-body human grasping of objects. In: ECCV (2020) 3

Tang, J., Wang, J., Ji, K., Xu, L., Yu, J., Shi, Y .: A unified diffusion framework for scene-aware human motion estimation from sparse signals. In: CVPR (2024) 3

Tendulkar, P., Surís, D., V ondrick, C.: FLEX: Full-body grasping without full-body grasps. In: CVPR (2023) 3

Tessler, C., Guo, Y., Nabati, O., Chechik, G., Peng, X.B.: MaskedMimic: Unified physicsbased character control through masked motion inpainting. arXiv preprint arXiv:2409.14393 (2024) 3

Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.: Motionclip: Exposing human motion generation to clip space. In: ECCV (2022) 3

Tevet, G., Raab, S., Cohan, S., Reda, D., Luo, Z., Peng, X.B., Bermano, A.H., van de Panne, M.: CLoSD: Closing the loop between simulation and diffusion for multi-task character control. arXiv preprint arXiv:2410.03441 (2024) 3

Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. arXiv preprint arXiv:2209.14916 (2022) 5, 6, 7, 9

Tian, J., Yang, L., Ji, R., Ma, Y ., Xu, L., Yu, J., Shi, Y ., Wang, J.: Gaze-guided hand-object interaction synthesis: Benchmark and method. arXiv preprint arXiv:2403.16169 (2024) 3

Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023) 2, 3, 4, 9, 10

Turk, A.M.: Amazon mechanical turk. Retrieved August 17 , 2012 (2012) 23

Wan, W., Dou, Z., Komura, T., Wang, W., Jayaraman, D., Liu, L.: Tlcontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311.17135 (2023) 3

Wan, W., Yang, L., Liu, L., Zhang, Z., Jia, R., Choi, Y.K., Pan, J., Theobalt, C., Komura, T., Wang, W.: Learn to predict how humans manipulate large-sized objects from interactive motions. IEEE Robotics and Automation Letters (2022) 3

Wang, J., Hodgins, J., Won, J.: Strategy and skill learning for physics-based table tennis animation. In: SIGGRAPH (2024) 3

Wang, J., Xu, H., Xu, J., Liu, S., Wang, X.: Synthesizing long-term 3d human motion and interaction in 3d scenes. In: CVPR (2021) 3

Wang, J., Rong, Y., Liu, J., Yan, S., Lin, D., Dai, B.: Towards diverse and natural scene-aware 3d human motion synthesis. In: CVPR (2022)

Wang, J., Yan, S., Dai, B., Lin, D.: Scene-aware generative network for human motion synthesis. In: CVPR (2021) 3

Wang, X., Li, G., Kuo, Y.L., Kocabas, M., Aksan, E., Hilliges, O.: Reconstructing actionconditioned human-object interactions using commonsense knowledge priors. In: 3DV (2022) 3

Wang, Y., Lin, J., Zeng, A., Luo, Z., Zhang, J., Zhang, L.: PhysHOI: Physics-based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04393 (2023) 3

Wang, Z., Chen, Y., Liu, T., Zhu, Y., Liang, W., Huang, S.: HUMANISE: Languageconditioned human motion generation in 3d scenes. In: NeurIPS (2022) 1, 3

Wang, Z., Li, D., Jiang, R.: Diffusion models in 3d vision: A survey. arXiv preprint arXiv:2410.04738 (2024) 1

Wang, Z., Wang, J., Lin, D., Dai, B.: InterControl: Generate human motion interactions by controlling every joint. arXiv preprint arXiv:2311.15864 (2023) 3

Wei, D., Sun, X., Sun, H., Li, B., Hu, S., Li, W., Lu, J.: Understanding text-driven motion synthesis with keyframe collaboration via diffusion models. arXiv preprint arXiv:2305.13773 (2023) 3

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. In: NeurIPS (2022) 2

Wu, P., Escontrela, A., Hafner, D., Abbeel, P., Goldberg, K.: Daydreamer: World models for physical robot learning. In: CoRL (2023) 5

Wu, Q., Shi, Y., Huang, X., Yu, J., Xu, L., Wang, J.: THOR: Text to human-object interaction diffusion via relation intervention. arXiv preprint arXiv:2403.11208 (2024) 1, 3

Wu, Q., Dou, Z., Xu, S., Shimada, S., Wang, C., Yu, Z., Liu, Y., Lin, C., Cao, Z., Komura, T., et al.: DICE: End-to-end deformation capture of hand-face interactions from a single image. arXiv preprint arXiv:2406.17988 (2024) 3

Wu, S., Liu, Y., Li, L., Bi, M., Zeng, W., Yang, X.: HIMO: A new benchmark for full-body human interacting with multiple objects. In: ECCV (2024) 3

Wu, Y., Wang, J., Zhang, Y., Zhang, S., Hilliges, O., Yu, F., Tang, S.: SAGA: Stochastic whole-body grasping with contact. In: ECCV (2022) 3

Wu, Z., Li, J., Liu, C.K.: Human-object interaction from human-level instructions. arXiv preprint arXiv:2406.17840 (2024) 1, 3

Xiao, Z., Wang, T., Wang, J., Cao, J., Zhang, W., Dai, B., Lin, D., Pang, J.: Unified humanscene interaction via prompted chain-of-contacts. arXiv preprint arXiv:2309.07918 (2023) 3

Xiao, Z., Wang, T., Wang, J., Cao, J., Zhang, W., Dai, B., Lin, D., Pang, J.: Unified humanscene interaction via prompted chain-of-contacts. In: ICLR (2024) 3

Xie, X., Bhatnagar, B.L., Pons-Moll, G.: Chore: Contact, human and object reconstruction from a single rgb image. In: ECCV (2022) 3

Xie, X., Lenssen, J.E., Pons-Moll, G.: InterTrack: Tracking human object interaction without object templates. arXiv preprint arXiv:2408.13953 (2024) 3

Xie, Y., Jampani, V., Zhong, L., Sun, D., Jiang, H.: OmniControl: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.08580 (2023) 3

Xie, Z., Starke, S., Ling, H.Y., van de Panne, M.: Learning soccer juggling skills with layer-wise mixture-of-experts. In: SIGGRAPH (2022) 3

Xie, Z., Tseng, J., Starke, S., van de Panne, M., Liu, C.K.: Hierarchical planning and control for box loco-manipulation. arXiv preprint arXiv:2306.09532 (2023) 3

Xu, S., Li, Z., Wang, Y.X., Gui, L.Y.: InterDiff: Generating 3d human-object interactions with physics-informed diffusion. In: ICCV (2023) 3, 6, 7, 8, 9, 10, 25

Xu, S., Wang, Y.X., Gui, L.Y.: Diverse human motion prediction guided by multi-level spatial-temporal anchors. In: ECCV (2022) 3

Xu, S., Wang, Y.X., Gui, L.: Stochastic multi-person 3d motion forecasting. In: ICLR (2023) 3

Xu, X., Joo, H., Mori, G., Savva, M.: D3D-HOI: Dynamic 3d human-object interactions from videos. arXiv preprint arXiv:2108.08420 (2021) 3

Xu, Z., Chen, Q., Peng, Y., Liu, Y .: Semantic-aware human object interaction image generation. In: ICML (2024) 3

Xue, K., Seo, H.: Shape conditioned human motion generation with diffusion model. arXiv preprint arXiv:2405.06778 (2024) 3

Yang, C., Kang, C., Kong, K., Oh, H., Kang, S.J.: Person in Place: Generating associative skeleton-guidance maps for human-object interaction image editing. In: CVPR (2024) 3

Yang, J., Niu, X., Jiang, N., Zhang, R., Huang, S.: F-HOI: Toward fine-grained semanticaligned 3d human-object interactions. arXiv preprint arXiv:2407.12435 (2024) 2, 3

Yang, Y., Zhai, W., Luo, H., Cao, Y., Zha, Z.J.: LEMON: Learning 3d human-object interaction relation from 2d images. In: CVPR (2024) 3, 22

Yang, Y., Zhai, W., Wang, C., Yu, C., Cao, Y., Zha, Z.J.: EgoChoir: Capturing 3d human-object interaction regions from egocentric views. arXiv preprint arXiv:2405.13659 (2024) 3

Yang, Z., Yin, K., Liu, L.: Learning to use chopsticks in diverse gripping styles. ACM Transactions on Graphics (TOG) 41 (4), 1-17 (2022) 3

Yao, H., Song, Z., Zhou, Y., Ao, T., Chen, B., Liu, L.: MoConVQ: Unified physics-based motion control via scalable discrete representations. arXiv preprint arXiv:2310.10198 (2023) 3

Yazdian, P.J., Liu, E., Cheng, L., Lim, A.: MotionScript: Natural language descriptions for expressive 3d human motions. arXiv preprint arXiv:2312.12634 (2023) 3

Ye, Y., Li, X., Gupta, A., De Mello, S., Birchfield, S., Song, J., Tulsiani, S., Liu, S.: Affordance diffusion: Synthesizing hand-object interactions. In: CVPR (2023) 3

Yuan, W., Shen, W., He, Y., Dong, Y., Gu, X., Dong, Z., Bo, L., Huang, Q.: Mogents: Motion generation based on spatial-temporal joint modeling. In: ECCV (2024) 3

Yuan, Y., Kitani, K.: DLow: Diversifying latent flows for diverse human motion prediction. In: ECCV (2020) 3

Zhang, C., Liu, Y., Xing, R., Tang, B., Yi, L.: Core4d: A 4d human-object-human interaction dataset for collaborative object rearrangement. arXiv preprint arXiv:2406.19353 (2024) 3

Zhang, H., Christen, S., Fan, Z., Zheng, L., Hwangbo, J., Song, J., Hilliges, O.: ArtiGrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. arXiv preprint arXiv:2309.03891 (2023) 3

Zhang, J.Y., Pepose, S., Joo, H., Ramanan, D., Malik, J., Kanazawa, A.: Perceiving 3d human-object spatial arrangements from a single image in the wild. In: ECCV (2020) 3

Zhang, J., Zhang, Y., An, L., Li, M., Zhang, H., Hu, Z., Liu, Y.: ManiDext: Hand-object manipulation synthesis via continuous correspondence embeddings and residual-guided diffusion. arXiv preprint arXiv:2409.09300 (2024) 3

Zhang, J., Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., Shan, Y.: Generating human motion from textual descriptions with discrete representations. In: CVPR (2023) 3

Zhang, J., Luo, H., Yang, H., Xu, X., Wu, Q., Shi, Y., Yu, J., Xu, L., Wang, J.: NeuralDome: A neural modeling pipeline on multi-view human-object interactions. In: CVPR (2023) 1, 3

Zhang, J., Zhang, J., Song, Z., Shi, Z., Zhao, C., Shi, Y., Yu, J., Xu, L., Wang, J.: Hoi-mˆ 3: Capture multiple humans and objects interaction within contextual environment. In: CVPR (2024) 1, 3

Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023) 22

Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., Liu, Z.: MotionDiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001 (2022) 3, 5, 6, 7, 9

Zhang, M., Guo, X., Pan, L., Cai, Z., Hong, F., Li, H., Yang, L., Liu, Z.: ReMoDiffuse: Retrieval-augmented motion diffusion model. In: ICCV (2023) 5, 6, 7, 9

Zhang, M., Li, H., Cai, Z., Ren, J., Yang, L., Liu, Z.: Finemogen: Fine-grained spatio-temporal motion generation and editing. In: NeurIPS (2024) 3

Zhang, W., Dabral, R., Leimkühler, T., Golyanik, V., Habermann, M., Theobalt, C.: ROAM: Robust and object-aware motion generation using neural pose descriptors. arXiv preprint arXiv:2308.12969 (2023) 3

Zhang, X., Bhatnagar, B.L., Starke, S., Guzov, V., Pons-Moll, G.: COUCH: Towards controllable human-chair interactions. In: ECCV (2022) 3

Zhang, X., Bhatnagar, B.L., Starke, S., Petrov, I., Guzov, V., Dhamo, H., Pérez-Pellitero, E., Pons-Moll, G.: FORCE: Dataset and method for intuitive physics guided human-object interaction. arXiv preprint arXiv:2403.11237 (2024) 1

Zhang, Y., Huang, D., Liu, B., Tang, S., Lu, Y., Chen, L., Bai, L., Chu, Q., Yu, N., Ouyang, W.: Motiongpt: Finetuned llms are general-purpose motion generators. arXiv preprint arXiv:2306.10900 (2023) 3

Zhang, Z., Liu, R., Aberman, K., Hanocka, R.: TEDi: Temporally-entangled diffusion for long-term motion synthesis. arXiv preprint arXiv:2307.15042 (2023) 3

Zhao, C., Zhang, J., Du, J., Shan, Z., Wang, J., Yu, J., Wang, J., Xu, L.: I'M HOI: Inertia-aware monocular capture of 3d human-object interactions. In: CVPR (2024) 1, 3

Zhao, K., Li, G., Tang, S.: DART: A diffusion-based autoregressive motion model for real-time text-driven motion control. arXiv preprint arXiv:2410.05260 (2024) 3

Zhao, K., Wang, S., Zhang, Y., Beeler, T., Tang, S.: Compositional human-scene interaction synthesis with semantic control. In: ECCV (2022) 3

Zhao, K., Zhang, Y., Wang, S., Beeler, T., Tang, S.: Synthesizing diverse human motions in 3d indoor scenes. In: ICCV (2023) 3

Zheng, J., Zheng, Q., Fang, L., Liu, Y., Yi, L.: CAMS: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. In: CVPR (2023) 3

Zhong, L., Xie, Y., Jampani, V., Sun, D., Jiang, H.: SMooDi: Stylized motion diffusion model. arXiv preprint arXiv:2407.12783 (2024) 3

Zhou, K., Bhatnagar, B.L., Lenssen, J.E., Pons-Moll, G.: Toch: Spatio-temporal object-to-hand correspondence for motion refinement. In: ECCV (2022) 3

Zhou, W., Dou, Z., Cao, Z., Liao, Z., Wang, J., Wang, W., Liu, Y., Komura, T., Wang, W., Liu, L.: EMDM: Efficient motion diffusion model for fast, high-quality motion generation. arXiv preprint arXiv:2312.02256 (2023) 3

In this Appendix, we include additional method details and experimental results: ( i ) We provide demo videos in the website, explained in Sec. A. ( ii ) We present additional details of interaction retrieval, world model, and optimization in Sec. B. ( iii ) We provide implementation details and additional information on the experimental setup in Sec. C. ( iv ) We provide additional qualitative experiments in Sec. D. ( v ) We provide some failure cases in Sec. E. ( vi ) We discuss the potential negative societal impact in Sec. F.

A Visualization Video

Beyond the qualitative results presented in the main paper, we include two demo videos that offer more detailed visualizations of the task, further illustrating the efficacy of our approach. These demos highlight ( i ) We conduct a qualitative comparison of our approach with existing text-to-HOI work [28, 91] within the framework of supervised learning. Note that as our setting contains no text supervision, it is unfair to compare our work with these approaches; we include the comparison here for additional reference. We evaluate our method by directly testing our trained model on the annotated data available from their websites, specifically retrieving their generated videos for direct comparison. Remarkably, even without training on these datasets, our method generates results that demonstrate high-quality interactions. It is even capable of synthesizing complex interactions involving dynamically-changing contact, such as the handover and throwing of objects.

B Additional Details of Methodology

B.1 Low-Level Control

In this section, we provide additional details on the retrieval based on handcraft rules, which is straightforward and does not require training. We also investigate a learning-based method without relying on handcrafted designs.

Handcraft Interaction Retrieval. In Sec. 3.2 of the main paper, we detail the construction of the interaction database and emphasize the use of body parts and object categories as keys to fetch semantically-aligned contact maps. Same as the main paper, we define a contact map as a list of K index pairs of vertices { ( d i h , d i o ) } K i =1 . This section delves into the methodology for outlining an optimization process to generate the object initial pose s 1 given contact maps and the initial human pose a 1 , and choose one pose based on a predefined metric.

Let v h 1 [ d o ] denote the vertex on the surface of the object, and v o 1 [ d h ] represent the corresponding vertex on the human body surface, where d o and d h are the indices of vertices. Specifically, to optimize s 1 , the overall optimization objective is given by,

where λ fit , λ cont , and λ pene are hyperparameters.

Fitting Loss. To project a contact map to an object pose, we minimize the L2 distance between the human vertices and the object vertices indicated by the contact map,

Contact Loss. We leverage a contact loss to encourage the body part to contact the object surface in addition to the fitting loss,

where T = { ˜ d h | min ˜ d o ∥ v o 1 [ ˜ d o ] -v h 1 [ ˜ d h ] ∥ 2 ≤ ϵ } includes the index of the body part that is close to the object vertex v o 1 [ ˜ d o ] , where ϵ is a hyperparameter, ˜ d h and ˜ d o are vertex indices for human and object, respectively, in addition to the contact map { ( d i h , d i o ) } K i =1 .

Penetration Loss. Given the signed-distance field of the human pose sdf h 1 , we employ a penetration loss to penalize the body-object interpenetration,

B.2 World Model

World Model for Initial States. In the particular instance where the timestep t = 1 , the state vector s 1 encapsulates a single frame. Consequently, we employ two distinct models for dynamics prediction. For predictions originating from the initial state, the history motion encompasses a single timestep H = 1 . In contrast, for predictions for subsequent states, the historical interval covering m timesteps, where m denotes the frame count per segment.

World Model for Implicit Geometry Encoding. The input to the world model includes the trajectories of the human vertices (represented by red small spheres in the top-right of Figure 2 of the main paper), along with the vertex-to-object surface vectors. By adding the vertex-to-object surface vectors to human vertices, one can easily obtain the object vertices (shown as blue small spheres in the top-right of Figure 2 of the main paper). Though the network of the world model does not receive this information directly, it can learn to combine these features to derive it when needed.

World Model for Novel Objects. The world model employs 'contact vertices' as an input, which includes features derived from the object distance field. These features encompass the human vertexto-object surface distance and the human vertex velocity relative to the nearest object vertex as introduced in Sec. 3.3 of the main paper, inherently including information related to the object's shape. This encoding is consistently applied to both training objects from the BEHAVE [7] dataset and novel objects from the CHAIRS [47] and OMOMO [66] datasets.

World Model for Non-Contact Objects. The network can process inputs without contact conditions by adopting an approach similar to ControlNet [171]. The network comprises two components: G that operates without contact vertex conditions, applicable in scenarios where no contact occurs, and F , akin to the control components in ControlNet, which incorporates contact vertex conditions into the object trajectory when contact is present. When there is no contact, only the unconditional network is utilized. The model is aware of past object motion and thus needs to learn how human interaction affects the object's state. This includes understanding how objects follow contact positions or normals by F , as well as how they move without contact by G . With the no-contact object motion data provided by BEHAVE [7], the world model (more specifically, G ) learns to infer whether the object should free-fall based on its previous velocity or remain on the ground based on its height.

B.3 Optimization

We provide detailed formulations of optimization objectives, complementing Sec. 3.4 in the main paper. For efficiency , we perform optimization sparsely only if the loss is above a threshold to improve the efficiency. Specifically, given the reference interaction sequence { h i } L i =1 and { o i } L i =1

The metric for determining the final pose selection is given by the expression 1 ( E pene = 0) /E cont . We sample a pose from the set generated by all contact maps, with higher metrics corresponding to higher selection probability.

Learning-Based Interaction Retrieval. Our interaction retrieval can also be achieved by integrating knowledge from several learning-based algorithms. Although being more complicated, the retrieval can be done without handcraft rules. Our framework can be divided into following. ( i ) Given the text prompt t and the initial human pose a 1 , we synthesize corresponding images via Stable Diffusion [101]. ( ii ) We follow [37] filter out images with low quality in interaction. ( iii ) An offthe-shelf model LEMON [156] is to employed to obtain object affordance and human contact, given the generated image paired with human pose a 1 and object template. The output { ( l i h ) K i =1 , ( l i o ) K i =1 } indicates the contact vertex indexes of human and object respectively, and the output T 1 indicates the estimated object translation, which is used for initialization in the optimization. ( iv ) To acquire the object pose, we utilize the optimization to minimize the Chamfer distance between the human vertices and the object vertices, indicated by the contact vertices obtained in the last step.

Instructions: 1. Segmentation and Identification of Semantic Units: Break down the interaction sequence into smaller subsequences, each consisting of tens to several hundreds of frames. Every

2.Annotationof Each Subsequence:Annotate each subsequence using thetemplate“Aperson[does some interaction],[with some object] [using some body parts].#start_frame#end_frame”

For example, if a sequence involves making a sandwich, one subsequence could be annotated as: “A person holds a large box up with their right hand, lowers their right arm, and pulls the box with left hand towards them.#59#255";“A person opens a book with both hands, flips through the pages using their fingers, and then points to a line of text.#59#255"

video_name=BEHAVE_Date03_Sub05_boxtiny_boxtiny.mp4

0744

Statetheactions

Submit

of arbitrary length L , derived from previous steps, we apply gradient descent to optimize human pose sequence { h ∗ i } L i =1 and object pose sequence { o ∗ i } L i =1 , using the loss function,

where λ fit , λ vel , λ cont , and λ pene are hyperparameters.

Fitting Loss. We minimize the L1 distance between the input and the reference,

Velocity Loss. We leverage a velocity loss to smooth the interaction sequence,

Contact loss. We leverage a contact loss to encourage the body part to contact the object surface, if they are close to each other in the initial interaction,

where v h ∗ i [ d h ] denotes the vertex on the human body surface, and v o ∗ i [ d o ] represents the corresponding vertex on the surface of the object. And T i = { d h | min d o ∥ v o i [ d o ] -v h i [ d h ] ∥ 2 ≤ ϵ } includes the index of reference human vertex v h i [ d h ] that is close to the reference object vertex v o i [ d o ] , where ϵ is a hyperparameter, d h and d o are vertex indices for human and object, respectively.

Penetration Loss. Given the signed-distance field of the human pose sdf h ∗ i , we employ a penetration loss to penalize the body-object interpenetration,

C Additional Details of Experimental Setup

Datasets. We include a screenshot of our annotation platform in Figure A. Our annotations are further diversified by GPT-4 [88]. The prompt used for this purpose is: I'm going to give

Youwill be given a sentence that describes the interaction between a person and an object.Youwill need to extract informationfrom the

descriptionand answer threequestionsbasedonthefollowingrules andexamples.

1.Extract the category of the object with which the person interacts.

Answers should be selected from the following objects:[trashbin, monitor,boxlarge, stool,boxsmall, backpack,boxlong, plasticcontainer,

tablesquare, yogaball, yogamat, toolbox, chairwood, chairblack, boxmedium, boxtiny, suitcase, tablesmall].

[start of rules]

· Pick the most similar object from the list if the description is not in the list.

[end of rules]

2. Infer the body part that contacts with the object at the beginning of the interaction.

Answers should be selected from the following body parts: [right hand, left hand, arm,back of thehip,upper leg,leg,upper back,lf foot, right foot,front body,back,

shoulder,no contact].

[start of rules]

·If the description does not specify which hand or foot, randomly choose from "left" or "right".

[end of rules]

3.Simplify and modify the description.

[start of rules]

·The subject of the sentence should be "a person".

·Modify words that indicate interaction with a simple, clear verb. For example, change "maintain possession" to "hold" or "grasp"; change "move

with force" to "pull" or "push.""

·Eliminate unnecessary verbs, and convert some obscure verbs into common verbs. For example, change "A person clenches a box with their left

hand and then passes it to their right hand." to "A person transfers a box from their left hand to their right hand.", remove the unnecessary verb "hold'

in this sentence. And "A person walks leisurely in a clockwise direction with an object slung over their right shoulder." to "A person walks in a

clockwise circle while holding an object on their right shoulder."

·If the sentence is ambiguous, such as "one hand," choose the most likely answer from "left hand," "right hand," or "both hands.""

·If contact is made with hands or feet, explicitly mention the body part in the description.

·Remember to replace the specific object in the sentence with “an object' or *something", unless the replacement will change the meaning of the

verb. For example, keep *"basketball" in “play basketball", but change “"a chair"' to “an object" in “lift a chair".

·Remember to remove unimportant details that are not very relevant to the action itself, especially when related to body part *"back"” and “shoulder",

such as changing “He holds a big object."' to “a person holds an object with right hand." *"A person with a backpack on his back turns left and walks

forward."'to *a person turns left and walks forward." ,and “A person with a backpack slung over the shoulder walks forward." to “A person walks

forward with an object on their shoulder.".

[end of rules]

[start of examples]

A person used their foot to apply force to the medium-sized box. Answer: boxmediumlleft footlA person kicks an object with left foot.

Someone moves the box by pushing and pulling it. Answer: boxlargelright handla person alternately pushes and pulls an object with right hand.

He touches the bottom of the box. Answer: boxtinylright handla person touches an object with right hand.

She clenches the upper body part of the chair. Answer: chairwood|right handla person holds an object with right hand.

A medium-size box is picked up by a man in black firmly using left hand. Answer: boxmediumleft handla person picks up an object with left hand.

He walks in a clockwise circle, and sits on a wooden chair. Answer: chairwood|back of the hiplA person walks in a clockwise circle and sits on something.

He with a yogabll on his hand, is walking forward. Answer yogaballright handla person holds an object with right hand walking forward.

A person holds a backpack while walking forward. Answer: backpacklright hand|A person holds an object with right hand while walking forward.

A person lies on a chair. Answer: chairblack|back|A person lies on something.

A person lies face down on the chair. Answer: chairwood|front body|A person lies face down on something.

A box is possessed by an individual. Answer: boxtinylleft hand|A person holds an object with their left hand.

She is playing basketball. Answer basketballright hand|A person is playing basketball with their right hand.

[end of examples]

The description of human-object interaction:

Aperson holds the tiny box with left hand and passes it to the right, then uses both hands to hold the opposite sides of the box and lifts it above their head.

A persongrabs the seat cushionofachair with theirleft hand and pulls thechair totheleft side.

A person grabs the backrest of a chair and pushes it backwards.

A person bends at the waist and uses their right hand to drag the suitcase on the ground.

A person's buttocks are against the small table, and then they push the small table backward.

A person uses right hand tolift the rectangular object in front of them to chest height.

A person places their left hand on the rectangular object that is on the ground, squats down, and then stands up again.

A person puts down a backpack that was slung over their right shoulder onto the ground.

A person inserts their right hand through the shoulder straps of the backpack and takes two steps backward.

A person starts by using their right hand to place the rectangular object horizontally in front of their eyes, then they use their left hand to take over the

rectangular object, and finally. the entire person turns to the left to face the back.

Please output the answer following the format of

"Answer:Answer1|Answer2|Answer3"

The answers:

Answer: [boxtinyleft handla person transfers an object to the right with left hand, then lifts it above their head with both hands.

Answer:|chairwood|left handla person pulls an object to the leftwith theirleft hand.

Answer: |chairwoodlright handla person pushes an object backwards with their right hand.

Answer: Isuitcaselright handla person bends and drags an object on the ground with their right hand.

Answer: tablesmallback of the hipla person sits against an object then pushes it backward.

Answer: [boxlonglright handla person lifts an object to chest height with their right hand.

Answer: [boxlonglleft handla person squats down and stands up while touching an object with their left hand.

Answer: Ibackpacklright shoulderla person puts an object from their right shoulder onto the ground.

Answer: backpacklright handla person puts on a backpack with their right hand and take two steps backward.

Answer: boxlonglright handla person places an object in front of their eyes using their right hand, then transfers the object to their left hand, and finally

turns left.

A person throws an object with right hand towards the ground.

Text-to-Motion

A person is carrying an object on the shoulder.

Text-to-Motion

Interaction Retrieval

Interaction Retrieval

you a description, and I would like to have three rewritten sentences with varying degrees of complexity, following the example: '...' The input text is '...' Please give me three texts that vary in complexity but keep the meaning of the sentence the same. This results in (i) less complexity : someone holds a backpack and steps left; (ii) middle complexity : a person holds a backpack in front of them with both hands and takes a step to the left; (iii) more complexity : with both hands, a person clutches a heavy backpack firmly and brings it close to their body, then steps to the left with their left leg.

Metrics. In Sec. 4.1, we introduce the metrics employed in this paper. This section details the formula for the proposed metric CMD. The formulations for other metrics are available in the existing literature [34, 148]. CMD quantifies the discrepancy between the contact maps of ground truth interactions and those synthesized one. In this context, a contact map is characterized by the proportion of time { p i } P i =1 each body part maintains active contact. Here, p i denotes the percentage of time during which the body part i is less than a threshold distance from the object. And the metric is defined as,

where p GT i is from the ground truth contact map, P is the number of body parts defined in SMPL [76], and we set the distance threshold as 0 . 03 m.

Implementation Details. The segment in the MDP contains m = 4 frames. The dynamics model, which includes 2 dynamics blocks as described in the main paper, is trained on the BEHA VE training set [7], with a batch size of 32, a latent dimension of 64, and for 500 epochs. For rollout after the initial step t > 1 , our dynamics model is trained to predict over a longer timeframe ( F = 3 × m = 12 ), exceeding the past motion duration ( H = m = 4 ). For the initial step t = 1 , we train a separate dynamics model to forecast a duration of F = 15 given the past motion over H = 1 frame, consistent with Sec. B.2. The optimization process is conducted over 300 epochs, utilizing a learning rate of 0 . 01 . The dynamic model is trained on an NVIDIA A40 GPU for a day. Our full log for high-level planning is presented in Figure B.

D Additional Qualtitative Results

Interaction Retrieval. We here visualize the intermediate retrieval results. Figure C depicts the results from learning-based retrieval, resulting in a diverse set of interactions that are both high-quality and semantically aligned.

Qualitative Experiments on OMOMO dataset. Figure D exemplifies our method that is able to generalize effectively to the OMOMO [66] dataset, despite our dynamics model not being trained on its object geometry or annotations.

Grab the top of the whitechair, swing the whitechair, and put down the whitechair.

Put the tripod horizontally down.

E Failure Cases

We present failure cases of our method in our website, consisting of (i) inconsistency of interaction with textual description, (ii) inconsistency of human actions with textual descriptions, and (iii) wrong object category inferred by LLM.

F Potential Negative Societal Impact

Some potential negative societal impacts include: ( i ) Our approach can synthesize realistic human motion interacting with objects, which could be misused to create deceptive or harmful content, such as portraying individuals in false situations. This could contribute to the spread of misinformation. ( ii ) Our method evaluates real behavioral information, raising potential privacy concerns. Although our model utilizes a processed representation (SMPL [76]) of human motion that retains minimal identifying details - unlike raw data or images - its ability to simulate human-object interactions could still be exploited for unauthorized surveillance or behavioral analysis. For instance, with photorealistic textures, it might be used to model and generate personal habits or movements without consent, posing risks of privacy violations. However, the use of a processed representation can be positively viewed as a privacy-enhancing feature, as it minimizes the exposure of personally identifiable details.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: in Sec. 1 and 4

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: in Sec. 5

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: this paper doesn't include proof and theory.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: in Sec. 4

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our code is not available at this time but will be released in the future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: in Sec. 4

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: in Sec. 4

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: in Sec. C

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: in Sec. F

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: in Sec. F

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not use or release pretrained language models trained by our own

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: in Sec. 4

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: in Sec. C

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: in Sec. F

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of text-conditioned 3D dynamic human-object interaction (HOI) generation, which has lagged behind advancements in text-conditioned human motion generation due to the scarcity of large-scale interaction data and detailed annotations. The authors propose InterDreamer, a framework that decouples interaction semantics and dynamics to generate realistic HOI sequences without relying on text-interaction pair data. By leveraging pre-trained large models for high-level semantic control and introducing a world model to understand low-level interaction dynamics, InterDreamer surpasses existing motion capture data limitations. Applied to the BEHAVE and CHAIRS datasets, InterDreamer demonstrates the ability to produce coherent and text-aligned 3D HOI sequences, showcasing its effectiveness through comprehensive experimental analysis.\n\nSTRENGTHS:\n- The authors introduce a novel task of synthesizing whole-body interactions with dynamic objects guided by textual commands, without relying on text-interaction pair data.\n- The types of HOI interactions are limited compared to the diversity of movement in the human body itself. So, the idea that decomposes semantics and dynamics and then integrates them makes sense to me. The main challenge lies in effectively understanding the positions where the human object interacts. The authors propose leveraging the power of LLMs to simplify this challenge effectively.\n- The technical details are sound, and the experimental results are good and demonstrate its zero-shot capability.\n- The paper is well written, with clear figures and typography.\n\nWEAKNESSES:\n- This work doesn't seem to be able to handle more complex long interactions, such as a person walking to a chair and then sitting down, or a person lifting a box on the floor and carrying it for a while before putting it on the ground.\n\nQUESTIONS:\n- I'm curious, can the Interaction Retrieval predict the accurate contact areas for an object that has never been seen before?\n- Is this post-processing optimization expensive, and how long does it take to optimize a sequence?\n\nLIMITATIONS:\n- The method still is unable to handle fine-grained human-object interactions (HOIs), such as dexterous manipulation using hands. Right now HOI generation is still a very challenging task, so I wouldn't consider this limitation to be a weakness of the manuscript, and expect that subsequent work will refine this point.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims at human-object interaction generation under less supervision. Since existing methods rely on large-scale interaction data, this paper attempts to propose a method that does not require paired data. The proposed method includes three stages, high-level planning by LLMs, low-level control by existing text-to-motion methods, and interaction retrieval, a world model based on existing human-object interaction generation methods. The proposed method is evaluated on the public datasets, BEHAVE and CHAIRS datasets.\n\nSTRENGTHS:\n1. The issue focused on in this paper is a worthy research topic. If the training does not require pair-wise data, it will greatly reduce the dependence of the human-object interaction generation on the training data.\n2. Given texts, the figures intuitively demonstrate visual representations of the generated human-object motions.\n\nWEAKNESSES:\n1. About novelty. While the issue focused on in this paper is valuable, the proposed method is a combination of existing methods. I appreciate the technical effort of the authors, but please highlight how each stage differs from existing approaches.\n2. Missing some details. \n--- In Line 165, how is the database built? If the data used to build the dataset includes the training set and the test set, it is not standard.\n---Does the low-level control need to be trained? As mentioned in Line 269, the text-to-motion model is the existing method which is pre-trained on HumanML3D. If no training is required, how does the model generalize on action types? \n--- As mentioned in Line 180, the authors claimed “this model trained on the 3D HOI dataset”, while the main work is that the proposed method does not require training on paired HOI data. It is inconsistent and confusing.\n--- What is the size of the object’ signed distance fields? How many vertices is the object represented by? What network to use to encode and decode object shapes?\n3. The motivation for the quantitative experiment (section 4.2) is unclear. Why compare different control conditions? \n4. In Line 316, what is the vertex-based control? Does it mean human vertex controls in Line 210? The definition should be given.\n5. In the caption of Figure 4, it is claimed that the proposed method can handle complex and long sequences. Is there a strategy in the proposed method for complex and long sequences? What is the longest?\n\nQUESTIONS:\nAs mentioned in weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitation of their method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInetrDreamer is a framework for synthesizing Human-Object Interactions (HOI) from textual queries. The key feature of InterDreamer is the ability to train without paired text and HOI motion data. To achieve this the work employs a multi-stage pipeline with LLM operating as a high-level planner that defines the parameters to infer the starting object pose and human motion sequences, which are brought together with the help of optimization on the next stage. The method is evaluated on the BEHAVE dataset which is additionally labeled with text as part of this work.\n\nSTRENGTHS:\n1) The key feature of InterDreamer is undoubtedly its biggest strength: the lack of requirement for paired data text-to-HOI data. This is a promising feature that in theory allows scaling the HOI modeling without tedious data-labeling by leveraging the advancements in LLMs.\n2) Another notable aspect is that the proposed planning can be adopted on top of other existing motion models improving their performance (as demonstrated in Table 2).\n\nWEAKNESSES:\n1) One of the key features of the proposed framework is a High-level Planning module that queries the LLM to extract necessary features for downstream modules. However the presented description of the query protocol is scarce, similar works (e.g. SINC by Athanasiou, Petrovich, et al., ICCV'23) also employ LLM within the framework and provide the full template of the query in the supplementary material (Section B) to ensure the reproducibility.\n\n2) Contribution formulation in the Introduction (Lines 71-72): the considered task is text-to-HOI modeling, while training without paired data is rather a feature of the method than a task itself. Further in the text, in the Conclusions section, the work claims to introduce the novel task of 3D HOI generation from text (Line 319). Both formulations do not reflect the actual contribution of the work.\n\n3) Evaluation is performed only on the self-labeled BEHAVE dataset, however, there exists at least one more dataset with text annotations: OMOMO \\[51] which is not used for evaluation.\n\nQUESTIONS:\n1) Appendix B.1 describes two approaches to Interaction retrieval. Which of the described techniques is used? How do they compare in terms of performance?\n2) Are the BEHAVE text annotations planned to be released upon acceptance? \n3) What is the reasoning behind omitting the OMOMO dataset from the evaluation?\n4) Lines 157-158: should it be $a^{*}$ as it is the final result after optimization?\n\nLIMITATIONS:\nThe work presents a substantial discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to address the text-conditioned human-object interaction (HOI) motion generation task. Unlike previous HOI generation approaches that rely on limited existing HOI datasets with text annotations for supervised learning, this work proposes a framework that decouples interaction semantics learning from interaction dynamics learning. This decoupling eliminates the need for large-scale HOI datasets with text annotations for training. The interaction semantics learning leverages an existing pretrained text-conditioned human motion generation model, while the interaction dynamics are optimized using a learned world model that predicts object states induced by human motions. The proposed framework can generate realistic HOI motions that align with the input text conditions.\n\nSTRENGTHS:\n* HOI modeling remains a challenging and understudied task compared to recent developments in human motion modeling. One major bottleneck is the lack of large-scale, high-quality interaction data. This paper offers an insightful solution to overcome the scarcity of large-scale interaction datasets.\n* Several key designs in this framework are reasonable, including (1) LLM-based high-level planning to reduce the distribution gap between input text instructions and the language used for text-to-motion model training dataset; (2) the world model trained with interaction dataset for human pose and object pose optimisation.\n* Comprehensive quantitative experiments results show the effectiveness of the proposed modules.\n\nWEAKNESSES:\n* Without the object geometry information encoded as input, how does the world model perform in generalizing to different objects? How does it perform when the object is out of the predefined list?\n * Efficiency of the autoregressive optimisation: as presented in figure 2, the optimisation of action and state is performed per step, which might result in inefficient inference, while the world model are trained to predict longer-horizon states. Could the author elaborate more on this design choice?\n * Regarding the world model training with N vertices sampled from contact area, what if the human pose is not in contact with object, for example, for the text instruction “the person throw away the ball“ where in most of frames the ball is flying in the air without contacting with the human, how does the object state forecasted and optimised with the world model.\n* From the visualisation results, there are no significant improvements over baselines HOI-Diff and CG-HOI in terms of the realism of human-object interaction. Is this mainly result from the not-perfect world model training and optimisation or the lack of hand pose? Could the author give some insights on the major challenges and bottlenecks presented in the current HOI understanding?\n\nQUESTIONS:\nSee the weakness section, and I am happy to discuss with authors during the rebuttal phase and adjust the score accordingly.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}