InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction
Authors: Sirui Xu
Ziyin Wang
Yu-Xiong Wang
Liangyan Gui
Venue: NeurIPS 2024 poster (2024)
Keywords: human object interaction
human motion generation
Primary area: generative_models

Abstract: Text-conditioned human motion generation has experienced significant advancements with diffusion models trained on extensive motion capture data and corresponding textual annotations. However, extending such success to 3D dynamic human-object interaction (HOI) generation faces notable challenges, primarily due to the lack of large-scale interaction data and comprehensive descriptions that align with these interactions. This paper takes the initiative and showcases the potential of generating human-object interactions without direct training on text-interaction pair data. Our key insight in achieving this is that interaction semantics and dynamics can be decoupled. Being unable to learn interaction semantics through supervised training, we instead leverage pre-trained large models, synergizing knowledge from a large language model and a text-to-motion model. While such knowledge offers high-level control over interaction semantics, it cannot grasp the intricacies of low-level interaction dynamics. To overcome this issue, we introduce a world model designed to comprehend simple physics, modeling how human actions influence object motion. By integrating these components, our novel framework, InterDreamer, is able to generate text-aligned 3D HOI sequences without relying on paired text-interaction data. We apply InterDreamer to the BEHAVE, OMOMO, and CHAIRS datasets, and our comprehensive experimental analysis demonstrates its capability to generate realistic and coherent interaction sequences that seamlessly align with the text directives.

InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction
Sirui Xu
Ziyin Wang
Yu-Xiong Wang Liang-Yan Yan Gui
University of Illinois Urbana- Sebana-Champaige
Equal Contribution $ Eqqual Advising
{siruixu2, ziyin, yxs, lgui)eillinoss.ede
https://siruixxu gru.g.thub.... io/InterDreamer/
Abstract
Text-conditioned human motion generation has experienced significant advance- ments with diffusion models trained on extensive motion capture data and corre- sponding textual annotations. However, extending such success to 3D dynamic human-object interaction (HOI) generation faces notable challenges, primarily due to the lack of large-scale interaction data and comprehensive descriptions that align with these interactions. This paper takes the initiative and showcases the potential of generating human- interactions without direct training on text-interaction pair data. Our key insight achieving this is that interaction semantics and dy- namics can be decoupled. Being unable to learn interaction semantics through supervised training, we instead leverage pre-trained large models, synergizing knowledge from large language model and text-to-motion model. While such knowledge offers high-level control over interaction semantics, it cannot grasp the intricacies of low-level interaction dynamics. To overcome this issue, introduce a world model designed comprehend simple physics, modeling how human actions influence object motion. By integrating these components, novel frame- work, InterDreamer, is able to generate text-aligned 3D HOI sequences without relying paired text-interaction data. We apply InterDreamer the BEHAVE, OMOMO, and CHAIRS datasets, and our comprehensive experimental analysis demonstrates its capability to generate realistic and coherent interaction sequences that seamlessly align with the text directives.
Introduction
Text-guided human motion generation has made unprecedented progress through advancements in diffusion models [41, 105, 106, 131], leading synthesis outcomes that realistic, diverse, and controllable. This progress has ignited an increased interest exploring expanded tasks related to text- guided human interaction generation, such social interaction [69] and human-scene interaction [44]. However, many of these explorations are limited in that the dynamics objects is not involved or text-guided. Aiming bridge such gap, this paper tackles more challenging task generating versatile 3D -ojant interactions (HOls) through language guidance, as illustrated in Figure 1.
Although direct solution, suggested by the concurrent work [28, 65, 91, 107, 136, 140], would be replicating the success observed in human motion generation and adopting a similar supervised approach for learning text-driven HOIs, it not scalable. As can be observed, generating social or scene interactions is heavily dependent on extensive collections of text-interaction pair data [34, 69, 83, 130], and scaling these methods to address more complex HOIs outlined in study could require datasets of comparable magnitude. Achieving this goal appears unattainable by merely annotating existing 3D HOI datasets [7, 30, 45, 47, 53, 66, 169, 170, 177, 180], which are relatively limited in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Semantics
Dynamics
person picka backpack, puts it on, and then walks forward clockwise.
How humans
backpacks?
Figure 1: InterDreamer generates vivid 3D human-object interaction sequences guided by text descriptions, by synergizing semantics and dynamics knowledge from large-scale text-motion data (upper left), a large language model (bottom left), human-object interaction data (upper middle), and prior knowledge (bottom middle) from simple physics. We visualize the generated text-guided interaction sequence (upper right), with the beginning of the sequence unfolded (bottom right).
size. Although recent studies [28, 65, 91, 155] have annotated some of these datasets, the volume of text-interaction pairs still lags behind that available for existing text-driven motion generation efforts.
An intriguing question naturally arises: given the limited annotations of the text, what is the potential of learning for lext-conditonee HOI generation without text supervision, which is the main focus of this paper. However, formulating the task in such a setting presents significant challenges, primarily due to the inability to directly learn the alignment between text and HOI dynamics. Our key observation is that interaction semantics and dynamics can be decoupled. That is, the high-level semantics of an interaction, aligned with its textual description, can be informed by human motion and the initial object pose. Meanwhile, the low-level dynamies of the interaction specifically, the subsequent behavior of the object is governed by the forces exerted by the human, within novel the constraints of physical laws. Motivated by these insights, we introduce InterDreamer framework that synergizes knowledge of interaction semantics and dynamics (Figure 1), both of which do not necessarily require learning from text-interaction pairs, if they are decoupled.
The semantics of interaction, although not available through direct supervised training, can be harnessed from prior knowledge without text-interaction pair datasets. Specifically, to acquire semantically aligned interaction, we first consult large language model (LLM), such as GPT-4 [88] and Llama [120], provide understanding including how humans typically use specific body parts in interactions with particular objects, by exploiting its in-context learning capability with few-shot prompting [10] and chain-of-thougih prompting [134]. The intermediate thoughts and the final thought are then used to (i) generate semantically aligned human motion with a treitrinnd text- to-motion model; and (ii) identify initial object pose that is harmonious with the generated human pose and text description, following a philosophy similar to retrieval-augmente generation [62].
While these large models offer high-level motion semantic modeling, they lack crucial low-level dynamics knowledge. Nevertheless, by decoupling interaction dynamics from semantics, a key advantage emerges our InterDreamer framework: interaction dynamics can be learned from motion capture data without the necessity of text amnotations. We instantiate this idea by developing world model, which predicts the subsequent state of an object affected by the interaction. The key here is to reach generalizable representations in different motion and objects. To do so, we exert control over the object through the motion of vertices on the human body. These vertices are solely sampled in regions where contact occurs, agnostic to the overall object shape and whole-bodi motion. Such abstraction empowers the model to learn the simple dynamics from publicly available 3D HOI dataset BEHAVE [7], and generalize naturally to other datasets [47, 66]. The plausibility of the generated interaction is further enhanced by a subsequent optimization procedure on the synthesized human and object motion.

---

To summarize, our contributions are: (i) We address the task of synthesizing whole-body interactions with dynamic objects guided by textual commands, achieving this notably without the need for paired text-interaction data, novel paradigm to the best of our knowledge. (īi) We introduce a framework that decomposes semantics and dynamies, and they can be easily integrated. Specifically, it harnesses knowledge from large language model (LLM) and text-to- to-ttoon model as external resources, alongside our proposed world model. Remarkably, the only component that requires additional training is the world model, which highlights the ease of use of our framework. (iii) Experimental results demonstrate that our framework, InterDreamer, is capable of producing semantically aligned and realistic human-object interactions, and generalizes beyond existing HOI datasets.
2 Related Work
Cext-Conditioned Human Motion Generation. Significant progress has been witnessed in human
motion synthesis tasks, given different kinds of external conditions, including action categories [2, 36, 61, 93], past motion [5, 17, 86, 110, 149, 150, 163], trajectories [31, 50, 51, 100, 122, 145], scene context [12, 29, 39, 44, 113, 114, 125-127, 130, 153, 175, 182, 183], and without condition [96]. Recently, human motion synthesis guided by textual descriptions [1. 6, 18, 24, 26, 34, 35, 49, 54, 57, 64, 71, 77, 81, 94, 95, 97, 104, 116, 133, 160, 162, 168, 172, 174, 178, 179, 181, 185, 187] is popular and extended to various applications, including the text-conditioned generation of multiple- person [33, 43, 63, 75, 132] and human- scene interaction [14, 21, 44, 48]. Our goal to model human and object dynamics concurrently guided by text.
Human-Object Interaction Generation. Synthesizing hand-object interactions [11, 15, 20, 68, 73, 79, 80, 119, 137, 161, 165, 167, 184, 186] and single-frame human-object interactions [25, 42, 56, 92, 128, 143, 152, 154, 166] are popular topics and extended zero-shot settings [52, 67, 156, 157J. Recently, researchers explore whole-body dynamic interaction generation, in kinematic-based approaches [22, 27, 32, 38, 58-60, 66, 84, 85, 99, 107-109, 111, 123, 136, 139, 140, 148, 151, 176] and physics-based approaches [4, 8, 16, 23, 40, 72, 74, 78, 87, 89, 115, 117, 124, 129, 142, 146, 147, 158]. Current methods in HOI synthesis are often restricted by a narrow scope of actions, the use of non-dynamic objects, and a lack of comprehensive whole-body motion. Our work aims to generate diverse whole-body interactions with various objects, and enables control through language input. Recent datasets [7, 30, 45, 47, 53, 66, 112, 138, 144, 155, 164, 169, 170, 180] provide the groundwork for research in this area, and concurrent efforts [28, 65, 91] demonstrate the feasibility of applying supervised learning methods via annotating datasets. However, the amount of data currently available fall short when compared to more extensive eext-motion datasets [34, 70, 83). This discrepancy in data volume limits the capability of supervised methods to capture the complexity of human-object interactions, motivating us to investigate the potential of zero-shot generation.
External Knowledge from LLMs. Large language models (LLMs) are being used for advanced visual tasks, such as editing images based on instructions [9]. In digital humans, they are used to reconstruct 3D human-object interactions [128] and generate human motion [3, 46, 159, 178] as well as human-scene interactions [141]. Our approach inspired by [128], which uses LLMs to infer contact body parts with given object for reconstructing 3D human-object interactions
task different from ours. Our approach utilizes GPT-4 [88] or Llama [120], to not only understand contact body parts but also narrow the distribution gap between different tasks, and provide knowledge for interaction retrieval. This is accomplished by utilizing the in-context learning capabilities of
LLMs [22] and their support for Attrieval-augmente generation [62].
3 Methodology
Problem Formulation. Our goal is to synthesize a sequence of 3D human-object interactions æ that satisfies a descriptive text p. This sequence is series of tuples [(,, 01), (22,02),..... (has 0s)) }] where h; represents the human pose parameters defined in the SMPL model [76], while the shape of the human is unified the same as [34]. o, defines the pose of the rigid object in terms of its 3D spatial position and orientation. The sequence length M is variable and is dynamically determined by our text-to-motion model based on the input text p. We do not require text supervision for training.
Overview. Our framework, illustrated in Figure 2, can be conceptualized as a Markov decision process (MDP). We begin by dividing the motion sequence into T segments, each with m frames,
3

---

High-Level Planning
Low-Level Control
World Model
interction batruse perns Yas willee givs reslace cotirut eekerak torket rnd sovver three quenker Dilering reko complee.
Interaction Retrieval
Dynamics Modeling
ag 8. 0.pl < 8/41
Eurant
whuch
haven las
Abstracting saloo (djec) and colero (aaner)
just slrcle)
Jui
the bob part tuat cuntuct the object the beplaale the
badk bit todk docias
wolch hand randiedly clease
controta to the symamicc mocel
Text-to-Motion ap... 4 d> 4u1
Juat afous futt aplle Seas Suz dloal bomrzaan betth price ddto
The coeerpotee hf en interttion:
Optimization
An overview of our InterDreamer. (i) Our high-level planning analyzes the description using LLMs and provides guidance to the low-level control. (ii) Our low-level control includes a text-to-motion model that translates text into human actions ar+11 and interaction retrieval model that extracts the object's 's first state 8). (iit) Our world model executes actions to output the next state $1+1. We achieve this by abstracting the problem as predicting the motion of contact vertices represented by red spheres for humans and blue spheres for objects on the top right using human vertices as controls for the prediction of object vertices. An optimization process is coupled with the dynamics model, projecting the state and action onto valid counterparts. Solid arrows mean that the process is performed iteratively.
Figure 2:
where M = T m. Object motion [o can be seen sequence of environmental states {st] and human motion } 1M is described as a sequence of actions {ar that interact with the environment. Under such an MDP setup, our framework starts with high-level planning L, which deciphers textual interaction description p by g L(p) (Sec. 3.1). Then, text-to- motion model x translates context g into human actions, modeled as a++1 ~ π xa St. (- -/-1)) (Sec. 3.2). The interaction retrieval R proposes an initial object state 8 ~ R(s, (a1.)). based on the initial action d1 and context y (Sec. 3.2). After that, a world model P is trained to predict future states 8 P(
M
[18t,1/a,,00 (a, St; 0++1) from the current action and state (Sec. 3.3). Our world model
incorporates an optimization process, for both state and action refinement (Sec. 3.4). Notably, the text-to-motion and world models are executed iteratively until text-to-motion generates an end frame.
3.1 High-Level Planning
Leveraging LLMs' strong reasoning capabilities and inherent common sense, our high-level plan- ning L yields interaction details g L(p) that cannot be naively extracted in textual descriptions The process undertaken by L encompasses three steps: (i) Determining the object: the LLM is employed translate described objects into corresponding categories from a predefined list. (fi) Determining initial human-object contact: the L.M infers the body parts involved in the interaction, drawing from list defined the SMPL model [76]. And most importantly, (iii) reducing the distribution gap: the LLM bridges the distribution gap between the free-form textual input and the language used within the training data of the -ox--to-motion model [34]. This involves standardizing syntax and content according to designed guidelines. In Figure 2, we demonstrate the prompt we used with the fhot prompting [10]. We define intermediate thoughts and the final thought, i.e., answers to three questions, as detailed information structuring the entire framework with a philosophy similar to retrieval-augmmented generation [62]. Our legel planning operates indirectly in the generation of interactions. Nonetheless, it narrows the vast range of possible interactions the real world into a more manageable distribution within the capabilities of our framework. We incorporate GPT-4 [88] and Llama-2 [120] for evaluation.
L(p), which guides the subsequent procedure,

---

3.2 Low-Level Control
With the information 9 derived from the description the low-level control aims to create sequence of human actions {a=){= by a text-to-motion model, and an initial state 81 by interaction retrieval, such that they correspond to the objectives outlined by g-
Text-to-Motion. We utilize text-to-motion model π to develop actions to be executed in the world model. At each timestep i, π receives the sequence of previous actions {ar]=z and the text tokens encoded from the rewritten description in g = L(p), and produces next action 1 which later in Sec. 3.4 will be adjusted through an optimization process that intertwines actions with the object state st. Thus, the overall process can be formally defined as a++1 IT( TT(a+1 St. (a 9). while the initial action a1 ~ x (a)l)) is influenced merely by context g without prior actions or states, which will be used in interaction retrieval. и builds upon existing rext-to-motion models, where we evaluate MDM [1 (188, MotionDiffuse [172], ReMoDiffuse [173], and MotionGPT [46].
Interaction Retrieval. The interaction retrieval component R. establishes the initial state $1
R(s [ 9). based on the initial action ai generated by the text-to-motion model. We propose user-friendly pipeline for this purpose built on handcrafted rules. First, we create databases by extracting HOI frames from the training sets of each target datasets - BEHAVE [7], OMOMO [66], and CHAIRS [47]. The indexing key for retrieval is tuple consisting of the body part in contact and the category of the involved object. Each retrieval value is a per-frame contact map, represented by a list of K vertex pairs {(d], d()) k Here, dj, refers to the contact vertex on the human surface, while d; refers the corresponding contact vertex on the object surface. This contact map is linked to its corresponding key, creating a searchable record of interactions. During the inference stage, using the body part and object information provided by the high-level planning (Sec. 3.1), we retrieve all relevant contact maps from the database. We then sample one map {(d), d',) d' and use it to establish the object state $1 R(s aa, g), thus initializing the interaction. Further details including how we ensure consistency between the sampled state and human action are provided in Sec. B.1 of the Appendix. We also discuss an alternative learning- baaning-based approach in Sec. B.
3.3
World Model
Our world model combines a dynamics model and the optimization process, dedicated to simulating state transitions affected by applied actions. While drawing inspiration from similar concepts utilized in robotics [103, 135] and autonomous driving systems [55], we use it here to generate HOI trajectories. This model, trained the training set of 3D HOI dataset such as BEHAVE [7], serves a similar role as a simulator but is much simpler - it takes the preceding object state 8, along with a pair of consecutive actions ag and a,+1, and predicts the subsequent object state s,+1. The interplay between the low-level control and the world model ultimately produces a coherent interaction rollout.
In designing the dynamics model, a naïve method would be directly taking raw actions, states, and object geometry as input. However, this suffers from a severe generalization problem during inference: the dynamics model is likely to encounter human actions and object geometry that do not exist in the training set, since our -ootinn model is not trained with object interaction data. To overcome this limitation, we instead focus on encoding interactions through the contact vertices on the object, which capture both the action and object geometry, as shown in Figure 2. This locality ensures that the dynamics model remains focused on interactions in the contact region, without being distracted by the motion of body parts and geometry details that are irrelevant to the interaction.
Input Representation. Specifically, at each timestep f, we abstract the past actions as Hl histor- ical vertex trajectories [[o]) and the future actions F future vertex trajectories Ho? y H+F " where non-fixed variable N is the number of sampled contact vertices, and m the length of segments as mentioned in the overview of Sec 3. Note that we train our dynamics model to forecast over a longer duration than the past motion (F > H), only the foremost future action will be used for autoregressive generation during the inference, as suggested in [19]. To determine these N vertices, we start with object's 's signed distance fields {sdf, ) over the past H frames, derived from the past state st. We then sample vertices that meet the following criteria: |sdf,
/!
(adf(()))) (w/) S sdf, (xf)) < 81 Vi 1,..., H, Vj. and ||ol ef"1 2 8g, Vj o k. where ón and &2 are two hyperparameters. The objective is to sparsely sample contact vertices while ensuring that HI+ they are sufficient to encompass the interaction. We characterize each vertex trajectory {u: Nell
5

---

Table 1: Quantitative results on evaluating the dynamics model, Our dynamics model with vertex-
based action generates interactions of the best quality.
Methods
Inttr--interacioon
Interaction Prediction [148)
CMD
Pene. 2%)
Trans. [mim]
(ad) 4
Penc. 2%)
w/o action
0.424
533
123
2.56
228
contact markers action (InterDiff (148))
0.219
484
123
226
164
human motion as action
0.325
957
129
265
218
contact vertices as action (ours)
0.151
443
119
221
156
Table 2: Quantitative results on human motion quality given our annotation on the BEHAVE [7] dataset. We show that our high-level planning effectively adapts single human generators into human- object interaction generation. To evaluate R- R-Prrcision, a batch size is selected.
Methods
Planning (Ours)
R- R-Precision"
FID
MM Dist
Multinodaliy!
Diversity
Top
Top 2
Top 3
Ground Truth
0,23745.00
0,39240.004
0.496=0.006
0.024=0.000
4.259+9.005
6.510
0.153400.00
0.279#0.026
0.30g=0.010
12.279#00227
5.351
7.604=0.100
7.59g=0.554
MDM [118]
-..33000000
4..07
0.402 300010
10.374
10.374 10.300
4.303
7,281 1t
7.47 4:2
0. 0.205-99.011
0,351
0.458
10.208
4.837
4.520
7.323
MotioaDifluse [172]
0.2 19.022
0.369
Loooz
9.015
-0.40s
4.649
4.99 1.9312 Lu.372
7.295/n.50
ReMoDilluse [173]
0.223 0.223-10.00
0. 196 196000
0.368 Lo.015
0.338
0.448
0.482 100011
0.448-10012
210022
5.237
6.385-1l.a11
4.855
4,784
130033
6.350
7,201 1.0.318
7.160-9.506
0.233 10.000
0.344 10.00s
0.457 100005
5.497
-..17-0
5,205 Lu00eT
1.062-0-211
MotionGPT [46]
0.23440.000
0,387
0.471
200000
4.751 99.191
4.995
1,337
7.106+9.188
with a feature f to provide (i) human vertex coordinates at T-pose, providing information about the position of the human vertex the body surface; (ii) the vertex-to-object surface vector, indicating vertex's 's impact on the object as well as inherently including information related to the object's 's shape; and (iii) the vertex's velocity relative its nearest object vertex. Thus, the model needs to learn how the features of human action f affect the evolution of the state of the object.
Architecture. As demonstrated in Figure 2, the network comprises two components: G that operates without contact vertex conditions, applicable scenarios where no contact oceurs, and F, which incorporates contact vertex conditions into the object trajectory when contact is present. The k- th layer of G can be initiated as (xx, θ). mapping the input feature map I& at the k-th layer to another feature map, with O denoting the MLP's parameters. To incorporate human vertex controls, we introduce second network Fr +.)) operating
N vertex features { where θ, is its parameters. With a cross- artss-attention layer Attn, a dynamics block is formulated as: 2k+1, (v 1 = Attn(Ga (x), θ), (Fr(y) ,,)); We stack multiple dynamics blocks to form the model. The initial input, Tu. corresponds to the previous state st. while each yd represents the feature of the vertex trajectory, containing both the trajectory {u] = and its associated feature vector f³, The output of this model is preliminary and subject to further optimization as introduced in Sec. 3.4, which will yield the final future state. We utilize the Mean Squared Error loss to train the dynamics model. For more explanations, please refer to Sec. B.2 of the Appendix.
3.4 Optimization
Optimization plays a role in introducing prior knowledge and avoiding the accumulation of errors. During inference, we input the action 0.+1 and state 81+1 and refine them. This refinement is achieved through gradient descent on the human and object pose parameters. Our optimization includes several loss terms: a fitting loss to align optimized results with their preliminary one, velocity loss for temporal smoothness, a contact loss to promote occurring contact, and a collision loss reduce penetration. We provide detailed formulations in Sec. B.3 of the Appendix.
Experiments
Extensive comparisons evaluate the performance of our InterDreamer across two motion-relevant tasks. Details of the evaluation settings are provided in Sec. 4.1. We present both quantitative (Sec. 4.2) and qualitative (Sec. 4.3) results for our approach. Additionally, we perform ablation studies to verify the efficacy of each component within our framework. These studies also cover the

---

Table 3: Quantitative results on human motion quality on OMOMO [66] dataset with their provided annotation. We show that our high-level planning narrows the distribution gap and adapts single human generators into human-object interaction generation. To evaluate R R-Precision, a batch size of 32 is selected.
Planning (Ours)
R-Precision"
Methods
FID4
MM Dis+
Mullimodiliiy'
Diversity
Top
Top
Top
Ground Truth
0.044#0.00
0.095
0.151=0.000
0.000=9.000
6.858#9.00
3.660#9.110
0.056=8.00
0.09640.077
0.135
16.638
7.110
2,44640-455
5.86240.500
MDM [118]
0.062#0.006
0.10920.004
0.155=0.008
15.735#0.250
6.889+0002
2.663=0.50
6.461#90641
0.04g=50.00
0.094
15. 15.4424902211
5,799
1.658
5,981 5.981 200.15
MotionDilTuse [172]
0.078 0.07g-10.00
0.141 10.015
0.189
10.815 10.083
5.91643008
1.677
5.718 10.1820.522
0.062-19.0g
L0.00s
100012
15.479 10.204
5.690-Loo0y Luodp
1.17910.345
6.032 6.032
ReMoDilTuse [173]
0.067=0000z
0.127
0.174
14,560
5.678
1,193-0
5,368-9.41
0.061 -.061-
0.1 141.000000
0.152
18. 18.472400550
6,358 LoooRs
4.583100.01
6.726
MotionGPT [46]
0,064
0.121 20.007
0.164 20.000
17.512
6.28750.04)
4.470=0.00
7,048=+0.00
person penson hand.
person lango right
person holds plastic contsiner with their right hand and moves different directions.
person moves a long box in their left hand downward, right and then to left.
A person transfers a plastic container from their left hand their right hand.
Figure 3: Qualitative results on free-form text input. The interaction sequences, with textures from [13], are presented through a time-series visualization.
interaction prediction task [148] to evaluate our dynamics model. Additional details and results are presented in Sec. C and Sec. D of the Appendix. Please refer to our website for video results.
4.1 Experimental Setup
Datasets. We BEHAVE [7], CHAIRS [47]. and OMOMO [66] datasets for quantitative evaluation. The BEHAVE dataset includes recordings of 8 individuals interacting with 20 everyday objects, and our analysis focuses on 18 objects for which interaction sequences are available at 30 Hz. The human pose is modeled using SMPL-- [102], with hand poses set to average pose due to the absence of detailed hand pose the dataset. We manually segment the long interaction sequences in the test set, and annotate them with descriptions well their starting and ending indices, leading 532
A person is seated on the small box with legs crossed.
person their right and steps on and over a wooden chair and steps down.
Figure 4: Qualitative results in more challenge scenarios with free-form input not from our an- notations, showing the ability our InterDreamer fit object sizes and handle complex and long sequences. Here, our synergized models are GPT-4 [88] and MotionGPT [46].
7

---

A seated person on chair 63 adjusts the sitting position repentedly. Someone can be seen on chair and adjusts their sitting position.
Figure 5: Qualitative results the CHAIRS [47] dataset. Our dynamics model trained on the BEHAVE [7] dataset generalizes well on the CHAIRS objects unseen in training. Frames are separately visualized. Here, our synergized models are GPT-4 [88] and MotionGPT [46].
Language
3. person carrying on objjctt the shoskder
shoulder:
Language Planning
à parton throws an olgect with right hand towards the ground.
hand
Test-t-Modion
Interaction Retrivvel
Test-todlation
Interaction Retrivall
Figure 6: Results from the interaction retrieval. We demonstrate that our proposed retrieval approach based on handcraft rules can extract diverse and realistic interactions.
subsequences for evaluation. For qualitative evaluation, we go beyond using annotations specifically created and employ free form text to demonstrate model's capability out-of-distribution inputs. To assess our model's performance with novel objects, we expand our retrieval database to include objects from the OMOMO [66] and CHAIRS [47] datasets, while we do not fine-tune the dynamics model on them-a direct qualitative evaluation without additional adaptation.
Metries. The evaluation metrics are divided into three categories: (i) Human motion quality: The Fréchet Inception Distance (FID) measures the distance between the generated motion and ground truth. The MultiModallity (Multimodality) and Diversity metrics assess the variance in generated human motion. R-Precision evaluates the consistency between the text and the generated human motion within the latent space. MultiModal distance (MM Dist) is the distance between the motion feature and the text feature. We follow [34] to generate motion and text features. (ii) Interaction quality: We propose CMD to measure the distance between contact maps of real interactions and those generated. The per-sequence contact map is defined by the percentage of time that each body part is actively in contact. The detailed formulation is provided in Sec. C of the Appendix. We also measure the collision (Pene. [148]), which calculates the average percentage of object vertices that have non-negative values in the human signed distance fields [90]. (iii) Object motion accuracy: The dynamics model's performance in the interaction prediction task [148] is evaluated by the accuracy of predicted object motion, including Trans. Err. the average distance between predicted and ground truth, and Rot. Err. the average distance between the predicted and ground truth.
Someone be sitting yogaball.
(a) Low-level control w/o planning v.s. w/ planning
(b) Text feature w/ planning v.s. w/o planning
Figure 7: (a) Ablation study on the high-level planning. On the left are results from MotionGPT [46] using free-form descriptions, and on the right are results with our planning additionally. Without planning, the motion generative model struggles interpret free-form HOI descriptions and generate mmnantically-aligne motion. (b) We visualize CLIP [98] features of text on HumanMI.3D [34] via t-SNE [82], raw HOI descriptions ("w/o planning"). and HOI descriptions processed through our high-level planning ("w/ planning"). See Table for quantitative measurements.

---

(a) human motion as action
(b) contact vertices as action
Figure 8: Ablation study on the dynamics model. Given the text description of "A person walks clockwise while holding a small box with left hand," our (b) vertex- -bsee control can synthesize consistent contacts, which (a) the baseline fails to do.
Baselines. Most recent research on text-to-HOI synthesis follows a supervised learning approach [28, 91], making direct quantitative comparisons unfair. Therefore, primarily focus on qualitative comparisons with these methods. To enable quantitative evaluation, we develop a range baselines to assess both the overall performance of framework and the effectiveness of its individual components. In the context of high-level planning, we utilize GPT-4 [88] and Llama-2 [120], illustrating the effectiveness of our prompts across different language models. For low-level motion generation control, our baselines include MDM [1 18) Motion [172]. ReMofuffese [173], and MotionGPT [46]. which span a range of text-to- to-tooooooonnn approaches trained on HumanML3D [34] and
show the generalizability of our framework. To evaluate the dynamics model, we include different design choices: (i) unconditional dynamics model which operates object dynamics independently of
human motion; (ii) using human marker features as actions to the dynamics model, similar to [148]; (iii) using unprocessed human motion and object pointcloud motion as input to the dynamics model; (iv) our proposed vertex- vertex-based actions where only the contacting vertices are used for control.
4.2 Quantitative Results
In Table 1, comparing our framework to baselines with unconditional dynamics model, Inter- Dreamer achieves better interaction quality in terms of CMD and penetration scores, showing the importance of human influence on object motion. Against methods that utilize direct raw human motion or markers for action features, method demonstrates enhanced performance by offering more fine-ge ame--gained guidance and extracting generalizable features for dynamics modeling. Tables and present comparative analysis of our approach of combining high-level planning with low-level control, where we adopt various text-to-motion models against their counterparts without high-level planning the BEHAVE and OMOMO datasets. Our approach consistently outperforms base- lines. Specifically, InterDreamer exhibits superior motion quality, reflected by lower FID, higher R-P R-Precision, and better diversity, highlighting the benefits of incorporating planning reduce the distribution gap for the motion generator to generalize in the HOI synthesis task.
plaaminn
4.3 Qualitative Results
Figure displays several results guided by the free-form text. Our method exhibits proficiency in interpreting the textual input and synthesizing dynamic, realistic interactions, despite the absence of training with text-interaction paired data. More importantly, as illustrated in Figure 4, we selectively use more complex sequences of interactive descriptions that are beyond the scope of the existing HOI dataset. Figure further exemplifies our method that able to generalize effectively to the CHAIRS dataset, despite our dynamics model not being trained on it. Figure 6 depicts the retrieval procedure, resulting in a diverse set of interactions that are both high-quality and semantically aligned. More experimental results and the user study are presented in Sec. D of the Appendix.

---

Table 4: Ablation study on the high-level planning. QI and
Table 5: Quantitative comparison
Q2 ask to identify the object category and the contact body part, respectively. We assess the accuracy by comparing the LLM's responses with labels we annotate. Note that the text input to LLMs may contain ambiguities; for example, the annotation is "hand" when the motion uses "right hand." We include Q1 Ace* and Q2 Ace excluding ambiguous text.
of text similarity. The text processed by high-level planning is more sim- ilar to text in HumanML3D [34] on
average, while addressing the distribu- tional gap significantly for challeng- ing out-of -iftdiittibuti descriptions, compared to text without planning.
[L of paramelers)
Q1 Ace
QT Acc
Acc
Q2Acc
GPI-4 [88]
0.801
0.997
0.733
0.961
Sim. [34]
Average
Out-ol- Dist.
Llama 2 (7B) [120]
0.073
0.147
0.436
0.689
Llama-2 (133)
0.232
0.319
0.682
0.853
wo planning
0.913
0,838
LJamm 2 (70B)
0.722
0.967
0.798
0.907
w/ planning
0.932
0.927
4.4 Ablation Study
Adaptability of High-Level Planning. Is our framework adaptable across different large language models (LLMs)? As illustrated in Table 4, our analysis contains two types of language models: GPT-4 [88], which is accessible through APIs and operates as a black box model; and Llama-2 [120], an open-source model. We demonstrate that language models with large parameters exhibit very high accuracy in responding to questions tailored to our prompts, validating the framework's 's adaptability.
Effectiveness of High-Level Planning with Low-Level Control. In consistency with Table Figure 7 offers qualitative comparison of text-to-motion results, contrasting results with and without L.L.-revised text descriptions. The comparison shows that motion generated without L.LMMenhanced descriptions often fails to correspond to the intended text, if the text is too challenging, e.g., not in the distribution of Human [34], which is used to train text-to-motion models. This underscores the LLM's critical role in bridging the distribution gap. In Figure 7(b). we visualize the CLIP [98] features of descriptions from HumanML3D, our raw annotations, and those processed by high-level planning. Quantitative evidence is provided in Table 5, Text processed through high-level planning demonstrates greater similarity to the HumanMML3D dataset. Additionally, we test on more challenging out-of- distribution text, selecting examples with an average cosine similarity to HumanML3D text of less than 0.85. High-level planning successfully rephrases these texts, significantly increasing their similarity. For example, in Figure 7(a), the text "Someone can be seen sitting on a yoga ball" has cosine similarity of 0.874 to the closest in-distribution text, while the rephrased text after planning, "A person is seated on an object," achieves similarity of 0.958.
Effectiveness of World Model. In the quantitative evaluation, we show that the performance of our framework is enhanced by the tailored design of our world model. Table I provides additional evidence of the effectiveness by integrating the proposed world model, as interaction correction within the InterDiff framework [148] in the interaction prediction task. This implementation demonstrates enhanced conditionality in the object dynamics modeling across two tasks, attributed to the vertex- level condition actions. Doing so effectively removes the whole-body complexity, most of which tends to be irrelevant to the interaction. Figure 8 further indicates that our vertex- condition can establish consistent interactions over time, while the condition by raw motion is not robust.
5
Conclusion
We tackle the task of text-guided 3D human-object interaction generation, aiming to accomplish this without relying paired text-interaction data. To this end, we present InterDreamer that decouples interaction dynamics from semanties, formulating the task as retrieval-ause -uggeetne generation and Markov decision process, where high-level planning and low-level control are introduced to generate semantically aligned human motion and initial object pose, while a world model is responsible for the object dynamics guided by the interaction. Our approach demonstrates effecctiveness in this novel task, suggesting its potential for various real-world applications.
The current utilization of dynamics modeling could be enhanced. A prospective improvement involves incorporating model-based learning techniques, which empower the agent to more effectively interact and learn a broader range of skills. The results may not be physically plausible and lead to artifacts in some cases, for example, foot skating. Hand poses are rough because they are missing from the dataset, but could be improved by integrating a physics simulator.
Limitations.
10

---

Acknowledgments
We thank Qiusi Zhan for supporting the implementation and evaluation of high-level planning. Jiangwei Yu and Xiyan Xu for their efforts in dataset annotation, and Xiang Li for the discussions. This work was supported in part by NSF Grant 2106825, NIFA Award 2020-67021- 60000-2222199, the IBM-IIlinois Discovery Accelerator Institute, the Toyota Research Institute, and the Jump ARCHES endowment through the Health Care Engineering Systems Center at Illinois and the OSF Foundation. This work used computational resources NCSA Delta and PTI Jetstream2 through allocations CIS220014, CIS230012, CIS230013, and CIS240311 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, and on TACC Frontera through the National Artificial Intelligence Research Resource (NAIRR) Pilot.
References
[1] Ahuja, C., Morency, L.P.: Language2pose: Natural language grounded pose forecasting. In: 3DV (2019) 3
[2] Athanasiou, N., Petrovich, M., Black, M.J., Varol, G.: Teach: Temporal action composition for 3d humans. In: 3DV (2022) 3
[3] Athanasiou, N., Petrovich, M., Black, M.J., Varol, G.: SINC: Spatial composition of 3d human motions for simultaneous action generation. In: ICCV (2023) 3
[4] Bae, J., Won, J., Lim, D., Min, C.H., Kim, Y.M.: Pmp: Learning to physically interact with
environments using part-wise motion priors. In: SIGGRAPH (2023) 3
[5] Barquero, G., Escalera, S., Palmero, C.: BeLFusion: Latent diffusion for behavior-driven human motion prediction. In: ICCV (2023) 3
[6] Barquero, G., Escalera, S., Palmero, C.: Seamless human motion composition with blended positional encodings. In: CVPR (2024)
[7] Bhatnagar, B.L.,, Xie, X., Petrov, I., Sminchisescu, C., Theobalt, C., Pons-Moll, G.: BEHAVE:
Dataset and method for tracking human object interactions. In: CVPR (2022) 1, 3, 6, 8, 22, 25
[8] Braun, J., Christen, S., Kocabas, M., Aksan, E., Hilliges, O.: Physically plausible full-body hand-object interaction synthesis. arXiv preprint arXiv:2309.07907 (2023) 3
[9] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning follow image editing instructions. In: CVPR (2023) 3
[10] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A.,
Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. In: NeurlPS (2020) 2,
[11] Cao, J., Liu, J., Kitani, K., Zhou, Y.: Multi-modal diffusion for hand-object grasp generation. arXiv preprint arXiv:2409.0456 04460 (2024)* 3
[12] Cao, Z., Gao, H.. Mangalam. K., Cai. Q.Z., Vo, M., Malik, J.: Long-term human motion prediction with scene context. In: ECCV (2020)* 3
[13] Casas, D., TrinnadTrinidad, M.: SMPLitex: A generative model and dataset for 3d human texture estimation from single image. In: BMVC (2023) 7
[14] Cen, Z., Pi, H., Peng, S., Shen, Z., Yang, M., Zhu, S., Bao, H., Zhou, X.: Generating human motion in 3d scenes from text descriptions. In: CVPR (2024)
[15] Cha, J.. Kim, J., Yoon, J.S., Back, S.: Tex12HOI: Text-guided 3d motion generation for hand-object interaction. In: CVPR (2024)
[16] Chao, Y.W., Yang, J., Chen, W., Deng, J.: Learning to sit: Synthesizing human-chair interac- tions via hicrarchical control. In: AAAI (2021)

---

[17] Chen, L.H., Zhang, J., Li, Y., Pang, Y., Xia, X., Liu, T.: HumanMAC: Masked motion completion for human motion prediction. In: ICCV (2023)
[18]
Chen, X., Jiang, B., Liu, W. Huang, Z., Fu, B., Chen, T., Yu, G.: Executing your commands via motion diffusion in latent space. In: CVPR (2023) 3
[19] Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., Song, S.; Diffusion policy: Visuomotor policy learning via action diffusion. In: RSS (2023) 5
[20] Christen, S., Hampali, S., Sener, F., Remelli, E., Hodan, T., Sauser, E., Ma, S., Tekin, B.: Diffh2o: Diffusion-based synthesis of hand-object interactions from textual descriptions. arXiv preprint arXiv:2403. 17827 (2024)* 3
[21] Cong, P., Dou, Z.W. Ren, Y., Yin, W., Cheng, K., Sun, Y., Long, X., Zhu, X., Ma, Y.: LaserHuman: Language-guidess scene-aware human motion generation in free environment. arXiv preprint arXiv:2403. 133307 (2024) 3
[22] Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F.: Context-aware human motion prediction. In: CVPR (2020) 3
[23] Cui, J., Liu, T., Liu, N., Yang. Y., Zhu, Y., Huang, S.: AnySkill: Learning open-vocabulary physical skill for interactive agents. In: CVPR (2024) 3
[24] Dabral, R., Mughal, M.H., Golyanik, V., Theobalt, C.: MoFusion: A framework for denoising- diffusion- motion synthesis. In: CVPR. pp. 9760-9770 (2023) 3
[25] Dai, S., Li, W., Sun, H., Huang, H., Ma, C., Huang, H., Xu, K., Hu, R.: InterFusion: Text- driven generation of 3d human-object interaction. In: ECCV (2024) 3
[26] Dai, W., Chen, L.H., Wang, J., Liu. J., Dai. B., Tang, Y.: Motionlem: Real-time controllable motion generation via latent consistency model. arXiv preprint 1iXx:2244411955 (2024) 3
[27] Daiya, D., Conover, D., Bera, A.: COLLAGE: Collaborative human-agent interaction genera- tion using hierarchical latent diffusion and language models, arXiv preprint arXiv:2409.20502
(2024) 3
[28] Diller, C., Dai, A.: CG-HOI: Contact-guided 3d human-object interaction generation. In: CVPR (2024) 2, 9,
[29] Diller, C., Funkhouser, T., Dai, A.: FutureHuman331: Forecasting complex long-term 3d human behavior from video observations. In: CVPR (2024) 3
[30] Fan, Z., Taheri, O., Tzionas, D., Kocabas, M., Kaufmann, M., Black, M.J., Hilliges, O.: ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In: CVPR (2023) 1,
[31] Feng, H., Ma, W., Gao, Q., Zheng, X., Xue, N., Xu, H.: Stratified avatar generation from sparse observations. In: CVPR (2024) 3
[32] Ghosh, A., Dabral, R., Golyanik, V., Theobalt, C., Slusallek, P.: IMoS: Intent-driven full-body
motion synthesis for human-object interactions. arXiv preprint arXiv:2212.07555 (2022) 3
[33] Ghosh, A., Dabral, R., Golyanik, V., Theobalt, C., Slusallek, P.: ReMoS: Reactive 3d motion synthesis for two-person interactions. arXiv preprint arXiv:2311. (2023) 3
[34] Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., Cheng, L.: Generating diverse and natural 3d human motions from text. In: CVPR (2022) 1, 3, 4, 8, 9, 10, 25
[35] Guo, C., Zuo, X., Wang, S., Cheng, L.: Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In: ECCV (2022) 3
[36] Guo, C., Zuo, X., Wang. S., Zou, S., Sun, Q., Deng, A., Gong, M., Cheng, L.: Action2motion: Conditioned generation of 3d human motions. In: ACMMM (2020) 3
[37] Han, S., Joo, H.: CHORUS: Learning canonicalized human-object spatial relations from unbounded synthesized images. In: ICCV (2023) 22
12

---

[38] Hassan, M., Ceylan, D., Villegas, R., Saito. J., Yang, J., Zhou, Y., Black, M.J.; Stochastic scene-aware motion prediction. In: ICCV (2021)3 3
[39] Hassan, M., Ghosh, P., Tesch, J.. Tzionas, D., Black, M.J.; Populating 3d scenes by learning human-scene interaction. In: CVPR (2021)
[40] Hassan, M., Guo, Y., Wang, T., Black, M., Fidler, S., Peng, X.B.: Synthesizing physical character-scene interactions. In: SIGGRAPH (2023)
[41] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurlPS (2020) 1
[42] Hou, Z., Yu, B., Tao, D.: Compositional 3d human-object neural animation. arXiv preprint
arXiv:2304.1400 14070 (2023) 3
[43] Huang, B., Li, C., Xu, C., Pan, L., Wang, Y., Lee, G.H.: Closely interactive human reconstruc- tion with proxemics and chysics-guided adaption. In: CVPR (2024) 3
[44] Huang, S., Wang, Z., Li, P., Jia, B., Liu, T., Zhu, Y., Liang, W. Zhu, S.C.: Diffusion-based generation, optimization, and planning in 3d scenes. In: CVPR (2023) 1,3
[45] Huang, Y., Taheri, O., Black, M.J., Tzionas, D.: InterCap: Joint markerless 3D tracking of humans and objects interaction. In: GCPR (2022) 1,3
[46] Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., Chen, T.: MotionGPT: Human motion as foreign language. In: NeurIPS (2023) 3, 6, 7, 8, 9,26
[47] Jiang, N., Liu, T., Cao, Z., Cui, J., Chen, Y., Wang, H., Zhu, Y., Huang, S.: CHAIRS: Towards full-body articulated humen-object interaction. In: ICCV (2023) 1, 1,2, 7, 7,
[48] Jiang, N., Zhang, Z., Li, H., Ma, X., Wang, Z., Chen, Y., Liu, T., Zhu, Y., Huang, S.: Scaling up dynamic human-scene interaction modeling. In: CVPR (2024)
[49] Jin, P., Li, H., Cheng, Z., Li, K., Yu, R., Liu, C., Ji, X., Yuan, L., Chen, J.: Local -etiodd motion diffusion model for text-to-motion generation. arXiv preprint 1tXXi:2407.10522 (2024)
3
[50] Karunratanakul, K., Preechakul, K., Suwajanakom, S., Tang, S.: GMD: Controllable human motion synthesis via guided diffusion models. In: ICCV (2023) 3
[51] Kaufmann, M.. Aksan, E., Song. Pece, F., Ziegler, R., Hilliges, O.: Convolutional autoen- coders for human motion infilling. In: 3DV (2020) 3
[52] Kim, H., Han, S., Kwon, P., Joo, H.: Zero-shot learning for the primitives of 3d affordance in general objects. arXiv preprint arXiv:2401.12978 (2024) 3
[53] Kim, J., Kim, J., Na, Joo, H.: ParaHome: Parameterizing everyday home activities towards 3d generative modeling of human-object interactions. arXiv preprint arXiv:2401.1022 (2024)
3,3
[54] Kim, J., Kim, J., Choi, S.: Flame: Free-form language-based motion synthesis editing. In: AAAI (2023) 3
[55] Kim, S.W., Zhou, Y., Philion, J., Torralba, A., Fidler, S.: Learning to simulate dynamic environments with gamegan. In: CVPR (2020) 5
[56] Kim, T., Saito, S., Joo, H.: NCHO: Unsupervised learning for neural 3d composition of humans and objects. In: ICCV (2023)
[57] Kong, H., Gong, K., Lian, D., Mi, M.B., Wang, X.: Priority-centric human motion generation in discrete latent space. In: ICCV (2023)
[58] Krebs, F., Meixner, A., Patzer, I., Asfour, T:: The kit bimanual manipulation dataset. In: Humanoids (2021) 3
13

---

[59] Kulkarni, N., Rempe, D., Genova, K., Kundu, A., Johnson, J., Foubey, D., Guibas, L.: NIFTY: Neural object interaction fields for guided human motion synthesis. arXiv preprint arXiv:2307.0751 (2023)
[60] Lee, J., Joo, H.: Locomotion-Actinn Cenirdtionnnttioo---aa-ae Action-Manipulation: Synthesizing human- interactions in complex 3d environments. In: ICCV (2023)*
[61] Lee. T. Moon, G., Lee, K.M.: Multiact: Long-term 3d human motion generation from multiple action labels. In: AAAI (2023)
[62] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H.,, Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nip tasks. In: NeurlPS (2020) 2, 3,4
[63] Li, B., Ho, E.S., Shum, H.P., Wang, H.: Two-person interaction augmentation with skeleton priors. In: CVPR (2024)
[64] Li. C., Chibane, J., He, Y., Pearl, N., Geiger, A., Pons-Moll, G.: Unimotion: Unifying 3d human motion synthesis and understanding. arXiv preprint arXiv:2409. 15904 (2024) 3
[65] Li. J., Clegg, A., Mottaghi, R., Wu, J., Puig, X.. Liu, C.K.: Controllable human-object interaction synthesis. arXiv preprint DEXXvv2312.03911 (2023) 2,
[66] Li, J., Wu, J., Liu, C.K.: Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG) 42(6), 1-11 (2023) 2, 3, 5, 7, 8, 22, 25, 26
[67] Li, L., Dai, A.: GenZI: Zero-shot 3d human-scene interaction generation. In: CVPR (2024)
[68] Li, Q., Wang, J., Loy, C.C., Dai, B.: Task-oriented human-object interactions generation with implicit neural representations. arXiv preprint arXiv:2303.1312 (2023)
[69] Liang, H., Zhang, W., Li, W. Yu, J., Xu, L.: InterGen: Diffusion- multi-human motion generation under complex interactions. arXiv preprint arXiv:2304.05684 (2023) 1
[70] Lin, J., Zeng, A., Lu, S., Cai, Y., Zhang, R., Wang, H., Zhang, L.:: Motion-X: large-scale 3d expressive whole-body human motion dataset. In: NeurlPS (2023) 3
[71] Liu, H., Zhan, X., Huang, S., Mu, T.J., Shan, Y.: Programmable motion generation for open-set motion control tasks. In: Proceedings of the IEBE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1399-1408 (2024) 3
[72] Liu, L., Hodgins, J.: Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Transactions on Graphics (TOG) 37(4), 1-14 (2018) 3
[73] Liu, S., Zhou, Y., Yang. J., Gupta, S., Wang, S.: Contactgen: Generative contact modeling for grasp generation. In: ICCV (2023)
[74] Liu, Y., Chen, C., Ding, C., Yi, L.: PhysReaction: Physically plausible real-time humanoid reaction synthesis via forward dynamics guided 4d imitation. arXiv preprint arXiv:2404.011 (2024) 3
[75] Liu, Y., Chen, C., Yi, L.: Interactive humanoid: Online full-body motion reaction synthesis with social affordance canonicalization and forecasting, arXiv preprint arXiv:2312.08983 (2023)
[76] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, MJJ.: SMPL: A skinned multi- person linear model. ACM transactions on graphics (2015) 3, 4, 25,
[77] Lu, S., Chen, I.H., Zeng, A., Lin, J., Zhang, R., Zhang, I.,, Shum, H.Y.: HumanTOOMATO: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978 (2023) 3
[78] Luo, Z.., Wang, J., Liu, K., Zhang, H., Tessler, C., Wang, J., Yuan, Y., Cao, J., Lin, Z.,, Wang, F., et al.: SMP1.Olmmpics: Sports environments for physically simulated humanoids. arXiv preprint arXiv:2407.000 00187 (2024)* 3
14

---

[79] Ma. J., Chen, X., Bao, W., Xu, J., Wang, H.: MADiff: Motion- aware mamba diffusion models for hand trajectory prediction on egocentric videos. arXiv preprint urXiv:2409.02633 (2024)
[80] Ma, J., Xu, J., Chen, X., Wang, H.: Dift-IP2D: Diffusion-based hand-object interaction
prediction egocentric videos. arXiv preprint arXiv:2405.04370 (2024) 3
[81] Ma, S., Cao, Q. Zhang, J., Tao, D.: Contact-aware human motion generation from textual descriptions. arXiv preprint arXiv:2403. (2024) 3
[82] Van der Maaten, I.., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(1)) (2008) 8
[83] Mahmood, N., Ghorbani, N., Troje, N.F., Pons-Moll, G., Black, M.J.: AMASS: Archive of motion capture surface shapes. In: ICCV (2019) 3,3
[84] Mandery, C., Terlemez, O., Do, M., Vahrenkamp, N., Asfour, T.: The kit whole-body human motion database. In: ICAR (2015)
[85] Mandery, C., Terlemez, O., Do, M., Vahrenkamp, N., Asfour, T:: Unifying representations and large-scale whole-body motion databases for studying human motion. IEEE Transactions Robotics (2), 796-809 (2016)
[86] Mao, W., Liu, M., Salzmann, M.: Generating smooth pose sequences for diverse human motion prediction. In: CVPR (2021) 3
[87] Merel, J., Tunyasuvunakool, S., Ahuja, A., Tassa, Y., Hasenclever, L., Pham, V., Erez, T.,
Wayne, G., Heess, N.: Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG) (9)4), 39-1 (2020)
[88] OpenAI: ChatGPT, https: //cht/://////t/t/ openai com/ (2023) 2, 3, 8, 9, 10, 23, 26
[89] Pan, L., Wang, J., Huang, B., Zhang, J., Wang, H., Tang, X., Wang, Y.: Synthesizing physically plausible human motions in 3d scenes. arXiv preprint arXiv:2308.0903: (2023) 3
[90] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove. S.: DeepSDF: Learning continu- ous signed distance functions for shape representation. In: CVPR (2019)
[91] Peng, X., Xie, Y., Wu, Z., Jampani, V., Sun, D., Jiang. H.: HOI-Diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:2312.06553 (2023) 1, 3,9, 21
[92] Petrov, I.A., Marin, R., Chibane, J., Pons-Moll, G.: Object pop-up: Can we infer 3d objects and their poses from human interactions alone? In: CVPR (2023) 3
[93] Petrovich, M., Black, M.J., Varol, G.: Action-conditioned 3d human motion synthesis with transformer vae. In: ICCV (2021) 3
[94] Petrovich, M., Black, M.J., Varol, G.: TEMOS: Generating diverse human motions from textual descriptions. In: ECCV (2022) 3
[95] Petrovich, M., Black, M.J., Varol, G.: TMR: Text-to-motion retrieval using contrastive 3d human motion synthesis. In: ICCV (2023)
[96] Raab, S., Leibovitch, I., Li, P., Aberman, K., Barkine-Hornung, O., Cohen-Or, D.: MoDi: Unconditional motion synthesis from diverse data. In: CVPR (2023) 3
[97] Raab, S., Leibovitch, I., Tevet, G., Arar, M., Bermano, A.H., Cohen-Or, D.: Single motion diffusion. arXiv preprint arXiv:2302.05905 (2023) 3
[98] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML (2021) 8. 10
[99] Razali, H., Demiris, Y.: Action-conditioned generation of bimanual object manipulation sequences. In: AAAI (2023) 3
15

---

[100] Rempe, D., Luo, Z., Bin Peng, X., Yuan, Y., Kitani, K., Kreis, K., Fidler, S., Litany, O.: Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In: CVPR (2023)
3
[101] Rombach, R., Blattmann, A., Lorenz, D., Esser, Ommer, B.; High-resolution image synthesis with latent diffusion models. In: CVPR (2022) 22
[102] Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics 36(6) (2017) 7
[103] Seo, Y., Hafner, D., Liu, H., Liu, F., James, S., Lee, K., Abbeel, P.: Masked world models for visual control. In: CoRL (2023) 5
[104] Shafir, Y., Tevet, G., Kapon, R., Bermano, A.H.: Human motion diffusion as generative prior. arXiv preprint arXiv:2303.014: (2023) 3
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning [105]
using nonequilibrium thermodynamies. In: ICML (2015)
[106] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
[107] Song, W., Zhang, X., Li, S., Gao, Y., Hao, A., Hou, X., Chen, C., Li, Qin, H.: HOIAnimator: Generating text-prompt human-object animations using novel perceptive diffusion models. In: CVPR (2024) 3,3
[108] Starke, S., Zhang, H., Komura, T., Saito, J.: Neural state machine for character-scene interac- tions. ACM Trans. Graph. 38(6), 209-1 (2019)
[109] Starke, S., Zhao, Y., Komura, T., Zaman, K.: Local motion phases for learning multi-contact
character movements. ACM Transactions on Graphics (TOG) 39(4), 54-1 (2020)
[110] Sun, J., Chowdhary, G.: Towards globally consistent stochastic human motion prediction via motion diffusion. Xiv preprint arXiv:2305. 12554 (2023) 3
[111] Taheri, O., Choutas, V., Black, M.J., Tzionas, D.: GOAL: Generating 4d whole-body motion for hand-object grasping. In: CVPR (2022)?
[112] Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: A dataset of whole-body human grasping of objects. In: ECCV (2020)
[113] Tang, J., Wang, J., Ji, K., Xu, L., J., Shi, Y.: A unified diffusion framework for scene-aware human motion estimation from sparse signals. In: CVPR (2024)
[114] Tendulkar, P.. Surís, D., Vondrick, C.: FLEX: Full-body grasping without full-body grasps. In: CVPR (2023)>
[115] Tessler, C., Guo, Y., Nabati, O., Chechik, G., Peng, X.B.: MaskedMimic: Unified physics-
based character control through masked motion inpainting, arXiv preprint arXiv:2409.1439 (2024)
[116] Tevet, G., Gordon, B., Hertz, A., Bermano, A.H., Cohen-Or, D.: Motionclip: Exposing human motion generation to clip space. In: ECCV (2022)
[117] Tevet, G., Raab, S., Cohan, S., Reda, D.. Luo, Z., Peng, X.B., Bermano, A.H., van de Panne, M.: CLoSD: Closing the loop between simulation and diffusion for multi-task character control. arXiv preprint arXiv:2410.03441 (2024)
[118] Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., Bermano, A.H.: Human motion diffusion model. arXiv preprint mtXiv:2209.14999 (2022) 5, 6, 7,9
[119] Tian, J., Yang, I.., Ji, R., Ma, Y., Xu, I., Yu, J., Shi, Y., Wang, Gaze-guided hand-object interaction synthesis: Benchmark and method. arXiv preprint arXiv:2403.16169 (2024)
16

---

[120] Touvron, H., Martin, L.. Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale. S., al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 (2023) 2, 3, 9, 10
[121] Turk, A.M.: Amazon mechanical turk. Retrieved August 17. 2012 (2012) 23
[122] Wan, W., Dou, Z., Komura, T., Wang, W., Jayaraman, D., Liu, L.: Ticontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:231 1.17135 (2023)
[123] Wan, W., Yang, L., Liu, L., Zhang, Z., Jia, R., Choi, Y.K., Pan, J., Theobalt, C., Komura, T., Wang, W.: Learn predict how humans manipulate large-sized objects from interactive motions. IEEE Robotics and Automation Letters (2022) 3
[124] Wang, J., Hodgins, J., Won, J.: Strategy and skill learning for physics- based table tennis animation. In: SIGGRAPH (2024)
[125] Wang, J., Xu, H., Xu, J., Liu, S., Wang, X.: Synthesizing long-term 3d human motion and interaction in scenes. In: CVPR (2021) 3
[126] Wang, J., Rong Y., Liu, J., Yan, S., Lin, D., Dai, B.: Towards diverse and natural scene-aware 3d human motion synthesis. In: CVPR (2022)
[127] Wang, J., Yan, S., Dai, B., Lin, D.: Scene-aware generative network for human motion synthesis. In: CVPR (2021)3
[128] Wang, X., Li, G., Kuo, Y.L., Kocabas, M., Aksan, E., Hilliges, O.: Reconstructing action- conditioned human-object interactions using commonsense knowledge priors. In: 3DV (2022) 3
[129] Wang, Y. Lin, J., Zeng, A., Luo, Z., Zhang, J., Zhang, L.: PhysHO1: Physics-based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.04333 (2023) 3
[130] Wang, Z., Chen, Y., Liu, T., Zhu, Y., Liang, W., Huang, S.: HUMANISE: Language-
conditioned human motion generation in 3d scenes. In: NeurIPS (2022) 1,3
[131] Wang, Z. Li, D., Jiang, R.: Diffusson models in 3d vision: A survey, arXiv preprint
arXiv:2410.04733. (2024) 1
[132] Wang, Z., Wang. J., Lin, D., Dai, B.: InterControl: Generate human motion interactions by controlling every joint, arXiv preprint 1rXiv:2311.15864 (2023) 3
[133] Wei, D., Sun, X., Sun, H., Li, B., Hu, S., Li, W., Lu, J.: Understanding text-driven motion synthesis with keyframe collaboration via diffusion models. arXiv preprint 1337 (2023)
[134] Wei, J., Wang. X., Schuurmans, D.,, Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., al.: Chain-of-thought prompting elicits reasoning in large language models. In: NeurlPS (2022) 2
[135] Wu, P., Escontrela, A., Hafner, D., Abbeel, P., Goldberg, K.: Daydreamer: World models for physical robot learning. In: CoRL (2023) 5
[136) Wu, Q., Shi, Y., Huang, X., Yu, J., Xu, L., Wang, J.: THOR: Text human-object interaction diffusion via relation intervention. arXiv preprint arXiv:2403.1 1208 (2024) 1,3
[137] Wu. Q.. Dou, Z., Xu, S., Shimada, S., Wang, C., Yu, Z., Liu, Y., Lin, C., Cao. Z., Komura, T., et al.: DICE: End-to-end deformation capture hand-face interactions from a single image. arXiv preprint arXiv:2406. 17988 (2024)* 3
[138] Wu, S., Liu, Y., Li, L., Bi. M., Zeng. W., Yang. X.: HIMO: A new benchmark for full-body human interacting with multiple objects. In: ECCV (2024) 3
[139] WI, Y., Wang. J., Zhang, Y., Zhang, S., Hilliges, O., Yu, F., Tang, S.: SAGA: Stochastic whole-body grasping with contact. In: ECCV (2022) 3
17

---

[140] Wu, Z., J., Liu, C.K.: Human-objec interaction from human-level instructions. arXiv preprint arXiv:2406. 17840 (2024) 1,
[141] Xiao, Z., Wang, T., Wang, J., Cao, J., Zhang, W., Dai, B., Lin, D., Pang, J.: Unified human- scene interaction via prompted chain-of-contacee arXiv preprint arXiv:2309.07916 (2023) 3
[142] Xiao, Z., Wang, T., Wang. J., Cao, J., Zhang, W., Dai, B., Lin, D., Pang. J.: Unified human- scene interaction via prompted
In: ICLR (2024) 3
[143] Xie, X., Bhatnagar, B.L., Pons-Moll, G.: Chore: Contact, human and object reconstruction from a single rgb image. In: ECCV (2022) 3
[144] Xie, X., Lenssen, J.E., Pons-Moll, G.: InterTrack: Tracking human object interaction without object templates. arXiv preprint arXiv:2408. 13953 (2024)
[145] Xie, Y., Jampani, V., Zhong, L.. Sun, D., Jiang, H.: OmniControl: Control any joint at any time for human motion generation. arXiv preprint arXiv:2310.0885 (2023)
[146] Xie, Z., Starke, S., Ling, H.Y., van de Panne, M.: Learning soccer juggling skills with layer-wise mixture-of-experts. In: SIGGRAPH (2022) 3
[147] Xie, Z., Tseng, J., Starke, S., van de Panne, M.. Liu, C.K.: Hierarchical planning and control for box loco-manipulation. arXiv preprint arXiv:2306.09532 (2023) 3
[148] Xu, S., Li, Z., Wang, Y.X., Gui, L.Y.: InterDiff: Generating 3d human-object interactions with physics-informed diffusion. In: ICCV (2023) 3, 6, 7, 8, 9, 10, 25
[149] Xu, S., Wang, Y.X., Gui, Y.: Diverse human motion prediction guided by multi-level spatial-temporal anchors. In: ECCV (2022) 3
[150] Xu, S., Wang, Y.X., Gui, L.: Stochastic multi-person 3d motion forecasting. In: ICLR (2023)
3
[151] Xu, X., Joo, H., Mori, G., Savva, M.: D3D-HOI: Dynamic 3d human-object interactions from videos. arXiv preprint arXiv:2108.08420 (2021) 3
[152] Xu, 7.,, Chen, Q., Peng. Y., Liu, Y:: Semantic-aware human object interaction image generation. In: ICML (2024) 3
[153] Xue, K., Seo, H.: Shape conditioned human motion generation with diffusion model. arXiv preprint 206700000000077788 (2024) 3
[154] Yang, C., Kang, C., Kong, K., Oh, H., Kang, S.J.: Person in Place: Generating associative skeleton-guidance maps for human-object interaction image editing. In: CVPR (2024) 3
[155] Yang, J., Niu, X., Jiang, N., Zhang, R., Huang, S.: F-HOI: Toward fine-grained semantic- aligned 3d human-object interactions. arXiv preprint arXiv:2407.1244 (2024) 2,
[156] Yang, Y., Zhai, W., Luo, H., Cao, Y., Zha, Z.J.: LEMON: Learning 3d human- interaction relation from 2d images. In: CVPR (2024) 3, 22
[157] Yang, Y., Zhai, W., Wang, C., Yu, C., Cao, Y., Zha, ZJJ: EgoChoir: Capturing 3d human-object
interaction regions from egocentric views. arXiv preprint arXiv:2405.13650 (2024) 3
[158] Yang, Z., Yin, K., Liu, L.: Learning to chopsticks in diverse gripping styles. ACM
Transactions on Graphics (TOG) 41(4). 1-17 (2022) 3
[159] Yao, H., Song, Z., Zhou, Y., Ao, T., Chen, B., Liu, L.: MoConVQ: Unified physics-based motion control via scalable discrete representations. arXiv preprint arXiv:2310. 101198 (2023)
[160] Yazdian, P.J., Liu, E., Cheng, L., Lim, A.: MotionScript: Natural language descriptions for expressive 3d human motions. arXiv preprint 1tXiv:2312.12633 (2023)
[161] Ye, Y., Li, X., Gupta, A., De Mello, S., Birchfield, S., Song, J., Tulsiani, S., Liu, S.: Affordance diffusion: Synthesizing hand-object interactions. In: CVPR (2023) 3
18

---

[162] Yuan, W., Shen, W., He, Y., Dong, Y., Gu, X., Dong, Z., Bo. L., Huang, Q.: Mogents: Motion generation based on spatial-temporal joint modeling. In: ECCV (2024)
[163] Yuan, Y., Kitani, K.: DLow: Diversifying latent flows for diverse human motion prediction. In: ECCV (2020) 3
[164] Zhang. C., Liu, Y., Xing, R., Tang, B., Yi, L.: Core4d: 4d human-object-hume human interaction dataset for collaborative object rearrangement. arXiv preprint arXiv:2406. 19353 (2024)
[165] Zhang, H., Christen, S., Fan, Z., Zheng, L., Hwangbo, J., Song, J., Hilliges, O.: ArtiGrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. arXiv preprint arXiv:230.03899 (2023)* 3
[166] Zhang. J.Y., Pepose, S., Joo, H., Ramanan, D., Malik, J., Kanazawa, A.: Perceiving 3d human-object spatial arrangements from a single image the wild. In: ECCV (2020) 3
[167] Zhang, J., Zhang, Y., An, L., Li, M., Zhang, H., Hu, Z., Liu, Y: ManiDext: Hand-object ma- nipulation synthesis via continuous correspondence embeddings and residual- -giiedd diffusion. arXiv preprint arxi::2409 .09300 (2024) 3
[168] Zhang, Zhang, Y., Cun, X., Zhang, Y., Zhao, H., Lu, H., Shen, X., Shan, Y.: Generating human motion from textual descriptions with discrete representations. In: CVPR (2023) 3
[169] Zhang, J., Luo, H., Yang, H., Xu, X., Wu, Q. Shi, Y., Yu, J., Xu, L., Wang. J.: NeuralDome: A neural modeling pipeline on multi-view human- object interactions. In: CVPR (2023) 1,3
[170] Zhang, J., Zhang, J., Song, Z., Shi, Z., Zhao, C., Shi, Y., Yu, Xu, L., Wang, J.: Hoi-m" 3: Capture multiple humans and objects interaction within contextual environment. In: CVPR (2024) 3,3
[171] Zhang, L.., Rao, A., Agrawala, M.: Adding conditional control text-to-image diffusion models. In: ICCV (2023) 22
[172] Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., Liu, Z.: MotionDiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001 (2022) 3, 5,
6,7, 7.9 9
[173] Zhang, M., Guo, X., Pan, L., Cai, Z., Hong, F., Li, H., Yang, L., Liu, Z.: ReMoDiffuse: Retrieval-augmented motion diffusion model. In: ICCV (2023) 6,
[174] Zhang, M., Li, H., Cai, Z., Ren, J., Yang. L., Liu, Z.: Finemogen: Fine-grained spatio-temporal motion generation and editing. In: NeurlPS (2024) 3
[175] Zhang, W., Dabral, R., Leimkühler, T., Golyanik, V., Habermann, M., Theobalt, C.: ROAM: Robust and object-aware motion generation using neural pose descriptors. arXiv preprint arXiv:2308. 12969 (2023)*
[176] Zhang, X., Bhatnagar, B.I., Starke, S., Guzov, V., Pons-Moll, G.: COUCH: Towards control- lable human-chan interactions. In: ECCV (2022)
[177] Zhang, X., Bhatnagar, B.L., Starke, S., Petrov, I., Guzov, V., Dhamo, H.. Pérez-Pellitero. E., Pons- Moll, G.: FORCE: Dataset and method for intuitive physics guided human-object interaction. arXiv preprint arXiv:2403.I 1237 (2024)
[178) Zhang, Y., Huang, D., Liu, B., Tang. S., Lu, Y., Chen, I.., Bai, I.., Chu, Q., Yu, N., Ouyang, W.: Motiongpt: Finetuned lims are general-purpose motion generators. arXiv preprint arXiv:2306. 10900 (2023)>
[179] Zhang, Z., Liu, R., Aberman, K., Hanocka, R.: TEDi: Temporally-entangled diffusion for long-term motion synthesis. arXiv preprint arXiv:2307.15043 (2023)
[180] Zhao, C., Zhang, J., Du, J., Shan, Z.., Wang, J., Yu, J., Wang, J., Xu, L.: I'M HOI: Inertia-aware monocular capture of 3d human-object interactions. In: CVPR (2024) 1,3
19

---

[181] Zhao, K., Li, G.. Tang, S.: DART: A diffusion- based autoregressivve motion model for real-time text-driven motion control. arXiv preprint arXiv:2410.05260 (2024)
[182] Zhao, K., Wang, S., Zhang, Y., Beeler, T., Tang, S.: Compositional human-scene interaction synthesis with semantic control. In: ECCV (2022) 3
[183] Zhao, K., Zhang, Y., Wang, S., Beeler, T., Tang, S.: Synthesizing diverse human motions in 3d indoor scenes. In: ICCV (2023) 3
[184) Zheng, J., Zheng, Q. Fang, L.., Liu, Y., L.: CAMS: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. In: CVPR (2023)
[185] Zhong, L., Xie, Y.. Jampani, Sun, D., Jiang, H.: SMooDi: Stylized motion diffusion model. arxiv preprint arXiv:2407.1278 127833 (2024) 3
[186] Zhou, K., Bhatnagar, B.L., Lenssen, J.E., Pons-Moll, G.: Toch: Spatio-temporal object-to-hand correspondence for motion refinement. In: ECCV (2022)
[187] Zhou, W., Dou, Z., Cao, Z., Liao, Z., Wang, J., Wang, W., Liu, Y., Komura, T., Wang, W., Liu, L.: EMDM: Efficient motion diffusion model for fast, high-quality motion generation. arXiv preprint ar 1.002556 (2023)* 3
20

---

In this Appendix, we include additional method details and experimental results: (i) We provide demo videos in the website, explained in Sec. A. (ii) We present additional details of interaction retrieval, world model, and optimization in Sec. B. (iii) We provide implementation details and additional information on the experimental setup in Sec. C. (iv) We provide additional qualitative experiments in Sec. D. (v) We provide some failure cases in Sec. E. (vi) We discuss the potential negative societal impact in Sec. F.
A Visualization Video
Beyond the qualitative results presented in the main paper, we include two demo videos that offer more detailed visualizations of the task, further illustrating the efficacy of our approach. These demos highlight
(i) We conduct qualitative comparison of our approach with existing text-to-HOI work [28, 91] within the framework of supervised learning. Note that as our setting contains no text supervision, it is unfair compare work with these approaches; here for additional reference. We evaluate our method by directly testing our trained model on the annotated data available from their websites, specifically retrieving their generated videos for direct comparison. Remarkably, even without training on these datasets, our method generates results that demonstrate high-quality interactions. It is even capable of synthesizing complex interactions involving dynamically-chamging contact, such as the handover and throwing of objects.
include the comparison
B
Additional Details of Methodology
B.1 Low-Level Control
In this section, we provide additional details on the retrieval based on handcraft rules, which is straightforward and does not require training. We also investigate learning-basee method without relying on handcrafted designs.
Handeraft Interaction Retrieval. In Sec. 3.2 of the main paper, detail the construction of the interaction database and emphasize the use of body parts and object categories as keys to fetch semantically-aligned contact maps. Same as the main paper, we define a contact map as a list of K index pairs of vertices {dd, d)) [ This section delves into the methodology for outlining an optimization process to generate the object initial pose 81 given contact maps and the initial human pose a1, and choose one pose based on predefined metric.
Let un, [do] denote the vertex on the surface of the object, and Vo, [dh] represent the corresponding vertex on the human body surface, where d, and d, are the indices vertices. Specifically, to optimize 81, the overall optimization objective is given by,
Eapt- = Aat Ent. + Aeone Eront + Apone Epeney
where Aat. Acont, and Xpene are hyperparameters.
Fitting Loss. To project a contact map to an object pose, we minimize the L,2 distance between the human vertices and the object vertices indicated by the contact map,
Eit.
||
[d.] Un, [da]ll2-
Contact Loss. We leverage contact loss to encourage the body part contact the object surface in addition to the fitting loss,
Ecout =
Σ
min || [do] - un (dh]lla. d
d,ET
where T= (dyl mina, I1oo, (d.)
un [dw]lla S }) includes the index of the body part that is close to the object vertex v., [d,), where e is a hyperparameter, d,, and d, are vertex indices for human and object, respectively, in addition the contact map ((d), d)){
Penetration Loss. Given the signed-distance field of the human pose sdf . we employ a penetration loss to penalize the boject interpenetration,
21

---

Pepene
Σ de
min(sdff, (No, [do]); 0)-
The metric for determining the final pose selection is given by the expression 1 (peue 0)/ Econt We sample a pose from the set generated by all contact maps, with higher metrics corresponding to higher selection probability.
Learning-Rasee Interaction Retrieval. Our interaction retrieval can also achieved by integrating knowledge from several learning-based algorithms. Although being more complicated, the retrieval can be done without handcraft rules. Our framework can be divided into following. (i) Given the text prompt and the initial human pose d1, we synthesize corresponding images via Stable Diffusion [101]. (ii) We follow [37] filter out images with low quality in interaction.
(iii) An off-
the-shelf model LEMON [156] is to employed to obtain object affordance and human contact, given the generated image paired with human pose d1 and object template. The output {()), (1.),
indicates the contact vertex indexes of human and object respectively, and the output T₁ indicates the estimated object translation, which is used for initialization in the optimization. (iv) To acquire the object pose, we utilize the optimization to minimize the Chamfer distance between the human vertices and the object vertices, indicated by the contact vertices obtained in the last step.
Eac Σ min Vo [t-1 U/, 12
(5)
B.2
World Model
World Model for Initial States. In the particular instance where the timestep & = 1, the state vector 81 encapsulates a single frame. Consequently, we employ two distinct models for dynamics prediction. For predictions originating from the initial state, the history motion encompasses a single timestep H In contrast, for predictions for subsequent states, the historical interval covering m timesteps, where m denotes the frame count per segment.
World Model for Implicit Geometry Encoding. The input to the world model includes the trajectories of the human vertices (represented by red small spheres in the top-right of Figure of the main paper), along with the vertex-to-object surface vectors. By adding the vertex-to-objeet surface vectors to human vertices, one can easily obtain the object vertices (shown as blue small spheres in the top-right of Figure 2 of the main paper). Though the network of the world model does not receive this information directly, it can learn to combine these features to derive it when needed.
World Model for Novel Objects. The world model employs "contact vertices" as input, which includes features derived from the object distance field. These features encompass the human vertex- to-object surface distance and the human vertex velocity relative to the nearest object vertex as introduced in Sec. 3.3 of the main paper, inherently including information related to the object's s shape. This encoding is consistently applied to both training objects from the BEHAVE [7] dataset and novel objects from the CHAIRS [47] and OMOMO [66] datasets.
World Model for Non-Contact Objects. The network can process inputs without contact conditions by adopting an approach similar to ControlNet [171]. The network comprises two components: G that operates without contact vertex conditions, applicable in scenarios where no contact occurs, and F. akin to the control components in ControlNet, which incorporates contact vertex conditions into the object trajectory when contact is present. When there is contact, only the unconditional network is utilized. The model is aware of past object motion and thus needs to learn how human interaction affects the object's state. This includes understanding how objects follow contact positions or normals by F, as well as how they move without contact by g. With the no-contact object motion data provided by BEHAVE [7], the world model (more specifically, 9) learns to infer whether the object should free-fall based on its previous velocity or remain on the ground based on its height.
B.3 Optimization
We provide detailed formulations of optimization objectives, complementing Sec. 3.4 in the main paper. For efficiency, we perform optimization sparsely only the loss is above a threshold to improve the efficiency. Specifically, given the reference interaction sequence {h. and (o.)
22

---

Instruciion:: Segmeration and léen Rlction of Semandic Units Breek down the intenction sey ence imo smalee sibeeguentes, each consising tens severa hundreds of frames Pvery aesequence rspessents single sema ilk unit capturing a distnct full interaction
Arnolation Subsequenc: mnniatt sach subuequence wsig templase 1s passnn
merectinn) soms object] (ssing body par partsjAstart, framokinn, frame"
For eomple sequence traskenn with hand towards em.5552255 10 person opens book with both tands, fllss through the pager using ther fingers, and then points to line text.1504255"
sentiihh vuboquuacee could armoteeed penan holds
thae hand, lowers right amm. and puits the
video, come-EEMMEE DateOr, Siboi, www.bbobtybbbeee bowttry
6764
Sabmit
Figure A: We use Amazon Mechanical Turk [121] build an annotation platform. We provide instructions to guide the annotator to split long sequence into several short sub-sequences with their start and end frames, and then annotate each sub-sequence. We inform annotators that our collected data are used for text-motion generation when they accept the job.
of arbitrary length L, derived from previous steps, apply gradient descent optimize human pose sequence (h and object pose sequence {o 2 using the loss function,
Eopt Ast Eatt + Avel Elyel + Acont Fecont + Fipnne
(6)
where Ast. Avel. Acont and Apene are hyperparameters.
Fitting Loss. We minimize the L1 distance between the input and the reference,
Emt > - h.ll1 + ||ا - o:ll2).
(7)
Velocity Loss. We leverage velocity loss to smooth the interaction sequence,
4-1
hi + +1o of Evel Σ (1
(8)
Contact loss. We leverage contact loss to encourage the body part to contact the object surface, if they are close to each other the initial interaction.
Econt
Σ E.ET.
min |/; [do] un; [da]]l2,
do
where un; [dy] denotes the vertex on the human body surface, and no [do] represents the correspond-
ing vertex on the surface the object. And T, {dxl ming, || [d.] vh, [d,]]2 S <) includes the index of reference human vertex Uh, [d,] that is close to the reference object vertex [dol, where e a hyperparameter, d, and d, are vertex indices for human and object, respectively.
Penetration Loss. Given the signed-distance field of the human pose sdfn;, we employ a penetration loss penalize the body-object interpenetration,
E Pene
Σ
Σ
min(sdlfn; (wo; [do]), 0).
(10)
i=1
C Additional Details of Experimental Setup
Datasets. We include screenshot of our annotation platform in Figure Our annotations are further diversified by GPT-4 [88]. The prompt used for this purpose is: I'm going to give
23

---

You will be given sentence that deserihes the interaction between person und an object. You will need to extract information from the description and anwer three questions based on the following rules und examples.
Extract the eategory the object which the person internets,
Answers should be sclected from the following objects: [trshhon, monioor boxlarge, stool, boxsmall, backpack, bodiogg, platiecontainer, tablesquare, yogaball, yogamar, toolbox, chairwood, chairblack, boxmedium, bestiny, suftcase. tablesmall).
[sturt rules]
Pieli the most similar object from the list if the description is not in the list.
rules]
2. Infer the body part that contacts with the object at the beginning of the interaction.
Answers should be selected from the following body parts: (right handl hand, back of the kip. appenlay leg, apper back, right fromt bady back
sbouldor, no contect).
[start of rules]
4f description not specify which hand foot, randomll choose "lefi" or 'right".
[end of culos]
3. Simplify and modifyy the description.
[sturt rukks]
Thee subject the sentence should "a person":
*Modify werds that indicate interaction with simple, cleur verb. For cxample, change "mointain possession" "hold" or "grasp" change "move foree" to "pull" "yush."
Eliminatte umnecessary verbs, and convert some obscure verbs into common verbs. For example, change "A person elenches box with their left right hand." "A person transfers box from their hand
hand
right hand,", ", remove the unnoecssary vorb 'hold this sentence. And "A person walks leisurely in clockwise dircction with an object slung over their right shockder," to "A person walks a clockoise circle while holding object their right shoulder."
passes
oIf the sentence ambiguous, such "one hand," choose the most likely answer from "left hand," "right hand," or "both hands"
If contact is mado with hands or feet, explicilly mention the body part in the description.
with "an object" "somethinng" unless the replaceneett will change the mosning the verb. For example. keep "basketball" in "play baskectball"" but change "a chair" to "an object" in "lift chair".
*Romember replace speeifie object in the
*Ronsenber remore unieppertant details that are not relovact the sction Rscif, espocially when rolated to body part "back" and "shookier" such as changing "He holds big object." la person holds object with right hand." ''A person with backpack on back
walks
forward." " to "a penson tums left and walks forward." and "A person with backpack slung over the shoulder walks forward." " to "A person walks forward with object their shoulder." (ond of rules]
[start of exampless]
person used their foot apply force the medium sized box, Answer: bosmedium/left foot penson kicks object with left foot.
Somoone moves the box by pushing and ling Answer boddigeiright handaa person altenitely pushes and pulls an object with right hand. He touches the bottom of the box. Answer: bostinyfright handaa person touches an object with right hand.
elenshes the upper body part the chuir. Amswr: chiirwoodrrdhtt handja person
an object right hand.
A medium--ize box pioked up by a mas black fimily using left hand. Answer: boxmediunjleft handla person picks up an object with left hand,
He walks clockewise citcle, and sits weoden chuir. Answer: chniwoodffack of the hiplA person walks in clockwise cirole and sits something.
He with a yogibll his hand, walking forward Anewer yogsballright handja person an object with right hand walking forward. A parson holds a backpack while walking forward. Answer: backpackfright hand person holds an object with right hand while walkinng forward. chaisblack(ssck. lA person
person
chair.
something
A parson lies face down on the chair. Asswer chairwoodd front bodyAA person lies face down on something. box possessed by individual. Answer: boxtiny left handlA person holds object with their left hand. She is playing basketball, Answer booketball) git hand& person playing basketball their right hand. (end of examples]
The tleccrptiion of human-object interaction:
A person holds the rime hav with left hand and passes it to the risht then uses both hands hold the recovite sites of the haw and lifts abvive their head. and nulls the choir to side.
A necionn grabs the sest cushion chair with their
A person orahn the bockrest af'a chuir and rushes it boolowedss
their right hand dong the saitaase the ground. A nemoon's butlocks are sminee the small table. and then they nush the smell table buckward.
pepooon bends the woist and
A personn uses richt hand to lift the recrangular object in from of them to chest height.
A. pecionn places
goound, squatt A person puts down a bicknack that was shonp over their right shoulder onto the eocund.
left bund the rectangular obtect that
and then stands up again.
personn insers their right hand thoough the shoulder staps of backnack und takos two stens hackward,
A person starts by using their richs hand to plae the rectanguler obiect horizontslyy by in front of their evess then they use their keft band to take over the
rectangular objecs, and finally, the entre person turns to the left to fice the bock.
Please outpot
following "Answert Answer Answer 2) Answer 3"
The
Answer bastinglleh feft handla person fransfers an obiect to the right with left hand, then lifes abose their head with both hands.
Answer chairwood left hardle person pulls an object to the lelt with their left hand.
Answer chainooodlight handla person pushes object backwards with their right hand.
Answer sibsassrich hanila persun hands and drags an object on the ground with their right Fansl
Answer: tablecnalljhcck of the hina penson against object than pushes badloaadd
Answer buxlonginght handla person lifl an object to chest height with their right hand.
Answer: boxlongleet handaa penson squats down and stands up while touching objeet with their left hand,
Answer backpackiright shoulderis person puts object from cheir right shoulder omta the goound.
Answer backmecklrsht handla nerson puts on buckruck with their right hand and take two steps buckward.
Answer: boxlonginght handja person places object front of their eyes using their right hand, then transfers. the object to their left hand. and finally
[urn Iunckl kil.
Figure B: Full log of our high-level planning.
24

---

person throws an object with right hand towards the ground.
A person is carrying an object on the shoulder.
Text-to-Motion
Interaction Retrieval
Text-to-Motion
Interaction Retrieval
Figure C: Qualitative results from the interaction retrieval. We demonstrate that our learning-basee interaction retrieval can extract diverse and realistic interactions.
you a description, and I would like to have three rewritten sentences with varying degrees of complexity, following the example: "..." The input text is "..." Please give me three texts that vary in complexity but keep the meaning of the sentence the same. This results in (i) less complexity: someone holds backpack and steps left; (ii) middle complexity: a person holds a backpack in front of them with both hands and takes step the left; (iii) more complexity: with both hands, person clutches heavy backpack firmly and brings it close to their body, then steps to the left with their left leg.
Metrics. In Sec. 4.1, we introduce the metrics employed in this paper. This section details the formula for the proposed metric CMD. The formulations for other metrics are available in the existing literature [34, 148]. CMD quantifies the discrepancy between the contact maps of ground truth interactions and those synthesized one. In this context, a contact map is characterized by the proportion of time {p, / each body part maintains active contact. Here, p; denotes the percentage of time during which the body part i less than a threshold distance from the object. And the metric is defined as,
P P
CMD
P.
P. 111,
(11)
where p is from the ground truth contact map, P is the number of body parts defined in SMPL [76]. and we set the distance threshold as 0.03
Implementation Details. The segment in the MDP contains m frames. The dynamics model, which includes 2 dynamics blocks as described in the main paper, is trained on the BEHAVE training set [7], with a batch size of 32, latent dimension of 64, and for 500 epochs. For rollout after the initial step& f >> 1, our dynamics model is trained to predict over longer timeframe (F = 3 x m 12), exceeding the past motion duration (H m = 4). For the initial step t = 1, we train a separate dynamics model to forecast duration of F = 15 given the past motion over // 1 frame, consistent with Sec. B.2. The optimization process is conducted over 300 epochs, utilizing a learning rate of 0.01. The dynamic model is trained on an NVIDIA A40 GPU for day. Our full log for high-level planning presented in Figure
D Additional Qualtitative Results
Interaction Retrieval. We here visualize the intermediate retrieval results. Figure C depicts the results from learning-basee based retrieval, resulting in diverse set of interactions that are both high-quality and semantically aligned.
Qualitative Experiments on OMOMO dataset. Figure D exemplifies our method that is able to generalize effectively to the OMOMO [66] dataset, despite our dynamics model not being trained on its object geometry annotations.
25

---

Grab top the whitechair, swing (bo whitechair, put down the whitechair.
Put trigod horizourally down.
Figure Qualitative results on the OMOMO [66] dataset. Our method generalizes well on the OMOMO objects and annotations unseen in training. Frames are separately visualized. Here, our synergized models are GPT-4 [88] and MotionGPT [46].
E
Failure Cases
We present failure cases of our method in our website, consisting of (i) inconsistency of interaction with textual description, (ii) inconsistency of human actions with textual descriptions, and (fii) wrong object category inferred by LLM.
Potential Negative Societal Impact
Some potential negative societal impacts include: (i) Our approach can synthesize realistic human motion interacting with objects, which could be misused to create deceptive or harmful content, such as portraying individuals in false situations. This could contribute the spread of misinformation. (ii) Our method evaluates real behavioral information, raising potential privacy concerns. Although our model utilizes a processed representation (SMPL. [76]) of human motion that retains minimal identifying details unlike raw data or images its ability to simulate human-object interactions could still be exploited for unauthorized surveillance photorealistic textures, it might be used to model and generate personal habits or movements without consent, posing risks of privacy violations. However, the use processed representation can be positively viewed as privacy-enhancing feature, as it minimizes the exposure of personally identifiable details.
behavioral analysis. For instance, with
26

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. [NA].
[NA] means either that the question is Not Applicable for that particular paper the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: in Sec. 1 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: in Sec. 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
27

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specing well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: this paper doesn't include proof and theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: in Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
28

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis-
sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our code is not available this time but will released the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
wwwsssiioniiciiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n //ips..coom httppssccpppubb///pppccp wwwp////uueeee/eeeeeeeeeeeeess /CodeSubmia ss Pollicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
29

---

Answer: [Yes]
Justification: in Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: in Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгrоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: in Sec. C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
30

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: in Sec. F
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: in Sec. F
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not use or release pretrained language models trained by our own
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
31

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: in Sec. 4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/daaasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: in Sec. C
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
32

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [Yes]
Justification: in Sec. F
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "Sfer5GWZbi", "reviewer": "Reviewer_Jz5F", "summary": "This work aims to address the text-conditioned human-object interaction (HOI) motion generation task. Unlike previous HOI generation approaches that rely on limited existing HOI datasets with text annotations for supervised learning, this work proposes a framework that decouples interaction semantics learning from interaction dynamics learning. This decoupling eliminates the need for large-scale HOI datasets with text annotations for training. The interaction semantics learning leverages an existing pretrained text-conditioned human motion generation model, while the interaction dynamics are optimized using a learned world model that predicts object states induced by human motions. The proposed framework can generate realistic HOI motions that align with the input text conditions.", "questions": "See the weakness section, and I am happy to discuss with authors during the rebuttal phase and adjust the score accordingly.", "limitations": "Yes, the authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* HOI modeling remains a challenging and understudied task compared to recent developments in human motion modeling. One major bottleneck is the lack of large-scale, high-quality interaction data. This paper offers an insightful solution to overcome the scarcity of large-scale interaction datasets.\n* Several key designs in this framework are reasonable, including (1) LLM-based high-level planning to reduce the distribution gap between input text instructions and the language used for text-to-motion model training dataset; (2) the world model trained with interaction dataset for human pose and object pose optimisation.\n* Comprehensive quantitative experiments results show the effectiveness of the proposed modules.", "weaknesses": "* Without the object geometry information encoded as input, how does the world model perform in generalizing to different objects? How does it perform when the object is out of the predefined list?\n * Efficiency of the autoregressive optimisation: as presented in figure 2, the optimisation of action and state is performed per step, which might result in inefficient inference, while the world model are trained to predict longer-horizon states. Could the author elaborate more on this design choice?\n * Regarding the world model training with N vertices sampled from contact area, what if the human pose is not in contact with object, for example, for the text instruction “the person throw away the ball“ where in most of frames the ball is flying in the air without contacting with the human, how does the object state forecasted and optimised with the world model.\n* From the visualisation results, there are no significant improvements over baselines HOI-Diff and CG-HOI in terms of the realism of human-object interaction. Is this mainly result from the not-perfect world model training and optimisation or the lack of hand pose? Could the author give some insights on the major challenges and bottlenecks presented in the current HOI understanding?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful and insightful comments. We address the concerns below:\n\n**W1: How the world model acts in novel objects**\n\n- The world model employs \"contact vertices\" as input, which includes features derived from the object distance field. These features encompass the human vertex-to-object surface distance and the human vertex velocity relative to the nearest object vertex (L205-206), **inherently including information related to the object's shape**. This encoding is consistently applied to both training objects from the BEHAVE dataset and novel objects from the CHAIRS dataset.\n\n- As discussed in (L64-68, L184-191), this vertex-based representation ensures that the world model concentrates on modeling the critical contact regions, and based on such contact modeling, the network learns to predict how object motion will be affected by these interactions. This approach prevents the model from overfitting to specific details, such as particular object shapes or body part motions. By focusing on high-level concepts—the principle that human-applied force on an object results in object acceleration, akin to Newton's law—the network can learn more generalizable patterns that apply across various contexts, e.g. different human actions and objects.\n\n- Empirically, our approach is more **effective** than encoding the entire object shape to the world model. The improved performance is demonstrated by the results in Table 1 and Figure 8, comparing vertex control (our proposed) v.s. raw control (full geometry and full human motion). Figure 5 further highlights the model's ability to generalize to **unseen objects from the CHAIRS dataset**.\n\n**W3: World model for non-contacting objects**\n\n- **How the network accepts the input without contact condition:**\nOur network can process inputs without contact conditions by adopting an approach similar to ControlNet (Zhang et al., Adding conditional control to text-to-image diffusion models, ICCV 2023). The network comprises two components: $\\mathcal{G}$ (L208) that operates without contact vertex conditions, applicable in scenarios where no contact occurs, and $\\mathcal{F}$ (L210), akin to the control components in ControlNet, which incorporates contact vertex conditions into the object trajectory when contact is present. When there is no contact, only the unconditional network is utilized.\n\n- **Why the network can learn to respond to the input without contact conditions:**\nThe model is aware of past object motion and thus needs to learn how human interaction affects the object’s state. This includes understanding how objects follow contact positions or normals by $\\mathcal{F}$ and how they move without contact by $\\mathcal{G}$. With the no-contact object motion data provided by BEHAVE, the world model (more specifically, $\\mathcal{G}$) learns to infer whether the object should free-fall based on its previous velocity or remain on the ground based on its height, as illustrated by the example at 02:27 in demo_2.mp4 (“a person throws a yoga ball towards the ground”) in the supplementary material.\n\n**W2: Efficiency of the autoregressive optimization and reason for long-term prediction**\n\n- As mentioned in the supplementary material (L693), optimization during autoregressive generation is performed selectively and only when the loss exceeds a certain threshold. In the Fig. 2 caption, we will clarify that while the overall framework is autoregressive, optimization is applied sparingly. This approach minimizes unnecessary computations, thereby maintaining computational efficiency.\n\n- The rationale behind training the world model for longer sequences is for effectiveness in capturing temporal dependencies. As noted in L197 of the main paper, we reference \"Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023,” which similarly found that predicting over a longer horizon promotes smoother and more coherent sequences for autoregressive generation, by planning actions ahead of time. However, longer predictions can increase the risk of error accumulation. To address this, we use only the early stage of long-horizon predictions, optimize them (while leaving some later parts unused), and use these optimized predictions in the next generation round. This strategy balances the advantages of long-term prediction with the need for accuracy and computational efficiency.\n\n**W4: Comparison with the baselines**\n\n- We would like to emphasize that **our model is trained without direct text supervision, unlike the baselines that rely on it**. Despite this, our model outperforms the baselines in text-interaction alignment to some extent, generating interactions that more accurately reflect the text instructions, even for long and complex descriptions, as demonstrated in the demo videos. Furthermore, our method achieves better realism to the baselines, with less penetration and jittering due to the optimization.\n\n- To further enhance realism, we acknowledge that integrating a hand model trained on hand-specific datasets (as full-body datasets like BEHAVE do not include hand poses) could address some of the limitations observed in our results. We agree with the reviewer that this addition could improve the fidelity of human-object interactions, which we leave as interesting future work.\n\n**W5: Major bottlenecks presented in the current HOI generation**\n\n- As discussed in the limitations section of our paper, one of the major challenges in human-object interaction (HOI) understanding is achieving physics realism, which requires a more advanced dynamics model or incorporating simulation. Another significant bottleneck is the lack of detailed hand pose representation in conjunction with full-body interaction in many datasets. This limitation hinders the accuracy of modeling interactions that involve fine motor skills and detailed object manipulation. We welcome the opportunity to discuss these challenges further with the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZGP7c3KHC", "reviewer": "Reviewer_R7nv", "summary": "InetrDreamer is a framework for synthesizing Human-Object Interactions (HOI) from textual queries. The key feature of InterDreamer is the ability to train without paired text and HOI motion data. To achieve this the work employs a multi-stage pipeline with LLM operating as a high-level planner that defines the parameters to infer the starting object pose and human motion sequences, which are brought together with the help of optimization on the next stage. The method is evaluated on the BEHAVE dataset which is additionally labeled with text as part of this work.", "questions": "1) Appendix B.1 describes two approaches to Interaction retrieval. Which of the described techniques is used? How do they compare in terms of performance?\n2) Are the BEHAVE text annotations planned to be released upon acceptance? \n3) What is the reasoning behind omitting the OMOMO dataset from the evaluation?\n4) Lines 157-158: should it be $a^{*}$ as it is the final result after optimization?", "limitations": "The work presents a substantial discussion of limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1) The key feature of InterDreamer is undoubtedly its biggest strength: the lack of requirement for paired data text-to-HOI data. This is a promising feature that in theory allows scaling the HOI modeling without tedious data-labeling by leveraging the advancements in LLMs.\n2) Another notable aspect is that the proposed planning can be adopted on top of other existing motion models improving their performance (as demonstrated in Table 2).", "weaknesses": "1) One of the key features of the proposed framework is a High-level Planning module that queries the LLM to extract necessary features for downstream modules. However the presented description of the query protocol is scarce, similar works (e.g. SINC by Athanasiou, Petrovich, et al., ICCV'23) also employ LLM within the framework and provide the full template of the query in the supplementary material (Section B) to ensure the reproducibility.\n\n2) Contribution formulation in the Introduction (Lines 71-72): the considered task is text-to-HOI modeling, while training without paired data is rather a feature of the method than a task itself. Further in the text, in the Conclusions section, the work claims to introduce the novel task of 3D HOI generation from text (Line 319). Both formulations do not reflect the actual contribution of the work.\n\n3) Evaluation is performed only on the self-labeled BEHAVE dataset, however, there exists at least one more dataset with text annotations: OMOMO \\[51] which is not used for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful and insightful comments. We address your concerns below:\n\n\n**W1:  full template of the query**\t\n- We appreciate the reviewer’s suggestion and have included our detailed query log in Fig. 1 of the rebuttal PDF file. We will discuss related work on this and add the log to the revision.\n\n\n**W2: Contribution formulation**\n- We believe that learning text-guided HOI generation from data without direct text supervision is a contribution to our work. We would like to mention that Reviewer Kjsy recognized the novelty of this task, noting that \"The authors introduce a novel task of synthesizing whole-body interactions with dynamic objects guided by textual commands, without relying on text-interaction pair data.\" We appreciate the reviewer’s suggestions and apologize for any confusion. To clarify our statements, we will revise them as follows:\n    * For Lines 71-72: We address the task of synthesizing whole-body interactions with dynamic objects guided by textual commands, achieving this without the need for paired text-interaction data—a novel approach to the best of our knowledge.\n    * For Line 319: We focus on the task of text-guided 3D human-object interaction generation, aiming to accomplish this without relying on paired text-interaction data.\n\n**W3&Q3: evaluation on OMOMO**\n- Our method is designed to be dataset-agnostic. We chose the BEHAVE dataset for training the world model because it provides sufficient HOI dynamics to develop an effective dynamics model. We then tested our method on **both the BEHAVE, as well as CHAIRS datasets for novel objects**, which we believe is a sufficient evaluation. \n- We appreciate the reviewer's suggestions. In response to it, we expanded our interaction planning and retrieval database to include the OMOMO dataset, similar to our approach with CHAIRS.\n- Table 1 in the rebuttal PDF demonstrates that our interaction planning effectively bridges the distributional gap between the text-to-motion model and OMOMO text. Additionally, qualitative results of the full pipeline on OMOMO, provided in Fig. 2 of the rebuttal PDF, further illustrate the effectiveness of our entire pipeline on novel objects. We will incorporate more experiments into the revision.\n\n**Q1: two approaches to interaction retrieval**\n- For our primary experiments, we primarily utilized the handcrafted approach to implement the full pipeline, as it is straightforward and does not require training. To explore the feasibility of retrieval without relying on handcrafted rules, we also investigated a learning-based method for interaction retrieval. As shown in Figure C of the supplementary material, our qualitative evaluation demonstrates that the learning-based retrieval effectively captures diverse and realistic interactions, producing results comparable to the handcrafted method. Notably, the learning-based approach does not require constructing a database during inference, offering a more flexible solution.\n\n**Q2: text annotation release**\n- Yes, we plan to release the text annotations for the BEHAVE dataset upon acceptance.\n\n**Q4: typo revision**\n- We appreciate the reviewer for catching this typo. Yes, the final result should indeed be denoted as $a^*$. We will correct this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5q4cWzqjgr", "reviewer": "Reviewer_aGg2", "summary": "This work aims at human-object interaction generation under less supervision. Since existing methods rely on large-scale interaction data, this paper attempts to propose a method that does not require paired data. The proposed method includes three stages, high-level planning by LLMs, low-level control by existing text-to-motion methods, and interaction retrieval, a world model based on existing human-object interaction generation methods. The proposed method is evaluated on the public datasets, BEHAVE and CHAIRS datasets.", "questions": "As mentioned in weaknesses.", "limitations": "The authors have discussed the limitation of their method.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The issue focused on in this paper is a worthy research topic. If the training does not require pair-wise data, it will greatly reduce the dependence of the human-object interaction generation on the training data.\n2. Given texts, the figures intuitively demonstrate visual representations of the generated human-object motions.", "weaknesses": "1. About novelty. While the issue focused on in this paper is valuable, the proposed method is a combination of existing methods. I appreciate the technical effort of the authors, but please highlight how each stage differs from existing approaches.\n2. Missing some details. \n--- In Line 165, how is the database built? If the data used to build the dataset includes the training set and the test set, it is not standard.\n---Does the low-level control need to be trained? As mentioned in Line 269, the text-to-motion model is the existing method which is pre-trained on HumanML3D. If no training is required, how does the model generalize on action types? \n--- As mentioned in Line 180, the authors claimed “this model trained on the 3D HOI dataset”, while the main work is that the proposed method does not require training on paired HOI data. It is inconsistent and confusing.\n--- What is the size of the object’ signed distance fields? How many vertices is the object represented by? What network to use to encode and decode object shapes?\n3. The motivation for the quantitative experiment (section 4.2) is unclear. Why compare different control conditions? \n4. In Line 316, what is the vertex-based control? Does it mean human vertex controls in Line 210? The definition should be given.\n5. In the caption of Figure 4, it is claimed that the proposed method can handle complex and long sequences. Is there a strategy in the proposed method for complex and long sequences? What is the longest?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful and insightful comments. We address your concerns below:\n\n**W1: Novelty**\n\nWe appreciate the reviewer's thoughtful suggestions. While our method incorporates elements adapted from existing approaches, we would like to clarify the novelty of these components, particularly in the context of human-object interaction (HOI) generation.\n\n- High-Level Planning: Our approach introduces a simple yet effective method to **bridge the distributional gap** (Line 138) between pre-trained text-to-human generation and free-form text-to-interaction generation. This is a key distinction from most existing work [102], which primarily utilizes LLMs only for understanding contact parts. We evaluate the significance of our design in Figure 6 and L300-309.\n\n- Low-Level Control: While we employ existing methods and pre-trained models for low-level control, which we do not claim as novel, our contribution lies in the seamless integration and evaluating the effectiveness of four established text-to-motion models within our framework, which show that **our framework is general.**\n\n- World Model: Unlike existing work that typically encodes the full state of the interaction (as shown in references [43, 83, 108] in the paper), our method introduces a novel approach to modeling contact vertices. As discussed in the paper (Lines 64-68, 184-191), this vertex-based representation ensures that the world model focuses on the critical contact regions, preventing overfitting to specific details, such as particular object shapes. This leads to improved performance, as demonstrated in Table 1, Figure 8, and Figure 5 (generalize to unseen objects from the CHAIRS dataset).\n\nOverall, HOI generation, as a challenging task with inherent data scarcity issues, requires a novel paradigm distinct from existing approaches. **Our perspective focuses on how to effectively reuse and harness large models, which is an emerging and crucial research problem.** While naive reuse can lead to poor performance (e.g., Figure 6), our approach fosters synergy among models, resulting in contributions tailored to this task. We welcome the reviewer's feedback on any specific related papers and would be happy to discuss further or clarify any similarities with these works.\n\n\n**W2.1: Database build for interaction retrieval**: \n- We apologize for the omission of details. To clarify, we used only the training set to build the database. We will revise in the revision.\n \n**W2.2: Generalization of low-level control**: \n\nThe low-level control does not require additional training. We would like to clarify how the model generalizes across action types from the following perspectives:\n- Generalization to Diverse Actions:\nThe text-to-motion model inherently possesses a degree of generalization across actions, similar to CLIP, where the compositionality of natural language allows for generalization. Semantics in the language space can be composed to represent different actions. By aligning language space [74] or discrete tokens [128, 35] with motion space/tokens, the motion model benefits from the generalization capabilities of the text compositionality.\n- Generalization to HOI Descriptions:\nWhile these models are not directly generalizable to free-form HOI descriptions, we bridge the distributional gap between free-form interaction descriptions and the text-to-motion model's distribution using chain-of-thought prompting, as demonstrated in Figure 6 and Lines 300-309, which constitutes one of our contributions.\n\n**W2.3: No need for pair data**: \n- To clarify, the proposed method **does not require training on paired HOI data**. However, it **does require training on HOI data without paired text (Lines 62-63)**, and only the world model needs to be trained with this data (Lines 76-77). We will revise.\n\n**W2.4: size of SDF**:\n- If the size refers to the resolution of the SDF, our approach directly calculates the distance from points to the object’s surface, theoretically providing infinite precision. If a different aspect of size was intended, we would appreciate further clarification.\n\n**W2.5: number of vertices**:\n- It depends on the specific objects used. For example, for the BEHAVE dataset, we use their provided fine-grained objects, each containing over 10,000 vertices, to calculate the SDF.\n\n**W3: Motivation for control based on contact vertices** \n- The motivation for our quantitative experiment in Section 4.2 stems from the need to validate a key contribution of our work: as outlined in the paper (L64-68, L184-191), the vertex-based representation ensures that the world model focuses on critical contact regions. This enables the network to predict how object motion is influenced by interactions at these vertices, reducing the risk of overfitting to specific object shapes or detailed body part movements. By emphasizing high-level concepts–the fundamental principle that human-applied force on an object leads to acceleration (akin to Newton's law)--the network learns more generalizable patterns, enhancing its performance in HOI modeling.\n- To substantiate this claim, we compared our method against various control conditions, including a raw control where all available information is fed into the network. The results clearly demonstrate that our approach is superior.\n\n\n**W4: vertex-based control**\n- Yes, we appreciate the reviewer for pointing this out. The vertex-based control refers to our method where only the contacting vertices are used for control. We will clarify this definition in the revision.\n\n\n**W2.6: object shape encoding**: \n- The object’s shape is inherently encoded using the human vertex-to-object surface distance, as described in L204-206. This information is then processed through dynamics blocks (MLPs) as outlined in L208 and L210, and further integrated with attention layers (Line 212) for interaction modeling.\n\t\nWe put the response to W5 in the global response of rebuttal because of the character limit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nMcERJwzT", "reviewer": "Reviewer_Kjsy", "summary": "The paper addresses the challenge of text-conditioned 3D dynamic human-object interaction (HOI) generation, which has lagged behind advancements in text-conditioned human motion generation due to the scarcity of large-scale interaction data and detailed annotations. The authors propose InterDreamer, a framework that decouples interaction semantics and dynamics to generate realistic HOI sequences without relying on text-interaction pair data. By leveraging pre-trained large models for high-level semantic control and introducing a world model to understand low-level interaction dynamics, InterDreamer surpasses existing motion capture data limitations. Applied to the BEHAVE and CHAIRS datasets, InterDreamer demonstrates the ability to produce coherent and text-aligned 3D HOI sequences, showcasing its effectiveness through comprehensive experimental analysis.", "questions": "- I'm curious, can the Interaction Retrieval predict the accurate contact areas for an object that has never been seen before?\n- Is this post-processing optimization expensive, and how long does it take to optimize a sequence?", "limitations": "- The method still is unable to handle fine-grained human-object interactions (HOIs), such as dexterous manipulation using hands. Right now HOI generation is still a very challenging task, so I wouldn't consider this limitation to be a weakness of the manuscript, and expect that subsequent work will refine this point.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors introduce a novel task of synthesizing whole-body interactions with dynamic objects guided by textual commands, without relying on text-interaction pair data.\n- The types of HOI interactions are limited compared to the diversity of movement in the human body itself. So, the idea that decomposes semantics and dynamics and then integrates them makes sense to me. The main challenge lies in effectively understanding the positions where the human object interacts. The authors propose leveraging the power of LLMs to simplify this challenge effectively.\n- The technical details are sound, and the experimental results are good and demonstrate its zero-shot capability.\n- The paper is well written, with clear figures and typography.", "weaknesses": "- This work doesn't seem to be able to handle more complex long interactions, such as a person walking to a chair and then sitting down, or a person lifting a box on the floor and carrying it for a while before putting it on the ground.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thoughtful and insightful comments. We address your concerns below:\n\n\n**W1: handle more complex long interactions**: \n- As shown in Figure 4 and in the examples after 00:43 in demo_1.mp4 of the supplementary material, our approach is capable of handling complex and extended interactions, such as “A person holds a medium box up with their right hand, lowers their right arm, and pulls the box with left hand towards them.” \n- However, we recognize the model's limitation on very long and highly complex sequences, particularly those involving multiple distinct phases, which we found to be difficult for existing work even with supervised training [21,50,71 in the paper]. To address this, future work could explore temporal compositionality in HOI, as well as incorporate more complex multi-phase interactions into the training data, to enhance the model's capability in these scenarios.\n\n**Q1: Accuracy of contact reasoning for novel objects**: \n- The handcrafted interaction retrieval works well with predefined objects but lacks generalizability, as it requires building a specific database. In contrast, the learning-based interaction retrieval (L671-681 in the supplementary material) can handle **novel objects** but may struggle in complex scenarios due to two main issues:\n\n     1. Stable Diffusion sometimes generates low-quality images in complex human-object interactions, leading to unnatural humans, incorrect object states, or unreasonable contact patterns.\n    2. These lower-quality images fall outside the distribution that LEMON (an off-the-shelf model that we used for estimating object affordance and human contact from images) is not able to predict accurate contact areas (affordance).\n        \n     Improving the text-to-image model for human-object interaction could enhance image quality, which we see as a promising direction for future work.\n\n**Q2: Efficiency of optimization**: \n- The optimization is expensive, and each step often takes a few seconds. Thus, for efficiency, as mentioned in the supplementary material (L693), optimization is only performed if the loss exceeds a certain threshold. This strategy prevents unnecessary computations, thus maintaining overall computational efficiency.\n\n**Limitation**\n- Thanks for your feedback. We agree with the reviewer that generating fine-grained human-object interactions (HOIs), such as dexterous hand manipulations, remains a challenging task. One potential solution would be to incorporate a dedicated hand model, trained specifically on the nuances of fine-grained hand interactions (e.g., OakInk 2[1]), and then integrate it with the body model.\n\n    [1] Zhan, et al. OAKINK2: A Dataset of Bimanual Hands-Object Manipulation in Complex Task Completion. CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive comments. We appreciate the recognition that our task is challenging (Reviewer Jz5F) and novel (Reviewer Kjsy), and that our solution is insightful (Reviewer Jz5F), sound (Reviewer Kjsy), and promising, particularly in its potential to scale HOI modeling without the need for tedious data labeling (Reviewer R7nv) and to reduce the dependency of training data (Reviewer aGg2). Additionally, we are excited that our experiments are regarded as comprehensive (Reviewer Jz5F) and good (Reviewer Kjsy), with figures that intuitively demonstrate the generated human-object motions (Reviewer aGg2) and clear visuals (Reviewer Kjsy). 

We will carefully revise and incorporate all suggestions in the revision. We address specific concerns in separate, individual responses. We primarily clarify how our model operates, particularly the details and novelty of the world model, and add additional experiments on the OMOMO dataset. The complete log for high-level planning and experiments on the OMOMO dataset is detailed in the PDF.

Due to the 6000-character limit, we include our response to Reviewer aGg2 on the fifth weakness, regarding the handling of complex and long sequences, in the global response.

**W5: handling complex and long sequence**:
- The capability for managing longer sequences arises from autoregressive generation, where the length of the sequence depends on the capacity of the text-to-motion model. For instance, MotionGPT, one of the text-to-motion models that we evaluate in the paper, can generate sequences of up to 196 frames at 20fps. We further interpolate these sequences to 30fps, resulting in the longest sequence in Figure 4 reaching 294 frames.

- In terms of handling complex interactions, our method excels due to two key factors:

   1. The LLM’s reasoning improves the text-to-motion model's ability to handle complex descriptions, leading to more intricate motion sequences.
    2. Our dynamic model adeptly handles complex scenarios, such as human motions that differ significantly from the training set on BEHAVE. This capability arises because, unlike human motion representations, the localized and canonicalized contact vertex representation remains consistent even for out-of-distributional human motion, allowing the network to effectively handle and generalize across complex conditions.

Meta-review: There were extensive discussions between the authors and reviewers. After the rebuttal, most of the major concerns have been addressed. The remaining issues are mainly about the presentation of the paper and quality of the visual results. On the other hand, the proposed approach has its own novelty, recognized by the reviewers, that it does not required paired text-human-motion data for training, which may inspire future work. As text to human motion generation is a relatively under-explored topic, expecting very high-quality visual results may overshadow and undervalue the emergence of new ideas in this direction.

Overall, the AC found the positive points slightly outweighs the negative side. The response to the ethics reviews are also satisfying. The AC thus recommends to accept the paper as a Poster.

The authors are highly encouraged to incorporate the new results and revise the presentation of the paper following reviewers' comments.
Decision: Accept (poster)