HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Models

## Abstract

Abstract We address the problem of generating realistic 3D human-object interactions (HOIs) driven by textual prompts.
To this end, we take a modular design and decompose the complex task into simpler sub-tasks.
We first develop a dual-branch diffusion model (HOI-DM) to generate both human and object motions conditioned on the input text, and encourage coherent motions by a cross-attention communication module between the human and object motion generation branches.
We also develop an affordance prediction diffusion model (APDM) to predict the contacting area between the human and object during the interactions driven by the textual prompt.
The APDM is independent of the results by the HOI-DM and thus can correct potential errors by the latter.
Moreover, it stochastically generates the contacting points to diversify the generated motions.
Finally, we incorporate the estimated contacting points into the classifier-guidance to achieve accurate and close contact between humans and objects.
To train and evaluate our approach, we annotate BEHAVE dataset with text descriptions.
Experimental results on BEHAVE and OMOMO demonstrate that our approach produces realistic HOIs with various interactions and different types of objects.

## 1 Introduction

Text-driven synthesis of 3D human-object interactions
(HOIs) aims to generate motions for both the human and object that form coherent and semantically meaningful interactions.
It enables virtual humans to naturally interact with objects, which has a wide range of applications in AR/VR, video games, and filmmaking, etc.

The generation of natural and physically plausible 3D HOIs involves humans interacting with dynamic objects in various ways according to the text prompts, thereby posing several challenges.
First,
the variability of object shapes makes it particularly challenging to generate semantically meaningful contact between the human and object to avoid floating objects.
Second, the generated HOIs should be faithful to the input text prompts as there are many plausible interactions between human and the same object (*e.g*., a person carries a chair, sits on a chair, pushes or pulls a chair).
Text-driven 3D HOI synthesis with a diverse set of interactions is still under-explored.
Third, the development and evaluation of 3D HOI synthesis models requires a high-quality human motion dataset with various HOIs and textual descriptions, but existing datasets lack either HOIs [, ] or textual descriptions [].

On one hand, recent methods [, , , , , , , ] can synthesize realistic human motions for HOIs for static objects only.
They usually synthesize the motion in the last mile of interaction, *i.e*. the motion between the given starting human pose and the final interaction pose, and overlook the movement of the objects when the human is interacting with them.
On the other hand, existing methods for motion generation with dynamic objects do not adequately reflect real-world complexity.
For instance, they focus on grasping small objects [], provide the object motion as conditioning [], predict deterministic interactions between the human and the same object without the diversity [, ],
consider only a small set of interactions (*e.g*. sit/lift [], sit/lie down [], sit [, , ], grasp [, ]), or investigate a single type of object (*e.g*. chair [, ]).

In this paper, we go beyond and propose HOI-Diff for 3D HOIs synthesis involving humans interacting with different types of objects in diverse ways, which are both physically plausible and semantically faithful to the textual prompt, as shown in Fig. 1.
Our key insight is to decompose 3D HOIs synthesis into three modules:
(a) coarse 3D HOIs generation that extends the human motion diffusion model [] to a dual-branch diffusion model (HOI-DM) to generate both human and object motions conditioning on the input text prompt.
To encourage coherent motions, we develop a cross-attention communication module, exchanging information between the human and object motion generation models;
(b) affordance prediction diffusion model (APDM) that estimates the contacting points between the human and object during the interactions driven by the textual prompt.
Our APDM does not rely on the results of the HOI-DM and thus can recover from its potential errors.
Moreover, it stochastically generates the contacting points to diversity the generated motions;
and (c) affordance-guided interaction correction that incorporates the estimated contacting information and employs the classifier-guidance to achieve accurate and close contact between humans and objects,
significantly alleviating the cases of floating objects.
Compared with designing a monolithic model, our proposed HOI-Diff disentangles motion generation for humans and objects and estimation of their contacting points, which are later integrated to form coherent and diverse HOIs, reducing the complexity and burden for each of the three modules.

To our best knowledge, HOI-Diff is one of the first text-driven approach s capable of generating realistic motions for various HOIs and different types of objects during the interactions.
While our approach pioneers this capability, we note the existence of concurrent works such as  [, , ], which also explore similar directions.
For both training and evaluation purposes, we annotate each video sequence in BEHAVE dataset [] with text descriptions, which mitigates the issue of severe data scarcity for text-driven 3D HOIs generation.
In addition, we evaluate our approach on the OMOMO dataset [], which focuses on the manipulation of two hands.
Extensive experiments validate the effectiveness and design choices of our approach, particularly for dynamic objects, thereby enabling a set of new applications in human motion generation.

## 2 Related Work

Human Motion Generation with Diffusion Models.
The denoising diffusion models have been widely used 2D image generations [, , ] and achieved impressive results.
Recent work [, , , , , , , , , , , , , , , , ] apply the diffusion model in the task of human motion generation.
While these methods have successfully generated human motion, they usually generate isolated motions in the free space without considering the objects the human is interacting with.
Our method is primarily focused on motion generation with human-object interactions.

Scene- and Object-Aware Human Motion Generation.
Recent works condition motion synthesis on scene geometry [, , , ].
This facilitates the understanding of human-scene interactions.
However, the motion fidelity is compromised due to the lack of paired full scene-motion data.
Other approaches [, , , , , ] instead focus on the interactions with the objects and can produce realistic motions.
However, they focus on interacting with static objects with limited interactions.
OMOMO [] proposes a conditional diffusion framework that can generate full-body motion from the object motion.
The object motion is needed as input in OMOMO, whereas our method can jointly synthesize human motion and object motion.
IMoS [] synthesizes the full-body human along with the 3D object motions from textual inputs.
Although they can generate realistic interactions, IMoS only focuses on grasping small objects with hands.
InterDiff [] predicts whole-body interactions with dynamic objects. However, it focuses on the motion prediction task, where the future motion needs to be forecasted based on past motion information and object information.
Note that the interaction type is deterministic, given the object and past motion information.
Different from this, we tackle the motion synthesis task, where the interaction with the same object can be controlled by the text prompt.
Recently, there has been a surge of interest in the text-driven synthesis of 3D human-object interactions for dynamic objects, resulting in the development of concurrent works [, , ].

Multi-Human Motion Generation.
Recent works [, , ] incorporate human-to-human interactions into the motion diffusion process.
They usually also adopt cross-attention to exchange information from different human motions.
Different from them, our goal is to generate coarse HOIs instead of doing so in a single run, which alleviates the burden of the model.

Affordance Estimation.
The affordance estimation on 3D point cloud is studied in [, , , , , , ].
[] formulates this task as the point cloud segmentation task from the extracted geometric features.
[] voxelizes the point cloud and creates affordance maps with interactive manipulation.
[, ] focuses on the object grasping affordances and some specific applications.
[] learns the affordance for object-object interactions.
[] addressed the task of open-vocabulary affordance learning.
Overall affordance learning is a very challenging task even the shape of point cloud is given.
Instead of predicting the point-wise contact labels, in our method, we propose to make some hypotheses to simplify human-object interactions, making it more tractable without significantly compromising accuracy.

## 3 Method

### 3.1 Overview

Motion Representations.
We denote a 3D HOI sequence as ${\bm{x}}=\{{\bm{x}}^{h},{\bm{x}}^{o}\}$.
It consists of human motion sequence ${\bm{x}}^{h}\in\mathbb{R}^{L\times D^{h}}$ and object motion sequence ${\bm{x}}^{o}\in\mathbb{R}^{L\times D^{o}}$, where $L$ denotes the length of the sequence.
For ${\bm{x}}^{h}$, we adopt the redundant representation widely used in human motion generation [] with $D^{h}=263$, which include pelvis velocity, local joint positions, velocities and rotations of other joints in the pelvis space, and binary foot-ground contact labels.
For the object motion sequence ${\bm{x}}^{o}$, we assume the object geometry is given as an input, and thus we only need to estimate its 6DoF poses in the generation, *i.e*., $D^{o}=6$.
We represent each object instance as a point cloud of 512 points ${\bm{p}}\in\mathbb{R}^{512\times 3}$.

Diffusion Model for 3D HOI Generation.
Given a prompt ${\bm{c}}=({\bm{d}},{\bm{p}})$, consisting of a textual description ${\bm{d}}$ and the object instance’s point cloud ${\bm{p}}$, a diffusion model $p_{\theta}({{\bm{x}}_{t-1}}|{{\bm{x}}_{t}},{\bm{c}})$(^1^11We use superscripts $h$ and $o$ to denote human and object sequence, respectively. Without a superscript, it means the 3D HOI sequence, containing both ${\bm{x}}^{h}$ and ${\bm{x}}^{o}$. Subscript is used for the diffusion denoising step.) learns the reverse diffusion process to generate clean data from a Gaussian noise ${{\bm{x}}_{t}}$ with $T$ consecutive denoising steps

$$ $p_{\theta}({{\bm{x}}_{t-1}}|{{\bm{x}}_{t}},\bm{c}):=\mathcal{N}({{\bm{x}}_{t-1 }},\mathbf{\mu}_{\theta}({{\bm{x}}_{t}},t,\bm{c}),(1-\alpha_{t})\mathbf{I}),$ (1) $$

where $t$ is the denoising step.
Following [], instead of predicting the noise $\mathbf{\epsilon}_{t}$ in each diffusion step, our diffusion model $M_{\theta}$ with parameters $\theta$ predicts the final clean motion
${\bm{x}}_{0}=M_{\theta}({{\bm{x}}_{t}},t,{\bm{c}})$.

Figure: Figure 2: Overview of HOI-Diff for 3D HOIs generation using diffusion models. Our key insight is to decompose the generation task into three modules: (a) coarse 3D HOI generation using a dual-branch diffusion model (HOI-DM), (b) affordance prediction diffusion model (APDM) to estimate the contacting points of humans and objects, and (c) afforance-guided interaction correction, which incorporates the estimated contacting information and employs the classifier-guidance to achieve accurate and close contact between humans and objects to form coherent HOIs.
Refer to caption: extracted/5474021/figures/overview.png

We sample $\mathbf{x}_{t-1}\sim\mathcal{N}({\bm{\mu}}_{t},\Sigma_{t})$ and compute the mean as in []

$$ $\displaystyle{\bm{\mu}}_{t}=\frac{\sqrt{\alpha_{t-1}}\beta_{t}}{1-\alpha_{t}}{ \bm{x}}_{0}+\frac{\sqrt{1-\beta_{t}}(1-\alpha_{t-1})}{1-\alpha_{t}}{{\bm{x}}_{ t}},$ (2) $$

where $\alpha_{t}=\prod_{s=1}^{t}(1-\beta_{s})$ and $\beta_{t}\in(0,1)$ are the variance schedule. $\Sigma_{t}=\frac{1-\alpha_{t-1}}{1-\alpha_{t}}\beta_{t}$ [] is a variance scheduler of choice. Similar to ${\bm{x}}_{t}$, ${\bm{\mu}}_{t}$ consists of ${\bm{\mu}}^{h}_{t}$ and ${\bm{\mu}}^{o}_{t}$, corresponding to human and object motion, respectively.

Figure: Figure 3: Illustration of HOI-DM architecture for coarse 3D HOIs generation. It has two branches designed for generating human and object motions individually. A mutual cross-attention is introduced to allow information exchange between two branches to generate coherent motions. The human motion model $M^{h}$ finetunes a pretrained human MDM [].
Refer to caption: extracted/5474021/figures/hoi_pipeline.png

Simply adopting the diffusion model described in Eq.(1) would impose a huge burden on the model, which requires joint generation of human and object motion and more critically, enforcement of their intricate interactions to follow the input textual description.
In this paper, we propose HOI-Diff for 3D HOIs generation, disentangling motion generation for humans and objects and estimation of their contacting points.
They are later integrated to form coherent and diverse HOIs, which reduces the complexity and burden for each of the three modules, leading to better generation performance as evidenced by our experiments.

Fig. 2 shows the overview of our proposed approach.
We introduce a dual-branch Human-Object Interaction Diffusion Model (HOI-DM), which can produce diverse yet consistent motions, capturing the intricate interplay and mutual interactions between humans and objects (Sec. 3.2).
To ensure physically plausible contact between humans and objects, we propose a novel affordance prediction diffusion model (APDM) (Sec. 3.3), whose output will be used as classifier guidance (Sec. 3.4) to correct the interactions at each diffusion step of human/object motion generation.

### 3.2 Coarse 3D HOIs Generation

First, we introduce a dual-branch Human-Object Interaction Diffusion Model (HOI-DM) to generate human and object motions that are roughly coherent.
As shown in Fig. 4, it consists of two Transformer models [], human motion diffusion model (MDM) $M^{h}$ and object MDM $M^{o}$, which work similar to [].
Specifically, at the diffusion step $t$, they take the text description and noisy motions ${\bm{x}}_{t}^{h}$ and ${\bm{x}}_{t}^{o}$ as input and predict clean human and object motions ${\bm{x}}_{0}^{h}$ and ${\bm{x}}_{0}^{o}$, respectively.

To enhance the learning of interactions of the human and object when generating their motion, we introduce a Communication Module ($CM$) designed for exchanging feature representations between the human MDM $M^{h}$ and the object MDM $M^{o}$.
$CM$ is a Transformer block that receives the intermediate feature ${\bm{f}}^{h},\bm{f}^{o}$ from both $M^{h}$ and $M^{o}$.
It then processes these inputs to generate refined updates based on the cross attention mechanism [].
The updated feature representations
$\tilde{{\bm{f}}_{h}}$ and $\tilde{{\bm{f}}_{o}}$ of the human and object are then conditioned on each other, which are then fed into the subsequent layers of their respective branches to estimate clean human and object motion ${\bm{x}}_{0}^{h}$ and ${\bm{x}}_{0}^{o}$, respectively.
The $CM$ is inserted at the 4th transformer layer for human MDM and the last layer for object MDM, which was empirically found to work better.

Given the limited data availability for 3D HOI generation, during training, the human motion model $M^{h}$ finetunes a pretrained human MDM [].
This fine-tuning is critical to ensure the smoothness of the generated human motions.
We ablate this design choice in Sec. 4.3.
Object MDM is trained from scratch. We modify the input and output linear layers to take in the object motion which has a different dimension from the human motion.
More methodology details of HOI-DM can be found in the supplementary material.

### 3.3 Affordance Estimation

Due to the complexity of the interactions between a human and object,
HOI-DM alone usually fails to produce physically plausible results, leading to floating objects or penetrations.
To improve the generation of intricate interactions, the problem that needs to be solved is to identify where the contacting areas are between the human and object.
InterDiff [] defines the contacting area based on the distance measurement between the surface of human and object.
This approach, however, heavily relies on the quality of the generated human and object motions and cannot recover from errors in the coarse 3D HOI results.
In addition, the contact area is diverse even with the same object and interaction type, *e.g*., “sit” can happen on either side of a table.
To this end, we introduce an Affordance Prediction Diffusion Model (APDM) for affordance estimation.
As illustrated in Fig. 4, the input includes a text description ${\bm{d}}$ and the object point cloud ${\bm{p}}$.
Our APDM doesn’t rely on the results of the HOI-DM and thus can recover from the potential errors in HOI-DM. In addition, it stochastically generates the contacting points to ensure the diversity of the generated motions.

Affordance estimation in 3D point clouds itself is a notably challenging problem [, , , , , , ], especially in the context of 3D HOI generation involving textual prompt.
In this paper, we consider eight primary body joints – the pelvis, neck, feet, shoulders, and hands – as the interacting parts in HOI scenarios.
It can effectively model common interactions such as grasping an object with both hands, sitting actions involving the pelvis and back, or lifting with a single hand.
We use binary contact labels to determine which joints are in contact with the object.
Subsequently, we predict eight corresponding contact points on the object surface, identified as the points closest to the selected body joints. Note that the binary contact label estimation for different body joints are independent, allowing us to handle complex HOIs.

Specifically, at each diffusion time step $n$ of APDM(^2^22We note that APDM and HOI-DM work independently. We thus use two symbols to denote the different diffusion time steps to avoid confusion.),
the noisy data consists of human contact labels representing the contact status for the eight primary body joints, denoted as $\bm{y}^{h}_{n}\in\{0,1\}^{8}$, and the eight corresponding contact points on the object surface, denoted as $\bm{y}_{n}^{o}\in\mathbb{R}^{8\times 3}$. The model is designed to predict both contact probabilities and contact positions. Subsequently, dynamic selection of contacting body joints is performed by considering predicted probabilities over a specific threshold $\tau$ (set to be 0.6). The corresponding contact points on the object are then determined based on the selected joints. APDM works similar to the diffusion denoising process described in Eq.(1).
Besides, we utilize a large language model (ChatGPT) to determine whether the object state ${\bm{y}}_{0}^{s}\in\{0,1\}$ should be set to static (${\bm{y}}_{0}^{s}=1$) based on the textual description, which can help us better process static objects when synthesizing 3D HOIs, as discussed in the following section.
All the clean affordance data is grouped as ${\bm{y}}_{0}=({\bm{y}}^{h}_{0},{\bm{y}}^{o}_{0},{\bm{y}}^{s}_{0})$.

### 3.4 Affordance-guided Interaction Correction

With the estimated affordance, we can better align human and object motions to form coherent interactions.
To this end, we propose to use the classifier guidance [] to achieve accurate and close contact between humans and objects, significantly alleviating the cases of floating objects.

Specifically, in a nutshell, we define an analytic function $G({\bm{\mu}}^{h}_{t},{\bm{\mu}}^{o}_{t},{\bm{y}}_{0})$ that assesses how closely the generated human joints and object’s 6DoF pose align with a desired objective.
In our case, it enforces the contact positions of human and object to be close to each other and their motions are smooth temporally.
Based on the gradient of $G({\bm{\mu}}^{h}_{t},{\bm{\mu}}^{o}_{t},{\bm{y}}_{0})$, we can perturb the generated human and object motion at each diffusion step $t$ as in [, , ],

$$ $\displaystyle\mathbf{{\bm{\mu}}}^{h}_{t}$ $\displaystyle=\mathbf{{\bm{\mu}}}^{h}_{t}-\tau_{1}\Sigma_{t}\nabla_{{\bm{\mu}} ^{h}_{t}}G({\bm{\mu}}^{h}_{t},{\bm{\mu}}^{o}_{t},{\bm{y}}_{0}),$ (3) $\displaystyle\mathbf{{\bm{\mu}}}^{o}_{t}$ $\displaystyle=\mathbf{{\bm{\mu}}}^{o}_{t}-\tau_{2}\Sigma_{t}\nabla_{{\bm{\mu}} ^{o}_{t}}G({\bm{\mu}}^{h}_{t},{\bm{\mu}}^{o}_{t},{\bm{y}}_{0}).$ (4) $$

Here $\tau_{1}$ and $\tau_{2}$ are different strengths to control the guidance for human and object motion, respectively.
Due to the sparseness of object motion features, we assign a larger value to $\tau_{2}$ compared to $\tau_{1}$.
This applies greater strength to perturb object motion, facilitating feasible corrections for contacting joints.
During the denoising stage, to eliminate diffusion models’ bias that can suppress the guidance signal, we iteratively perturb $K$ times in the last denoising step, as illustrated in Algorithm 1.

How can we define the objective function $G({\bm{\mu}}^{h}_{t},{\bm{\mu}}^{o}_{t},{\bm{y}}_{0})$? We consider three terms here.
First, in the generated 3D HOIs, the human and object should be close to each other on the contacting points.
We therefore minimize the distance between human contact joints and object contact points

$$ $\displaystyle G_{con}=\sum_{i\in\{1,2,...,8\}}\left\lVert R\big{(}{\bm{\mu}}^{ h}_{t}(i)\big{)}-V\big{(}{\bm{y}}^{o}_{t}(i)\big{)}\right\rVert^{2},$ (5) $$

where ${\bm{\mu}}^{h}_{t}(i)$ and ${\bm{y}}^{o}_{t}(i)$ denote the $i$-th available contacting joint indexed by ${\bm{y}}^{h}_{0}$ and $i$-th object contact point, respectively.
$R(\cdot)$ converts the human joint’s local positions to global absolute locations, and $V(\cdot)$ obtains the object’s contact point sequence from the predicted mean of object pose ${\bm{\mu}}^{o}_{t}$.

Second, the generated motion of dynamic objects typically follows human movement.
However, we observe that when the human interacts with a static object, such as sitting on a chair, the object appears slightly moved.
To address this, we immobilize the object’s movement in the generated samples if the state is static (${\bm{y}}_{0}^{s}=1$), ensuring that proper contact is established between the human and the static object.
The objective is defined as

$$ $\small{G_{sta}={\bm{y}}_{0}^{s}\cdot\sum_{l=1}^{L}\left\lVert{\bm{\mu}}_{t}^{o }(l)-\bar{{\bm{\mu}}}_{t}^{o}\right\rVert^{2},}$ (6) $$

where ${\bm{\mu}}_{t}^{o}(l)$ denotes the object’s 6DoF pose in the $l$-th frame. $\bar{{\bm{\mu}}}_{t}^{o}=\frac{1}{L}\sum_{l}{\bm{\mu}}_{t}^{o}(l)$, which is the average of predicted means of the object’s pose.

Third, we define a smoothness term $G_{smo}(\mu)$ for the object motion to mitigate motion jittering during contact.
Due to the space limit, we explain it in the supplementary material.

Finally, we combine all these goal functions to as the final objective

$$ $\displaystyle G=G_{cont}+\alpha G_{sta}+\beta G_{smo},$ (7) $$

where $\alpha$ and $\beta$ are weights for balance.

## 4 Experiments

### 4.1 Setup

Dataset.
Since the data designed for studying text-driven 3D HOIs generation is severely scarce,
we manually label interaction types, interacting subjects, and contact body parts on top of the BEHAVE dataset  []. We then use ChatGPT to assist in generating three text descriptions for each HOI sequence, increasing the diversity of the data.
Specifically, BEHAVE encompasses the interactions of 8 subjects with 20 different objects. It provides the human SMPL-H representation [], the object mesh, as well as its 6DoF pose information in each HOI sequence. To ensure consistency in our approach, we follow the processing method used in HumanML3D [] to extract representations for 22 body joints.
All the models are trained to generate $L=196$ frames in our experiments.
In the end, we have 1451 3D HOI sequences along with textual descriptions to train and evaluate our proposed approach.
We follow the official train/test split on BEHAVE.
We provide more details of the dataset, our annotation process, and annotated textual examples in the supplementary material.

In addition, we evaluate our approach on OMOMO dataset [].
OMOMO focuses on full-body manipulation with hands. It consists of human-object interaction motion for 15 objects in daily life, with a total duration of approximately 10 hours.
It provides text descriptions for each interaction motion.
We utilize their object split strategy for both training and evaluation, ensuring the objects between the training and testing sets are different. Additionally, we preprocess human and object motion, similar to our way for the BEHAVE dataset.
More details of the OMOMO dataset are in the supplementary material.

Evaluation metrics.
We first assess different models for human motion generation using standard metrics as introduced by [], namely Fréchet Inception Distance (FID), R-Precision, and Diversity.
FID quantifies the discrepancy between the distributions of actual and generated motions via a pretrained motion encoder. R-Precision gauges the relevance between generated motions and their corresponding text prompts. Diversity evaluates the range of variation in the generated motions. Additionally, we compute the Foot Skating Ratio to measure the proportion of frames exhibiting foot skid over a threshold (2.5 cm) during ground contact (foot height $<$ 5 cm).

To evaluate the effectiveness of HOIs generation, we report the Contact Distance metric, which quantitatively measures the proximity between the ground-truth human contact joints and the object contact points.
Ideally, we should develop similar metrics, *e.g*., FID, to evaluate the *stochastic* HOI generation.
However, due to the limited data available in BEHAVE [], training a motion encoder would produce biased evaluation results.
To mitigate this issue, we resort to user studies to quantify the effectiveness of different models. Details will be introduced later.

### 4.2 Comparisons with Existing Methods

Baselines. Our work introduces a novel 3D HOIs generation task not addressed by existing text-to-motion methods, which focus exclusively on human motion generation without accounting for human-object interactions.
To compare with existing works, we mainly focus on evaluating human motion generation.
We then design different variants of our models for comparing 3D HOIs generation.
Specifically, we adopt the prominent text-to-motion methods MDM [] and PriorMDM* [] with the following settings.
(a) MDM${}^{finetuned}$: In this setup, we fine-tune the original MDM model [] on the BEHAVE dataset [].
(b) MDM*: This variant involves adapting the input and output layers’ dimensions of the MDM model [] to accommodate the input of 3D HOI sequences. This adjustment allows for the simultaneous learning of both human and object motions within a singular, integrated model.
(c) PriorMDM* []: We adapt the ComMDM architecture proposed in [], originally designed for two-person motion generation, to suit our needs for HOIs synthesis by modifying one of its two branches for object motion generation.
(d) InterDiff []: While InterDiff is not designed for text-driven synthesis of 3D HOI, we added text conditioning to InterDiff as the baseline.
More details in the supplementary material.

**Table 1: Quantitative results on the BEHAVE [] dataset. We compare our method with baselines adapted from existing models. MDM${}^{finetuned}$: fine-tune the original MDM model on the BEHAVE dataset. MDM*: adapting the input and output layers’ dimensions of the MDM model [] to accommodate both human and object motions. PriorMDM*: We adapt the ComMDM architecture proposed in []. The right arrow $\rightarrow$ means closer to real data is better.**
|  | FID $\downarrow$ | R-precision $\uparrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Diversity $\rightarrow$ | Contact Distance $\downarrow$ | Foot Skate Ratio $\downarrow$ | User Study $\uparrow$ |  |  |
|  |  |  |  | (preferred) |  |  |
| Real | 0.096 | 0.324 | 7.244 | - | - | - |
| MDM${}^{finetuned}$ [] | 3.832 | 0.198 | 7.976 | - | - | - |
| MDM* [] | 3.169 | 0.206 | 7.606 | 0.448 | 0.184 | 0.87% |
| PriorMDM* [] | 4.352 | 0.215 | 8.186 | 0.416 | 0.270 | 16.53% |
| InterDiff [] | 7.192 | 0.170 | 7.686 | 0.506 | 0.218 | 0.00% |
| Ours (Full) | 1.501 | 0.235 | 7.391 | 0.347 | 0.182 | 82.60% |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: My main concerns lie in the motivation of the paper, the claim of the introduction and the physical constraints of the generated results. If the concerns are well addressed, I am willing to raise my rating.
soundness: 3
strengths: 1. The topic of text-driven HOI synthesis is of large importance in the community;
2. The annotation on BEHAVE is beneficial for the community;
3. Experimental results show the effectiveness of the proposed method;
4. The paper is well written and organized.
summary: This paper address the problem of text-driven 3D human-object interaction synthesis. The task is decomposed into simpler sub-tasks, i.e., human and object motion generation, affordance estimation and affordance-guided interaction correction. A dual-branch diffusion model (DBDM) is developed to generate the coarse human and object motion. Then, an affordance prediction diffusion model is designed to predict the contacting area between the human and object. Finally, the estimated contacting points are incorporated into the classifier-guidance to achieve accurate and close contact between humans and objects. Experiments on BEHAVE and OMOMO show realistic HOI generation results.
weaknesses: 1. The motivation of decomposing HOI synthesis into simpler sub-tasks is not clearly elaborated, which should be discussed. Why the model generate a roughly coherent HOI motion at first rather than directly generate a high quality HOI motion?
2. The authors claim that the text-driven 3D HOI synthesis with a diverse set of interactions is under-explored and existing datasets lack either HOIs or textual descriptions. However, many works have been proposed for this task such as CG-HOI and CHOIS. They also annotate the BEHAVE dataset with textual descriptions or build 3D HOI dataset with textual descriptions, such as OMOMO. The claim should be modified in the introduction section;
3. The authors claim that the model can produce physically plausible results, yet no physical constraints are integrated to guarantee that. And based on the visualization results, the physical constraint cannot be kept, mainly due to the dataset.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In the Line 256~257, why consider these 8 body joints? And which eight? Only five are listed in the paper.

2. In the experiment section, most of the metrics are general human motion generation metrics, only the contact distance is for human-object generation. Why not use the metrics of InterDiff for human-object generation, such as MPJPE-H, Trans. Err, Rot. Err, MPJPE-O. Otherwise, it is difficult to measure the effect of the method on human-object generation.

3. Why are the hands barely moving? They look flat.
soundness: 2
strengths: Here I highlight the strengths of the paper:

1. The modular design that starts with a coarse grain and refines to a fine grain is valuable for human-object interaction generation.
2. The ablation study is sufficient to prove the effectiveness of the proposed method.
3. The presentation of this paper is clear, especially the pipeline of the proposed method.
summary: This paper presents a text-driven human-object interaction generation method with a modular design. The proposed method includes three steps, a dual-branch diffusion, an affordance prediction diffusion model, and the refinement stage. Firstly, the dual-branch generates a coarse human-object motion. Then the affordance prediction diffusion model predicts the contacting points. Finally, the refinement stage refines the coarse human-object motion by the contacting points. In the experimental section, the proposed method is evaluated on public datasets, BEHAVE and OMOMO.
weaknesses: Although the coarse-to-fine modular design is valuable, this paper only considers the contact between body parts and objects. This method neglects the important contact between hand and object in human-object interaction. Here, I highlight my concerns.

1. Lack of citations and discussion of related work. Most of the methods cited and compared are those before 2023. There are a lot of related works that appear in 2024, such as [1,2,...]. These new works should be discussed and compared.

[1]CG-HOI: Contact-Guided 3D Human-Object Interaction Generation
[2]HOIAnimator: Generating Text-prompt Human-object Animations using Novel Perceptive Diffusion Models
[3]InterFusion: Text-Driven Generation of 3D Human-Object Interaction
[4]Controllable Human-Object Interaction Synthesis
[5]F-HOI: Toward Fine-grained Semantic-Aligned 3D Human-Object Interactions

2. The hand-object interaction modeling is neglected. The proposed method only models the contacting points between body parts and the object, which is very coarse-grained for human-object interaction. This causes the method to generate effect distortions in the contact area between hand and object. 

3. The compared methods are almost all human motion generation methods; only InterDiff is a human-object interaction generation method. There is a lack of comparison of human-object interaction generation methods, making it difficult to evaluate the effect of the proposed method in this domain.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: None
soundness: 2
strengths: The use of a dual-branch diffusion model combined with an affordance prediction model allows for accurate and diverse generation of human-object interactions, addressing the complexity of such tasks effectively.

The model demonstrates superior performance on standard metrics against baseline models, particularly in generating interactions with unseen objects, which showcases its robustness and ability to generalize across different scenarios.
summary: The paper focuses on generating realistic 3D human-object interactions (HOIs) driven by textual prompts using a dual-branch diffusion model (DBDM) and an affordance prediction diffusion model (APDM). It introduces a novel method to estimate contact points and ensure accurate interactions between humans and objects, enhancing the realism and diversity of the generated interactions. The approach is tested on the BEHAVE and OMOMO datasets, demonstrating its effectiveness in handling dynamic objects and generating physically plausible interactions that align closely with textual descriptions.
weaknesses: The model's performance heavily relies on the quality of datasets like BEHAVE and OMOMO. If these datasets are limited or biased towards specific types of interactions, the model's generalization ability may be compromised. I request authors to test the model with completely unseen objects not included in the BEHAVE or OMOMO datasets. For example, using a text-to-3D related model [1] to create a new object and showing inference results for that object.
[1] DreamFusion: Text-to-3D using 2D Diffusion, ICLR 2023

The qualitative results show that, while the output aligns with text better than the baselines compared with, the generated result can still be physically implausible despite the introduction of Affordance Prediction Diffusion Model, for eg, objects can be seen floating/being carried without enough contact to hold them in place. Also significant penetration is visible between the body and object, but to a much lesser extent than the methods compared against. I'd like to ask authors if they have any plan to add a loss function (e.g. contact loss) especially designed to this issue. Also, I think more sophisticated metric for measuring the penetration, such as "Intersection Volume," needs to be included to properly check the "physical plausibility of the motions". The measure is basically calculated for penetrated volumes; for details on relevant metrics, consider referring to the paper: Physics-aware Hand-object Interaction Denoising. CVPR 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	What’s the feature extractor you used in the evaluation? Did you re-train them or just use the original feature extractor (Guo et al.,  2022)?
2.	Is the affordance in APDM estimated for the whole HOI sequence or for every single frame?
3.	In the affordance estimation stage, you only consider eight body joints for contact. Is this enough for modeling the HOI process? For example, one may kneel on the yoga ball with his knees, which are not included.
4.	Does the model generate constant motion length (as mentioned in L335)?
soundness: 4
strengths: 1.	The novel decomposition of the complex task into sub-tasks allows for specialized models to handle different aspects of the problem, and the integration of them brings better performance in generating HOIs.
2.	The authors manually annotate the BEHAVE dataset with text label, which extends the HOI dataset with language modality and enables downstream tasks such as text-to-hoi generation, motion captioning, etc.
3.	The authors conduct various experiment to validate the effectiveness of the method, and also perform generalization test on the unseen OMOMO dataset.
4.	The paper is well-written and easy to understand.
summary: This paper introduces a novel approach for generating realistic 3D human-object interactions(HOIs) based on textual prompts and object geometry. The authors propose a modular design that decomposes the task into simpler sub-tasks: a dual-branch diffusion model(DBDM) for generating human and object motions, an affordance prediction diffusion model(APDM) for estimating contact information, and classifier-guidance mechanism for correcting interactions to ensure accurate contacts. Additionally, the authors annotate the BEHAVE dataset with text labels to enable the training of models. The method is evaluated on both the BEHAVE and OMOMO datasets, with results showing the ability to produce realistic and diverse HOIs with different types of objects.
weaknesses: 1.	The R-precision, FID and Diversity metric only evaluate the human motion with the text prompt, rather than the whole generated HOI, which cannot comprehensively measure the quality of the generated HOI. The Contact Distance alone only measures the quality of generated contact. Is there any additional metric that could assess the full HOI? For example, whether the human penetrates with objects could be evaluated by Interpenetration Ratio metric. I hope the authors could provide more effective metrics and perform some evaluation on them. 
2.	Also, the R-precision value is far too low in comparison with the ground truth(and also other T2M works, for example, MLD achieves 0.796 in R-Precision@3 in HML3D test set), which may indicate the inconsistency between text prompt and generated HOI. Would you please explain why this occurs and some possible solutions? And I’d like to see some comparisons with more recent Text2HOI works such as Controllable HOI(Jiaman Li, et al).

Meta-review: This paper proposes a method for synthesizing 3D hand-object interactions from text.  

The reviewers appreciate the good results as well as extension of existing datasets with text annotations.

Several weaknesses and questions were raised, concerning the motivation, clarity and experimentation.  There are several questions regarding the evaluation metrics and lack of comparisons to qualified baselines.  These points are somewhat addressed in the author responses, but not in a very convincing manner.  

The final scores are 3 borderline reject and 1 borderline accept (reviewer EXrp).  Reviewer EXrp also states explicitly a lack of support for the paper.  After reading the paper, review and author responses, the AC recommends that the paper be rejected.  Authors are recommended to revise their papers based on the reviewer questions and responses.  Furthermore, they are recommended to adjust their experiments and evaluation measures to further demonstrate the effectiveness of their approach and improvements of existing works.

Decision: Reject