CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects
Authors: Huaijin Pi
Zhi Cen
Zhiyang Dou
Taku Komura
Venue: NeurIPS 2025 poster ()
Keywords: Whole-body motion generation
human-object interaction
articulated object manipulation
Primary area: applications

Abstract: Synthesizing whole-body manipulation of articulated objects, including body motion, hand motion, and object motion, is a critical yet challenging task with broad applications in virtual humans and robotics.
The core challenges are twofold.
First, achieving realistic whole-body motion requires tight coordination between the hands and the rest of the body, as their movements are interdependent during manipulation. 
Second, articulated object manipulation typically involves high degrees of freedom and demands higher precision, often requiring the fingers to be placed at specific regions to actuate movable parts.
To address these challenges, we propose a novel coordinated diffusion noise optimization framework.
Specifically, we perform noise-space optimization over three specialized diffusion models for the body, left hand, and right hand, each trained on its own motion dataset to improve generalization.
Coordination naturally emerges through gradient flow along the human kinematic chain, allowing the global body posture to adapt in response to hand motion objectives with high fidelity.
To further enhance precision in hand-object interaction, we adopt a unified representation based on basis point sets (BPS), where end-effector positions are encoded as distances to the same BPS used for object geometry.
This unified representation captures fine-grained spatial relationships between the hand and articulated object parts, and the resulting trajectories serve as targets to guide the optimization of diffusion noise, producing highly accurate interaction motion.
We conduct extensive experiments demonstrating that our method outperforms existing approaches in motion quality and physical plausibility, and enables various capabilities such as object pose control, simultaneous walking and manipulation, and whole-body generation from hand-only data.
The code will be released for reproducibility.

CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects
Huaijin Pi
Zhi Cen
Zhiyang Dou'
Taku Komura
The University of Hong Kong ZZejiang University
https://phj128 .et..ggtt............ po/page/CoA
(a) Text ("a person uses mixer") ) whole-body
(b) Keyframe object pose whole-body
(c) Hand only trajectories whole-body motion
Walking and opening the box simulaneously
Our approach enables: (a) generating whole-body manipulation of articulated objects from text input (e.g.. "a person uses the mixer"): (b) manipulating the object to target pose and articulation (the blue object the target pose): (c) synthesizing whole-body motion guided by trajectories from hand-only data; (d) generating motions involving simultaneous walking and object manipulation (e.g., opening box while walking).
Figure 1:
Abstract
Synthesizing whole-body manipulation of articulated objects, including body mo- tion, hand motion, and object motion, is a critical yet challenging task with broad applications in virtual humans and robotics. The core challenges are twofold. First, achieving realistic whole-body motion requires tight coordination between the hands and the rest the body, as their movements are interdependent during manipulation. Second, articulated object manipulation typically involves high degrees of freedom and demands higher precision, often requiring the fingers to be placed at specific regions to actuate movable parts. To address these challenges, propose a novel coordinated diffusion noise optimization framework. Specifically, we perform noise-space optimization over three specialized diffusion models for the body, left hand, and right hand, each trained on its own motion dataset to improve generalization. Coordination naturally emerges through gradient flow
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

along the human kinematic chain, allowing the global body posture to adapt in response to hand motion objectives with high fidelity. To further enhance precision in hand-object interaction, we adopt unified representation based on basis point sets (BPS), where end-effector positions are encoded as distances to the same BPS used for object geometry. This unified representation captures tineggriiedd spatial relationships between the hand and articulated object parts, and the resulting trajec- tories serve largets to guide the optimization of diffusion noise, producing highly accurate interaction motion. We conduct extensive experiments demonstrating that our method outperforms existing approaches in motion quality and physical plausi- bility, and enables various capabilities such as object pose control, simultaneous walking and manipulation, and whole-body generation from hand-only data. The code will be released for reproducibility.
1 Introduction
Human-objer interaction (HOI) motion generation [47, 54] has broad applications in virtual reality, character animation [5. 68, 48], and robotics. These interactions range from simple activities like sitting on chair [54, 47] to more complex tasks involving articulated object manipulation [6, 21], such as opening box or a microwave. This paper focuses on the challenging setting of whole-body manipulation of articulated objects. Given initial pose of the human and the object, along with textual instruction, our goal is to synthesize realistic, physically plausible interaction sequences that involve coordinated body, hand, and articulated object motion.
Most prior works on HOI generation [16, 47, 74, 4, 28, 29, 8] suffer from two key limitations. First, they typically focus on either body-only motion [47, 28, 29] or hand-only manipulation [73, 81, 6, 21, 8]. Although hand-only methods can produce plausible contact behaviors in short- range scenarios, they fail to capture important whole-body dynamics such as bending down, reaching forward, or walking while manipulating objects. Such whole-body behaviors are essential for generating realistic human-object interactions, especially when manipulation is not restricted to fixed space. Second, most existing works target rigid objects [70, 29, 8], while articulated objects introduce more complex motion patterns and require continuous hand adjustments.
Whole-body manipulation of articulated objects highly challenging. First, demands coordinated motion between the body and hands to reflect natural physical behaviors. Body movement affects how the hands approach and manipulate objects, and conversely, hand-object interactions can influence global posture. Second, precise control of finger positions is essential to maintain accurate, physically plausible contact throughout the sequence. This is especially important for articulated objects, where the manipulation often requires placing the fingers at specific regions to actuate the articulation while avoid colliding with other parts.
To address these challenges, we propose a novel framework called CoDA (Coordinated Diffusion noise optimization for whole-body manipulation of Articulated objects), which jointly synthesizes the motions of the human body, hands, and articulated objects. Our core idea is to optimize the input noise vectors of three specialized diffusion models, which independently model the body, left hand, and right hand, to generate coordinated whole- body motion. This decoupled design allows each component to be trained on its own data source, such as using large-scale datasets like AMASS [38] for body motion, manipulation datasets like ARCTIC [13] and GRAB [56] for hand motion, thereby improving generalization across diverse motions. Coordination naturally emerges during optimization, as gradients from hand motion objectives flow through the human kinematic chain, allowing the global posture to adapt in response to fine-graine hand motion. This optimization further enables precise control over hand-object contact, while the diffusion noise space [25] provides strong motion priors to preserve naturalness in the generated sequences.
To enable precise manipulation while accounting for object geometry and articulation, we adopt a basis point set (BPS) representation [49, 81] encode both the object surface and end-effector trajectories in a unified form. Specifically, we represent the positions of the end-effectors. namely the wrists and fingertips, by their distances to the same BPS used for encoding the object geometry. The unified representation captures the relative spatial relationship between the hand and the object geometry as well as its articulation during complex manipulation tasks. The generated trajectories, based on this representation, provide continuous target signal for optimizing whole-body motion.

---

We evaluate our approach on both the ARCTIC [13] dataset of articulated object manipulation and the GRAB [56] dataset of rigid object interactions. Our method achieves state-of of-the-art performance on both benchmarks, outperforming existing approaches in motion quality and physical plausibility. Beyond benchmark evaluation, our framework enables several compelling capabilities, as illustrated in Figure 1. It supports object pose control at specific times, and coordinated whole-body behaviors involving simultaneous locomotion and manipulation, which are absent from the ARCTIC dataset. In addition, our framework allows to leverage hand-only datasets [2] to generate whole-body motion, enabling broader data usage and generalization. To the best of our knowledge, this is the first work to jointly generate body, hand, and articulated object motions for whole-body manipulation tasks.
2 Related work
Human-object interaction. Human-object interaction (HOI) generation [54, 28, 9] has received increasing attention due to its potential to enable virtual humans to perform various actions in 3D environments. Early works focus on generating static interactions such as sitting or lying on furniture [54, 16, 82, 84, 80), using either auto-regressive pipelines or whole-sequence generation [61, 62, 40, 1, 86]. Recent methods explore diffusioo-bbsse models [22, 47, 27, 4, 74, 23] and apply guidance techniques [11, 19] to improve human-scene contact quality. Beyond static objects, several works consider dynamic objects [70, 71] or generate human motion conditioned on given object trajectories [28, 10J. For example, OMOMO [28] proposes two-stage framework that first generates wrist trajectories and then completes body motion accordingly. Other approaches [45, 70, 29, 12, 53, 72] jointly generate body and object motion, and incorporate contact-aware guidance into the diffusion process to improve the quality. Another line of research [17. 42, 69, 59, 43, 63] enables physically simulated characters to perform scene-level interactions by learning control policies through environment interaction. These methods mainly focus on navigation and interactions with large-scale objects such as furniture obstacles. While generating plausible body motion, they ignore finger motion, which is crucial for -reiggraied manipulation.
Hand-object interaction. ManipNet [77] synthesizes object manipulation given wrist and object trajectories, using multiple representations to model the hand-object relationship. GRIP [58] design temporal hand-object spatial feature for stable grasping. Some works [87, 33] address the task of denoising noisy hand motion to recover clean interaction sequences. While these methods explore various representations for modeling hand-object spatial relationships, they rely on access to predefined wrist and object trajectories. [85, 81, 79] explore settings where only the object trajectory is provided. CAMS [85] introduces a canonicalized representation to enable precise contact generation. [81, 79] generate manipulation by predicting contact maps as intermediate representations. Other works generate hand and object motion jointly, without relying on predefined trajectories. DiffH20 [8] applies grasp guidance to diffusion models for more coherent hand-object interactions.. Text2HO1
[6] employs cascaded diffusion to iteratively refine the results. HOIGPT [21] leverages separate codebooks for hands and objects, and jointly predicts motion and text. Physics-based approaches
[7, 78] generate grasping motions through reinforcement learning in simulated environments. Despite their differences, all these methods ignore the body context, resulting in floating hand motions.
Whole-body interaction. Although there are several whole-body manipulation datasets [56, 13, 24, 23, 76, 37, 34], only few works consider body and hand interaction simultaneously. [57, 66] assume the object is static and only synthesize approaching and grasp motion. IMoS [14] demonstrates full-body manipulation with given finger motion; it generates body motion auto-regressively and optimizes object trajectories by assuming a static hand-object contact frame. TOHO [30] synthesizes whole-body interactions using implicit representations [18], relying on the same contact assumption to recover object motion. DiffGrasp [83] generates whole-body motion conditioned on given object trajectories using diffusion models, and introduces hand-object guidance to improve interaction quality. Wu et.al. [67] employs LLM [41] to analyze the scene and plan motions for grasping and relocating rigid objects. Other works [64, 65, 75] employ physics- based tracking to mimic manipulation behaviors. [3, 36, 31] explores humanoid grasping, but the generated motions remain unnatural and do not involve complex manipulation. Most of the above methods focus exclusively on rigid object interaction and do not address articulated objects. Compared to rigid object interaction, articulated object manipulation is more complex, as it often requires placing the fingers at specific regions to actuate the articulation.
3

---

Objjcct
person uses the box."
BPS
Initial
Text
Object Trajectory Dillusion
Right Hand Diffusion
Body Dillussion
Left Hand Diffusion
Moving Object
Forward Kinematics
BPS
Object Trajectoy
End-effector Trajectory Diflusion
End-cfllettr BPS
...
C deefice + Apen + Areg Erey
Fnd-effictor Trojectory
(a) Object and End-effector Generation
(b) Whole-body Motion Optimization
Forward
Figure 2: Pipeline overview. (a) Given the initial human pose, object pose, and text, we first generate the articulated object trajectory and the corresponding end-effector trajectories via two conditional diffusion models. (b) We then optimize the latent noise inputs of three decoupled diffusion models by propagating gradients through the kinematic chain, guided by end-effector tracking, penetration, and regularization losses. Finally, we forward the optimized noise through the diffusion models to synthesize coherent whole-body motion aligned with generated object motion.
3 Preliminary
In this section, we define the input and output in this paper. Given the initial pose of a human and an articulated object, along with textual instruction, our goal is to generate full sequence including the whole-body human motion (body and fingers) and the articulated object motion over time.
Object representations. The objects from the ARCTIC [13] dataset are two-part articulated objects with 7 degrees of freedom. We use S, = {To, R., a} to indicate the object pose, where the object state S, RR consists of object translation T, R object rotation R. R and the angle of the rotational joint E R between the two parts of the object.
use SMPL-X [44], which is a parametric human body model to represent the whole body, including the face and fingers. SMPI.-X is differentiable function that takes input shape, pose, and expression parameters and outputs 3D mesh with 10, 475 vertices and 20, 908 triangles. The vertices posed with linear blend skinning with rigged skeleton which is learned from the data. As we focus on the body motion with two hands, we remove the face related parameters. + {0, t} is the pose parameters to drive the SMPL-X model, where O E 3222x3 represents joint angles and t E R the root translation.
Motion representations.
Text descriptions. In the ARCTIC [13] and the GRAB [56] dataset, each sequence is annotated with an action label. Following previous work [14, 6], we construct the text description using the template "A person <action> the <object>." For example, "A person the box.".
Method
The overview of our pipeline is shown in Figure 2. We first generate the motion of the articulated object (Section 4.),, then predict the end-effector trajectories (Section 4.2), and finally synthesize the whole-body motion by optimizing the noise of decoupled diffusion models (Section 4.3).

---

4.1 Object motion generation
Given the initial object pose and the textual instruction, we train a diffusion model [60] to generate the object future trajectory. The input includes the CLIP [50] feature of the text, the initial object pose, and the object geometry embedding. We represent the object geometry using the normalized part-based BPS descriptor [81], which will be formally defined in Section 4.2 and Figure 3. The output a sequence of object states over time.
4.2 End-effector trajectory generation
Given the generated object trajectory, we extract its geometry representation and combine with the trajectory itself and the textual instruction as input to a diffusion model that predicts end-effector trajectories. Instead of directly predicting 3D joint coordinates [28], we design a distance- based representation that encodes end-effector positions in the same space as the object geometry.
Unified BPS-based representation for object and end-effectors. We first present the object geometry representation. Following previous work [81], adopt the normalized part BPS [49] to represent the object geometry. Specifically, the object mesh is first normalized to the unit scale by dividing all vertex coordinates by the maximum distance from the object origin to any vertex. Then pre-defined fixed set of basis points P E I2 x3 shared across all objects, are uniformly sampled within the unit sphere centered at the object origin. The BPS representation distances from each basis point to the nearest vertex on each of the two rigid object parts, resulting in an object geometry vector O RRR Kx2
computed as the
We then introduce end-effector BPS, a distance-based representation tailored for encoding the positions of end-effectors in the object coordinate system. The end-effectors include both wrists and fingertips, comprising total of 12 joints (2 wrists and 10 fingertips). As shown in figure 3, at each frame, for each of the 12 end-effectors, we compute K -ddimensional vector of Euclidean distances to the basis points. We use the same pre-defined set of basis points P E IR K x3 in object geometry representation [8]] 1. This results in a (12 x K) -ddimensional end-effector BPS vector per frame. The diffusion model outputs a sequence of end-effector BPS over time, along with binary contact labels for each fingertip, indicating whether it is close the object surface.
Given the generated end-effector BPS sequence, we recover the end-effector trajectories by solving a sim- ple optimization problem. For each end-effector at each frame, minimize the following loss infer its 3D position:
P arg C(p.) Pe
(1)
C(p.)= = - Ip. Pyll2 dzl12.
Object BPS
End-effector BPS
(2)
The illustration of the end- effector BPS. (a) the object BPS [81]. (b) is the proposed end-effector BPS represen- tation. Gray points denote the basis points; pink/yellow are two object parts; blue indi- cates a fingertip. Only one end-effector and 64 basis points are visualized for simplicity.
Figure 3:
where Pe is the optimized 3D position and d, is the predicted distance to the j-th basis point P.j. By sharing the basis point set with the object BPS rep- resentation, method provides a consistent spatial reference frame that facilitates geometric alignment between end-effectors and object parts.
RoPF-based object motion encoding. To better encode the object trajectory, we adapt the idea of CaPE [26], which encodes relative camera pose information via RoPE [55]. In our case, each object pose is also represented as a 4 X 4 transformation matrix. Inspired by CaPE [26]. we use the object pose to transform the query and key features in each attention layer. This enables the model encode the relative object motion within local temporal window, providing temporally-awee conditioning for the generation. We refer readers the supplementary material for more details.
4.3 Whole-body motion generation
The goal of this stage is to generate coherent whole-body motion that aligns with the predicted end-effector trajectories and articulated object motion. Rather than directly predicting whole-body
5

---

poses conditioned on end-effectors [28], we adopt an optimization based approach inspired by DNO [25]. Specifically, we optimize the noise input the diffusion models (Figure 2 (b)), and then forward the optimized noise through the diffusion models to generate the final motion. To further improve motion quality, we decouple the body into three components: body, left hand, and right hand, and train separate diffusion models for each. This decoupled design enables us to train each module using individual data, such as training the hand models using the ARCTIC [13] and GRAB [56]. and the body-only model without hands on the AMASS [38]. Such specialization improves generalization by allowing novel combinations of finger and body motion to be synthesized. Moreover, this formulation facilitates gradient flow through the kinematic chain during optimization, which improves coordination between the body and hands.
Decoupled motion diffusion model. We adopt decoupled human representation for whole-body motion, dividing the human pose into three components: body, left hand, and right hand. Formally, for each frame i, the whole-body pose O is represented
x= {xb, Xth; Xrh
(3)
fire 1², r pa 0-
(4)
X
[0
(5)
X, = (
(6)
where H6 denotes the body component, including root velocities 1" E R (projected on the XZ- plane), root height r2 E R. angular velocity j" E R. and body joint rotations O, E R6x,,, while Ou. E R and 4 6x represent the left and right hand joint rotations, respectively. All joint rotations are encoded using the 6D representation [88], with Js = 22, Ju, = Jrh = 15 joints for the body and each hand. We train three separate diffusion models, M,. M and M, model the motion manifolds of the body and hands individually.
Optimization over diffusion noise. Given the trained diffusion models for body, left hand, and right hand, we optimize the noise vectors z = {za, Zth, Zrn J to generate whole-body motion as shown in Figure (b). Let fz) denote the process that maps the input noise to global joint positions through diffusion models and forward kinematics:
(z) (2)) Mn. (Zu)), My (2,a));
(7)
where FK(()) converts root translation and local joint rotations into global joint positions. We formulate motion generation as minimizing a loss L over the diffusion noise:
z = arg (z))
(8)
x
The overall loss function consists of three components with different weights Aees Apen. and Xvegi
L = Xee + Apen + Areg
where Lee, Epen. and Creg are the end-effector tracking. penetration, and regularization losses.
We encourage the generated global fingertip positions py to follow the predicted trajectories Py from the previous stage. We also constrain the relative fingertip positions to the wrist joints:
Lee p p |b"
p/111, -071-1
(10)
where pfj and pyj denote the relative fingertip positions with respect to the wrist.
To reduce hand-object interpenetration. we penalize fingertip joints that fall inside the object mesh:
Cpen Σ min (SDF(J*) - 0.01, 0.00)
(11)
where SDF(J') is the signed distance the j-th hand joint, assuming 1cm finger thickness.
We add regularization term to discourage foot floating and foot sliding:
Greg |lmin (J - 0.02]], + 1 || - -J' J',ll, + 1 right || - -J 11
(12)
where J denotes the height of all joints in the body, and JK and J denote the 3D positions of the left and right foot joints at frame i, respectively. The binary indicators denote whether the left or right foot is in contact with the ground, based height threshold of 0.02 meters.

---

We adopt DDIM [52] sampling to efficiently generate motion sequences during optimization following DNO [25]. The loss computed on the final output, and gradients are propagated back through the DDIM solver to update the noise. After optimization, we pass the optimized noise into the decoupled diffusion models to generate the final whole-body motion. Combined with the previously generated object trajectory, this yields a complete human-object manipulation sequence. This noise- space optimization avoids high-dimensional pose regression, reduces artifacts, and produces natural whole-body motions aligned with the object manipulation process.
5
Experiments
5.1
Implementation details
We adopt a transformer- based diffusion architecture similar to MDM [60] for all models in our framework. During inference, we perform noise optimization using DDIM [52] with T=10 for 800 steps and a assine-decayed learning rate, following the DNO [25] strategy. All experiments are conducted on single NVIDIA A A100 GPU. More training details are in the supplementary material.
5.2 Dataset and evaluation metrics
Dataset. We evaluate ARCTIC [13] for artieulated object manipulation and on GRAB [56] for rigid object interaction. ARCTIC contains around 2 hours of motion data featuring 10 subjects interacting with 11 articulated objects, including complex motions such bimanual grasps and in-hand manipulation. Following the protocol in [81], we randomly sample 4 sequences per object category to construct the test set. The GRAB dataset covers about hours of interaction from 10 subjects with 51 rigid objects, focusing primarily on grasping and simple lifting actions. Similar to [14], we use data from the last subject as the test set. For training object motion and end-effector trajectories generation, ARCTIC is used for articulated objects, and GRAB is used for rigid objects. The body motion model trained on ARCTIC, GRAB, and AMASS [38], while the two hand motion models are trained on ARCTIC and GRAB.
Evaluation metrics. Similar to [8, 6], we evaluate the motion quality using the following metries: (I) Frechet Inception Distance (FID) measures the feature-level distance between generated and real motions, using a motion feature extractor trained on the dataset following [15]. (2) R-Precision quantifies the alignment between generated motion and the corresponding textual prompt, measured using Top-3 accuracy. (3) Diversity reflects the variation among generated motion samples. (4) Foot skating indicates motion realism by detecting undesired foot sliding, following the computation in [32, 47]. We additionally report physical realism metrics following [8]: (5) Interpenetration volume (IV) computes the number of hand vertices that penetrate the object mesh. (6) Interpenetration depth (ID) measures the maximum penetration depth of hand vertices into the object. (7) Contact ratio (CR) is defined the average proportion of hand vertices within 5 mm of the object surface. We also conduct a user study involving 16 participants to evaluate the generated motion sequences.
5.3 Comparison with baselines
Baselines. As there is no existing method that jointly generates body, hand, and articulated object motion, we adapt several representative methods our task: IMoS [14], MDM [60], OMOMO (28), Text2HO1 [6], and CHOIS [29]. IMoS is a CVAE-based [51] auto-regressive model, while MDM is full-sequence diffusion-based [20] model. Tex12HO1 is originally designed for hand-object interaction with multiple diffusion models for iterative refinement. CHOIS is a diffusion-based model that incorporates contact guidance during inference. We extend them to jointly generate whole-body motion and object motion. OMOMO first generates wrist motion and then synthesizes body motion. We extend it to a three-stage model: first generating object motion, then predicting fingertip and wrist trajectories. and finally producing shole-body motion. OMOMO+DNO further extends OMOMO by using its diffusion model latent prior and applying DNO [25] to refine the generated results.
Quantitative results. We report quantitative results on ARCTIC and GRAB in Table 1 and Table 3, and user study results in Table 2. Our method achieves the best performance on nearly all metrics across both datascts. While it ranks slightly lower in diversity, it significantly outperforms all
7

---

Table 1: Comparison on the ARCTIC [13] dataset. The right arrow - means the closer to real motion the better. IV, ID, and CR denote interpenetration volume, interpenetration depth, and contact ratio. The best and second-best results are highlighted green and yellow, respectively.
Methods
FIDL
R- R-Precision"
Diversity
Foot skating.
IV.
IDI
CRT
Real
0.531
8.664
0.002
4.68
11.47
0.085
IMOS [14]
6.686
0.305
6.144
1.469
14.28
13.24
0.010
MDM [60]
3.972
0.209
8.167
0.027
16.90
15.85
0.033
Tex22OO [6]
6.654
0.234
5.923
0.028
12.72
17.14
0.010
OMOMO [28]
3.710
0.406
6.110
0.028
13.77
15.16
0.061
OMOMO + DNO [25]
2.873
0.391
7.004
0.022
8.95
13.30
0.075
CHOIS [29]
3.758
0.367
7.423
0.023
17.19
15.84
0.030
Ours
2.283
0.477
7.208
0.002
5.25
12.87
0.086
Table 2: User study on the ARCTIC [13] dataset.
Metries
Ours
CHOIS [29]
OMOMO [28]
Tex12HO1 [6]
Best Motion Realism Rate f
88.7%
1.1%
9.9%
0.3%
Best Physical Plausibility Rate
87.3%
1.4%
10.2%
1.1%
baselines in the study, indicating superior perceptual quality and physical plausibility. To assess the physical feasibility of our method, we conduct mimic-based evaluation following (46, 64), where humanoid policy is trained reproduce the generated motions in the IsaacGym [39] simulator. We use eight generated sequences (each 10 seconds long) involving boxes and microwaves, and measure the tracking duration-the time (in seconds) during which both object and joint position errors remain below a 10 cm threshold. Results in Table 4 show that our motions lead to longer tracking durations compared OMOMO, demonstrating improved physical plausibility and better compatibility with downstream humanoid execution. In addition, to evaluate the generalization capability of our method, we conduct an additional experiment on the ARCTIC dataset by holding out the box object for testing and training on the remaining objects. As shown in Table 5, our method achieves substantially lower FID compared to the baseline, demonstrating better object-level generalization.
Qualitative results. As demonstrated in Figure 4, method achieves significantly better hand-object contact compared to baselines. We provide more results in the supplementary material.
5.4 Ablation study
We ablate key components of our framework to understand their impact on overall performance: (a) A single model to jointly predict object motion and end-effector trajectories. (b) Predicting relative coordinate of end-effectors the object center without end-effector BPS. (c) Using object velocity and rotational velocity as the trajectory input without RoPE-based representation. (d) Removing the optimization process and using a conditional diffusion model with fingertip trajectories as input. (e) Using single diffusion model for the entire body without the decoupled body-hand representation. (f) Excluding the AMASS [38] dataset during training the body motion model. (g) replacing the end-effector representation with a distance field [77], where the trajectory is encoded as fixed grid
of distances in the object's local coordinate frame, while keeping the object geometry encoded using
Table 3: Comparison the GRAB [56] dataset.
Methods
FIDI
R-Precisiont
Diversity
Foot skatingl
IV_
ID4
CR1
Real
0.727
15.045
0.010
5.81
13.11
0.049
IMoS [14]
52.290
0.180
8.374
0.152
11.57
20.35
0.000
MDM [60]
26.734
0.289
8.627
0.109
12.96
16.03
0.001
Tex12HO1 [6]
30.101
0.320
10.302
0.086
12.52
14.55
0.000
OMOMO [28]
25.017
0.391
9.294
0.094
11.03
14.03
0.004
CHOIS [29]
25.835
0.320
9.887
0.055
9.31
14.37
0.002
Ours
21.544
0.438
9.387
0.046
4.93
10.23
0.040

---

Table 4: Physical feasibility evaluation on the ARCTIC [13] dataset.
Metrics
OMOMO [28]
Ours
Tracking Duration
4.75
8.75
Table 5: Comparison on the ARCTIC [13] dataset with held-out box object.
Methods
FIDL
R-Precision?
Diversity->
Foot skating]
IV
ID]
CR1
OMOMO [28]
44.009
0.547
6.234
0.028
26.92
8.23
0.116
Ours
16.091
0.547
4.964
0.002
26.24
8.56
0.128
BPS. (h) conditioning the hand motion diffusion model on object trajectories. As shown in Table 6, each component contributes the performance improvement.
5.5 More discussions
Generalization to different object geometry. To further validate generalization to unseen object geometries of the same category, we train the object motion and end-effector trajectory models on the hand-only dataset [85], using 7 training and 3 testing objects. Despite the dataset containing only hand motion, our method successfully generates whole-body motion, as shown Figure
Various capabilities. Our approach enables various capabilities. First, it allows control over keyframe object poses by setting them as opti- mization targets for object trajectory generation. Second, it can synthesize whole-body motions that involve simultaneous locomotion and ma- nipulation, even though such combinations are present the training dataset [13]. Third, enables generating whole-body motion guided by hand-only datasets [2], using wrist and fin- gertip trajectories as optimization targets. We provide more qualitative results in the supple- mentary material.
Figure 5: Generalization to different object ge- ometry. We train the object motion and end- effector trajectory models on hand-only data [85] with diverse object geometries. These models integrated into our framework to provide optimiza- tion targets, enabling realistic whole-body motion synthesis for unseen object geometry.
Deployment on simulated humanoids As shown Figure 6, our generated whole-body motion serve reference for controlling humanoids in physics-based simulators. We apply physical motion tracking methods [46, 35, 64] to track the synthesized motions. The humanoid is able to physically interact with objects and perform coordinated manipulation behaviors in the simulated environment.
Inference speed We report the inference time of each module in our pipeline, measured single NVIDIA A100 GPU for generating 300-frame motion sequence. The object motion model
Table 6: Ablation study the ARCTIC [13] dataset.
Methods
FID,
R-Precisiont
Diversity-+
Foot skatingl
IDJ
CRt
Real
0.531
8.664
0.002
11.47
0.085
(a) w/o separate models
3.790
0.438
6.939
0.002
8.21
13.16
0.103
end-effector BPS
4.069
0.453
0.002
13.54
RoPE motion
2.714
0.469
7.021
0.002
6.12
12.66
0.093
(d) w/o optimization
4.883
01.114
6.406
0.030
16.39
16.13
0.095
w/o decoupled
2.699
0.438
7.142
0.008
12.45
16.29
0.082
(f) w/o AMASS
3.305
0.453
6.859
0.003
5.46
13.04
0.089
(g) w/ SDF
2.350
0..22
7.064
0.003
6.69
13.97
0.086
(h) w/ extra condition
2.998
0.453
7.212
0.002
8.95
13.30
0.075
Ours
2.283
0.477
7.208
0.002
5.25
12.87
0.086

---

OMOMO
CHOIS
Oun
Figure Qualitative comparison. Given the text "A person uses the ketchup.", our method generates the whole-body motion with better hand-object contact compared to baselines.
Figure 6: Deployment on simulated humanoids. We apply existing motion tracking techniques deploy the generated motion simulated humanoid. The articulated object physically manipulated by the humanoid within the physics simulator [39].
requires approximately 0.52 seconds, the end-effector model takes about 3.66 seconds, and the whole- body motion optimization, which involves iterative diffusion sampling and gradient-based updates, takes around 16.9 minutes. Most of the computation time is spent on the whole- body optimization stage. Although slower than feed-forward approaches such as CHOIS [29], this optimization process produces motions with substantially higher quality and physical plausibility.
Limitations. First, the optimization process is slower than other generative methods [60], limiting real-time applications. Second, due to the limited object diversity existing datasets [13], the model struggles to generalize to novel object categories. Third, our framework only focuses single-object manipulation; extending it to handle multiple interacting objects or multi-step sequential interactions remains an open direction. Finally, enabling both the body and fingers reason about and avoid obstacles in complex scenes, such as surrounding geometry or other objects, is still difficult problem.
6 Conclusion
In this paper, we present a coordinated diffusion noise optimization framework for synthesizing whole- body manipulation of articulated objects. By optimizing over the noise space of separately trained diffusion models for the body, left hand, and right hand, our method enables natural coordination between the body and hands. We introduce unified distance-batt based representation built on basis point sets to generate end-effector trajectories, facilitating precise hand-object interactions. Extensive experiments demonstrate that our approach achieves state-of-the-an performance in motion quality and physical plausibility, al supports various capabilities such as object pose control, simultaneous manipulation and locomotion, and whole-body motion generation from hand-only data.
10

---

7 Acknowledgments
This work partly supported by the Innovation and Technology Commission of the HKSAR Government under the ITSP--Plaffrm grant (Ref: ITS/333/23FP) and the InnoHK initiative (TransGP project). Part of the research was conducted in the JC STEM Lab of Robotics for Soft Materials, funded by The Hong Kong Jockey Club Charities Trust.
References
[1] Joao Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. Circle: Capture rich contextual environments. In CVPR, 2023. 3
[2] Prithviraj Banerjee, Sindi Shkodrani, Pierre Moulon, Shreyas Hampali, Shangchen Han, Fan Zhang, Linguang Zhang, Jade Fountain, Edward Miller, Selen Basol, Richard Newcombe, Robert Wang, Jakob Julian Engel, and Tomas Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. CVPR, 2025. 3,9
[3] Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Physi- cally plausible full-body hand-object interaction synthesis. In International Conference on 3D Vision (3DV), 2024. 3
[4] Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang. Zhu Shuai, Hujun Bao, and Xiaowei Zhou. Generating human motion in 3d scenes from text descriptions. In CVPR, 2024.
[5] Zhi Cen, Huaijin Pi, Sida Peng. Qing Shuai, Yujun Shen, Hujun Bao, Xiaowei Zhou, and Ruizhen Hu. Ready-to-react: Online reaction policy for two-character interaction generation. In ICLR, 2025.
[6] Junuk Cha, Jihyeon Kim, Jae Shin Yoon, and Seungryul Baek. Text2hoi: Text-guided 3d motion generation for hand-object interaction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1577-1585, 2024. 2, 3, 7,
[7] Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20577-20586, 2022. 3
[8] Sammy Christen, Shreyas Hampali, Fadime Sener, Edoardo Remelli, Tomas Hodan, Eric Sauser, Shugao Ma, and Bugra Tekin. Difth2o: Diffusion-based synthesis of hand-object interactions from textual descriptions. In SIGGRAPII Asia 2024 Conference Papers, 2024. 2, 3,
[9] Peishan Cong, Ziyi Wang. Zhiyang Dou, Yiming Ren, Wei Yin, Kai Cheng, Yujing Sun, Xiaoxiao Long, Xinge Zhu, and Yuexin Ma. Laserhuman: Language-guided scene-aware human motion generation in free environment. arXiv preprint arXiv:2403.13300 2024. 3
[10] Peishan Cong, Ziyi Wang, Yuexin Ma, and Xiangyu Yue. Semgeomo: Dynamic contextual hu-
man motion generation with semantic and geometric guidance. arXiv preprint arXiv:2503.001291 2025.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021. 3
[12] Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction genera- tion. Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 19888 19901, 2024. 3
[13] Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2, 3, 4, 6, 7,

---

[14] Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In Computer Graphics Forum, 2023. 3, 7,8
[15] Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, 2022.
[16] Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J. Black. Stochastic scene-aware motion prediction. In ICCV, 2021. 2,
[17] Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scene interactions. ACM SIGGRAPH 2023 Conference Proceedings, pages 1-9, 2023.
[18] Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. Nemf: Neural motion fields for kinematic animation. In NeurlPS, 2022. 3
[19) Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. arXiv e-prints, art. arXiv:2207.12555 12598, July 2022. 3
[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabiliistc models. In NeurIPS, 2020. 7
[21] Mingzhen Huang, Fu-Jen Chu, Bugra Tekin, Kevin J Liang, Haoyu Ma, Weiyao Wang. Xingyu Chen, Pierre Gleize, Hongfei Xue, Siwei Lyu, Kris Kitani, Matt Feiszli, and Hao Tang. Hoigpt: Learning long sequence hand-object interaction with language models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2025. 2,
[22] Siyuan Huang, Zan Wang. Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chum Zhu. Diffusion-basee generation, optimization, and planning in 3d scenes. In CVPR, 2023. 3
[23] Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, and Yixin Zhu. Au- tonomous character-scene interaction synthesis from text instruction, 2024. URL https:
/ waaxiv.org/abs///00
03187.
[24] Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1737-1747, 2024.
[25] Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwa- janakorn, and Siyu Tang. Optimizing diffusion noise can serve as universal motion priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1334-1345, 2024. 2, 6, 7,8
[26) Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew Davison. Eschernet: A generative model for scalable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9503-9513, 2024.
[27] Nilesh Kulkarni, Davis Rempe, Kyle Genova, Abhijit Kundu, Justin Johnson, David Fouhey, and Lconidas Guibas. Nifty: Neural object interaction fields for guided human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 947-957, 2024. 3
[28] Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 12(6):1-11, 2023. 2, 3, 6, 8,9
[29] Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object interaction synthesis. In European Conference on Computer Vision, pages 54 72. Springer, 2024. 2, 3, 7, 8,
[30] Quanzhou Li, Jingbo Wang, Chen Change Loy, and Bo Dai. Task-oriented human-object interactions generation with implicit neural representations. In Proceedings of the IEFF/CVF Winter Conference Applications of Computer Vision, pages 3035-3044, 2024. 3
12

---

[31] Yitang Li, Mingxian Lin, Zhuo Lin, Yipeng Deng, Yue Cao, and Li Yi. Learning physics- based full-body human reaching and grasping from brief walking references. arXiv preprint urXiv:2503.07481, 2025. 3
[32] Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. ACM Truns. Graph., 2020. 7
[33] Xueyi Liu and Li Yi. Geneoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion. In The Twelfth International Conference on Learning Representations, 2024.
[34] Jintao Lu, He Zhang, Yuting Ye, Takaaki Shiratori, Sebastian Starke, and Taku Komura. Choice: Coordinated human-object interaction in cluttered environments for pick-and-plas actions. arXiv preprint arXiv:24/2.0000 2024. 3
[35] Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 10895-10904, 2023.
(36) Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris M. Kitani, and Weipeng Xu. Omnigrasp: Simulated humanoid grasping on diverse objects. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL pettps://openrvvil net/forum? iidimmmiiiddlltee t3x00133. 3
[37] Xintao Lv, Liang Xu, Yichao Yan, Xin Jin, Congsheng Xu, Shuwen Wu, Yifan Liu, Lincheng Li, Mengxiao Bi, Wenjun Zeng, and Xiaokang Yang. Himo: A new benchmark for full-body human interacting with multiple objects, 2024. URL https://aarxii org/abs//////oeeooooooooooeee ab4/ 12371.
3
[38] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive motion capture as surface shapes. In ICCV, 2019. 2, 6, 7,8
[39] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo. Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.1044 2021. 8, 10
[40] Aymen Mir, Xavier Puig. Angjoo Kanazawa, and Gerard Pons-Moll. Generating continual human motion in diverse 3d scenes. In 2024 International Conference on 3D Vision (3DV), pages 903-913. IEEE, 2024. 3
[41] OpenAl. Openai: Introducing chatgpt. https:/ httpa////pppaaa..ocoooooo/aaaagp 2022.
[42] Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang. Synthesizing physically plausible human motions 3d scenes. 2024 International Conference 3D Vision (3DV), pages 1498-1507. IEEE, 2024. 3
[43] Liang Pan, Zeshi Yang. Zhiyang Dou, Wenjia Wang, Buzhen Huang, Bo Dai, Taku Komura, and Jingbo Wang. Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization. CVPR, 2025.
[44] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3d hands, face, and body from single image. In CVPR, 2019. 4
[45] Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. arXiv preprint arXiv:23/2.0655: 2023. 3
[46] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1-14, 2018. 8,
13

---

[47] Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, and Hujun Bao. Hierarchical generation of human-object interactions with diffusion probabilistic models. In ICCV, 2023. 2, 3.
[48] Huaijin Pi, Ruoxi Guo, Zehong Shen, Qing Shuai, Zechen Hu, Zhumei Wang, Yajiao Dong, Ruizhen Hu, Taku Komura, Sida Peng, et al. Motion-2-to-3: Leveraging 2d motion data to boost 3d motion generation. arXiv preprint wrxXv:2442.131111 2024. 2
[49] Sergey Prokudin, Christoph Lassner, and Javier Romero. Efficient learning on point clouds with basis point sets. In Proceedings of the IEEF/CVF international conference on computer vision, pages 4332 4341, 2019. 2,5
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[51] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In NearIPS, 2015. 7
[52] Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL ttte/forummid-ttlgiarce
[53] Wenfeng Song, Xinyu Zhang, Shuai Li, Yang Gao, Aimin Hao, Xia Hou, Chenglizhao Chen, Ning Li, and Hong Qin. Hoianimator: Generating text-prompt human-object animations using novel perceptive diffusion models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 81 1 1-880, June 2024. 3
[54] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character- scene interactions. ACM Trans. Graph., 2019. 2,
[55] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[56] Omid Taheri, Nima Ghorbani, Michael Black, and Dimitrios Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part IV 16, pages 581-600. Springer, 2020. 2, 3, 4, 6, 8
[57] Omid Taheri, Vasileios Choutas, Michael J. Black, and Dimitrios Tzionas. Goal: Generating 4d whole-body motion for hand-object grasping. In CVPR, 2022. 3
[58] Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, and Michael J. Black. GRIP: Generating interaction poses using latent consistency and spatial cues. In International Conference on 3D Vision (3DV), 2024. URL httpt:///gritt is tue mpg. de. 3
[59] Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG), 43(6):1-21, 2024.
[60] Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or and Amit Haim Bermano. Human motion diffusion model. In ICLR, 2023. 5, 7, 8, 10
[61] Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. In CVPR, 3
[62] Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. CVPR, 2022.
[63] Wenjia Wang, Liang Pan, Zhiyang Dou, Zhouyingcheng Liao, Yuke Lou, Lei Yang. Jingbo Wang, and Taku Komura. Sims: Simulating human-scene interactions with real world script planning, arXiv preprint arX1::2411.19921, 2024. 3
14

---

[64] Yinhuai Wang, Jing Lin, Ailing Zeng. Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics-based imitation of dynamic human- human-object interaction. arXiv preprint wrX1:2332..43993. 2023. 3, 8,9
[65] Yinhuai Wang, Qihan Zhao, Runyi Yu, Ailing Zeng, Jing Lin, Zhengyi Luo, Hok Wai Tsui, Jiwen Yu, Xiu Li, Qifeng Chen, et al. Skillmimic: Learning reusable basketball skills from demonstrations. arXiv preprint arXiv:2408. 15270, 2024.
[66] Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, and Siyu Tang. Saga: Stochastic whole-body grasping with contact. In ECCV, 2022. 3
[67] Zhen Wu, Jiaman Li, Pei Xu, and C Karen Liu. Human-object interaction from human-level instructions. arXiv preprint arXiv:2406. 17840, 2024. 3
[68] Lixing Xiao, Shunlin Lu, Huaijin Pi Ke Fan, Liang Pan, Yueer Zhou, Ziyong Feng, Xiaowei Zhou, Sida Peng, and Jingbo Wang. Motionstreamer: Streaming motion generation via diffusion- based autoregressive model in causal latent space. arXiv preprint arXiv:2503. 2025. 2
[69] Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. In The Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.o www.teeeunteeeee met:/forumm.dd-tt/ayyykkk
[70] Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Yan Gui. Interdiff: Generating 3d human-object interactions with physics-iffrrmed diffusion. In ICCV, 2023. 2,3
[71] Sirui Xu, Yu-Xiong Wang, Liangyan Gui, al. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. Advances in Neural Information Processing Systems, 37:52858- 52890, 2024. 3
[72] Mengqing Xue, Yifei Liu, Ling Guo, Shaoli Huang, and Changxing Ding. Guiding human- object interactions with rich geometry and relations. arXiv preprint 2rX1v:2503.21172 2025.
3
[73] Lixin Yang. Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. OakInk: A large-scale knowledge repository for understanding hand-object interaction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2
[74] Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. In European Conference on Computer Vision, pages 246-266666 263. Springer, 2025. 2,3
[75] Runyi Yu. Yinhuai Wang, Qihan Zhao, Hok Wai Tsui, Jingbo Wang. Ping Tan, and Qifeng Chen. Skillmimic-v2: Learning robust and generalizable interaction skills from sparse and noisy demonstrations. arXiv preprint Ntl::255500094. 2025.
[76] Xinyu Zhan, Lixin Yang. Yifei Zhao, Kangrui Mao, Hanlin Xu, Zenan Lin, Kailin Li, and Cewu Lu. Oakink2: A dataset of bimanual hands-object manipulation in complex task completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 445 456, June 2024. 3
[77] He Zhang, Yuting Ye, Takaaki Shiratori, and Taku Komura. Manipnet: Neural manipulation synthesis with hand-object spatial representation. ACM Trans. Graph., 2021. 3, 8
[78] Hui Zhang, Sammy Christen, Zicong Fan, Luocheng Zheng, Jemin Hwangbo, Jie Song, and Otmar Hilliges. Artigrasp: Physically plausible synthesis of bi-manual dexterous grasping and articulation. In 2024 International Conference on 3D Vision (3DV), pages 235-246, IEFE, 2024. 3
[79] Jiajun Zhang, Yuxiang Zhang, Liang An, Mengcheng Li, Hongwen Zhang, Zonghai Hu, and Yebin Liu. Manidext: Hand-object manipulation synthesis via continuous correspondence embeddings and residual-guided diffusion. arXiv preprint arXiv:2409.09330 2024. 3
15

---

[80] Wanyue Zhang, Rishabh Dabral, Thomas Leimküihler Vladislav Golyanik, Mare Habermann, and Christian Theobalt. Roam: Robust and object-aware motion generation using neural pose descriptors. In 2024 International Conference on 3D Vision (3DV), pages 1392-1402, IEEE, 2024.
[81] Wanyue Zhang, Rishabh Dabral, Vladislav Golyanik, Vasileios Choutas, Eduardo Alvarado, Thabo Beeler, Marc Habermann, and Christian Theobalt. Bimart: A unified approach for the synthesis of bimanual interaction with articulated objects. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025, 2, 3,
[82] Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. Couch: Towards controllable human-chair interactions. In ECCV, 2022. 3
[83] Yonghao Zhang, Qiang He, Yanguang Wan, Yinda Zhang, Xiaoming Deng, Cuixia Ma, and Hongan Wang. Diffgrasp: Whole-bodi grasping synthesis guided by object motion using diffusion model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10320, 2025. 3
[84] Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In ICCV, 2023. 3
[85] Juntian Zheng, Qingyuan Zheng, Lixing Fang, Yun Liu, and Li Yi. Cams: Canonicalized manipulation spaces for category-level functional hand-object manipulation synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 585-594, 2023. 3,9
[86] Yang Zheng. Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C Karen Liu, and Leonidas Guibas. Gimo: Gaze-informed human motion prediction context. In ECCV, 2022.
3
[87] Keyang Zhou, Bharat Lal Bhatnagar, Jan Eric Lenssen, and Gerard Pons-Moll. Toch: Spatio- temporal object-to-hand correspondence for motion refinement. European Conference on Computer Vision, pages 1-19. Springer, 2022.
[88] Yi Zhou, Connelly Barns,, Lu Jingwan, Yang Jimei, and Li Hao. On the continuity of rotation representations in neural networks. In CVPR, 2019. 6
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: To jointly generate body, hand, and articulated object motion, we introduce a coordinated diffusion noise optimization framework equipped with a unified BPS-based representation. We conduct experiments on public datasets and show various results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed in the Section 5.5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
We do not include theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe our method in detail and we will release the code and models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our code will be released. While the code not included the submission, the implementation details are provided in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httppsscccppbpcccccceecce gugggdiessssse CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are provided in Section 5.1. More details are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report our results by averaging 10 runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
19

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes)
Justification:
The compute resources are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /wwrrrerrrpp..e/ cc Seppubicc/Ehiccsuuute
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts are discussed in the supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release models that have high risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The user study instructions are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: We conducted user study in which participants were asked to compare and rate generated motion sequences. The study posed minimal risk and no personal information collected.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs in our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http:s://neuuree http///rppss.cccoeeeeeeeeeeeeeee /LLM) for what should or should not be described.
23


Reviews:
[{"review_id": "jOF9yFPVSt", "reviewer": "Reviewer_3mBR", "summary": "This paper proposed a novel coordinated diffusion noise optimization framework for human-objection interactions generation. The specialized hands, body, object diffusion models are trained on its own datasets separately. The proposed unified representation captures fine-grained spatial relationships between the hand and articulated object parts, obtaining highly accurate interaction motion. Extensive experiments show the superiority of the proposed method in quality, physical plausibility, and capabilities.", "questions": "1. Will the separate models limit the potential to learn inter-correlation between parts, such as how the object types/movements affect the body motion. For example, if a person will have a less probability to run if he is operating a larger items like microwave, compared with some small items like phone.\n2. Is it pipeline limieted to hand manipulation? Is that possible to extend it to whole-body contact?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "+ The proposed coordinated diffusion noise optimization framework is interesting. It allows to train specialized diffusion models on its own datasets and can merge the expertises together to generate whole-body interactions.\n+ The proposed unified BPS-based representation for object and end-effectors is novel and seems effective. It facilitates geometric alignment between end-effectors and object parts.\n\n- Computational Cost and Efficiency. The multi-stage pipeline, especially the optimization process with diffusion, is significantly more computationally intensive than standard generative approaches.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed most of my concerns, and I have raised my score to accept accordingly.", "rebuttal": "## **Response to Reviewer 3mBR**\nThank you for your thoughtful feedback and valuable suggestions. We are glad that you found our framework $\\textcolor{blue}{\\text{interesting}}$ and the proposed unified BPS-based representation $\\textcolor{blue}{\\text{novel and effective}}$.\n\n### **Weakness**\n**Computational efficiency.**\n\nWe acknowledge that our method is more computationally intensive than existing approaches, as discussed in $\\textcolor{green}{\\text{L297}}$ of the main paper. However, as the first work targeting whole-body manipulation of articulated objects, our primary focus is more on generating high-quality and physically plausible motion. Our method significantly outperforms prior baselines in terms of motion quality, as shown in $\\textcolor{green}{\\text{Table 1}}$ and $\\textcolor{green}{\\text{Table 2}}$ of the main paper. We believe that improving computational efficiency is definitely an important and promising direction for future work.\n\n### **Questions**\n**Will the separate (body, right/left hand) models limit the potential to learn inter-correlation between parts?** \n\nWhile our model adopts a decoupled architecture for the body, left hand, and right hand, it does not limit the capacity to learn inter-part coordination. On the contrary, coordination naturally emerges as a result of the paired training data and the inference through our joint optimization process in noise space.\n\nSpecifically, although the diffusion models are trained separately, they are trained on paired body–hand data from GRAB and ARCTIC and are optimized jointly during inference. The optimization propagates gradients through the kinematic chain, ensuring that fine-grained hand motion objectives influence the global body posture, and vice versa. This design enables natural coordination between the body and the hands.\n\nFurthermore, the end-effector tracking loss plays a central role in enforcing this coherence. Since the hand model controls the high-frequency finger details and the body model influences low-frequency global motion, successfully matching end-effector trajectories requires tight harmonization between all three models. As shown in our experiments ($\\textcolor{green}{\\text{Tables 1, 3, and 4}}$ of the main paper), this leads to motions with superior physical plausibility, contact quality, and realism, outperforming baselines. We also note that our ablation study ($\\textcolor{green}{\\text{Table 4, row (e)}}$ of the main paper) shows a performance drop when replacing the decoupled model with a single unified model, confirming that our separate model design, when paired with coordinated optimization, is both effective and beneficial.\n\nSimilarly, the inter-correlation between the object and the body is implicitly learned through the paired data. Manipulating a large object like a microwave often results in simple and stationary object motion and end-effector trajectories, leading to relatively static body motion. In contrast, smaller objects like a phone typically involve more dynamic object motion and end-effector trajectories, which in turn produce more active whole-body responses. \n\nThe advantage of using separate models rather than a single model as the motion prior is that it allows us to leverage more diverse and part-specific datasets. As shown in $\\textcolor{green}{\\text{Figure 1\\(c,d\\)}}$ of the main paper, this design supports a richer combination of motion patterns and enables compositional generation of whole-body behaviors.\n\n**Is the pipeline limited to hand manipulation? Is it possible to extend it to whole-body contact?** \n\nOur pipeline is not limited to hand manipulation. While we currently use the fingertips as the end-effectors, this choice is due to the nature of the ARCTIC dataset, which primarily contains standing manipulation tasks. In principle, the end-effector trajectory model can be extended to include other interacting joints such as elbows, shoulders, knees, or feet. Our optimization framework is highly flexible and agnostic to the number and type of joints selected as control targets. Any set of joints can be designated as optimization targets, and gradients from their tracking losses will propagate through the kinematic chain to guide the whole-body motion generation in a coordinated and physically plausible manner."}, {"review_id": "ScBSWuPl4y", "reviewer": "Reviewer_fHcd", "summary": "In this paper, the authors introduce CoDA, a method for generating full-body motion of articulated object manipulation. The method can be used for text-to-motion generation, motion generation using object key frames, and motion generation from hand trajectories. The key insight is to model motion generation of the two hands and body separately, allowing to be supervised by each component's own data source. This is the first method for full-body articulated-object motion generation. Results show superior performance against baselines.", "questions": "See \"Cons\".", "limitations": "The authors discussed the positive impacts but I think a discussion on the negatives could benefits in a long term. For example, when combined with human pose conditioned video generation methods, this proposed method can be used to generate videos of real individuals (e.g., public figures) performing actions that they did not do. This can raise concerns on misinformation.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Pros\n\n**The results are impressive**: They are both quantitatively and qualitatively significantly better than the baselines. \n\n**Comprehensive experiments**: The authors provide detailed comparison with many existing baselines (see Table 3) in terms of FID, diversity, and physical metrics like foot skating and interpenetration volume etc on both ARCTIC and GRAB. Detailed ablation is provided in Table 4 to ablate the effect of invididual components such as end-effector BPS and AMASS. \n\n**Good presentation**: The writing is easy to understand. The video is intuitive.\n\n## Cons\n\n**Comparison/clarification on interaction sensors**: Overall, I quite like the paper. The authors introduced \"end-effector BPS\", a distance-based features to represent the positions of joints in the object coordinate systems. In general, there are similar \"sensors\" or \"interaction encoder\" in the motion generation line of work. It would be better if the authors could clarify if the choice of such representations actually make a big difference in the results? For example, in ManipNet, they introduced  \"Proximity sensor\" and \"Signed Distance Sensor\", how do they differ in performance compared to \"end-effector BPS\"?\n\n**Quantitative evaluation on unseen object generalization**: Both ARCTIC and GRAB are a bit limited in terms of object diversity. This means, generalization to unseen object is important for future work. I understand that this is an extremely challenging task given the limited data. However, it is best for the authors to provide some quantitative measures for future work to compare against. \n\n## Justification\n\nOverall, I think the paper is well-written, easy to understand. The experiments are quite comprehensive. The only suggestions are the clarification on existing interaction representations (do they really make much difference?). Further, generalization to unseen objects is an important future direction, and it should be comparable.", "was_revised": true, "final_justification": "After reading the rebuttal and other reviews, I decide to raise the score and champion this paper because the authors have addressed my concerns. \n\nI acknowledge the comments from other reviewers related to limited evaluation (such as limitations in ARCTIC objects). This is the first method that tackles the full-body motion generation with articulated objects. Accepting this paper therefore will encourage future work for more complex interaction setting and thus grows our community for human-object motion generation. \n\nAlso, while other reviewers pointed out about robotic applications, this work focuses more on generating animation. So I still think this paper should be accepted.", "rebuttal": "## **Response to Reviewer fHcd**\nThank you for your insightful suggestions and feedback. We are glad that you find our $\\textcolor{blue}{\\text{results impressive}}$, $\\textcolor{blue}{\\text{with comprehensive experiments}}$ and a $\\textcolor{blue}{\\text{good presentation}}$. We also appreciate that you $\\textcolor{blue}{\\text{quite like the paper}}$.\n\n\n### **Weakness**\n**Comparison/clarification on interaction representations.** \n\n| Methods      | FID ↓ | R-Precision ↑ | Diversity → | Foot skating ↓ | IV ↓  | ID ↓  | CR ↑  |\n|--------------|--------|----------------|--------------|----------------|-------|-------|--------|\n| w/ SDF | 2.350 | 0.422 | 7.064 | 0.003 | 6.69 | 13.97 | 0.086 |\n| Ours     | **2.283** | **0.477**   | **7.208**    | **0.002**      | **5.25** | **12.87** | **0.086** |\n\nWe construct a variant that uses a distance field representation similar to the sensors in ManipNet. Specifically, we represent the end-effector trajectory as a 10×10×10 grid of distances in the object’s local coordinate frame, where each voxel stores its distance to the end-effector. We keep the object geometry encoded using BPS and only replace the end-effector representation. As shown in the above table, our design outperforms this variant.\n\nBoth representations offer advantages over directly regressing 3D coordinates, as distance-based encodings better capture the spatial relation between the object and the end-effector. However, our approach further improves performance by aligning the end-effector and object representations in the same BPS space, enabling more consistent and effective interaction learning. The grid-based variant lacks this alignment, leading to slightly lower performance.\n\n\n**Quantitative evaluation on unseen object generalization.**\n\n| Methods      | FID ↓ | R-Precision ↑ | Diversity → | Foot skating ↓ | IV ↓  | ID ↓  | CR ↑  |\n|--------------|--------|----------------|--------------|----------------|-------|-------|--------|\n| OMOMO | 44.009 | 0.547 | 6.234 | 0.028 | 26.92 | **8.23** | 0.116 |\n| Ours     | **16.091** | **0.547**   | **4.964**    | **0.002**      | **26.24** | 8.56 | **0.128** |\n\nTo evaluate the generalization capability of our method, we conduct an additional experiment on the ARCTIC dataset by holding out the box object for testing and training on the remaining objects. As shown in the above table, our method achieves much better FID compared to the baseline.\n\n### **Social impact**\n**Concerns about misinformation.**\n\nWe agree with the reviewer that generated motion could potentially be used as input to generate fake videos of individuals. We will discuss this concern in the final version."}, {"review_id": "upNe6F0Knm", "reviewer": "Reviewer_JkmY", "summary": "The paper aims to address a new task that generates whole-body human-object interaction motions with articulated objects. Leveraging human-object interaction and human-body locomotion data, the paper proposes to split the interaction generation problem into loss function generation and whole-body diffusion noise optimization. Experiments show that the method outperforms several existing methods, and they demonstrate various method applications.", "questions": "1) Can the idea of the proposed method apply to the existing human-object interaction generation tasks (e.g., whole-body interaction with rigid objects, hand-object interaction with articulated objects)? If so, the contribution of the paper could increase.\n\n2) In Lines 291-292, the paper points out that the method allows control over keyframe object poses. The object trajectory generation is a simple conditional diffusion model. What is setting optimization targets?\n\n3) What happens if we set the object trajectory or the text as the condition of the hand diffusion models? Is it better than the existing method design?\n\n4) What is the time cost of the proposed method?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1) The paper is solving a new and challenging task. Existing interaction generation methods either focus on interacting with rigid objects or neglect human body movements.\n2) The proposed method is straightforward.\n3) The evaluation metrics comprehensively reflect motion naturalness, motion diversity, and physical plausibility.\n4) The ablation study demonstrates the value of individual method designs.\n5) The writing is clear and easy to follow.\n\nWeaknesses:\n\n1) The paper claims that the method allows the use of hand-only datasets to generate whole-body motion, whereas the body diffusion model requires motion priors from existing human-body datasets.\n2) Head orientation is often unnatural in the qualitative results of the method (e.g., Figure 1(d)). This could be due to the data used to train the body diffusion model is not fully interaction-aware. It can be challenging to generate coordinated and natural bodies and hands using only the gradient descent without the coordination of motion semantics.\n3) The time consumption of the method could be significantly large. This limits the real application of the method.\n4) Traditional diffusion-based methods (e.g., OMOMO) directly perform one forward diffusion process during inference time, which does not take good physical plausibilities into consideration. The paper's method defines physical loss functions and performs test-time optimization, which ought to achieve better physical performance. Comparing the method with physics-aware baselines (e.g., OMOMO + physics-based post optimization, OMOMO + diffusion noise optimization) is necessary.\n5) In the existing baseline methods, only IMoS aims to solve the whole-body interaction generation problem. The paper misses more advanced whole-body interaction generation baselines such as [1-2].\n6) Some method names (e.g., \"HOIFHLI\" in Line 115) are unofficial and could be misleading. It is better to use author names (e.g., \"Wu et.al.\").\n\nReferences:  \n[1] Wu Z, Li J, Xu P, et al. Human-object interaction from human-level instructions[J]. arXiv preprint arXiv:2406.17840, 2024.  \n[2] Tendulkar P, Surís D, Vondrick C. Flex: Full-body grasping without full-body grasps[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 21179-21189.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed most of my concerns, and I raised my score to borderline accept accordingly.", "rebuttal": "## **Response to Reviewer JkmY**\nThank you for your valuable feedback and helpful suggestions. We are glad you found our paper $\\textcolor{blue}{\\text{clear}}$, $\\textcolor{blue}{\\text{comprehensive evaluations}}$ and $\\textcolor{blue}{\\text{well-designed ablations}}$, addressing a $\\textcolor{blue}{\\text{new and challenging task}}$ with a $\\textcolor{blue}{\\text{straightforward approach}}$.\n\n### **Weakness**\n**Using hand-only datasets to generate whole-body motion.**\n\nAs noted in $\\textcolor{green}{\\text{L295}}$ and $\\textcolor{green}{\\text{L289}}$ of the main paper, our method supports generating whole-body motion from hand-only datasets. Specifically, we refer to two cases:\n1. $\\textcolor{green}{\\text{L295}}$ shows that ground-truth hand-only end-effector positions can be directly used to generate whole-body motion through our pipeline. \n2. $\\textcolor{green}{\\text{L289}}$ describes that once our model is fully trained, a new hand-only dataset can be used to train only the end-effector trajectory generation model, without re-training the separate diffusion models responsible for whole-body optimization.\n\nOur framework supports the integration of new hand-only data either as direct input or by training an end-effector model. Since the motion models used in the optimization stage remain fixed, the method is highly reusable and extensible. As shown in $\\textcolor{green}{\\text{Figure 1\\(c\\)}}$ and $\\textcolor{green}{\\text{Figure 5}}$ of the main paper, our approach can be easily applied to such hand-only datasets without requiring additional full-body supervision. We will clarify this point in the final version.\n\n**Unnatural head orientation and the coordination between the body and hands.**\n\nThis issue occurs only rarely in our results. Moreover, it is easy to address within our framework by incorporating a simple head orientation loss that encourages the head to face the object. We find that adding this term significantly improves head alignment. We thank the reviewer for pointing out this issue.\n\nRegarding the concern on coordination between the body and hands, although the diffusion models are trained separately, they are trained on paired body–hand data from GRAB and ARCTIC and are optimized jointly during inference. The optimization propagates gradients through the kinematic chain, ensuring that fine-grained hand motion objectives influence the global body posture, and vice versa. This design enables natural coordination between the body and the hands. \n\nFurthermore, the end-effector tracking loss plays a central role in enforcing this coherence. Since the hand model controls the high-frequency finger details and the body model influences low-frequency global motion, successfully matching end-effector trajectories requires tight harmonization between all three models. As shown in our experiments ($\\textcolor{green}{\\text{Tables 1, 3, and 4}}$ of the main paper), this leads to motions with superior physical plausibility, contact quality, and realism, outperforming baselines. We also note that our ablation study ($\\textcolor{green}{\\text{Table 4, row (e)}}$ of the main paper) shows a performance drop when replacing the decoupled model with a single unified model, confirming that our separate model design, when paired with coordinated optimization, is both effective and beneficial.\n\nWhile preserving the coordination, the separate model design also enables compositional generation of diverse and flexible whole-body motions, which would be difficult to achieve with a single model.\n\n**Time efficiency.**\n\nWe acknowledge that our method is slower than existing approaches, as discussed in $\\textcolor{green}{\\text{L297}}$ of the main paper. However, our method produces substantially better results than baselines, as demonstrated in $\\textcolor{green}{\\text{Table 1}}$ and $\\textcolor{green}{\\text{Table 2}}$ of the main paper. As the first work addressing whole-body manipulation of articulated objects, our primary focus is on achieving high-quality motion. Improving efficiency is an important direction for future work.\n\n\n**Comparison with physics-aware baselines.**\n\n| Methods      | FID ↓ | R-Precision ↑ | Diversity → | Foot skating ↓ | IV ↓  | ID ↓  | CR ↑  |\n|--------------|--------|----------------|--------------|----------------|-------|-------|--------|\n| OMOMO | 3.710 | 0.406 | 6.110 | 0.028 | 13.77 | 15.16 | 0.061 |\n| OMOMO+DNO | 2.873 | 0.391 | 7.004 | 0.022 | 8.95 | 13.30 | 0.075 |\n| Ours     | **2.283** | **0.477**   | **7.208**    | **0.002**      | **5.25** | **12.87** | **0.086** |\n\nFollowing your suggestion, we apply diffusion noise optimization (DNO) to OMOMO using the same loss functions as in our method. The results are shown in the above table. While noise optimization improves OMOMO, OMOMO+DNO is still worse than our full pipeline. This is because DNO is applied to a whole-body motion model, which we find cannot always produce arm motions consistent with the generated hand trajectory. In contrast, our decoupled design, with separate motion models for body and hands, can leverage more data and achieve better performance. This is also supported by our ablation in $\\textcolor{green}{\\text{Table 4(e)}}$ of the main paper.\n\n**Comparison with more whole-body interaction baselines [1, 2].**\n\n1. Wu et al. [1]: The code was not available at the time of submission. Although it was released last week, it remains incomplete, lacking full pipeline training scripts and checkpoints. We will include comparisons once the code is fully released.\n2. FLEX [2]: This method focuses on whole-body grasp generation rather than interaction motion. Many of its components are specifically designed for achieving stable grasps and are not directly applicable to articulated-object manipulation tasks like ours.\n\n**Reference method naming.**\nThe name “HOIFHLI” was borrowed from the original project website of the paper. We will revise it to “Wu et al.” in the final version.\n\n### **Questions**\n**Applicability to existing human-object interaction generation tasks.** \n\nWe have evaluated our method on the GRAB dataset for whole-body interaction with rigid objects, as shown in $\\textcolor{green}{\\text{Table 3}}$ of the main paper. The visualization results can also be found in the supplementary material. Our method is easily extensible to hand-object interaction settings, which can be seen as a special case of whole-body interaction. We will explore this further in future work.\n\n**Details of controlling keyframe object poses.**\n\n$\\textcolor{green}{\\text{Section E.2}}$ in the supplementary material presents details related to object motion control. Here, we additionally provide the specific optimization loss used in our setup. We apply diffusion noise optimization using the object motion generation model. Given a set of keyframes where the object pose $\\mathbf{S}_j$ is known at certain frames $j$, we define the loss:\n\n$$\\mathcal{L}_{\\text{keyframe}} = \\sum^{j \\in \\mathcal{K}} \\| \\hat{\\mathbf{S}}_j - \\mathbf{S}_j \\|^2.$$\n\nHere, $\\hat{\\mathbf{S}}_j$ denotes the predicted object pose at frame $j$, and $\\mathcal{K}$ is the set of keyframe indices. This loss is used to optimize the initial noise of the object motion diffusion model. Once optimized, the generated object trajectory conforms to the specified keyframes, and the resulting end-effector trajectories and whole-body motion follow through our pipeline, producing consistent whole-body behavior for the given keyframe constraints. We will include this explanation in the final version.\n\n**Using object trajectory or text as a condition for the hand diffusion model.**\n\n| Methods      | FID ↓ | R-Precision ↑ | Diversity → | Foot skating ↓ | IV ↓  | ID ↓  | CR ↑  |\n|--------------|--------|----------------|--------------|----------------|-------|-------|--------|\n| w/ condition | 2.998 | 0.453 | **7.212** | 0.002 | 8.95 | 13.30 | 0.075 |\n| Ours     | **2.283** | **0.477**   | 7.208    | **0.002**      | **5.25** | **12.87** | **0.086** |\n\nWe experimented with conditioning the hand motion diffusion model on object trajectories. However, as shown in the table, this leads to higher FID. We observe that adding more conditions to the hand model tends to cause overfitting to specific object trajectories, which makes the optimization more difficult. In contrast, our unconditional model provides a stronger and more generalizable motion prior.\n\n**Time cost.**\n\nWe report the runtime in $\\textcolor{green}{\\text{Table 4}}$ of the supplementary material. Our full pipeline takes approximately 17 minutes to generate a 10-second motion clip. While this is slower than the baselines, the improvement in physical realism and motion quality justifies the cost. As the first work tackling whole-body manipulation of articulated objects, our priority is high-quality generation.\n\n---\n[1] Wu Z, Li J, Xu P, et al. Human-object interaction from human-level instructions[J]. arXiv preprint arXiv:2406.17840, 2024.\n\n[2] Tendulkar P, Surís D, Vondrick C. Flex: Full-body grasping without full-body grasps[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 21179-21189."}, {"review_id": "RWuzNFF6Uf", "reviewer": "Reviewer_4399", "summary": "This paper proposes a novel framework for synthesizing whole-body manipulation of articulated objects by coordinating body, hand, and object motions. The approach leverages three separately trained diffusion models for the body, left hand, and right hand, and performs joint noise-space optimization to enable coordination through the human kinematic chain. A unified object-hand representation using basis point sets (BPS) enhances fine-grained control and precision in interaction. The method supports capabilities such as object pose control, walking during manipulation, and motion synthesis from hand-only data. Experiments show improved motion quality and physical plausibility over prior work.", "questions": "1. Articulated object manipulation has been extensively studied in the robotics community. Could the authors elaborate on how their approach relates to prior work in robotics, particularly those involving interaction with complex mechanisms (e.g., PartNet-Mobility, AKB-48, DexArt, AdaManip)? A more thorough discussion would help clarify the contribution relative to existing manipulation pipelines.\n\n2. Given the limitations of the ARCTIC dataset and the use of static HOI datasets and perceptual metrics (e.g., FID, R-Precision), the current evaluation may not fully reflect the model’s effectiveness in physically grounded scenarios. Have the authors considered evaluating their method in interactive simulation environments such as SAPIEN or Isaac Sim? Benchmarking the success rate of generated motions in these settings would provide more robust validation. As noted in the limitation section of the paper, extending the framework to support sequential interactions remains an open challenge—one that datasets like AdaManip are designed to capture.\n\n3. Have the authors considered validating their method on a real-world humanoid robot? While simulation is a useful intermediary, demonstrating that the generated motions can transfer to physical platforms would significantly strengthen the practical relevance of this work.", "limitations": "See the Questions section. The paper would benefit from additional evaluation on more diverse and realistic simulation benchmarks and real-world experiments to better demonstrate the method's generalizability and practical applicability.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper tackles the problem of whole-body human-object interaction, which is more realistic than hand-only interaction. The proposed method decomposes the task into object trajectory generation, end-effector trajectory generation, and coordinated diffusion noise optimization via the kinematic chain. This modular design improves joint control and generalization. The use of forward kinematics enables automatic optimization and adaptability across different humanoid configurations.\n\n2. The paper is well-written and clearly presented, with high-quality visualizations that effectively convey the model’s capabilities.\n\n3. The authors provide comprehensive comparisons against relevant baselines and include meaningful ablation studies to validate key design choices.\n\n**Weaknesses**\n1. The evaluation is limited to static HOI datasets. The work would be strengthened by demonstrating performance in simulation environments (e.g., SAPIEN, Isaac Sim) or on real-world humanoid robots to assess applicability in dynamic, embodied settings.\n\n2. The articulated objects used (from the ARCTIC dataset) are relatively simple two-part structure, which may not capture the complexity of real-world articulated objects. Evaluation on more diverse datasets such as PartNet-Mobility[1], AKB-48[2], DexArt[3], or AdaManip[4] would improve the generality of the conclusions. This limitation is also acknowledged by the authors.\n\n3. Several components of the proposed framework (e.g., the BPS-based representation, RoPE-style encoding from CaPE, and DDIM sampling from DNO) are adaptations of existing techniques. While the integration is novel, the method may be viewed as an incremental combination rather than a fundamentally new paradigm.\n\n[1] Xiang, Fanbo, et al. \"Sapien: A simulated part-based interactive environment.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Liu, Liu, et al. \"Akb-48: A real-world articulated object knowledge base.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Bao, Chen, et al. \"Dexart: Benchmarking generalizable dexterous manipulation with articulated objects.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Wang, Yuanfei, et al. \"Adamanip: Adaptive articulated object manipulation environments and policy learning.\" arXiv preprint arXiv:2502.11124 (2025).", "was_revised": true, "final_justification": "The authors' rebuttal addresses most of my concerns, and I am raising my rating to 4: Borderline Accept. However, a higher rating is not warranted at this stage, as the evaluation of more realistic and complex articulated object manipulation, both in simulation and on real humanoid robots, are left for future work.", "rebuttal": "## **Response to Reviewer 4399**\nThank you for your thoughtful feedback and constructive suggestions. We are glad that you find our paper $\\textcolor{blue}{\\text{well-written and clearly presented}}$, with $\\textcolor{blue}{\\text{high-quality visualizations}}$, $\\textcolor{blue}{\\text{comprehensive comparisons}}$, and $\\textcolor{blue}{\\text{meaningful ablations}}$.\n\n### **Weakness**\n**Demonstrating performance in simulation environments.**\n\n| Methods      | Tracking Duration↑  |\n|--------------|---------------------|\n| OMOMO        | 4.75 secs           |\n| Ours         | 8.75 secs           |\n\nWe appreciate the suggestion to evaluate in interactive simulators. We have included deployment on simulated humanoids visualization in $\\textcolor{green}{\\text{Section E.6}}$ in the supplementary material. To further assess the physical feasibility of our method, we conduct a mimic-based [5, 6] evaluation, where a humanoid policy is trained to mimic the generated motions in the IsaacGym [10] simulator. We use 8 generated sequences (each 10 seconds long) involving boxes and microwaves, and track how long the humanoid controller can successfully mimic the motion. Following [5, 6], the tracking duration is the time (in seconds) that the object and joint error are lower than a threshold of 10 cm. Compared to OMOMO, our motions lead to longer durations, indicating improved physical plausibility and better compatibility with downstream humanoid execution.\n\n**Limited articulation complexity in ARCTIC.**\n\nThank you for pointing out these articulated object benchmarks from the robotics community. While datasets such as PartNet-Mobility [1], AKB-48 [2], and DexArt [3] contain more complex object geometries, they lack paired human motion data, which is essential for our learning-based framework. As our method relies on motion capture data for training, we are currently limited by the available human-object interaction datasets. We hope future benchmarks will include richer articulated objects along with corresponding human motion data.\n\n**Novel integration without fundamentally new paradigm.**\n\nOur method presents a novel framework tailored for the new and challenging task of generating whole-body manipulation motions for articulated objects. The way we formulate and tackle this problem is itself novel. To meet the unique demands of this setting, we design a tailored framework that integrates existing components in new and effective ways:\n1. We introduce BPS not just for static object geometry encoding, as in prior work [7,8], but to represent time-varying end-effector trajectories. This allows for fine-grained geometric alignment over the entire motion sequence.\n2. Our joint noise optimization is not a direct reuse of existing strategies [9]. We coordinate separate diffusion models for the body and both hands by linking their gradients through the human kinematic chain. This enables whole-body motion composition with precise interaction control, which is not explored in previous diffusion-based methods [7, 8, 9].\n\nThis perspective is shared by $\\textcolor{blue}{\\text{Reviewer 3mBR}}$, who described our framework as $\\textcolor{blue}{\\text{interesting}}$ and our unified representation as $\\textcolor{blue}{\\text{novel and effective}}$.\n\n### **Questions**\n**Relation to articulated object manipulation in robotics community.**\n\nOur task setting is fundamentally different from manipulation pipelines in robotics benchmarks such as PartNet-Mobility [1], AKB-48 [2], DexArt [3], and AdaManip [4]. Specifically:\n1. **Goal**: These works [1, 2, 3, 4] primarily focus on physical execution, ensuring that a robotic agent can complete manipulation tasks in the real world. In contrast, our focus is on content generation [7, 8]: producing diverse, high-quality, and physically plausible whole-body motions that reflect natural human behavior. These goals are complementary: while robotics emphasizes control and execution, our generated motion can be used as kinematic trajectories or reference motions to guide humanoid control and serve as data for downstream planning and policy learning.\n2. **Task setting**: These works focus on single-arm grippers [1,2,4] or robotic hands [3], often on fixed bases or wheeled robots. In contrast, our method operates on full human bodies with 52 articulated joints, each with 3 degrees of freedom, and a 6-DoF root, resulting in a 159-dimensional motion space. This enables not only bimanual dexterous manipulation but also coordinated whole-body behaviors such as locomotion during interaction.\n3. **Data source**: These approaches rely on reinforcement learning [1,2,3] or rule-based expert policies [4] in simulation to collect training data. However, due to the high-dimensional action space and complex coordination in whole-body manipulation, it is difficult to design effective rewards or rules for our task. Instead, we leverage motion capture data from real humans to learn realistic behaviors.\n\n**Evaluation in simulation environments and sequential interactions.**\n\nAs noted above, we have conducted simulation-based evaluations using IsaacGym. Regarding sequential interactions, we agree this is a valuable direction and have also discussed it in $\\textcolor{green}{\\text{L304}}$ of the main paper. While AdaManip [4] focuses on single-arm grippers and uses rule-based policies that are not directly applicable to natural human motion, its formulation of sequential tasks offers useful inspiration. Although it is challenging to generate realistic whole-body human motions for such tasks using handcrafted rules, we hope future benchmarks will provide sequential human-object interaction motion capture data, which would greatly facilitate progress in this direction.\n\n**Extention to real-world humanoid robots.**\n\nWe appreciate the suggestion. While our current focus is motion generation, the synthesized whole-body motions can be used as reference motions for real-world humanoid robots. This opens opportunities for integration with existing humanoid control systems, and we see this as an exciting direction for future work.\n\n---\n[1] Xiang, Fanbo, et al. \"Sapien: A simulated part-based interactive environment.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[2] Liu, Liu, et al. \"Akb-48: A real-world articulated object knowledge base.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Bao, Chen, et al. \"Dexart: Benchmarking generalizable dexterous manipulation with articulated objects.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[4] Wang, Yuanfei, et al. \"Adamanip: Adaptive articulated object manipulation environments and policy learning.\" arXiv preprint arXiv:2502.11124 (2025).\n\n[5] Luo, Zhengyi, et al. \"Perpetual humanoid control for real-time simulated avatars.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[6] Wang, Yinhuai, et al. \"PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction.\" arXiv preprint arXiv:2312.04393 (2023).\n\n[7] Li, Jiaman, et al. \"Object motion guided human motion synthesis.\" ACM Transactions on Graphics (ToG), 2023.\n\n[8] Li, Jiaman, et al. \"Controllable human-object interaction synthesis.\"  European Conference on Computer Vision, 2024.\n\n[9] Karunratanakul, Korrawe, et al. \"Optimizing diffusion noise can serve as universal motion priors.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[10] Makoviychuk, Viktor, et al. \"Isaac gym: High performance gpu-based physics simulation for robot learning\" arXiv preprint arXiv:2108.10470 (2021)."}]

Author rebuttal:

Meta-review: This paper proposes CoDA, the first method for synthesizing whole-body manipulation of articulated objects by coordinating body, hand, and object motions through joint noise-space optimization of three separate diffusion models.

The authors' rebuttal effectively addressed reviewers' concerns, leading to improved scores. Both reviewers and the AC recognize the significance of this novel task in advancing human-object interaction research, and expect this paper to encourage future work in more complex interaction scenarios, thereby expanding our community's capabilities in motion generation.

While reviewers noted concerns about computational cost and efficiency, we believe these limitations are outweighed by the method's innovative contributions and potential for future optimization. The paper's strong technical merits and the importance of the addressed problem justify acceptance.
Decision: Accept (poster)