Omnigrasp: Grasping Diverse Objects with Simulated Humanoids
Authors: Zhengyi Luo
Jinkun Cao
Sammy Christen
Alexander Winkler
Kris M. Kitani
Weipeng Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Physics Simulation
Humanoid Control
Dexterous Manipulation
Primary area: robotics
TL;DR: Full body and dexterous humanoid grasping and object manipulation.

Abstract: We present a method for controlling a simulated humanoid to grasp an object and move it to follow an object's trajectory. Due to the challenges in controlling a humanoid with dexterous hands, prior methods often use a disembodied hand and only consider vertical lifts or short trajectories. This limited scope hampers their applicability for object manipulation required for animation and simulation. To close this gap, we learn a controller that can pick up a large number (>1200) of objects and carry them to follow randomly generated trajectories. Our key insight is to leverage a humanoid motion representation that provides human-like motor skills and significantly speeds up training. Using only simplistic reward, state, and object representations, our method shows favorable scalability on diverse objects and trajectories. For training, we do not need a dataset of paired full-body motion and object trajectories. At test time, we only require the object mesh and desired trajectories for grasping and transporting. To demonstrate the capabilities of our method, we show state-of-the-art success rates in following object trajectories and generalizing to unseen objects. Code and models will be released.

Omnigrasp: Grasping Diverse Objects with Simulated Humanoids
Zhengyi Luo
Jinkun Cao Sammy Christen Alexander Winkler
Kris Kitani
Weipeng Xu²
'Camegie Mellon University; 'Reality Labs Research, Meta; *ETH Zurich https://zhengyilee github. ieriii//mmmii o0Omnigrasp
Qurpy Curriy
Figure 1:
We control simulated humanoid to grasp diverse objects and follow complex trajectorics. (Top): picking and holding objects. (Bottom): green dots reference trajectory; pink dots object trajectory.
Abstract
We present a method for controlling a simulated humanoid to grasp an object and move it follow object's 's trajectory. Due the challenges controlling a humanoid with dexterous hands, prior methods often use disembodied hand and only consider vertical lifts short trajectories. This limited scope hampers their applicability for object manipulation required for animation and simulation. To close this gap, we learn controller that can pick up large number (>1200) objects and carry them to follow randomly generated trajectories. Our key insight is leverage humanoid motion representation that provides human-lik, motor skills and significantly speeds up training. Using only simplistic reward, state, and object representations, our method shows favorable scalability on diverse objects and trajectories. For training, we do not need dataset of paired full-body motion and object trajectories. At test time, we only require the object mesh and desired trajectories for grasping and transporting. To demonstrate the capabilities of our method, we show state-of-the success rates following object trajectories and generalizing to unseen objects. Code and models will be released.
Equal Contribution
Equal Advising
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Given an object mesh, we aim to control a simulated humanoid equipped with two dexterous hands to pick up the object and follow plausible trajectories, as shown in Fig.1. This capability could be broadly applied to creating human-object interactions for animation and AV/VR, with potential extensions to humanoid robotics [27]. However, controlling a simulated humanoid with dexterous hands for precise object manipulation poses significant challenges. The bipedal humanoid must maintain balance enable detailed movements of the arms and fingers. Moreover, interacting with objects requires forming stable grasps that accommodate diverse object shapes. Combining these demands with the inherent difficulties of controlling humanoid with a high degree of freedom (e.g.
153 DoF) significantly complicates the learning process.
These challenges have led previous methods of simulated grasping to employ a disembodied hand [16, 17, 61, 85] to grasp and transport. While this approach can generate physically plausible grasps, employing floating hand compromises physical realism: the hands' root position and orientation controlled by invisible forces, allowing it to remain nearly perfectly stable during grasping. Moreover, studying the hand in isolation does not accurately reflect its typical use, which is when it is attached to a mobile and flexible body. A naive approach to supporting hands is to use existing full-body motion imitators [42] provide body control and train additional hand controllers for grasping. However, the presence of a body introduces instability, limits hand movement, and requires synchronizing the entire body to facilitate finger motion. State-of-thee tfrtthrartt (SOTA) full-body imitators also have an average 30mm tracking error for the hands, which can cause the humanoid to miss objects. Due to the above challenges, previous work that studies full-body object manipulations often limits its scope to only one sequence of object interaction [78] and encounters difficulties in trajectory following [6], even when trained with highly specialized motion priors.
Another challenge of grasping is the diversity of the object shapes and trajectories. Each object may require a unique type of grasping, and scaling to thousands of different objects often requires training procedures such as generalist-specialist training [85] or curriculum [75, 101]. There is also infinite variability in potential object trajectories, and each trajectory may necessitate precise full-body coordination. Thus, prior work typically focuses on simple trajectories, such as vertical lifting [16, 85], or on learning a single, fixed, and pre-recorded trajectory per policy [17]. The flexibility with which humans manipulate objects to follow various trajectories while holding them remains unobtainable for current humanoids, even in simulations.
In this work, we introduce a full-body and dexterous humanoid controller capable of picking up and following diverse object trajectories using Reinforcement Learning (RL). Our proposed method, Omnigrasp, presents a scalable approach that generalizes to unseen object shapes and trajectories. Here. "Omni" refers to following any trajectory in all directions within reasonable range and grasping diverse objects. Our key insight lies in using a pretrained universal dexterous motion representation as the action space. Directly training a policy on the joint actuation space using RL results in unnatural motion and leads to severe exploration problem. Exploration noise in the torso can lead to a large deviation in the location of the arm and wrist as the noise propagates through the kinematic chain. This can lead to the humanoid quickly knocking the object away, which hinders training progress. Prior work has explored using a separate body and hand latent space trained using adversarial learning [6]. However, as the adversarial latent space can only cover small-scale and curated datasets, these methods do not achieve a high grasping success rate. The separation of hands and body motion prior also adds complexity to the system. We propose using a unified universal and dexterous humanoid motion latent space [41]. Learned from a large-scale human motion database [45], our motion representation provides a compact and efficient action space for RI. exploration. We enhance the dexterity of this latent space by incorporating articulated hand motions into the existing body-only human motion dataset.
Equipped with a universal motion representation, our humanoid controller does not require any specialized interaction graph [78, 102] to learn human-object interactions. Our input to the policy consists only of object and jojjetory-followwie body motion. For training, we use randomly generated trajectories and do not require paired full-body human-object motion data. We also identify the importance of pre-grasps [17] (the hand pose right before grasping) and utilize it in our reward design. The resulting policy can be directly applied to transport new objects without additional processing and achieve a SOTA success rate on following object trajectories captured by Motion Capture (MoCap).
information and is devoid of any grasp or reference

---

To summarize, our contributions are: (I) we design a dexterous and universal humanoid motion representation that significandy increases sample efficiency and enables learning to grasp with simple yet effective state and reward designs; (2) we show that leveraging this motion representation, one can learn grasping policies with synthetic grasp poses and trajectories, without using any paired full-body and object motion data. (3) we demonstrate the feasibility of training a humanoid controller that can achieve a high success rate in grasping objects, following complex trajectories, scaling up to diverse training objects, and generalizing to unseen objects.
2 Related Works
Simulated Humanoid Control. Simulated humanoids can be used to create animations [26, 36, 54,
55, 56, 57, 80, 94, 102], estimate full-body pose from sensors [23, 30, 33, 40, 43, 79, 92, 93, 95], and transfer to real humanoid robots [20, 27, 28, 59, 60]. Since there are no ground truth data for joint actuation and physies simulators are often non-differentiable, model-
control [29], trajectory optimization [36, 83], and deep RL [13, 54] are used instead of supervised learning. Due to its flexibility and scalability, deep RL has been popular among efforts in simulated humanoids, where policy/controller is trained via trial and error. Most of the previous work on humanoids does not consider articulated fingers, except for few [3. 6, 36, 49]. A dexterous humanoid controller is essential for humanoids to perform meaningful tasks in simulation and in the real world.
Dexterous Manipulation. Dexterous manipulation is an essential topic in robotics [7, 8, 11, 12, 15, 16, 19, 37, 62, 75, 85, 96, 97, 98] and animation [2, 6, 34, 1his.This
This task usually involves pick-and- place [7, 8]. lifting [75, 85, 97], articulating objects [98], and following predefined object trajectories [6, 9, 17J. Most of these efforts use a disembodied hand for grasping and employ non-physical virtual forces control the hand. Among them, D-Grasp [16] leverages the MANO [66] hand model for physically plausible grasp synthesis and 6DoF target reaching. UniDexGrasp [85] and its followup [75] use the Shadow Hand [1]. PGDM [17] trains a grasping policy for individual object trajectories and identifies pre-grasp initialization (initializing the hand in a pose right before grasping) as crucial factor for successful grasping. For the works that consider both hands and body, PMP [3] and PhysHOI [78] train one policy for each task or object. Braun el al. [6] studies similar setting to ours but relies on MoCap human-object interaction data and only uses one hand. Compared to prior work, Omnigrasp trains one policy to transport diverse objects, supports bimanual motion, and achieves a high success rate in lifting and object trajectory following.
Kinematic Grasp Synthesis. Synthesizing hand grasp can be widely applied in robotics and animation. A line of work [5. 10, 10, 18, 21, 38, 47, 51, 84, 89] focuses on reconstructing and predicting grasp from images or videos, while others [52, 90] study hand grasp generation to help image generation. Among them, Manipnet and CAMS [99] predict finger poses given a hand object trajectory. TOCH [103] and GeneOH [39] denoise dynamic hand pose predictions for object interactions. More research in this area focuses on generating static or sequential hand poses with a given object as the condition [31. 70, 88). For synthesizing body and hand poses jointly, there are limited MoCap data available [71] due to difficulties capturing synchronized full-body and object trajectories. Some generative methods [22, 35, 69, 72, 73, 82, 91] can create paired homan-object interactions, but they require initialization from the ground truth [22, 69, 82], or only predict static full-body grasps [73]. In this work, we use GrabNet [70] trained on object shapes from OakInk [86] to generate hand poses as reward guidance for our policy training.
Humanoid Motion Representation. Due to the high DoF of a humanoid and the sample inefficiency of RL training, the search space within which the policy operates during trial and error is crucial. A more structured action space such as motion primitives [24, 25, 48, 63] or motion latent space [56. 74] can significantly increase sample efficiency since the policy can sample coherent motion instead of relying on random "jittering" noise. This is especially important for humanoids with dexterous hands, where the torso motion can drastically affect the hand movement and lead to the humanoid knocking the object away. Thus, prior work in this space utilizes part-based motion priors [3, 6] trained on specialized datasets. While effective in the single task setting where the humanoid only needs to perform actions close to the ones in the specialized datasets, these motion priors can hardly scale to more free-formed motion, such as following randomly generated object trajectories. We extend the recently proposed universal humanoid motion representation, PULSE [41], to the dexterous humanoid setting and demonstrate that a 48-dimensional, hall-body-and-hene motion latent space can be used to pick up and follow randomly generated trajectories.
3

---

(6) Acquiring Dexterous Motion Reprecentation/Sii SSkills
(b) Training Grasp and Trajectory Following Policy
Objent
Latent Code
Gool
Fature Ohject
Orangraac
Action
Body
Dexterous Motion
Object States
(bear Sty1))
Modion Imiitation
Pectes
Desterous Motion Initator (PHC-X)
Disnilation
of
Plyyks (Jancc Gymi)
Inference
Freuiis
Training
Inieiin Gutk
grangae grop
Proaas
Raward
Learsing
Prontocppticc
Universal
Representation (PULSE-X)
Trajetrary
Figure 2: Omnigrasp trained in two stages. (a) universal and dexterous humanoid motion representation trained via distillation. (b) Pre-grasp guided grasping training using a pretrained motion representation.
3
Preliminaries
We define the human pose q, - (Ot.P.). consisting of 3D joint rotation 0, JR and position P, E Jx3 of all J links on the humanoid (hands and body). using the 6 degree-of-freedom (DOF) rotation representation [104]. To define velocities d1:r, we have á, (w,, ve) as angular w, E / and linear velocities v, R/x3, For objects, we define their 3D trajectories dt using object position P: orientation 0thi linear velocity ohhh and angular velocity with As notation convention, use - to denote the kinematic quantities from Motion Capture (MoCap) or trajectory generator and normal symbols without accents for values from the physics simulation. Ô refers to a dataset of diverse object meshes.
Goal-conditioned Reinforcement Learning for Humanoid Control. We define the object grasp- ing and transporting task using the general framework of goal-conditioned RL. Namely, goal- conditioned policy ı is trained to control a simulated humanoid to grasp an object and follow object trajectories 41.1 using dexterous hands. The learning task is formulated as Markov Decision Process (MDP) defined by the tuple M (S,A, A, R, of states, actions, transition dynamics, reward function, and discount factor. The simulation determines the state 8, S and transition dynamics T, where a policy computes the action a. The state 8, contains the proprioception sf and the goal state sf. Proprioception defined as s.. & (q., q.: c.), which contains the body pose q.- velocity q,. and contact forces on the hand. The goal state sf is defined based on the states of the objects. When computing the states o and s.. all values are normalized with respect to the humanoid heading (yaw). Based on proprioception sf and the goal state sf. we define a reward r; =R( sf) for training the policy. We use proximal policy optimization (PPO) [68] to maximize discounted reward E
>" Our humanoid follows the kinematic structure of SMPL-XX [53] using the mean shape. has 52 joints, of which 51 are actuated. 21 joints are body joints, and the remaining 30 joints are for two hands. All joints have 3 DoF, resulting an actuation space of a, R x3 Each degree of freedom is actuated by proportional derivative (PD) controller, and the action specifies the PD target.
Omnigrasp: Grasping Diverse Objects and Follow Object Trajectories
To tackle the challenging problem of picking up objects and following diverse trajectories. we first acquire a universal dexterous humanoid motion representation in 1. Using this motion representation, we design hierarchical RL framework (Sec. 4.2) for grasping objects using simple state and reward designs guided by pre-grasps. Our architecture is visualized in Figure 2.
Here, the "simple reward" refers to not needing paired full-boc fill-body-and aaa--and-aaa hand MoCap data when computing the reward, which increases complexity.

---

4.1 PULSE-X: Physics-based Universal Dexterous Humanoid Motion Representation
We introduce PULSE-X that extends PULSE [41] to the dexterous humanoid by adding articulated fingers. We first train a humanoid motion imitator [42] that can scale to large-scale human motion dataset with finger motion. Then, we distill the motion imitator into a motion representation using a variational information bottleneck (similar to a VAE [32]).
Data Augmentation. Since full-body motion datasets that contain finger motion are rare (e.g., 91 % of the AMASS sequences do not have finger motion), we first augment existing sequences with articulated finger motion and construct a dexterous full-body motion dataset. Similarly to the process in BEDLAM [4], we randomly pair full-body motion from AMASS [45] with hand motion sampled from GRAB [71] and Re: [50] to create dexterous AMASS dataset. Intuitively, training on this dataset increases the dexterity of the imitator and the subsequent motion representation.
PHC-X: Humanoid Motion Imitation with Articulated Fingers. Inspired by PHC [42], we design PHC-X трис- for humanoid motion imitation with articulated fingers. For the finger joints, we treat them similarly the rest of the body (c.g. toe or wrist) and find this formulation sufficient to acquire the dexterity needed for grasping. Formally, the goal state for training тPнс- with RL is sf
gg-mimic A (0.41 1 e 0,,P:: - P:; 0z+1 Vt @++1 Wi, O Pc+1). ). which contains the difference between proprioception and one frame reference pose â;-1-
Learning Motion Representation via Online Distillation. In PULSE [44], an encoder PUI.E.X decoder DrUI SE.x. and prior PP. are learned to compress motor skills into latent representation. For downstream tasks, the frozen decoder and prior will translate the latent code to joint actuation.
Formally, the encoder PULSE ((2 [s",, sf 4-mimic) computes the latent code distribution based on current input states. The decoder PULLSE- aaaise-((a,)) (a ss., 24) produces action (joint actuation) based on the latent code zt. The prior P PrULSE-X (z, defines a Gaussian distribution based on proprioception and replaces the unit Gaussian distribution used in VAEs [32]. The prior increases the expressiveness of the latent space and guides downstream task learning by forming a residual action space (see Sec.4.2). We model the encoder and prior distribution diagonal Gaussian:
St g-mimic) = -N pu of), P P = -NZzap(,,
To train the models, we use online distillation similar to DAgger [67] by rolling out the encoder- decoder in simulation and querying TrFC- for action labels alI-. For more information and evaluation of PHC-X and PULSE-X, please refer to the Appendix B.
4.2 Pre-grasp Guided Object Manipulation
Using hierarchical RL and PULSE-X's ' trained decoder DPULSE-X and prior PPULSE-x. the action space for our object manipulation policy becomes the latent motion representation z,, Since the action space serves strong human-like motion prior, we use simple state and reward design and do not require any paired object and human motion to learn grasping policies. We use only hand pose before grasping (pregraps), either from generative method or MoCap, to train our policy.
State. To provide the task policy Tmmnigrasp with information about the object and the desired object trajectory, we define the goal state as
of A (P++itto Pi
0 14414 @ 0thh gobj
lit+ oi obj E+lit+o W. ,P: ubj
Pt
obj
Pt
hand
) (2)
which contains the reference object pose and the difference between the reference object trajectory for the next é frames and the current object state. ob E R512 is the object shape latent code computed using the canonical object pose and Basis Point Set (BPS) [58]. Pt p. phamd is the difference between the current object position and each hand joint position. All values are normalized with respect to the humanoid heading. Notice that the state of does not contain body pose, grasp, or phase variables [6], which makes our method applicable to unseen objects and reference trajectories at test time.
Action. Similar to downstream task policies in PULSE, we form the action space of Tomnigrasp as the residual action with respect to prior's mean μº and compute the PD target a,:
a2 = DPULSE-X (Toniggmsp (zt amnigrasp/ |56, sf + 14²)
(3)
5

---

Algo 1: Learn Omnigrasp
Function TrainOmiggaap (rrasp(yyuLie (DyULSE.. PreLseXX TTdeiiggpp- O );
Input: Pretrained PULSE decoder D and price PrusEx Object mesh dataset O 3D Inajectory Generalor T while converged
M. @ initializee sampling memoryy
while M. not da da
04 gr of ~ rundomly sample initial object state, pre-grusp, and humanoid state;
du sample reference object trajectory using T
for
+- 1...I do
9
KOuniguap (5, s jo
(f)
// use pretraiaed latent space as action space:
M o +- (PULSEE(=)1())
I/ computa prior latent coda;
11
+ DrLLSEx (a, (a,lo,,, =t
I/ decode actica using pretrained decoder;
#1+1 (( [se, a.)
// simulation; coapute revard;
Ree of))
store (se, #1
2quriguns Ps.. S6+1) into memory M
15
Rommyrap + PPO update using experiences collected in M
16
Oudi f- Eval and pick hard ohject subset to train on.
17
relurn TOnnieap
where jel is computed by the prior P x The policy #Omnigrap computes 2, tmmigrasp E R-18 instead of the target a E R51x3 directly, and leverages the latent motion representation of PULSE-X to produce human-like motion.
Reward. While our policy does not take any grasp guidance or reference body trajectory as input, we
utilize pre-grasp guidance in the reward. We refer pre-grasp qite & (pime-grusp 0 grusp
as
single frame of hand pose consisting hand translation p and rotation gor-grasp PGDM [17]
shows that initializing floating hand to pre-graspi help the policy better reach objects and initiate manipulation. As we do not initialize the humanoid with the pre-grasp pose as in PGDM, we
design stepwise pre-grasp reward:
appeooch
pre-gmpp p gnou
-D;
hand || > 0.2 and <
omnigrasp
pro-goop
Pt S 0.2 and <<
band
(4)
based on time and the distance between the object and hands. Here, À 1.5s indicates the frame in which grasping should occur, and p. hand indicates the hand position. When the object is far away from the hands
( (11gmi--grasp bind 2|2 0.2),
approach reward r ropprouc similar to point-goal [42, 81] reward ppprooch = I| grap - p! ||2 - grasp phamd ll2,, where the policy is encouraged to get close to the pre-grasp. After the hands are close enough (5 0.2m), we use more precise hand imitation reward: r p.r-gga Whee" prm-pp -p." lla x1 (1)
p. 125022) + waee 100 gremm - -/tn*lla to encourage the hands to be close
to pre-grasps. For grasps that involve only one hand, we use an indicator variable 1 {Ilp) grepp pihh ||2 S 0.2] to filter out hands that are too far away from the object. After timestep X, we use only the object trajectory
following reward:
(Woppe 100 1 12 4 mie 1001 a 8 wine
Pgeld of 11212 4om
51 w of ( wr. (5)
abj computes the difference between the current and reference object pose. which is filtered by an indicator variable 1 (C} that is set to true if the object is in contact with the humanoid hands, The reward I [[C} w. encourages the humanoid's 's hand to have contact with the object. Hyperparameters can be found in Appendix
Object 3D Trajectory Generator. As there limited numher of ground-truth object trajectories [17], either collected from MoCap or animators, we design a 3D object trajectory generator that can create trajectories with varying speed and direction. Using the trajectory generator, our policy can be trained without any ground-trude object trajectories. This strategy provides better coverage of potential object trajectories. and the resulting policy achieves higher success in following unseen trajectories (see Table 1). Specifically, we extend the 2D trajectory generator used in PACER [65, 76] to 3D, and create our trajectory generator "g""(g""))
Ti.T- Given initial
aobj.
object pose ohj TD30 can generate a sequence of plausible reference object motion obyy We limit the z-direction trajectory to between 0.03m and 1.8m and leave the direction unbounded. For more information and sampled trajectories, please refer to Appendix C.

---

Table 1: Quantitative results on object grasp and trajectory following on the GRAB dataset.
GRAB-Gial-Tdd (CrossCllyect 140 sequenecs, 5 uuscen obgects)
GRAB-IM TTas (Cros-Sulyect, 92 segueneos 44 abgects)
Mathod
Traj
Su T
Suntaj 1
TTR 7
Fpe 3
Fix
Feur
Fut I
Sucegeep
Suntaj
TTR
Fym
Fut J
Fux J
Fi4
PPO 10B
Gen
98.415
55.9%
97.5%
36.4
0.4
21.0
14.5
96.8%
$3.25
97.9%
35.6
0.5
19.5
13.9
PRC [42]
MaCap
3.61
11.4%
81.1%
66.1
0.3
13
3.8
0%
3.3%
97.45
56.5
0.3
1.4
2.9
AMP [57]
Gen
90.616
16.5%
93.0%
40.7
0.6
5.3
5.3
95.8 %
49.2%
96.55
34,9
0.5
6.2
6.0
Draun et al. [6]
MoCap
79%
85%
66%
635
Omnignapp
MpCap
91.6%
84.8%
98.7%
28.0
0,3
43
95. 85.8%
35.4%
99 89.8%
27.5
0.6
5.0
50
Omnignap
Gen
100%
94.1%
99.6%
30.2
0.90
5.4
4.7
98.9%
90.5%
99.8%
27.9
0.97
0.3
5.4
Training. Our training process is depicted in Algo One of the main sources of performance improvement for motion imitation hard-negative mining [42, 43], where the policy is evaluated regularly to find the failure sequences to train on. Thus, instead of using object curriculum [75, 85, 101], we use a simple hard-negative mining process pick hard objects Orand to train on. Specifically, let s, the number of failed lifts for object j
over all previous runs, The probability of choosing object j among all objects is P(j) = E/ -
Object and Humanoid Initial State Randomization. Since objects can have diverse initial positions and orientations with respect to the humanoid, it is crucial to have the policy exposed to diverse initial object states, Given the object dataset Ô and the provided initial states (either from MoCap or by dropping the object in simulation) do we perturb do ohi by adding randomly sampled yaw vaw-direction rotation and adjusting the position <bj We do not change the pitch and yaw of the object's 's initial pose some poses are invalid in component qu
simulation. For the humanoid, a standing T-pose there no paired data.
the initial state from the dataset if provided (e.g. GRAB dataset [71]), and
Inference. During inference, the object latent code p: ahj trajectory T.T all that is required, without any dependency on pre-grasps or paired kinematic human pose.
a random object starting pose To and desired object
5
Experiments
Datasets. We use the GRAB [71], OakInk [86], and OMOMO [34] to study grasping small and large objects. The GRAB dataset contains 1.3k paired full-body motion and object trajectories of 50 objects (we remove the doorknob as it is not movable). Since the GRAB datasct provides reference body and object motion, we them to extract initial humanoid positions and pre-grasps. We follow prior art [6] in constructing cross-object (45 for training and for testing) and cross-subject (9 subjects for training and I for testing) train-test sets. On GRAB, we evaluate on following MoCap object trajectories using the mean body shape humanoid. The OakInk dataset contains 1700 diverse objects of 32 categories with real-world scanned and generated object meshes. We split them into 1330 objects for training. 185 for validation, and 185 for testing. Train-test splits are conducted within categories, with train and test splits containing objects from all categories. Since no paired MoCap human motion or grasps exists for the OakInk dataset, we use an off the-shelf grasp generator [86] to create pre- pre-grasps. The OMOMO dataset contains 15 large objects (table lamps, monitors, etc.) with reconstructed mesh, and we pick 7 of them that have cleaner meshes. Due to the limited number of objects from OMOMO, we only test. lifting the objects used for training verify that our pipeline can learn to move larger objects. On OMOMO and OakInk, we study vertical lifting (30cm) and holding (3s) as the trajectory for quantitative results.
Implementation Details. Simulation is conducted in Isaac Gym [46], where the policy run at 30 Hz and the simulation at 60 Hz. For PULSE-X X and PHC-X, cach policy is a 6-layer MLP. For the grasping task, we employ a GRU [14] based recurrent policy and use a GRU with a latent dimension of 512, followed by a 3-layer MLP. We train Omnigrasp for three days collecting around 10" samples on a Nvidia A100 GPU. PHC-X and PULSE-XX are trained once and frozen, which takes around 1.5 weeks and days. Object density is 1000 kg/m². The static and dynamic friction coefficients of the object and humanoid fingers are set to For reference object trajectory, we use @ = 20 future frames sampled at 15Hz. For more details, please refer to Appendix C.
Metrics. For the object trajectory following, we report the position error Eges (mm), rotation error Evs: (radian), and physics-based metrics such as acceleration error Exe (mm/frame") and velocity error Ext (mm/frame). Following prior art in full-body simulated humanoid grasping [6]. we report the grasp success rate SucCgrup and Trajectory Targets Reached (TTR). The grasp success rate SucCyrop deems a grasp successful when the object is held for at least 0.5s in the physies simulation without dropping. TTR measures the ratio the target position (< 12cm away from the target position) reached over all the time steps in the trajectory and is only measured on successful trajectories. To measure the complete trajectory success rate, we also report Succmaj. where a trajectory following is unsuccessful if, any point in time, the object is > 25cm away from the reference.
7

---

CoJoRAB
Ob Oakink
do) OMOMO
Figure 3: Qualitative results. Unseen objects are tested for GRAB and OakInk. Green dots: reference traeectories. Best seen in videos supplemie ement site.
5.1 Grasping and Trajectory Following
As motion is best seen in vidcos, please refer to supplement site for extended evaluation on trajectory following, unseen objects, and robustness. Unless otherwise specified, all policies are trained on their respective dataset training split, and conduct cross-dataset experiments GRAB and OakInk. All experiments are 10 times and averaged as the simulator yields slightly different results for each due e.g. floating-point error. full-body simulated humanoid grasping relatively new task with limited number of baselines. we use Braun et [6] as our main comparison. We also implement AMP [57] and PHC [42] as baselines. We train AMP with a similar state and reward design (without using PULSE-X's latent space) and task and discriminator reward weighting of 0.5 and 0.5. PHC refers to using imitator for grasping, where we directly feed ground- -ttuh kinematic body and finger motion a pretrained imitator grasp objects. Since PHC and PULSE-X require pre-training, we also include PPO-1OB, which is trained using RL without PULSE-X for a month (~~ 10 billion samples).
GRAB Dataset (50 objects). Since Braun et al. do not use randomly generated trajectories, we train Omnigrasp using two different settings for fair comparison: one trained with MoCap object trajectories only, and one trained using synthetic trajectories only. From Table 1, we see that our method outperforms prior SOTA and baselines all metries, especially on success rate and trajectory following. Since all methods are simulation- based, we omit penetration/font sliding metrics and report the precise trajectory tracking errors instcad. Training directly using PPO without PULSE-X leads to performance that significantly lags behind Omnigrasp. even though it has used similar aggregate samples (counting PHC-X and PULSE-X training). Compared to Braun al., Omnigrasp achieves high success the motion imitator, PHC, yields low success rate even when the ground- -tuthh kinematic pose is provided, showing that the imitator's 's errоr (on average 30mm) is too large to overcome for precise object grasping. The body shape mismatch between MoCap and our simulated humanoid also contributes to this crror. AMP leads to a low trajectory success rate, showing the importance of using motion prior in the actions space. Omnigrasp track the MoCap trajectory precisely with average error of 28mm. Comparing training on MoCap trajectories and randomly generated ones, we can see training on generated trajectories achieves better performance on success rate and position error, though worse on rotation error. This is due to our 3D trajectory generator offering good coverage on physically plausible 3D trajectories, but there is gap between the randomly generated rotations and MoCap object rotation. This can be improved by introducing more rotation variation on the trajectory generator. The gap between trajectory Sucema and grasp success Succ shows that following the full trajectory a much harder task than just grasping, and the object can dropped during trajectory following. Qualitative results can be found in Fig. 3.
both object lifting and trajectory following. Directly using
OakInk Dataset (1700 objects). On the OakInk dataset, we scale our grasping policy to >100 objects and test our generalization to unseen objects. We also conduct cross datsset experiments, where we train the GRAB dataset and test the OakInk dataset. Results are shown Table 3. We out the 1330 objects are trained be picked up. and the whole lifting process also has high success rate. We ohserve similar results on the test split. Upon inspection, the failed ohjects are usually either too large or too small for the humanoid to establish a grasp. The large number of objects also places a strain on the hard-negative mining process. The policy trained both GRAB and OakInk shows the high- GRAB, there bi-manual pre-grasps, and the policy learned
that 1272
est success
both hands.

---

Table 3: Quantitative results OakInk with method. We also Omnigrasp cross-dataset, where a policy trained on GRAB tested on the OakInk dataset.
OukInk- Train (1330 objects)
OakInk-Tes Test (185 objects)
Training
Suotguup
Sucting
Egs 1
Exx 4
Ege
Ex
Sactineg
Suctine
Epe 1
Em 1
Exc 4
Eat
7.7
94.3%
87.5%
7.6
GRAB
$4.5%
75.2%
99.9%
22.4
0.4
6.8
5.7
81.95
72.1%
99.9%
22.7
0.4
7.1
5.8
GRAB + Oaktnk
95.6%
92.0%
100%
21,0
0.6
5,4
4.8
93.5%
89.0%
100%
21.3
0.6
5.4
4,8
Table 4: Ablation on various strategies of training Omnigrasp. PULSE-X: whether to use the latent motion representation. pre-grasp: pre- pre-grasp guidance reward. DeX-AMASS, whether to train PULSE-X on the dexterous AMASS dataset. Rand-pose: randomizing the object initial pose. Hard-neg: hard-negative mining.
GRAB Goal Test (Cross Object, 140 sequesees, unseen objects)
PULSEX
pro-goup
DESSAMASS
Rand-pose
Hand-nep
Succegup
Stictie
1
Pig
Fs. L
Fax 4
Fest
97. 97.0%
33.6%
92.8%
43.5
0.5
10.6
94.4%
77.3%
09.39
30.5
/
x
92.9%
79.9%
99.25
31.4
1.1
94.0%
71.0%
98.4%
32.3
1.3
6.2
5.7
/
/
100%
94.1%
99.6%
36.2
0.9
Using both hands significantly improves the success rate on some larger objects, where the humanoid can scoop up the object with one hand and carry with both. As OakInk only has pre-grasps using one hand, it cannot learn such a strategy. Surprisingly. train- ing only GRAB achieves high success rate OakInk, picking up more than 1000 objects without training on the dataset, showcasing the robustness of our grasping policy on unseen objects.
Table 2: Quantitative results on the OMOMO dataset.
OMOMO (7 objocts)
SuoC
Sucenu
TTR f
Fyss
Rvo
777
7/7
100%
22.8
0.2
3.1
3.3
OMMMO Dataset (7 objects). On the OMOMO dataset, we train policy to show that our method can learn to pick up large objects. Table 2 shows that our method can successfully leamn to pick up all the objects, including chairs and lamps. For larger objects, the pre-grasp guidance is essential for guiding the policy learn bi-manual manipulation skills (as shown Fig 3)
5.2 Ablation and Analysis
Ablation. In this section, we study the effects of different components of our framework using the cross-object split of the GRAB dataset. Results are shown in Table 4. First, we compare our method trained with (Row 6) or without (R1) PULSE-X'S X's action space. Using the same reward and state design, the universal motion prior significantly improves success rates. Upon inspection, using PULSE-X also yields human-lik like motion, while not using it leads to unnatural motion (see in supplement site). R2 vs. R6 shows that the pre-grasp guidance is essential in learning grasps that are stable for grasping objects, but without it, some objects can still be grasped successfully. The difference hetween R3 and R6 is whether to train using the dexterous AMASS dataset. R3 vs R6 shows that without training dataset that has diverse hand motion and full-body motion. the policy can learn to pick up objects (high grasp success rate), but struggles in trajectory following. This is expected as the motion prior probably lacks the motion of "holding the object while moving". R4 and R5 show that object position randomization and hard-negativin mining are crucial for learning robust and successful policies. Ablations on the object latent code, RNN policy, etc. can be found in the Appendix C.
see that using
Analysis: Diverse Grasps. In Fig. 4, we visualize the grasping strategy used by our method. We can see that based the object shape, our policy uses diverse set of grasping strategies hold the object during the trajectory following. Based on the trajectory and object initial pose, Omnigrasp discovers different grasping poses for the same object, showcasing the advantage of using simulation and laws of physics for grasp generation. We also notice that for larger objects. our policy will resort to using two hands and non- non-prehensile transport strategy. This behavior is learned from pre-grasps in GRAB, which utilize both hands for object manipulation.
Analysis: Robustness and Potential for Sim-to-real Transfer. In Table 5, we add uniform random noise [-0.01, 0.01] to both task observation (positions, object latent codes, etc.) and proprioception. A similar scale (0.01) of random noise is used in sim-to-real RL tackle noisy input real-world humanoids [28]. We see that Omnigrasp is relatively robust to input noise, even though it has not been trained with noisy input. Performance drop is more prominent in the acceleration and velocity metrics. Adding noise during training can further improve robustness. We do not claim that Omnigrasp currently ready for real- world deployment, but

---

Figure 4: (Top rows): grasping different objects using both hands. (Bottom) diverse grasps the same object.
Table 5: Study how noise affects pretrained Omnigrasp Policy
GRAD- TeokD---Go---ess Ohjee-Ohjeer. 140 sequences, 5 unseen objects)
GRAB -MMs- Test (Cross Subject, 92 sequences. 64 objects)
Meshod
Nouse Scale
Soccywo
Succue
TTR 1
Ege. 1
Ere
Ber I
But
Suo0
Succeg
TTR
Epm
Ey
Cur
Exg
Omnigrasp
109%
94.1%
99.6%
30.2
0,93
544
4.7
98.956
90.5% $6.2%
99.8%
27.9
0.97
6,3
5.4
Omnigrasp
0.01
100%
91.4%
99.2%
34.8
1.1
15.6
11.5
99.5%
99.615
32,5
1.0
17.9
13.2
believe that a similar system design plus sim-to-real modifications (e.g. domain randomization. distilling into vision-based policy) has the potential. We conduct more analysis on the robustness of our method with respect initial object position, object weight, and object trajectories supplement site.
Limitations, Conclusions, and Future Work
Limitations. While Omnigrasp demonstrates the feasibility of controlling simulated humanoid to grasp diverse objects and hold them to follow diverse trajectories, many limitations remain. For example, though the 6DoF input is provided in the input and reward, the rotation error remains to support precise in hand manipulations. The success rate on trajectory following can improved, as objects can be dropped or not picked up. Another area of improvement is to achieve specific types of grasps on the object, which may require additional input such desired contact points and grasp. Human-level dexterity, even simulation, remains challenging. For visualization of failure cases, supplement aite.
further improved. Omnigrasp has yet
Conclusion and Future Work. In conclusion, present Omnigrasp. humanoid controller capable of grasping 1200 objects and following trajectories while holding the object. It generalizes unseen objects similar sizes, utilizes hi-manual skills, and supports picking up larger objects. universal humanoid motion representation, grasping can be learned using simplistic reward and state designs. Future work includes improving trajectory following success rate, improving grasping diversity, and supporting object categories. Also, improving upon the humanoid motion representation is promising direction. While we utilize simple yet effective unified motion latent space, separating motion representation for hands and body [3, 6] could lead to further improvements. Effective object representation also an important future direction. How to formulate an object representation that does not rely on canonical object pose and generalizes vision-basee systems will be valuable help the model generalize more objects.
demonstrate that by using pretrained
Acknowledgement. Zhengyi Luo is supported by the Meta AI Mentorship (AIM) program.
10

---

References
[1] Dexterous hand series. http://wwwwww.shoooottttooo wwt.b//ttt.ooo/eeeee.eoooo....e. hanprtterruu-aan---ee haad-aandd-rr-e-- series/. 19 Sept. 2023.
Accessed: 2024-5-13.
[2] I. Akkaya, M. Andrychowiez, M. Chociej, M. Litwin, B. McGrew, A. Petron. A. Paino. M. Plappert.
G. Powell. R. Ribas, al. Solving rubik's cube with robot hand. arXiv preprint 11110:110..07113.
2019.
[3] J. Bac, Won, D. Lim, C.-H. Min, and Y. M. Kim. Pmp: Learning to physically interact with environments using part-wise motion priors. ACM SIGGRAPH 2023 Conference Proceedings, pages 1-10, 2023,
[4] M. Black, Patel, Tesch, and Yang. Bedlam: synthetic dataset of bodies exhibiting detailed lifelike animated motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8726 8737, 2023.
[5] S. Brahmbhatt, C. Ham, C. C. Kemp, and J. Hays. ContactDB: Analyzing and predicting grasp contact via thermal imaging. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
2019.
[6] J. Braun, S. Christen, M. Kocabas, E. Aksan, and O. Hilliges. Physically plausible full-body hand-object interaction synthesis. International Conference on 3D Vision (3DV), 2024.
[7] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding. D. Driess, A. Dubey, models transfer web knowledge to robotic control. arXiv
Finn, et al. Rt-2: Saiion-anguage-eeee
preprint ar..v:2307..555550
2023.
[8] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman,
A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint
arXiv:2212.06817, 2022.
[9] V. Caggiano, S. Dasari, and V. Kumar. Myodex: Generalizable representations for dexterous physiological
manipulation. 2022.
[10] Z. Cao. I. Radosavovic, A. Kanazawa, and J. Malik. Reconstructing hand-object interactions in the wild.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12417-12426.
2021.
[11] T. Chen, M. Tippur, Wu, V. Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand reorientation
novel and complex object shapes. Science Robotics, r(84)xaa,) 2023.
[12] T. Chen, Xu, and P. Agrawal. A system for general in-hand object re-orientation. Conference on Robot Learning, 2021.
[13] N. Chentanez, M. Müller, M. Macklin, V. Makoviychuk, and S. Jeschke. Physics- motion capture imitation with deep reinforcement learning. In Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interuction and Games, pages 1-10, 2018
[14) K. Cho, B. van Merrienboer, Çaglar Gülçehre. D. Bahdanau, F. Bougares, H. Schwenk, and Y. Ben- gio. Learning phrase representations using -ncoder-dccoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, 2014.
[15] S. Christen, L. Feng, W. Yang. Y.-W. Chao, O. Hilliges, and J. Song. Synh2r: Synthesizing hand-object motions for learning human-to-roon
handovers. 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 3 3168-312 -375. IEEE, 2024.
[16] S. Christen, M. Kocabas, E. Aksan, J. Hwangbo, J. Song, and O. Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20577-20586, 2022.
[17] S. Dasari, A. Gupta, and V. Kumar. Learning dexterous manipulation from exemplar object trajectories and pre-grasps. In 2023 IEEF International Conference on Robotics and Automation (ICRA), pages 3889-3896. TERE, 2023.
[18] Z. Fan, M. Parelli, M. E. Kadoglou. M. Kocabas, X. Chen, M. J. Black, and O. Hilliges. Hold: Category- agnostic 3d reconstruction of interacting hands and objects from video. arXiv preprint arXiv:2311.188 2023.
[19] H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Y. Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023. [20] Z. Fu, X. Cheng. and D. Pathak. Deep wody control: Learning a unified policy for manipulation and locomotion. arXiv preprint arxiv:2210, 1222110100444 10044, 2022.
[21] G. Garcia-Herrnadd, S. Yuan, S. Back, and T.-K. Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In Proceedings of the IEEE conference on computer vision and
patlern recognition, pages 409-419, 2018
[22] A. Ghosh, R. Dabral, V. Golyanik. C. Thcobalt, and P. Slusallek. Imos: Intent-driven full- boll-body motion synthesis for human-object interactions. In Earographics, 2023.
[23] K. Gong. B. Li. J. Zhang, T. Wang. Huang, M. B. Mi, J. Feng, and X. Wang. Posetriplet: Co-evolving
3d human pose estimation, imitation, and hallucination under self-supervision. CVPR, 2022.
[24] T. Haarojja, K. Hartikainen, P. Abbeel, and S. Levine. Latent space policies for hicrarchical reinforcement
lcarning. arXiv preprint Xrv:v8880 800,02208, 2018
[25] Hasenclever. Pardo. R. Hadsell, N. Heess, and Merel. CoMie: Complementary task learning&
mimicry for reusable skills. In H. D. li and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages
4105 4115, PMLR, 2020.
11

---

[26] M. Hassan, Y. Guo. T. Wang. M. Black, Fidler, and X. B. Peng. Synthesizing physical character-scene interactions. In ACM SIGGRAPH 2023 Conference Proceedingy, pages 1-9, 2023.
[27] T. Hc, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. Kitani, C. Liu, and G. Shi. Omnih2o: Universal
and dexterous suman-to-humanoid whole-body teleoperation and learning. In arXiv, 2024.
[28] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning ho-man-t-hhumnoid real-time
whole-body teleoperation. 2024.
[29] T. Howell, N. Gílcadi,
Tunyasuvunakool, K. Zakka, T. Erez, and Tassa. Predictive Sampling: Real-time Behaviour Synthesis with MuloCo. dee 2022.
[30] Huang, L. Pan. Y. Yang. Ju, and Y. Wang. Neural mocon: Neural motion control for physically
plausible human motion capture, arXiv preprint (tti::2203.14066 2022.
[31] H. Jiang, S. Liu, J. Wang, X. Wang. Hand-object contact consistency reasoning for human grasps
generation. In ICCV, 2021.
[32] D. P. Kingma and M. Welling, Auto-encoding variational bayes. 2nd International Conference on
Learning Representations, ICLR 2014 Conference Track Proceedings, pages 1-14, 2014, [33] S. Lee, S. Starke, Y. Ye, J. Won, and A. Winkler. Questenvsim: Environment-aware simulated motion tracking from sparse sensors. arXiv preprint arXiv:2306.0 2023.
[34] J. Li. J. Wu, and C. K. Liu. Object motion guided human motion synthesis. ACM Transactions on
Graphics (TOG), 42(6):1-11, 2023.
[35] Q.Li. J. Wang. C. C. Loy, and B. Dai. Task-oriented human-object interactions generation with implicit neural representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3035-3044, 2024.
[36] Liu and J. Hodgins. Learning basketball dribbling skills using trajectory optimization and deep reinforcement learning. ACM Transactions on Graphics (TOG), (7(:):1-14 2018.
[37] P. Liu, Orru, C. Paxton, N. M. M. Shafiullah, and L. Pinto. Ok-robot: What really matters in integrating open-knowledge models for robotics. arXiv preprint arXiv:2401.
2024.
[38] S. Liu, S. Tripathi, S. Majumdar, and X. Wang. Joint hand motion and interaction hotspots prediction from egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3282-3292. 2022.
[39] X. Liu and L. Yi. Geneoh diffusion: Towards generalizable hand-object interaction denoising via denoising diffusion. In ICLR, 2024.
[40] Z. Luo, J. Cao, R. Khirodkar, A. Winkler, K. Kitani, and W. Xu. Real-time simulated avatar from head-mounted sensors. arXiv preprint urXiv:2403. 06862, 2024.
[41] Z. Luo, J. Cao, Merel, A. Winkler, J. Huang, K. Kitani, and W. Xu. Universal humanoid motion representations for physics- based control. arXiv preprint arXiv:2310.044 00.08,2, 2023.
[42] Z. Luo, J. Cao, W. Winkler, K. Kitani, and W. Xu. Perpetual humanoid control for real-time simulated
avatars. International Conference on Computer Vision (ICCV), 2023.
[43] Z. Luo, R. Hachiuma, Y. Yuan, and K. Kitani. syynamics-reuultte kinematic policy for egocentric pose
estimation. NeurlPS,
2021.
[44] Z. Luo, Y. Yuan, and K. M. Kitani. From universal humanoid control to automatic physically valid
character creation. arXiv preprint (xXi:2206.099880 2022.
[45] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons- Moll,Mol,, and M. J. Black. Amass: Archive of motion
capture as surface shapes. Proceedings of the IEEE International Conference on Computer Vision,
99-0ccthh5441154550 2019.
[46] V. Makoviychuk. L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire.
A. Handa, and Gavriel State. Isaac gym: High performance gpu- based physies simulation for robot lcarning. arXiv preprint a00iv:2108. 10470, 2021.
[47] P. Mandikal and K. Grauman. Dexvip: Learning dexterous grasping with human hand pose priors from video. Conference on Robot Learning, pages 651-661. PMLR, 2022.
[48] J. Merel, Hasenclever, A. Galashov, A. Ahuja. V. Pham, G. Wayne, W. Teh, and Heess. Neural probabilistic motor primitives for humanoid control, 2018.
[49] J. Merel, S. Tunaasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, Erez, G. Wayne, and
N. Heess. Catch and carry: Reusable neural controllers for vision-guided whole-body tasks. ACM Trans.
Graph., 2020.
[50] G. Moon, S. Saito, W. Xu, R. Joshi, J. Buffalini, H. Bellan, N. Rosen, J. Richardson, M. Mallorie, P. Bree,
T. Simon, B. Peng, S. Garg, K. McPhail, and T. Shiratori. A dataset of relighted 3D interacting hands. In
NeurlPS Track on Datasets and Benchmarks, 2023.
[51] T. Nagarajan, C. Feichtenhofer, and K. Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1688-8697,, 2019.
[52] Narasimhaswamy, U. Bhattacharya, X. Chen, I. Dasgupta, S. Mitra, and M. Hoai. Handiffuser: Text-to-image generation with realistic hand appearances. arXiv preprint arXiv:2403.0 2024.
[53] G. Pavlakos. V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from a single image. Proceedings of the IEFF Computer Society Conference Computer Vision and Pattern Recognition, 2019 109m:::99977-17777
2019.
[54] X. B. Peng, P. Abbecl, S. Levine, and M. van de Panne. Deepmimic. ACM Trans. Graph., 37:1-14, 2018.
12

---

[55] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. Mcp: Learning composable hierarchical
control with multiplicative compositional policies. arXiv preprini arXiv:1905.0988 009800 2019.
[56] X. B. Peng. Y. Guo, L. Halper, S. Levine, and S. Fidler. Ase: Large-scale reusable adversarial skill
embeddings for physically simulated characters. arXiv preprint arXiv:2205. Stt::2205.01966, 2022.
[57] X. B. Peng. Z. Ma, P. Abbeel. S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized
physics-based character control. ACM Trans, Graph., pages 1-20, 2021.
[58]
Prokudin, Lassner, and Romero. Efficient learning on point clouds with basis point sets. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4332-4341, 2019. [59] I. Radosavovic, Xiao, B. Zhang. T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, ((89)ccd 2024.
[60] I. Radosavovic, B. Zhang. B. Shi, J. Rajasegaran, S. Kamat, T. Darrell, K. Sreenath, and J. Malik. Humanoid locomotion as next token prediction. arXiv preprint arXiv:2402. 19469, 2024.
[61] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforeement learning and demonstrations. arXiv preprint arXiv:1709 10087, 2017.
[62] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, arXiv preprint
arXiv:1709. 10087, 2017.
[63] D. Rao, F. Sadeghi, L.. Hasenelever, M. Wulfmeier, M. Zambelli, G. Vezzani, D. Tirumala, Y. Aytar,
J. Merel, N. Heess, and R. Hadsell. Learning transferable motor skills with hicrarchical latent mixture (tfi:22122000665 2021.
policies. arXiv preprint
[64] D. Rempe, T. Birdal, A. Hertzmann, J. Yang. S. Sridhar, and L. J. Guibas. Humor: 3d human motion model for robust pose estimation, arXiv preprint 105.:2200..04668, 2021.
[65] D. Rempe, Z. Luo, X. B. Peng, Y. Yuan, K. Kitani, K. Kreis, S. Fidler, and O. Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. arXiv preprint arXiv:2300.0/8 893,
2023.
[66] J. Romero, D. Tzionas, and M. J. Black. Embodicd hands: Modcling and capturing hands and bodies together. arXiv preprint arXiv:2201.0661
2022.
[67] S. Ross, G. J. Gordon, and J. A. Bagnell. A reduction of imitation learning and structured prediction preprint no-regret online learning. arXiy preprint arXiv: /011000866 2010.
[68] J. Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms,
2017.
[69] O. Taheri, V. Choutas, M. J. Black, and D. Tzionas. GOAL: Generating 4D whole-body motion for
hand-object grasping. In CVPR, 2022.
[70] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas. GRAB: A dataset of whole-body human grasping of objects. In ECCV, 2020.
[71] O. Taheri, N. Ghorbani, M. J. Black, and D. Tzionas. Grab: A dataset of whole-body human grasping of objects. In Computer Rision-ECCT 2020: 16th European Conference. Glasgow, UK. August 23-28, 2020. Proceedings, Part IV 16. pages 581-600. Springer, 2020.
[72] O. Taheri, Y. Zhou, D. Tzionas, Y. Zhou, D. Ceylan, S. Pirk, and M. J. Black. GRIP: Generating interaction poses using latent consistency and spatial cues. In International Conference 3D Vision (3DV), 2024.
[73] P. Tendulkar, D. Surís, and C. Vondrick. Flex: Full-body grasping without full-bodi grasps. In CVPR,
2023.
[74] C. Tessler, Yoni Kasten, I. Y. Guo, and C. Nvidia. Calm: Conditional adversarial latent models for
directable virtual characters.
[75] W. Wan, H. Geng, Liu, Z. Shan, Y. Yang. L. Yi, and H. Wang. Unidexgrasp++: Improving dexterous grasping policy lcarning via coometry-aware curriculum and iterative generatist-specialist learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3891-3902, 2023, [76] J. Wang. Z. Luo,, Y. Yuan, Y. Li, and B. Dai. Pacer+: On-demand pedestrian animation controller in
driving scenarios. arXiv preprint arXiv:2404. 1404:9922 2024.
[77] J. Wang, Y. Yuan, Z. Luo, K. Xie, D. Lin, U. lqbal, S. Fidler, S. Khamis, H. Kong, and Mellon University, Carnegic. Learning human dynamics in autonomous driving scenarios. International Conference on Computer Vision, 2023, 2023.
[78] Y. Wang. J. Lin, A. Zeng, Z. Luo, J. Zhang, and L. Zhang. Physhoi: dlysics-based imitation of dynamic human-object interaction. arXiv preprint arXiv:2312.0453 2023.
[79] A. Winkler, J. Won, and Y. Ye. Questsim: Human motion tracking from sparse sensors with simulated avatars. arXiv preprint arXiv:2209.09330
2022.
[80] J. Won, D. Gopinath, and J. Hodgins. A scalable approach to control diverse behaviors for physically
simulated characters. ACM Trans. Graph., 39, 2020.
[81] J. Won, D. Gopinath, and Hodgins. Physics-based character controllers using conditional vaes. ACM
Trans. Graph., 41:1-12, 2022.
[82] Y. Wu, J. Wang. Y. Zhang. S. Zhang, O. Hilliges, F. Yu, and S. Tang. Saga: Stochastic whole-body
grasping with contact. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[83] K. Xie, T. Wang, U. Iqbal. Y. Guo, S. Fidler, and E. Shkurti. Physics-based human motion estimation and 2021.
synthesis from videos. arXiv preprint urXiv:2109.099))
[84] X. Xie, B. L. Bhatuagar, and G. Pons- Moll. Visibility aware numan-objed interaction tracking from single rgb camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
13

---

pages 4757-4768. 2023.
[85] Y. Xu, W. Wan, J. Zhang, H. Liu, Z. Shan, H. Shen, R. Wang, H. Geng, Y. Weng, J. Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal- conditioned policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4737-4746, 2023.
[86] Yang. K. Li, X. Zhan, F. Wu. A. Xu, L. Liu, and Lu. OakInk: A large-scale knowledge repository for understanding hand-object interaction. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[87] L. Yang. K. Li. X. Zhan, F. Wu, A. Xu, L. Liu, and C. Lu. Oakink: A large-scale knowledge repository for understanding hand-object interaction. In Proceedings of the IEEE/CVT Conference Computer Vision and Pattern Recognition, pages 20953-20962, 2022.
[88] Y. Ye, A. Gupta, K. Kitani, and S. Tulsiani. G-hop: Generative hand-object prior for interaction
reconstruction and grasp synthesis, In CVPR, 2024.
[89] Y. Ye. A. Gupta, and S. Tulsiani. What's your hands? 3d reconstruction of generic objects in hands. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3895-3905,
2022.
[90] Ye, X. Li, A. Gupta, De Mello, S. Birchfield, Song, S. Tulsiani, and S. Liu, Affordance diffusion:
Synthesizing hand-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22479-22489, 2023.
[91] Y. Ye and C. K. Liu. Synthesis of detailed hand manipulations using contact sampling. ACM TOG,
1-00 2012. [92] Y. Yuan and K. Kitani. 3d ego-pose estimation via imitation learning. In Computer Vision - ECCV 2018,
volume 1220 LNCS, pages 763-778. Springer International Publishing. 2018.
[93] Y. Yuan and K. Kitani. Ego-pose estimation and forecasting as real-tit -tme pd control. Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob:10088 10081-10091 2019.
[94] Y. Yuan, J. Song. U. lqbal, A. Vahdat, and J. Kautz. Physdiff: Physics-guided human motion diffusion
model. arXiv preprint arXiv:2212.025500 2022.
[95] Y. Yuan, S.-E. Wei, Simon, K. Kitani, and Saragih. Simpoe: Simulated character control for 3d
human pose estimation. CVPR, 2021.
[96] A. Zeng. S. Song, K.TT. Yu, E. Donlon, R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-ant likk-and-pacce of novel objects in clutter with multi- aulti-afffrdaace grasping and cross-domain image matching. The International Journal of Robotics Research, 41(():690-700 2022.
[97] H. Zhang, S. Christen, Z. Fan, O. Hilliges, and J. Song. GraspXL: Generating grasping motions for diverse objects at scale. arXiv preprint arXiv:2403. 19649. 2024.
[98] H. Zhang. S. Christen. Z. Fan, L. Zheng, J. Hwangbo, J. Song, and O. Hilliges, ArtiGrasp: Physically
plausible synthesis of bi-manual dexterous grasping and articulation. In International Conference on 3D Vision (3DV), 2024.
[99] H. Zhang, Y. Ye, T. Shiratori, and T. Komura. Manipnet: neural manipulation synthesis with hand-object
spatial representation. ACM TOG, 40(4):1-14, 2021.
[100] H. Zhang. Y. Yuan, V. Makoviychuk, Y. Guo, S. Fidler, X. B. Peng, and K. Fatahalian. Learning physically simulated tennis skills from broadcast vidcos. ACM Trans. Graph., 42:1-14, 2023.
[101] Zhang, A. Clegg. S. Ha, G. Turk, and Y. Ye. Learning to transfer in-hand manipulations using greedy
shape curriculum. In Computer Graphics Forum, volume 42, pages 25 25-36. Wiley Online Library. 2023. [102] Zhang, D. Gopinath, Ye, J. Hodgins, G. Turk, and Won. Simulation and retargeting complex multi-character interactions. In ACM SIGGRAPH 202.3 Conference Proceedings, pages 1-11, 2023. [103] K. Zhou, B. L. Bhatnagar, J. E. Lenssen, and G. Pons-Moll. Toch: Spatio-temporal object-to-hand correspondence for motion refinement. In ECCV. Springer. October 2022.
[104] Y. Zhou, C. Barnes, J. Lu, J. Yang. and H. Li. On the continuity of rotation representations in neural networks. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Jun:: 2019.
14

---

Appendix
A Introduction
15
B Details about PHC-X and PULSE-X
15
B.1 Training and Architecture
15
B.2 Evaluation
16
Details about Omnigrasp
16
C.1 Object Processing
16
C.2 Training Details
16
C.3 Additional Ablations
17
C.4 Per-object Successrate breakdown
17
Additional Discussions
17
D.I Alternatives to PULSE-X
17
Broader social impact.
18
A Introduction
In this document, include additional details about Omnigrasp that omitted from the main paper due the page limit. Sec.B, we include additional information about training and evaluating the performance our humamoid motion representation, PULSE-X. In Sec. C, we include details about Omnigrasp, such as the trajectory generator and training procedures.
Extensive qualitative results are provided at the project page as well as the supplementary zip files (which contain lower-resolution videos due file size limitations). As motion is best in videos, highly encourage readers view them to judge the capabilities of method better. Specifically, controller to trace characters "Omnigrasp"
visualize using
while holding unseen objects during training. This complex trajectory is never seen during training. We also visualize the policy on GRAB [71], OakInk [87], and OMOMO [34] datasets, both for training and testing objects. On the GRAB dataset, we follow MoCap trajectories. while for the OakInk and OMOMO datasets, training. demonstrate robustness to different object poses, weights, and directions, also test our method by varying these variables and show that it can pick up objects. Interestingly, we notice that our method prefers to use both hands to pick and hold the object as the weight of the object increases. We also include motion imitation and random motion sampling for PHC-X and PUT -E-.X Further, visualize constructed dexterous AMASS dataset and the motion imitation result. Last, include failure for grasping and trajectory following.
showcase randomly generated trajectories for
B Details about PHC-X and PULSE-X
Data Cleaning. To train both PHC-X and PULSE-X. X. we follow PULSE's [41] procedure in filtering on implausible motion. This process yields 14889 motion sequences from the AMASS dataset for training humanoid motion representation. Out all 14889 sequences, only 9% of the sequences contain hand motion, and training on bias motion imitator to limited dexterity. Thus, we construct the dexterous AMASS dataset by pairing hand-only motion with body-only motion and demonstrate its effectiveness learning a motion representation that enables object grasping.
B.1 Training and Architecture
The state, action, and rewards for PHC-X and PULSE-X follow the implementation choices of PULSE with the only modifications on the training data (dexterous AMASS) and humanoid (SMPL-X), X). PHC-X trained 1.5 week while PULSE-X X takes days. We use the same-sized networks: 6-layer MLP of units [2048, 1536, 1024, 1024, 512, 512] for PHC-X and 3-layer MLP of units [3096, 2048. 1024] for PULSE-X0 X's encoder and decoders. We notice that due the increase in DoF from SMPL (69) SMPL-X (153), simulation is ~-2 times slower.
15

---

Table Hyperparameters for Omnigrasp, PHC-X, and PULSE-X. o: fixed variance for policy. T discount factor. c: clip range for PPO.
Method
Batch Size
Learning Rate
# of samples
PHC-X
3072
10
0.05
0.99
0.2
10
Batch Size
Learning Rate
Latent size
# of samples
PULSE-X
3072
x 10-4
48
10
Batch Size
Learning Rate
Wop
Wer
WFv
Meow
Mec
of samples
Omnigrasp
3072
x 10"4
0.05
0.99
0.2
0.5
0.3
0.05
0.05
0.1
~ 10
B.2
Evaluation
We evaluate PULSE-X and PHC-X on our constructed dexterous AMASS dataset. the mean per- joint position error (mm) both global Eg-iip and local Empipe (root-relative) set- tings. We also report acceleration and velocity errors, similar to the object tra- jectory following the setting but averaged across all body joints. From Table training data while maintaining a low per-joint error. Distilling from PHC-X to PULSE-X, we observe similar degradation in imitation performance as in PULSE, akin to the reconstruction error in training VAEs [32].
Table Imitation result on dexterous AMASS (14889 sequences).
The metrics
Dexterous AMASS Train
Method
Succ f
F papae
Fugir
4
Eig.
PHC-X PULSE-X
99.9
29.4
31.0
4.1
99.5
42.9
46.4
4.6
6.7
can see that PHC-X PULSE-X achieve a high success rate on
C Details about Omnigrasp
C.1 Object Processing
Since the simulator requires convex objects for simulation, we use the built-in v-haed function to decompose the meshes into convex geometries. The parameters we use for decomposition can found in Table 7. To compute object latent code, use 512-d BPS [58] by randomly sampling 512 points unit sphere and calculating their distances points the object mesh. As some object meshes have large number of vertices, we also perform quadratic decimation on the mesh contains more than 50000 vertices.
C.2 Training Details
Early Termination. During training, we terminate the episode whenever the object is more than 12cm away from its desired reference trajectory at time step || P: 1
| > 0.12.
Table Removal. Since the GRAB and OakInk datasets are table-top top objects, we use table the beginning of the episode to support the object. However, since our randomly generated trajectory can collide with the table and the humanoid has no environmental awareness except for the object, remove the table after certain timestamps (1.5s) during training.
Contact Detection. As IsaacGym does not provide easy access contact labels and only provides contact forces, there way of differentiating between contact with the table, humanoid body, or objects. Thus, resort to couristic-based way to detect contact. Specifically. if object within 0.2m from the hands, non-zero contact forces, and has a non-zero velocity, we deem it to have contact with the hands.
Trajectory Generator. Randomly generated trajectories can seen on our supplement site on the OakInk and OMOMO dataset, as there is no paired MoCap object motion for these datasets. We sample a random velocity and delta angle each time step and aggregate the velocities to produce full trajectories. We bound the velocity of our randomly generated trajectories to be between 2] m/s and bound the angles to between [0, 1] radian. With a probability the trajectories can not be too high or low, we bound the z-direction translation to be between [0.1, 2.0]. For orientation, we sample random ending orientation at the end of the trajectory and interpolate it between the object's initial trajectory obtain sequence of target rotations.
0.2, sharp turn could happen where the angle is between [0, 2T.. As
16

---

Table Additional ablations: Object- Object-latent latent refers to whether to provide the object shape latent code Thh to the policy. RNN refers to either using an RNN-based policy or an MLP-based policy. Im-obs refers to whether to provide the policy with ground truth full-body pose 4:-; input.
GRAB Test (Cross Object, 140 sequences, 5 unseen objects)
idx
Object Latent
RNN
Im-obs
Succ
SucCen f
TTR f
Eps
E 4
Exs:
E,d
x
100%
93.2%
99.8%
28.7
1.3
6.1
2
x
99.9%
89.6%
99.0%
33.4
1.2
4,4
3
/
95.2
77.8%
97.9%
32.2
0.9
3.2
3.9
4
/
x
100%
94.1% %
99.6%
30,2
0.9
5.4
4.7
C.3
Additional Ablations
In Table 8, we provide additional ablations left out due to space limitations. Comparing Row 1 (R1) and R4, we see that on the GRAB dataset cross-object test set, a policy trained without the object shape latent code a² can be on par with a policy with access to it. This because the humanoid learned general "grasping" for small objects, and the testing objects do not deviate too much from these strategies. Also, upon inspection, RI learns to rely on bi-manual manipulation and using two hands when it cannot pick it up with one hand, which point the object shape longer affects the grasping posc much. As result, RI suffers higher rotation error F On the GRAB cross-subject test (44 objects). RI has trajectory success rate of Sucomi 84.2%, worse than R4's s 90.5%. R2 vs. R4 shows that the RNN policy is more effective than the ML.P-based policy, confirming our intuition that some form of memory is beneficial for a sequential task, such as grasping and omnidirectional trajectory following. R3 studies the scenario where we provide ground truth full-body pose 4, to the policy at all times, similar the setting in PhysHO1 [78] (though without the contact graph). Results show that this strategy leads to worse performance, and also prevents us from training on objects that not have paired MoCap full-body motion. This indicates that the contact graph is needed to imitate human-objeet interaction precisely. Omnigrasp provides a flexible interface to support learning and testing on novel objects without needing paired ground-trufe -auun full-body motion.
C.4 Per-object Successrate breakdown
Table 9: Per-object breakdown on the GRAB- G-aa (cross-object) split.
Table we break down the per-object success rate on the cross-object split of the GRAB dataset. Of the 5 novel objects, our model finds it hardest pick up the toothpaste, which has elongated surface. Upon inspection, we find that Omnigrasp will slip on the round edges of the toothpaste surface and fail to grasp the object. Compared to previous SOTA [6], Omnigrasp outperforms in all metrics and objects.
Object
Brun al. 16]
Ohunigrasp
Succ
TTR 1
SSoC 1
Suctu
TTR
Apple
95%
91%
100%
99.6%
99.9%
Binoculuss
54%
83%
100%
91.55
99.6%
Cumera
95%
85%
100%
97.7%
99.7%
Mag
89%
74%
100%
07.95%
99.8%
Tooihoue
64%
94%
100%
80.9%
99.0%
D
Additional Discussions
D.1
Alternatives to PULSE-X
One alternative way for reusing the motor skills from motion imitator like PHC-X is train kinematic motion latent space to provide reference motion to drive PHC-X. Such gencral-purposs kinematic latent space has been used in physics- based control for pose estimation [77] and animation [100]. However, few have been extended to include dexterous hands. These latent spaces, like HuMoR [64], model motion transition using an encoder q, (ze å.. q.-1) and decoder po (q, 22, q.-1) where á, is the pose time step and is the latent code. 94 and py are trained using supervised learning. The issue with applying such a latent space to simulated humanoid control is twofold:
The output q. of the VAE model, while representing natural human motion, does not model the PD-target (action) space required to maintain balance. This is shown in prior art [77, 100], where an additional motion imitator is still needed to actuate the humanoid by imitating q, instead of using á, as policy output (PD-target).
q, and Pa are optimized using MoCap data, whose q, values are computed using ground truth motion and finite difference (for velocities). As a result, q and P, handle noisy humanoid states from simulation poorly. Thus. [77] runs the kinematic latent space in an open-loop auto- regressive fashion without feedback from physics simulation (e.g. using a.-1 from the previous time step's output rather
17

---

than from simulation). The lack of feedback from physics simulation leads to floating and unnatural artifacts [77], and the imitator heavily relies residual force control to maintain stability.
E Broader social impact.
Our method can be used to create realistic grasping policy for humanoids, generate animation, synthesize stable grasps. While the state designs have access privileged information, the overall system design methodol- ogy (plus sim-to-real transfer techniques real humanoid robot. Thus, it has potential positive social impact, as can create content or help build the next generation of home robots.
as domain randomization) has the potential be transferred to
18

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose framework to train simulated humanoid to pick diverse objects and follow omnidirectional trajectories, and showcase our performance on popular datasets and video results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations, failure cases, and visulizations failure cases are disuessed in the main paper and supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings. model well- sell-specification
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice
what implications would The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
19

---

All the theorems, formulas, and proofs in the paper should be numbered and cross cross-reference..
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core
paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe our method full detail and will provide code and trained models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible others to replicate the model with same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the casc of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. registered users), but it should be possible for other rescarchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described supplemental material?
Answer: [Yes]
Justification:
Code and instructions are included in the supplement.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see NeurIPS code and data submission guidelines (lttps::/nips.c// cctbbliic/c for more details.
guides/ guides/CodeSutee (CodeSubmissionPo
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
20

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines httpp:///niaa..cpuutieese quidee//Coeeuubiissiii cuatmisiionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Detais
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experimental settings are explained in detail for training and testing in the main paper and supplement.
Guidelines:
The answer means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, the appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report all of our experiments by averaging results from 10 runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state they have 96% CI, if the hypothesis of Normality errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide training time and requirements.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics https:/ fttps://neurip.... it catechicaGeeeeeees
Answer: [Yes]
Justification:
We adhere to the ethics guideline.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special considcration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss hoth potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss social impact in the supplment.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
foundational research not tied to particular
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release models that can cause potential harm.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model implementing safety filters.
22

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We do not any existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-pac re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: dp not involve crowdsourcing nor research with human subjects.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
23

---

Answer:
[NA]
Justification: not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
24


Reviews:
[{"review_id": "qqKUUa1sRs", "reviewer": "Reviewer_rKeN", "summary": "This paper proposes an approach to learning a humanoid controller that can manipulate objects to follow trajectories. It first assembles a dataset of human bodies and hands motions, and learns a control policy from the state transitions in the dataset. Then, they distill this policy using a VAE to obtain an action decoder that models realistic human action distributions. After that, they use this distribution as the action space to learn an RL policy whose reward is based on object trajectory following. With this action space, the neural network policy can output realistic actions and avoid exploration difficulties. The authors demonstrate several impressive, yet natural and physically plausible motions for a simulated humanoid to grasp and move objects according to trajectories.", "questions": "Regarding the robustness of the system: How would the policy perform in noisy situations? For example, when the joint positions are not accurate (real robots will have imperfect joint encoders). And how accurate does the object mesh need to be?\n\nHow important is the morphology similarity between the humanoid and the hand? For example, do they need to have the same number of joints or links? I also observed even in the current system, it seems the simulated humanoid does not perfectly match human kinematics. Is this correct? If so, how do you bridge this gap?\n\nRegarding the simple baseline alternatives:\n* Instead of learning actions from human trajectory sequences between two timesteps, why not directly using a VAE to reconstruct human poses? This VAE will directly learning a distribution of human joint positions. Then in the RL policy training, you can map the policy output to natural human joint positions using this VAE.\n* Even when one wants to learn the “actions” instead of “joint positions”, one could use a encoder-decoder structure (either VAE or vanilla auto-encoder) as the policy backbone and directly learn in the PULSE-X phase, instead of separating it into two stages.\nI’m curious if the authors tried these alternatives before? Or does this comparison exist in the literature? Are they performing not well? It would be much helpful to analyze this problem or provide references in the text.\n\nA more comprehensive discussion on the failure cases would be helpful. You show several videos of the failure cases, which I think is very helpful. But why do they fail? Is it because the objects themselves are quite difficult? Regarding this point, it would also be much helpful to have a per-object accuracy analysis.", "limitations": "The authors describe the limitations mainly as not performing dexterous in-hand manipulation. The authors could consider using recent learning-based dexterous in-hand manipulation work as the “mocap dataset” in your formulation and learn action distribution using it.\n\nI think there is another limitation, which is that the current formulation does not demonstrate capability when the object needs to interact with the environment.\n\nWhile I acknowledge that the above two points are out of scope and will not affect my assessment at all, they might be interesting future directions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The results are generally good, and the motion is relatively natural and physically plausible.\n\nThe demonstration of extending this approach to high-dimensional control is important.\n\nThe authors also provide several insightful observations, such as how to assemble the dataset to extend PULSE to human hand motion and how to achieve diverse grasping behaviors. These contributions are all important to the community.", "weaknesses": "While I definitely think this paper is good, I believe there are several critical aspects that should be more carefully evaluated, either qualitatively or quantitatively. I will provide the summary comments in this section and write the specific questions that I would like answers to during the discussion phase in the next section.\n\nFirst, I’m generally curious about the robustness of the proposed system. The object trajectory following policy takes a few inputs without noise from the simulator (joint positions, object latent code, proprioception). However, in reality, these quantities are far from perfect.\n\nSecond, this approach assumes the humanoid is similar enough to the mocap data. I agree this is a reasonable assumption. However, in reality, humanoid robots and their hands are usually not perfect replicas of humans. How similar do their morphologies need to be?\n\nRegarding the learning approach, it first learns a policy to output actions given the human sequence data. Then it distills this policy network using VAE to learn the distributions as the action space. There should be an alternative approach that can serve as a simple baseline (details below). This is not discussed (or I missed the reference?).\n\nThe current reward contains several terms with different reward weights and hyper-parameters. I would not say this is a simple reward, especially without a concrete comparison to previous work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the helpful feedback and comments. We have revised the paper to include robustness tests, a discussion about kinematic latent space, and failure case analysis. To address your concerns:\n\n---\n\n> **Robustness**\n\nIn Table 1 of the global PDF, we add uniform random noise [-0.01, 0.01] to both task observation $s_t^g$ (positions, object latent codes, etc.) and proprioception $s_t^p$. A similar scale (0.01) of random noise is used in sim-to-real RL to tackle noisy input in real-world humanoids [1]. We see that Omnigrasp is relatively robust to input noise, even though it has not been trained with noisy input. Performance drop is more prominent in the acceleration $E_{\\text{acc}}$ and velocity $E_{\\text{vel}}$ metrics. Adding noise during training can further improve robustness. We do not claim that Omnigrasp is currently ready for real-world deployment, but we believe that a similar system design **plus sim-to-real modifications** (e.g. domain randomization, distilling into a vision-based policy) has such potential.\n\n> **Morphology**\n\nThe AMASS [2] dataset has been successfully used in real-world humanoid control [1, 3, 4]. Retargeting techniques can be used to transfer human motion to the humanoid, even when they do not have the same morphology or joint numbers. Similarly, human hand motion in Mano [6] format can be transferred to robotic hands [7, 8].\n\nOne of the main strengths of Omnigrasp is its independence from paired full-body grasping and object trajectory data, which is expensive to capture and therefore only available in small-scale datasets. Since the retargeting process could introduce errors that cause human-object interaction to be imprecise, retargeted interaction data can be hard to use for methods that rely on demonstrations. Our two-stage process (first obtain motion prior and then learn grasping policy) can leverage imperfectly retargeted motion, since as long as the motion is human-like, we can acquire motor skills by imitating it. \n\n\n> **Baseline Alternative**\n\nExcellent advice! Such a general-purpose kinematic latent space has been used in physics-based control for pose estimation [10] and animation [11], though few have been extended to include dexterous hands. These latent spaces, like HuMoR [9], model motion transition using an encoder $q_\\phi (z_t| x_t, x_{t-1})$ and decoder $p_\\theta(x_t | z_t, x_{t-1})$ where $x_t$ is the pose at time step t and $z_t$ is the latent code. $q_\\phi$ and $p_\\theta$ are trained using supervised learning. The issue with applying such a latent space to simulated humanoid control is twofold:\n\n- The output $x_t$ from the VAE model, while representing natural human motion, does not model the PD-target (action) space required to maintain balance. This is shown in prior art [10, 11], where an additional motion imitator is still needed to actuate the humanoid by imitating $x_t$ instead of using $x_t$ as policy output (PD-target). \n- $q_\\phi$ and $p_\\theta$ are optimized using MoCap data, whose $x_t$ values are computed using ground truth motion and finite difference (for velocities). As a result, $q_\\phi$ and $p_\\theta$ handle noisy humanoid states from simulation poorly. Thus, [10] runs the kinematic latent space in an open-loop auto-regressive fashion without feedback from physics simulation (e.g. using $x_{t-1}$ from the previous time step's output rather than from simulation). The lack of feedback from physics simulation leads to floating and unnatural artifacts [10], and the imitator heavily relies on residual force control [12] to maintain stability. \n\nPULSE directly models the action distribution instead of the kinematic pose and does not need a motion imitator during inference.  Directly learning the latent space without distillation is provided as an ablation in the PULSE paper (Section C.2 Table 6), and it shows that training using RL does not converge to good performance. Random sampling for the variational bottleneck together with random sampling for RL leads to noisy gradients, which hinders policy learning.\n\nWe will add additional discussion to elaborate on these points.\n\n> **Reward Complexity**\n\nWe will clarify this in the text. The “simple reward” here refers to not needing paired full-body-and-hand MoCap data in the reward, which increases complexity. Prior art often involves graph-based [13, 14] or style rewards [5] that depend on paired data.\n\n> **Failure Analysis**\n\nWe categorize the failures into two types: failure due to grasping and due to dropping the object after during transport. The “restabilizing the object in hand” during transport behavior might require further dexterity and training rewards to master. Appendix Section C.3 provides a per-object breakdown of the GRAB-Goal split. We can see that toothpaste and binoculars have the lowest trajectory following success rates (80.9% and 90.5%). In our experience, objects that can roll (such as toothpaste) and large objects (such as binoculars) are more difficult.\n\n**Reference**\n\n> [1] Learning human-to-humanoid real-time whole-body teleoperation  \n[2] AMASS: Archive of motion capture as surface shapes  \n[3] Expressive whole-body control for humanoid robots  \n[4] HumanPlus: Humanoid Shadowing and Imitation from Humans  \n[5] Physically plausible full-body hand-object interaction   synthesis  \n[6]  Embodied hands: Modeling and capturing hands and bodies together  \n[7] Kinematic Motion Retargeting for Contact-Rich Anthropomorphic Manipulations  \n[8]  Task-oriented hand motion retargeting for dexterous manipulation imitation  \n[9] Humor: 3d human motion model for robust pose estimation  \n[10] Learning human dynamics in autonomous driving scenarios  \n[11] Learning Physically Simulated Tennis Skills from Broadcast Videos  \n[12] Residual force control for agile human behavior imitation and extended motion synthesis  \n[13] Simulation and retargeting of complex multi-character interactions  \n[14] Physhoi: Physics-based imitation of dynamic human-object interaction", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oLYYPFk59e", "reviewer": "Reviewer_gXSs", "summary": "The paper introduces a method for controlling a simulated humanoid robot to grasp and move objects along a trajectory with the use of a dex hand, This approach enables the robot to handle diverse objects with diverse trajectories. The key contribution is a humanoid motion representation that enhances training efficiency and human-like manipulation skills and applies it to grasping tasks. The method achieves a reasonable success rate in completing object trajectories and generalizes well to unseen objects.", "questions": "* Considering most of the human body motions and hand motions are randomly paired from different datasets, why not train the model separately (PHC for body motion and another model for hand motion)? What additional benefits does modeling them together provide?", "limitations": "The authors include a reasonable discussion of the limitation and no obvious potential negative societal impact needs to be discussed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* This work extends previous research on PHC with dexterous hand capabilities, enabling whole-body manipulation in simulation with a high success rate. The results, including supplemental videos, demonstrate human-like motion for object grasping and trajectory following.\n* Experiments show that the proposed controller exhibits reasonable robustness and generalization ability across objects of different scales.\n* Extensive ablation studies validate the effectiveness of various components, providing plausible analysis and explanations for comparisons, such as why experiments without object shape still achieve reasonable grasping performance.\n* The paper is well-written, easy to follow, and provides sufficient technical details, including supplemental code, for the community to reproduce the results.", "weaknesses": "* Despite extensive ablation studies, the work lacks some important baselines, such as fully end-to-end RL with similar compute (R1 in ablation), considering that training PHC-X and PULSE-X takes half a month. Baselines utilizing human motion prior data in different ways, such as AMP or ASE, are also missing.\n* More details about the dexterous hand should be included, given its significance over previous work on PHC. Comparisons with experiments using PHC but not PHC-X, such as using PHC for body control and raw action space for the hand, are needed. The ablation study compares with motion prior to training without Dex-AMASS, but it’s unclear how the hand joints are controlled, and more details are necessary.\n* [Minor - Nothing significant]Though the authors claim the work can be extended to real robots in supplementary material, it seems unlikely due to the low simulation frequency (leading to low simulation fidelity), unrealistic robot methodology design, and the use of privileged information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions and feedback. We have revised the paper to provide additional baselines (AMP and PHC), add details about dexterous hands, and discuss decoupled body and hand prior. To address your questions: \n\n---\n\n> **Additional Baselines**\n\nIn Table 2 of the global PDF, Row 2, we train the policy without PULSE-X to collect $10^{10}$ samples for 1 month (Omnigrasp uses $10^{9}$) and observe its performance still significantly lags behind Omnigrasp. Training with only RL (even with AMP) also leads to non-human-like motion (see supplement videos). Row 3 reports training our method with AMP but not PULSE-X, which does not lead to a high success rate. Row 4 reports results from Braun et al. [1], which uses ASE-style latent space **trained specifically for the grasping task** on GRAB data. Compared to Braun et al., we achieve a much higher success rate in terms of grasping and trajectory following and can follow complex trajectories. In comparison to ASE, PULSE-X's latent space is trained on AMASS and covers much broader types of human motion. ASE's discriminative latent space, while effective when learning from specialized and curated datasets, struggles to learn from large-scale unstructured datasets such as AMASS, as shown in [2]. \n\n> **Details about the Dexterous Hand**\n\nWe will add more details about how the hands are controlled in the revised manuscript. The hands are treated the same as the rest of the body (e.g. toe or wrist) and actuated using PD controllers (Section 4.1).\n\nUsing PHC for body control and raw action space for the hand is an excellent suggestion. However, this approach requires we learn a policy to output kinematic poses as \"actions\" for PHC to imitate. As observed in [2], kinematic motion is a poor sampling space for RL due to its high dimension and lack of physical constraints (e.g. a small change in root position can lead to a large jump in motion). Thus, prior art that uses kinematic motion as the action space (e.g. kinematic policy [3, 4]) uses supervised learning to train the motion generator instead of RL. Supervised learning would require paired full-body human grasping data, which is scarce and limited in diversity. One of the main advantages of Omnigrasp is its ability to learn grasping policy **without paired full-body grasping data**, enabling it to scale to many objects and diverse trajectories. \n\nFurther, even if a policy could output ground-truth kinematic pose for PHC, small errors in imitation can lead to the hand missing the objects. To demonstrate this point, we use **ground truth** MoCap as input to a pretrained PHC-X policy (\\~30mm imitation error on average) for grasping, using sequences from GRAB. The result from Table 2 (global PDF) Row 1 indicates that the accuracy of a trained imitator does not support object grasping. To use PHC for the grasping task, we will need to fine-tune PHC with object awareness and pair it with a strong kinematic motion generator. Such an approach has been explored for box loco-manipulation [6] without hands, but it only supports moving boxes for now. \n\n\n> **Training with Dex-AMASS**\n\nWe apologize for the confusion. When comparing training with or without Dex-AMASS for PULSE-X, the only difference is the **training data**. In both cases, the hand joints are controlled using PD controllers to output PD targets, but one with regular AMASS and one with our Dex-AMASS as training data. We will further clarify this. \n\n\n> **Extending to Real Robots**\n\nWe will revise the sentence to \"While the state has access to privileged information and the current humanoid has no real-world counterpart, the overall system design methodology has the potential to be transferred to a real humanoid robot, similar to how the motion imitation task is applied in recent humanoid work [5, 7, 8].\" We believe that the framework of first learning a universal motion representation, then learning grasping policy in simulation, followed by conducting sim-to-real modifications (e.g. domain randomization, distilling into a vision-based policy), **has the potential** to be applied to real-world humanoids. \n\n> **Train Separate Body and Hand Models**\n\nAs mentioned in Section 6, while we utilize a simple yet effective unified motion latent space, separate motion representation for hands and body could lead to further improvements. We completely agree that training a separate model *could* be beneficial. Braun et al [1] use decoupled body and hand prior, and achieve a lower success rate. We hypothesize that since the hand is tethered to the body and performs actions based on different wrist movements (which affects gravity), training a hand-only model that performs well when combined with the body is non-trivial. Since we observe accurate hand-tracking results when training a motion imitator jointly for hands and body, we proceed with the joint latent space design and find it is sufficient for achieving good grasping results. We are actively exploring using decoupled hand and body latent space, though it has not yet shown real benefit. \n\n\n**References**\n\n> [1] Physically plausible full-body hand-object interaction synthesis  \n[2] Universal humanoid motion representations for physics-based control  \n[3] Learning predict-and-simulate policies from unorganized human motion data  \n[4] Dynamics-regulated kinematic policy for egocentric pose estimation  \n[5] Learning human-to-humanoid real-time whole-body teleoperation  \n[6] Hierarchical planning and control for box loco-manipulation  \n[7] Expressive whole-body control for humanoid robots  \n[8] HumanPlus: Humanoid Shadowing and Imitation from Humans", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pysLMfHfY9", "reviewer": "Reviewer_z5bR", "summary": "The paper introduces Omnigrasp, a method for controlling a simulated humanoid with dexterous hands to grasp and manipulate a wide variety of objects along complex trajectories. The authors leverage a universal humanoid motion representation to improve training efficiency and scalability. This method achieves state-of-the-art success rates in object manipulation tasks and generalizes well to unseen objects and trajectories. Key contributions include a dexterous motion representation, the allowance for simple state and reward designs for training, and high success rates in diverse object manipulation scenarios.", "questions": "1. How do you plan to bridge the gap between simulation and real-world applications? Given that the input assumes access to the object pose, velocity, and mesh, which are impossible to obtain in the real world.\n2. How do you envision the way your method could be adapted for, or at least benefit, downstream tasks? E.g., more fine-grained dexterous manipulation.", "limitations": "1. The approach is validated only in a simulated environment, which may not capture all real-world complexities and variabilities.\n2. The method assumes accurate state estimation for object meshes, poses, and velocities, which is challenging to achieve in real-world scenarios.\n3. Fine-grained dexterous manipulation is yet to be achieved by this method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The challenging yet important topic regarding\n\n1. a full-body dexterous grasping, which considers the physical constraints and unstability of the body;\n2. omnidirectional movement after grasping, i.e., moving the object along any directions within a reachable range;\n\nis really worth the exploration. The paper not only presents an effective algorithm that does not heavily rely on existing object trajectories but also conducts experiments on extensive objects to evaluate grasping success rates.", "weaknesses": "While the paper marks a significant advancement in humanoid dexterous grasping, some inherent flaws in the setting hinder its seamless transfer to real-world scenarios. I.e., some state variables, such as the object mesh, pose, and velocity, cannot be accurately accessed in the real world. This discrepancy between simulation and reality poses a challenge for the practical deployment of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments! We have revised the paper to provide more discussion for sim-to-real transfer and downstream tasks. To address your concerns and questions: \n\n---\n\n> **Transfer to Real Humanoid**\n\nWe acknowledge that Omnigrasp in its current form could not be applied to the real world. Its potential for real-world deployment lies in the scalable learning framework (first learn motion representation and then train grasping and object trajectory following). To deploy in the real-world and avoid using privileged information, the popular practice in sim-to-real is to distill a teacher policy into a student policy that does not use privileged information [1, 2, 3]. We imagine an input space similar to [3] (8 point bounding-box) could be used in the real-world for grasping. The mesh information could be replaced with vision or point cloud [1]. Similar to the transfer observed from the motion imitation task to real humanoid motion imitation [4, 5, 6], we believe that a framework similar to Omnigrasp could be deployed to the real world. \n\n> **More Downstream Task Like Fine-grained Manipulation**\n\nWe believe that if we train *specifically* for the object manipulation task for one object, our framework is capable of learning fine-grained manipulation. Our early results indicate that we can *overfit* to fine-grained object movement and handovers using the same motion latent space. This indicates that the motor skills learned in PULSE-X could support fine-grained manipulation. Once we add more diverse objects and trajectories, the learning problem becomes significantly harder and we observe the policy struggle with more precise trajectory tracking. While we focus on scaling the grasping task using simulated humanoids, we believe that tuning the reward for the fine-grained manipulation task on a smaller number of objects can lead to promising results. How to learn a general policy that can do both is an interesting yet challenging future direction. \n\n\n**References**\n\n> [1] DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation  \n[2] Robot parkour learning  \n[3] DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics  \n[4] Expressive whole-body control for humanoid robots  \n[5] HumanPlus: Humanoid Shadowing and Imitation from Humans  \n[6] Learning human-to-humanoid real-time whole-body teleoperation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response

The authors would like to thank the reviewers for their time and constructive feedback. We hope that our responses clarify and address their concerns. We are glad that the reviewers find our work a "significant advance" (z5bR), our results "achieve a high success rate" (gXSs, z5bR), and our motion "impressive, natural, and human-like" (rKeN, gXSs). Here, we briefly address some common questions. 

> **Transfer to the Real-world**

We fully acknowledge that the current system cannot be transferred to the real world without modification. In this work, we focus on using *simulated humanoids* to grasp diverse objects (>1200) and follow diverse trajectories, a capability that has yet to be attained in simulation with humanoids. We hope that we are taking a step towards real humanoid capabilities. The privileged state used in Omnigrasp could be replaced with values easier to access in the real world (point cloud, pose estimation, etc.), and the humanoid we use can be replaced with a humanoid robot [1]. By first learning a policy that has access to all the available information that can be provided, we hope that we can enable new capabilities for simulated humanoids and design systems that can be modified for real-world deployment (via sim-to-real transfer, teacher-student distillation, etc.)


> **Provided Global PDF**

In the global PDF, we provide two tables containing additional experiments and baselines. Table 1 demonstrates the robustness of a pretrained Omnigrasp policy under artificial noisy conditions. While this experiment will not show Omnigrasp's performance when deployed in the real world, it demonstrates the system's potential to undergo sim-to-real modification for deployment. Table 2 provides additional baselines including PHC, AMP, and training with only RL.


**References**

> [1] Learning human-to-humanoid real-time whole-body teleoperation

Meta-review: The manuscript presents a novel pipeline, called Omnigrasp, that is able to learn controllers for humanoids with dexterous hands such that they can manipulate a wide range of objects along complex paths. By employing a universal motion representation for humanoids (called PULSE-X), the authors enhance training efficiency and scalability. Omnigrasp demonstrates state-of-the-art success rates in object manipulation tasks and strong generalization to new objects and trajectories.

The authors' responses were clear, concise, and offered valuable discussions. Overall, their comments reflect a strong grasp of the literature. Additionally, all reviewers had a positive opinion of the work. The authors also discussed about the limitations of their approach, and I expect that the text will be enhanced with the discussions/comments by the reviewers. For those reasons, I recommend acceptance (poster).
Decision: Accept (poster)