HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid
Authors: Xinyu Xu
Yizheng Zhang
Yong-Lu Li
Lei Han
Cewu Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Human-Scene Interaction; Object Rearrangement; Vision-Language-Action Model; Physical Humanoid
Primary area: reinforcement_learning
TL;DR: We introduce HumanVLA, which performs a varity of object rearrangement tasks directed by vision and language by physical humanoid.

Abstract: Physical Human-Scene Interaction (HSI) plays a crucial role in numerous applications. 
    However, existing HSI techniques are limited to specific object dynamics and privileged information, which prevents the development of more comprehensive applications.
    To address this limitation, we introduce HumanVLA for general object rearrangement directed by practical vision and language. 
    A teacher-student framework is utilized to develop HumanVLA.
    A state-based teacher policy is trained first using goal-conditioned reinforcement learning and adversarial motion prior.
    Then, it is distilled into a vision-language-action model via behavior cloning.
    We propose several key insights to facilitate the large-scale learning process.
    To support general object rearrangement by physical humanoid, we introduce a novel Human-in-the-Room dataset encompassing various rearrangement tasks.
    Through extensive experiments and analysis, we demonstrate the effectiveness of our approach.

Human VLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid
Xinyu Xu 12+
Yizheng Zhang**
Yong-Lu Li
Lei Han
Cewu Lult
Shanghai Jiao Tong University
Tencent Robotics X
(xuxinyu2000,, yonglu_li, luceru)ssjtu.
eew.ttu.eddccoo edu. cn Kyizhenzhang, 1xhan]@tencent.com
want to work. Place laptop the desk.
Move the chair front of the desk.
I'm cleaning the room. Please push the small brown table next to the bed.
Figure 1: VlumanVLA performs various object rearrangement tasks directed by the egocentric vision and natural language instructions.
Abstract
Physical Hlman--Scent Interaction (HSI) plays crucial role numerous applica- tions. However, existing HSI techniques are limited to specific object dynamics and privileged information, which prevents the development more comprehensive applications. To address this limitation, we introduce Human VI.A for general object rearrangement directed by practical vision and language. teacher-student framework utilized to develop Human A state-based teacher policy is trained first using goal-conditioned reinforcement learning and adversarial mo- tion prior. Then, it is distilled into assion-language- cloning. We propose several key insights to facilitate the large -scale learning pro- cess. To support general object rearrangement by physical humanoid, introduce a novel Human-in-the- Room dataset encompassing various rearrangement tasks. Through extensive experiments and analysis, demonstrate the effectiveness of the proposed approach.
model via behavior
Introduction
Learning human-scene interaction (HSI) in realistic physical environments is a vital requirement of many applications, including computer graphics, embodied AI, and robotics. In this field, many
Equal contribution. advissin.
advising.
The code is available at htltg://ggthacommm///e////uuaaaa
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

previous efforts have been made to promote expressive humanoid control 36
46
27
56
static
physical scene interaction 43 L 50 33 and manipulating a specific object 47 15 works have achieved great success in synthesizing plausible HSI controls.
These
Nonetheless, significant challenges persist in the realm of more extensive HSI applications and two primary issues need to be solved. Firstly, the current techniques are limited to static objects, such as sitting on chair [50] 33 or specific object dynamics, such as carrying box [15) and throwing a ball 47. However, in a more complicated real-world environment, humans demonstrate exceptional skills in manipulating a diverse range of objects with different geometries, poses, and weights. poses a challenging requirement on the varied dynamics of objects in HSI synthesis. i.c., universal manipulation policy. Secondly, ground -trut object and goal states are necessary to direct humanoid controls in previous works. However, without the help of external localization devices, this privileged information difficult to access in a real-world transfer. It prohibits practical real-worde extensions like humanoid robots and necessitates an easily deployable perception method.
complig
Our work takes a step forward in the above two challenges. We investigate the concept of general- purpose object rearrangement performed by physically interactive humanoid. The whole-body physical humanoid is instructed to carry out daily object loco-manipulations in an indoor room setting. The tasks involve human-like motion controls, interaction with diverse objects, and following desired object dynamics. Moreover, considering the unavailability of privileged information about object and goal states in real-world humanoid applications, we delve into humanoid controls directed by practical vision and language. Compared to privileged states, vision- -anguugg modalities are more accessible and offer new potential for practical applications. It also presents an ultimate vision of the research community on humanoids: human-like agent capable of understanding language instructions, perceiving its environment, and executing daily tasks to assist humans. Fig. [ provides intuitive examples of our work, where the humanoid agent can push a table, carry a laptop, and pull chair, all directed by vision and language. Comparisons of our work with previous studies are available in Tab, II
Our work starts with learning state-based based teacher policy and then distills the policy into a vision- language-action model. In the first stage, we train the policy using goal-conditioned reinforcement learning and adversarial motion priors (AMP) (36), within a generative adversarial imitation learn- ing [16] paradigm. The discrimination reward plus task-conditioned reward encourages humanoids to generate realistic motions and complete the task. However, interacting with diverse objects remains challenging for vanilla AMP. We introduce improved techniques to facilitate general manipulation, in-context navigation, and prioritized task completion. In the second stage, we distill the policy into a student network, named Human V LA, an end-to-end vision-language-action model for physical humanoid. Behavior cloning (2) is used to train the student Human VLA, i.e., cloning the teacher action at each step. challenge of learning VLA models is the poor perception quality of the unconstrained camera pose. We propose novel active rendering technique to improve gaze intention.
To support Human four different room layouts: bedroom, livingroom, kitchen, and warehouse. Each layout is populated with separated, instantiable, and replaceable objects from HSSD 22 assets to create diverse scenes. The humanoid agent is placed in the scene with an instruction to rearrange the room. Statistically, the HITR dataset consists of 50 static objects and 34 movable objects. In our extensive experiments, we train Human VLA in IsaacGym 228 with tasks from HITR. Results demonstrate the effectiveness of our method in generalized object rearrangement and vision- laaguae perception.
we create a novel dataset named Human-in-the-Roon (HITR). consists of
In summary, our contributions include: (1) humanoids. Several advanced techniques are introduced interact with diverse objects. (2) We propose Human the first lision-lnggaaee-cttonn tasks directed by egocentric vision and natural language instruction. (3) We propose the HITR dataset to facilitate research in this field. Comprehensive experiments are conducted in HITR to validate the effectiveness of our method.
We study general object rearrangement by physical
model on physical humanoids to complete
2 Related Works
Motion Synthesis is a long-term research topic in graphics, vision, and robotics. It can be divided
into two streams:
kinematic motion synthesis
43
42
17
14
24
18
5
20
54
19
25
7 and
physics-based motion synthesis [34] 36
35
21
50
27
46
56
51
33
47
26
Kinematic

---

Table I: Comparisons between Human and past works
Object Interaction
Object Dynamics
Language Instruction
# Static Objects
Movable Objects
Methods
Physics
Ego- Vision
NSM 45
25
2
SAMP 14
7
OMOMO 24
19
PADL 2
InterPhys
15
350
1
InterScene
135
57
UniHSI 50
40
HumanVLLA(Ours)
50
34
methods aim at synthesizing visually plausible motions with less penetration, floating, and being semantically faithful. It leverages generative neural networks like VAřs [14 7]. Transformers [ 19).
or Diffusions 24 18, 20] to predict next state. Our work belongs to the physics-based methods, which physical plausibility. It follows control-then-modee paradigm where the control is typically achieved by a learning algorithm, and the model is constrained by physics simulator. DeepMimic [4]] uses reinforcement learning plus imitation learning to track motion references and perform versatile motion controls. NCP [56] advances motion tracking with discrete latent prior. Adversarial Motion Prior (AMP) 36 uses generative adversarial imitation learning to learn natural state transition from unstructured motion data. It is further extended with a reusable controller [35]. high-level language (21), expressive control [27] and latent conditions (46) Recently, there has been an increasing emphasis on the synthesis of interactive motions. InterPhys [T5 uses task-conditioned reward plus stylized adversarial reward perform HSI tasks such sitting, lying, and box carrying. InterScene (33 extends the paradigm to synthesize long-horizon static interactions. UniHSI [50] leverages the vast knowledge of the language model provide unified interface for static HSL. However, previous works are limited to static objects or specific movable objects but fail to interact with various objects. In contrast, our research studies general object rearrangement in daily room, posed with challenges in diverse object geometries, positions, and weights.
have an additional requirement
Room Rearrangement is crucial application of embodied AI, where an instructed agent is placed in room search, navigate, and interact with desired objects. Recent efforts [88
have proposed various platforms and benchmarks to facilitate room rearrangement research. Visual room rearrangement [48 takes the agent transverse both goal and initial scenes recover object states
44
53
23
9
based on visual observations. OVMM (53) presents open-vocabulary pick-and-place manipulation challenges in pursuit of extreme generalization capability. Recent algorithms 44 leverage commonsense knowledge in large language models to plan rearrangements. However, these works are designed for simple embodiments, such as disc-shaped mobility and gripper manipulation. They are limited to moving on smooth terrain and handling only small-sizes objects. In contrast, our work pioneers the design of rearrangement tasks in a complex human- like embodiment. It benefits from bipedal locomotion and stronger interaction motors. For example, our model is capable of carrying 20kg objects, which is beyond the capabilities of traditional stretches.
Vision-L ancuuge-Actioonn (VLA) Model maps practical vasion-language input to generate action controls. It has demonstrated impressive results in the fields of embodied AI and robotics [12 57 Thanks to the robust scalability of the vision and language modalities, VLAs also benefit from large-scale training [57 32] opening up the potential for more general-purpost applications. However, existing VLAs are designed for simple embodiments, such as desktop gripper manipulation. The exploration of VLAs for more complex, high- -igeediinnsional humanoids is still in its early stages. Our work is the first to develop humanoid controls directed by practical vision and language.
32 55
3 Approach
In this section, we introduce the learning process of Human Training Human directly through large-scale reinforcement learning (RL) presents significant challenges, including high- dimensional action space, composite state space, slow rendering speed, and other common issues associated with large-scale RI. To this end, we utilize teacher-student framework to train Human- VLA, which has been validated in applications like dexterous re-orcintation
(8) and grasping [52).
3

---

Environment State (st)
Environment
Waypoint
Next State (St+1)
at
Task goal
Policy
It
Object
rr
Proprioception
Motion Dataset
Motion Discriminator
Train & Inference
Train Only
Figure 2: An overview of learning state-based Human VLA-Teache policy using goal-conditioned
reinforcement learning and adversarial motion prior.
consists of two phases. the first phase (Sec. 3.1). I we leverage goal-conditioned reinforcement
learning and adversarial motion priors (36] train HumanVL.A-Teacher It is presented with the oracle scene state, including precise object pose, geometry, navigation waypoint, and goal coordinate. more practical setting, where the egocentric vision is tasked with perceiving the scene, and natural language instruction is used to specify the goal. In blueprint of real-world humanoid robots, observations used by Human VLA all accessible in real-world deployment. Human VLA is trained via behavior cloning [2] from Human LLA-Teacher, where the pre- pre-triined teacher policy significantly reduces the compute demands of the process.
In the second phase (Sec. B.2). we operate
3.1 State-based Teacher Policy Learning
We train Human -eaehhr with complete knowledge of the scene state enable a variety of object rearrangement tasks. The rearrangement task is formulated as reinforcement learning process. To elaborate, each time step given state 8, and goal g. Human action from policy distribution Tech (0x (ul8t,00 g). The action dę is processed by physics simulator (8++1 [0, 8t) to generate the next state 81+1- The learning objective
Tech predicts an
maximize the accumulated reward R (Tach) = Σ r'r, where is the discount factor and r is the step reward at time L.
To train robust policies that enable humanoids interact with objects and achieve various goals g in life-like manner, it crucial for the humanoids to learn from authentic human motions and generalize across different tasks. To this end, the agent to complete the task and style reward S ($:++1 ) to imitate human motion prior.
goal-conditioned task reward y / $t, St+1) ) to encourage
We employ adversarial motion prior (AMP) [36] to model the style reward, which incorporates an adversarial discriminator D discriminate motions from simulated synthesis or tracked motion dataset. It is trained with the objective:
argmin (sue4.)) [log( (D (( Egg (Sueeee)) Dog(1 D(st.t++ )) D
(I)
uw Egi (serss))
v (())
where d (St:4+=-) and d= (sta+<e) are distributions of t' -frame motion clips from dataset M and policy æ. The first two items Eq. T are discriminate motions while the last item with coefficient woP regularizes the gradient penalty (30 in adversarial training. The style reward S to encourage realistic motion synthesis is then formulated
g (8.111) log(1 D( D(
We uniformly conceptualize goal-conditioned object rearrangement as three processes: locomotion towards the object, contacting the object, and relocating the object to the goal. These steps are accomplished by unified progressively increasing task rewards.

---

Despite the powers of goal- conditioned reinforcement learning and adversarial motion prior, gen- eralized object rearrangement tasks by humanoids still pose significant challenges. Previous works [1 33 50 have been limited to simple tasks such as static sitting, lying, or carrying specific box. We propose new techniques to overcome challenges generalized object rearrange- ment. Generalized object interaction involves geometrically various objects. However, tracking motion data for each individual object is labor-intensive, and infeasible to tackle novel objects. We expect RL to enable automatic object generalization. Thus, we encode object geometry to learn geometry-aware policy and design a carry curriculum to facilitate the learning. Due to the misalign- ment of objects in human motion data and the task, propose style reward clipping to prioritize high-level task execution. Navigating in a complex room requires high-level planning to avoid collisions, we use in-context path planning to enable efficient locomotion. More detailed explanations of our improved techniques are described in the following:
Geometry Encoding. Object state is crucial in HSI synthesis. Previous studies [15 47 primarily encode object position, rotation, and linear and angular velocities to act on certain objects like boxes or balls. A general policy for interactions with diverse objects should incorporate geometric information. Thus, we augment the teacher policy with geometric object representations via Basis Point Set (BPS) 37 encoding. A shared set of basis points ís randomly sampled from unit sphere and encodes object geometry using delta vectors from each basis point to the nearest object point. In contrast to geometries encoded by neural net [38) BPS encoding is computationally efficient and accelerates policy learning. Consequently, we use object geometry, position, rotation, and linear and angular velocities to form comprehensive object observation, thereby facilitating more expressive policy control.
Carry Curriculum Pre-training, Object rearrangement is conceptualized as three-step process in the aforementioned paragraph. However, directly learning the entire three-step rearrangement task from scratch is challenging due to the long task horizon. Besides, physics-based object movement presents greater challenges compared to kinematic object movement [24], primarily because the object state is not directly editable. Instead, it requires indirect control of the physical humanoid to interact. To this end, we draw inspiration from the curriculum learning [4] and design an easy carry curriculum to pre-train the policy. The carry curriculum only includes the first two of three steps: locomotion towards the object and carrying up the object for an in-the-air holding. The carry curriculum has shorter horizon and is empirically easier converge. Furthermore, the pre-trained in-the-air carry prior significantly benefits the subsequent object relocation. For the carry curriculum, use objects excluding those on the ground, such as tables and chairs, which are easier move by pushing and pulling along the ground without a lift. The carry curriculum shares the same learning paradigm with the rearrangement task, except for different two-stage reward design.
Style Reward Clipping. General object rearrangement involves manipulating novel objects that are not recorded in tracked motion data. This creates a misalignment in optimization directions: strictly imitating reference motion or ensuring high-level task execution. Previous work [15) balanced two items by a weighted sum between task reward and style reward in motion-aligned tasks. However, in our general object rearrangement setting, goal-conditioned task exploration progress can be stagnant and the policy may learn actions devoid of task semantics following the logarithmic gradient in Eq. 2 For instance, when the object is difficult to lift, the policy tends to mimic insignificant hand swings in the motion data, rather than exploring carry-up actions. We insert a style reward clipping to prioritize task execution, formulated as follows:
& = max(r (raxxrC(gg 8t, 1)4 Tr = w G (g, 81, 8141) + w (5)+))))
(4)
where w S are coefficients, and is the upper bound for the style reward. This formulation prioritizes goal-conditioned task execution over motion imitation in reward maximization. In addition, we use minimum upper bound Emin. to ensure basic motion imitation during the early stages when the task reward is near zero
In-context Path Planning. Navigating populated room requires high-level knowledge since the dense object cluster may collide with the humanoid agent and obstruct natural locomotion. We use in-context path planning to guide the navigation. Point clouds of all objects in the scene make up the spatial occupancy. We perform top-down point projection and grid discretization to derive a 2D obstacle map of 20cm 20cm grids. We then plan a navigable path from the starting position to the
5

---

Loss
w/o active rendering
Fch
ar
Teacher Policy
Action Decoder
Environment $1
Text Encoder
Image Encoder
w/ active rendering
Proprioception History
Move the pot from the central table the left of the sink the countertop.
Figure 3: Left:: An overview of learning Human VLA by mimicking teacher action and active rendering action. Right: Comparison between w/ and w/o active rendering. Active rendering leads to a more informative perception of human-object relationships.
object, and subsequently to the goal using A algorithm [33, represented as a series navigation waypoints guide locomotion at each step.
Incorporating all the above features, we present an overview of training Human - in Fig. Navigation waypoint, task goal, object state, and humanoid proprioception are sent to the policy network to derive an action. The learning process guided by task reward and motion discrimination reward. Further details about the learning process can be found in the appendix.
3.2 Distilling into Vision-Languagg-Actiin Model
While the Human VLA-Teachee Wech leverages privileged information such object state, goal state, and waypoint in the global coordinates, our biggest goal is towards practical humanoid control free of privileged information and real-world deployable. To this end, we replace privileged states with flexible egocentric vision and natural language instruction. Notably, the proprioception observation the only kept item from Human WLA-Teachee Teacher HumanVLA: which is represented in the local coordinate and can be obtained via forward kinematics and temporal differentiation. The history action also used in observation. To obtain egocentric vision, we mount mobile camera on the head of the human. It renders a 256 x 256 image with field of view spanning 90 degrees at each step. An overview of training Human VLA is illustrated in Fig.
Behavior Cloning, We train Human VLA Tula using a teacher-student framework to distill the via behavior cloning 2 Human employs Efficient/et- B0 45 for image encoding and a frozen bert-base-urcased (TO] for language encoding, whose features, along with proprioception, last action, are sent the action decoder to derive an action, At each time step t, Human --Teechher leverages privileged state St and to derive an action Tech (de(s.. while Human derives an action Teta (a: PP: az-1; Ve, based on proprioception Pt. last action 04-1, egocentric image Ut, and language instruction l. Behavior cloning bridges distributions between Teta (a: Pt. 0t-1, Vt. I) and Tech (a, (t, (t.g), which can be directly implemented via supervised learning. In the empirical training process. we observe severe covariate shift problem offline behavior cloning. Thus, we opt for DAgger 33 framework to train Human VLA which alleviates the problem via online learning.
knowledge from VIAmannVAAA-Teeee
Active Rendering. Though Human VIA-TTachhrr possesses comprehensive knowledge in versatile control, naive policy distillation still suffers from the gap of observation expressiveness, For instance, while egocentric vision used perceive objects, the humanoid gaze might not properly focus on the target object but renders less informative image of the background. Consequently, the perception quality of Human VLA significantly affected by the camera pose. However, an optimal camera pose, determined by the head pose, is not guaranteed in the teacher policy, which only imitates life-like head motion but ignores the vision quality. We propose an active rendering technique to encourage the camera to focus on the object. We analytically calculate the seaddto--bjeect direction in the global coordinate and then derive a head orientation. Inverse kinematics is used to obtain active rendering actions at" for the neck joint. It is used to derive mixed supervision op in conjunction with the teacher action ofch formulated as
ula 0,
(1 w )ar Ach + w at aaa
(5)

---

Table 2: Results in box rearrangement. it denotes our implementation.
Success Rate (%) 1
Precision (cm)
Execution Time (s) 4
InterPhys 15
94.3
8.3
9.1
InterPhys
15
97.8
12.6
5.3
Human ALA-nnLLATTTtttte Teacher
98.1
4.2
4.6
where war is the coefficient for active rendering. Notably, this is only applied to the neck joint, while other joints only follow the teacher action.
4 Human-in-the-Room Dataset
for object rearrangement are primarily designed for stretches with disc- shaped mobility and gripper manipulation. Human-like embodiment has different physical attributes. such as stronger motors to handle large furniture like chairs and tables. Besides, we follow [36 15 33
Existing datasets 48 53
50 to use humanoid model with spherical hands, which can struggle with manipulating small-sizen objects, such picking up towel. To address these issues, introduce novel Human-in-the Room (HITR) dataset, designed to facilitate Assonnlanguage directed object rearrangement tasks on humanoid. The HITR dataset includes carefully designed objects of various sizes, ranging from 2 21 to 126cm, and provides a variety of rearrangement tasks in various rooms. Each task involves separated, instantiable, and replaceable objects with defined initial and goal states. Additionally, each task is accompanied by a natural language instruction generated by a Large Language Model (LLM).
In constructing the HITR dataset, we reference common objects used in room designs from 23 44 53 and utilize object models from HSSD [22] to create basic assets. Object assets are manually resized to ensure the interaction friendliness. We adopt the procedural generation pipeline from 9 to generate diverse scenes. First, we manually design four room layouts: bedroom, livingroom, kitchen, and warehouse, then randomly populate replaceable objects within these layout templates to establish the scene, as well as the goal state, Next, we randomly relocate an object in the scene, either to the ground another receptacle. This relocated object is what the physical humanoid is tasked to rearrange. We concatenate two rendered images of the initial and goal scenes and use the composite image to prompt gpt-4-vision [31] generate instruction. The LLM is asked to distinguish between two states and provide an instruction to guide the state transition. However, the LLM still struggles with understanding complex spatial relationships, such left-right errors. To ensure the quality of instructions, we manually review and revise them as necessary. Ultimately, we build the HITR dataset of 615 tasks, with average of 6.5 objects per task. There are 50 static objects like bed and countertop, as well as 34 movable objects like pillow and vase. More details are in the appendix.
5 Experiments
5.1 Settings
Datasets. Our experiments are conducted on the HITR dataset. is split into train and test subsets at ratio of 9:1, containing 552 and 63 tasks respectively. The test subset is used to evaluate the generalizability of our method unseen tasks. For the motion dataset used training, we utilize OMOMO 24 and a locomotion subset from SAMP [14]. OMOMO provides variety of short-range motions involving moving different objects, while locomotion motions from SAMP enhance the locomotion aspect of dataset. We use 30-minute motions in total. The source motion dataset features object rearrangement involving only seven different objects, which is far less than those in the HITR dataset. Despite this, we anticipate that our method can generalize to different objects.
10-second running time limit and follow [15] evaluate methods with three metrics. (1) Success Rate: the proportion of tasks that are successfully rearranged within an error margin of 0. (2) Precision: the distance of the final object position to the goal. (3) Execution Time the average time taken to complete run. For the Success Rate, higher value indicates better performance,
Metrics. We adhere
for Precision and Execution Time, the lower the better. All experiments evaluated using 10 repeat runs. For baste methods, we follow [15] to set 0 20cm.
7

---

Table 3: Ablation study.
Privileged State
Success Rate (%)
Precision (em))
Execution Time (8)
Human VLA- geumetry encoding
85,9
14,4
4.5
64.5
43.4
5.5
w/o carry curriculum
66.3
73.4
5.3
w/o style clipping path planning
79.9
27.5
4,3
67.8
37.2
5.5
HumanVLA
x
74.8
42.6
5.1
w/o active rendering
X
67.9
55.6
5.6
w/o online learning
X
15.3
145.0
8.2
Table 4: Results in unseen tasks.
Privileged State
Success Rate (%) f
Precision (cm))
Execution Time (s) J
InterPhys 15
Human Teacher
79.3
19.3
4.6
Offine GC-BC 29
X
10.2
152.3
8.5
Human
X
60.2
57.0
5.8
w/o active rendering
X
56.7
65.5
5.9
For Aision-language based methods, where the goal is specified via coarse instructions rather than precision goal coordinates, we set 40cm. This criteria relaxation is also adopted in past works (8] for evaluating policies with different observations.
5.2
Implementation Details
We conduct experiments in parallel environments simulated using IsaacGym [28], with neural networks implemented via PyTorcc. Our physical humanoid model, following previous works [15
33 50), comprises 15 rigid bodies and 28 joints, each actuated by a PD-Controller. The simulator runs at 60Hz, and the policy is queried at 30Hz. The teacher policy is optimized using Proximal Policy Optimization [40] and takes two days on eight Tesla V100 GPUs to converge. The student policy trained using DAgger 39] and takes one day on two GPUs. We provide comprehensive details about hyperparameters,, neural architectures, observation space, and more in the appendix.
5.3 Comparisons in Box Loco-Manipulation
While our work pioneering in the exploration of vision-lanuugge-direeeed general object rear- rangement, direct comparisons with previous studies are difficult. The work most similar to ours InterPhys [5), which delved into state-based box loco-manipulation. Due to the unavailability of training data, motions, and codes of InterPhys [15], instead refer box rearrangement subset in HITR to conduct experiments. We train a state-based Human box rearrangement tasks, along with implementation of the InterPhys baseline. Results of box rearrangement are reported in Tab. 2 Given that the box is the simplest object to interact with, both methods achieve high success rates. However, 4.2 cm, outperforming both the 8.3 cm in the official report [5] and the 12.6 cm our implementation. We use the standard deviation evaluate the statistical significance of Human VLA-Teacher in 10 repeated runs. The values are 0.02, 0.004, and 0.04 for Success Rate, Precision, and Execution Time respectively. With high task completion rates and low variance, demonstrate the effectiveness and robustness of our method in this first trial.
using only
method exhibits superior precision, with result of
5.4 Ablation Study
We conduct comprehensive ablation studies on the train split to validate each design choice, with results presented in Tab. 3 Firstly, we evaluate the impact of improved techniques in HlumanVLA, Teacher training, which achieves a success rate 85.9% and a precision eliminating any component leads to decline in performance. The inclusion of geometry encoding and carry curriculum enables the model to manipulate a variety of objects effectively. Without either of these components, the success rate experiences drop of approximately 20%. Style reward
14.4cm. However,

---

Instruction: Move the blue pot to the left of the sink.
Instruction:
Move the blue white vase to the center table.
Figure 4: Qualitative results. The color transitions from green to yellow as the task progresses.
clipping prioritizes task execution, whose absence results in a 6% decrease in the success rate. Path planning helps humans navigate complex scenes. Its removal leads to significant 18.5% decrease in the success rate. Subsequently, we validate design choices in training Human directed by vision and language. The default Human VLA achieves a success rate of 74.8% with precision of 42.66 6cm. However, the absence of active rendering results a substantial 6.9% success rate drop, emphasizing the importance of perception quality. We implement offline behavior cloning baseline using ten off-the-shell teacher trajectories per task for training. Without online learning, the system suffers from severe covariate shift and performs poorly.
5.5 Generalizing to Unseen Tasks
We use the test split of HITR to evaluate the generalizability of methods. The unseen data, which includes novel scene compositions, visual appearances, and language instructions, poses significant challenge to our method. Results are presented in Tab. 4 The state-based Human tends be more robust in unseen data. Relatively small drops in success rate (6.6%) and precision (4.9cm) demonstrate strong generalizability of RL when using privileged information. Moreover, consistently outperforms the InterPhys [15]] baseline all metrics. Applying Human VLA unseen tasks turns out to be more challenging due to the complexity of vision and language modalities. The success rate of Human VLA decreases to 60.2%, and the precision drops to 57.0cm. However, Human VLA still consistently outperforms baselines without active rendering and the offline goal- conditioned behavior-cloning (Offine GC-BC) 29 method.
5.6 Qualitative Results
We provide qualitative visualizations of how Human performs object rearrangement tasks in Fig. More results are available in the appendix. We demonstrate that Human is capable of moving pot, vase, chair, and box based on language instructions.

---

Conclusion
We investigate vision-language directed object rearrangement by physical humanoids in this work, fundamental technique for HSI synthesis and real-world humanoid robots. Our system is developed using a sachee--ttudet distillation framework. We propose key insights to facilitate teacher policy learning with privileged states and introduce novel active perception technique to favor vision- language-action model learning. We present a novel HITR dataset to support our task. In extensive experiments, our HlumanVLA model demonstrates superior results in both quantitative and qualitative evaluations. Future works include dexterous manipulation by physical humanoids and long-horizon multi-object interaction.
Acknowledgments
This work was supported by the National Key Research and Development Project of China (No. 20222D0160102), National Key Research and Development Project of China (No. 2021ZD01 10704), Shanghai Artificial Intelligence Laboratory, and XPLORER PRIZE grants.
References
[1] Joao Pedro Araujo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Deepak Gopinath, Jiajun Wu, Alexander Clegg, and C. Karen Liu. Cirele: Capture in rich contextual environments, 2023.
[2] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 109-129, 1995.
[3] Suneel Belkhale, Tianli Ding, Ted Xiao, Pierre Sermanet, Quon Vuong, Jonathan Tompson, Yevgen Chebotar, Debidatta Dwibedi, and Dorsa Sadigh. Rt-h: Action hicrarchics using language, 2024.
[4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th annual international conference machine learning, pages 41-48, 2009.
[5] Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2022.
[6] Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Physically plausible full-body hand-object interaction synthesis. arXiv preprint arXiv:2309.07907, 2023.
[7] Yujun Cai, Yiwei Wang, Yiheng Zhu, Tat-Jen Cham, Jianfei Cai, Junsong Yuan, Jun Liu, Chuanxia Zheng, Sijie Yan, Henghui Ding, al. unified human motion synthesis model via conditional variational auto-encoder. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 11645-11655, 2021.
[8] Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation, 2021.
[9] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982-5994, 2022.
[10] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirec- tional transformers for language understanding, 2019.
[11] Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for object rearrangement. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2086-2092. IEEE, 2023.
[12] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi. Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet. Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-c: An embodied multimodal language model. In arXiv preprint arXiv:2303.03377 2023.
[13] Peter E Hart, Nils J Nilsson, and Bertram Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions Systems Science and Cybermetics, 4(2):100-10 107, 1968.
10

---

[14] Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In Proceedings of the International Conference Computer Vision 2021, October 2021.
[15] Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthe- sizing physical character-scene interactions. 2023.
[16] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, 2016.
[17] Daniel Holden, Taku Komura, and Jun Saito. functionnd neural networks for character control. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
[18] Siyuan Huang, Zan Wang, Puhao Li. Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion- based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[19] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. Advances in Neural Information Processing Systems, 36, 2024.
[20] Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, 7an Wang. Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. arXiv preprint arXiv:2403. 2024.
[21] Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Padl: Language-directed physics-hased character control. SIGGRAPH Asia 2022 Conference Papers, SA '22 New York, NY, USA, 2022. Association for Computing Machinery.
[22] Mukul Khanna* Yongsen Mao Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexan- der Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat Synthetic Scenes Dataset (HSSD-200)): An Analysis of 3D Scene Scale and Realism Tradeoffss for ObjectGoal Navigation. arXiv preprint, 2023.
[23] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d enviromment for visual ai. arXiv preprint arXiv:/712.054774 2017
[24] Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Trans. Graph., 42((), 2023.
[25] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: large-scale 3d expressive whole-body human motion dataset. Advances Neural Information Processing Systems, 2023.
[26] Hung Yu Ling, Fabio Zinno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. ACM Transactions on Graphics, 39(4), August 2020.
[27] Zhengyi Luo, Jinkun Cao, Alexander W. Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars. In International Conference on Computer Vision (ICCV), 2023.
[28] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021.
[29] Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang. Rohun Kulkarni, Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. 2021.
[30] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge', 2018
[31] OpenAI. Gpt-4 technical report, 2024.
[32] Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Authony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0888. 2023.
[33] Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Haofan Wang. Xu Tang. and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes. In International Conference on 3D Vision (3DV), 2024.
11

---

[34] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph., 3744::43117 July 2018.
[35] Xue Bin Peng. Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Trans. Graph., 41(4), July 2022.
[36] Xue Bin Peng. Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Trans. Graph., 40(4), July 2021.
[37] Sergey Prokudin, Christoph Lassner, and Javier Romero. Efficient learning on point clouds with basis point sets, 2019.
[38] Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation, 2017.
[39] Stephane Ross. Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. 2011.
[40] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[41] Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Picter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv Preprint arxiv:2403. 10506, 2024.
[42] Sebastian Starke, Ian Mason, and Taku Komura. Deepphase: Periodic autoencoders for learning motion phase manifolds, ACM Transactions on Graphics (TOG), 41(4);1-13, 2022.
[43] Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. ACM Trans. Graph., 38(6), nov 2019.
[44] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, al. Habitat 2.0: Training home assistants to rearrange their habitat, Advances in neural information processing systems, 34:251- 266. 2021.
[45] Mingxing Tan and Quoe V. Lc. Efficientmet: Rethinking model scaling for convolutional neural networks, 2020.
[46] Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Condi- tional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, New York, NY, USA, 2023. Association for Computing Machinery.
[47] Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics- imitation of dynamic human-object interaction, 2023.
[48] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrangement. In IEFF/CV Conference Computer Vision and Pattern Recognition (CVPR), June 2021.
[49] Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song. Jeannette Bohg, Szymon Rusinkiewicz, and Thomas Funkhouser. Tidybot: Personalized robot assistance with large language models. Autonomous Robots, 47(8))1087 1102, 2023.
[50] Zeqi Xiao, Tai Wang, Jingho Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-comtacts. In The Twelfth International Conference on Learning Representations, 2024.
[51] Zhaoming Xie. Jonathan Tseng, Sebastian Starke, Michiel van de Panne, and C Karen Liu. Hierarchical planning and control for box loco- manipulation. Proceedings of the ACM on Computer Graphics and Interactive Technigues, 6(3):1-18, 2023.
[52] Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, Tengyu Liu, Li Yi. and He Wang. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy, 2023.
[53] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang. Mukul Khanna, Theophile Gervet, Tsung- Yen Yang. Vidhi Jain, Alexander William Clegg, John Turner, et al. Homerobot: Open- vocabulary mobile manipulation. arXiv preprint arXiv:2306. 11565, 2023.
12

---

[54] Kaifeng Zhao, Yan Zhang, Shaofei Wang. Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In International conference on computer vision (ICCV), 2023.
[55] Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-via: 3d saion-languageactiinn generative world model. arXiv preprint arXiv22003.09631 2024.
[56] Qingxu Zhu, He Zhang, Mengting Lan, and Lei Han. Neural categorical priors for physics-based character control. ACM Trans. Graph., 42(6), dec 2023.
[57] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, al. Rt-2: lasoonlanquuge-actton models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165-2183, PMLR, 2023.
13

---

Template Layout
Goal State
Initial State
Spawn object
Relocate object
Spawn human
Concatenated Image
gpt-4-piston
Task
Prompt
You robot in nill give concatenated about the initial geal of the The Left the initial while objact rearnanged. position object rearrangesent. should spatial. faithful given object. give you image, noving object 'Laptop".
Instruction: Nave the laptop to the desk
different positions going
object
Figure 5: The task generation process of HITR dataset.
A Limitations
This work inherits the humanoid model from 36
with spherical hands. is hard to manipulate small-sized objects. Dexterous hands can be equipped to facilitate object manipulation in future works.
35 15
50
As the first work on general object rearrangement, our task settings only include one object movement each time. Long-horizon object rearrangement left for future work.
The current version of our system does not contain explicit memorizing, planning, navigation, and multi-agent collaboration modules. We leave more ad-hoc designs to future work.
B Broader Impacts
We study simulated physical humans in the work, whose technique holds the potential for extension to real-world humanoid robots. This could have a significant positive societal impact, as humanoid robots have the potential to assist humanity in various ways. However, it is crucial carefully consider safety concerns associated with the use of humanoid robots.
C Licenses
We use assets from ASE [36]. HSSD 222 OMOMO (24, and SAMP [4] in this work. ASE is
released under the NVIDIA license. HSSD is released under the CC BY-NC 4.0 license. OMOMO does not have specified license. SAMP is released with its license on its GitHub repository]
D Dataset
D.1 Generation Process
Figure 5 provides an illustration of how we construct the Human-in-the-Room (HITR) dataset, following the procedural generation pipeline (9]. First, we manually design four distinct room layouts: bedroom, livingroom, kitchen, and warehouse. These template layouts subsequently populated with various object models to create set of diverse scenes, which serve as the goal states
14

---

Figure 6: Different rooms in HITR dataset.
Figure 7: Movable objects in HITR dataset.
for the rearrangement tasks. To generate the initial state of each task, randomly relocate object to different receptacle or to the ground. The initial position of the humanoid randomly sampled from the navigable areas. As for the initial orientation, the humanoid heads the object. Our dataset guarantees the object visibility in the first view and also covers the target position visibility in 89% of tasks. Notably, our dataset does not specify the initial humanoid pose; instead, is sampled from the training motion dataset. We then concatenate images of the goal and initial states to create self-contained information carrier for each task. It is sent large language model, specifically gpt-4-vision, along with language prompts, to generate the corresponding instruction.
D.2 Statistics
The HITR dataset contains 615 unique tasks in various rooms, examples of which are depicted in
Fig. 6 The dataset includes 34 movable objects and 50 static objects, all of which are sourced from HSSD [22]. All movable objects are shown Fig. and span a wide range of categories such as chair, pillow, plant, coffeemaker, among others. To facilitate successful interaction with our humanoid
15

---

model, we manually adjust the scale of object models and assign suitable weights. The sizes of the objects vary from 21em 126cm, and their weights range from 5kg 20kg. On average, there are 6.5 objects present in each scene.
E
Details of the Approach
We describe the complete details of our approach in this section.
E.1
Training Human VLA-Teacher
E.1.1 Observation Space
The observation space HumanVLL- --Tehcheer includes proprioception, object, goal, and waypoint.
The 223- 223-dimensional proprioception includes:
root height E R
root rotation E R
root linear velocity E R3
root angular velocity R
link position E R11x3
link rotation E R
link linear velocity E R44x3
link angular velocity R 14xx
The object state includes object position (R"), rotation (R²), linear velocity (R3), angular velocity (R2) and BPS [37] geometry 3) encoded by delta vectors of 200 basis points.
The goal state includes the goal position (R3) and rotation (R%).
The waypoint is denoted by uP E R3
We follow the default AMP 36 to use a projected observation space for the discriminator, They
include:
root height R!
root rotation R
root linear velocity E R3
root angular velocity E R3
joint rotation R 12x6
joint velocity E 28
end-effector positions of left/right hand/foot, and head R5x
object position E R
We send 10 consecutive frames to the discriminator; thus the total dimension is R 10x 131
All these features represented in the local coordinate of the humanoid model. Rotations encoded using a 6-D normal-tangent representation.
E.1.2
Action Space
The action space (R28) of Human A-ATeechhrr consists of the target positions for 28 Proportional- Derivative controllers. Predicted actions are then utilized by the controllers to generate joint torques for effective control.
E.1.3 Network Architecture
We adopt MLPs as the basic networks for Human VLA-eeaheer. Each linear layer is followed by ReLU activation. The 600-dimensional BPS feature compressed using an MLP of 5512,512,128 layers to generate low-dimensional representation. This representation then concatenated with all other observations to derive the action and value. The actor, critic, and discriminator networks are all separate MLPs with hidden layers of [1024, 1024, 512]. The actor and critic networks trained using default PPO
40 losses. The discriminator trained using cross-entropy loss via adversarial
learning.
E.1.4 In-context Path Planning
Navigating through intricate scenes is challenging due to the potential for unexpected object collisions. To mitigate this, we employ in-context path planning to facilitate collision-free locomotion, as depicted in Fig. 8 This process involves two substreams: planning path from the humanoid to the object, and then from the object to the goal. Initially, we sample point clouds from all objects the scene. These points are then projected top-down onto the ground and divided into 50cm x 50cm grids to construct
16

---

Aggregate
Point Projection Discretization
A Planning
Aggregate
Figure 8: marks denote the path from the initial position to the object. Green marks denote the path from the object the goal.
An overview of the path planning process. The blue mark denotes the initial position. Red
navigation map. We utilize the A algorithm 13 to plan two paths: one from the humanoid to the object, and another from the object to the goal. Each path generated by the A algorithm is represented by series of densely packed waypoints. We consolidate waypoints that share consistent moving direction to create a sparse waypoint set. During task execution, the humanoid model is guided by sequence of these waypoints and is encouraged to move toward the waypoint at each step. Once the humanoid reaches the waypoint within 50cm distance, the waypoint proceeds to next. The last waypoint is the goal position. The waypoint is used in reward computation to guide the movement, described in the following sections.
E.1.5 Carry Curriculum Pre-training
We uniformly conceptualize object rearrangement as a three-step process: locomotion towards the object, making contact with the object, and then relocating the object to achieve the goal. However, directly training these three processes from scratch can exceptionally challenging. The main difficulty lies in the fact that physical interactions with objects necessitate contacting prior, such as robust object holding, to enable subsequent object dynamics. Through empirical experiments, we have found that carrying objects to other receptacles is significantly more difficult than pushing and pulling objects on the ground. This underscores the need learn a carry prior to boosting the subsequent object relocation.
Inspired by the curriculum learning paradigm [4, we design a carry curriculum pre-training scheme to learn the carry prior. It encompasses the initial two steps, namely locomotion and contacting, which form an casier curriculum compared to the difficult three-step rearrangement task. We exclude on-ground objects in carry curriculum pre-training, whose initial height and goal height are both smaller than 1m. These objects, such small table, chair, can be relocated by direct on-ground pushing and pulling without a lift. The goal of the carry curriculum is to enable the agent to walk towards the object and establish a robust contact, allowing to securely hold the object in the air; thus, we define reward combination of walking reward and contacting reward:
Pt walk contoct
where wealk encourages the humanoid to walk to the object. Specifically, it encourages a closer distance between root position x; and object position xt as well walking to the waypoint direction x7
up at target speed u*=1.5m/s:
.aaakk rt
1 + 0.2 0.2 exxeexp2221) ot |x obj
root
xx up
|| o
x root
0.5
0.3
otherwise
17

---

Table 5: Hyperparameters for
training.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
Num. envs
16,384
Max episode length
300
Num. epochs Observation clip
30,000
Discount factor
0.99
GAE parameter
0.95
5
Action clip
Optimizer
Adam
Learning rale
3e-5
Actor loss weight
Critic loss weight
5
Discriminator loss weight
Num. rollouts per PPO update
32
PPO clip
0.2
PPO miniepochs
Num, batches per miniepoch
AMP consecutive frames 1"
10
Gradient penalty way
Task reward weight of
Style weight 8
Min style clipping bound Eniin
contact encourages the humanoid hands arhand to contact with the object si followed by lifting ?
the object up for Ah = 0.3m from an initial height obj is formulated as:
T contact 0.2 112eexp(110pp obj
o hand + 0.5 clip(h, - - 0, Ah)/Ah
(8)
E.1.6 Rearrangement Learning
The complete rearrangement task consists of the whole three-step process. The reward is formulated as combination of walking reward, contacting reward, and relocation reward:
Ft rt woalk
rt contactt ri relocation
where ryaat r.ontact have similar formulation but different weights and conditions compared to the carry curriculum pre-training, specifically:
waalk
0.1 0.1 |2;²) 0.2
x x
I]) + +.11exp(-211e oooot x4
122)
|| obj
Food|
>> 0.5
rt
otherwise
(10)
contact
(0.1 1000
xt ahandl)) + 0.1 clip(h; b h 0, Ah
|| obj
x gont
>> 0.5
0.2
otherwise
(11)
In the context of the rearrangement task, we do not set explicit lifting condition, but instead require the contact prior to enhance subsequent dynamics. As a result, the height change, Ah is reduced to 0m for objects that going to be carried, and to zero for objects that consistently the ground.
pelloction comprises four components, namely rt ri .rpar, and rf rp encourages moving f
the object to the next waypoint " far encourages a close distance to the next waypoint. reear
encourages meet the goal position gooal rf encourages meet the rotation position Ggoat. They are formulated as:
rt relocation O..r + 0.2r
0.2r + 001rfoo
(12)
eexp(-2|lv" phhi u0 112)
||=2 goall >> 0.5 otherwise
(13)
faar = exp( - -1100 "P [])
(14)
near exp( (5 Iggoal II)
(15)
y = exp( - 2199
[bo ggoai Il)
(16)
E.1.7
Hyperparameter Setting
We provide hyperparameter table for Human VLA-VLAATe training in Tab. 5 it shared for both the carry curriculum pre-training and rearrangement learning.
E.2
Training Human
E.2.1
Observation Space
The observation space of Human VLA includes proprioception, last action, egocentric image, and language instruction.
18

---

Table 6: Hyperparameters for HumanVLA training.
Hyperparameter
Value
Hyperparameter
Value
Hyperparameter
Value
Num. envs
585
Max episode length
300
Num. epochs
20,000
Observation clip
5
Action clip
I
Optimizer
Adam
Learning rate
5c-4
Num. rollouts per epoch
Num. train steps per epoch
Batch size
DAgger Ao
DAgger Active rendering w
0.998
Camera resolution
256,256
Camera FoV
90
0.2
The proprioception adheres to the 223-dimensional feature defined in the Human VLA-TTechhe (Sec. F.I.T.
The last action is denoted by a E R228 An all-zero feature is used for it at the first step.
The egocentric image has 256 256 pixels with field of view spanning 90 degrees. We mount camera on the head of the humanoid model, with the camera position offset from the head being [0.103, 0, 0. 0.175]. The camera direction aligns with the forward direction of the head.
The natural language instruction specifies the task goal.
E.2.2
Action Space
The action space for Human VLA aligns with the 28-dimensional ClumaVLLA L action space defined in Sec. E.2
E.2.3 Network Architecture
The HumanVLa network consists of an image encoder, a text encoder, and an action decoder. The image encoder is an EfficientNet-B0 45] while the text encoder frozen bert-base-uncased [[O] Our primary design choices for these two models are their fast inference speed. The 1280-dimensional EfficientNet- BO feature is passed through a linear layer to yield compressed 128- 228-dimensional feature. An MLP with [512, 122] hidden layers is used to compress the 768 di8-dimensional bert-base-une unceeed feature down 128 dimensions. The compressed image and text features are then concatenated with the proprioception and last action, and this combined feature is sent into the action decoder. The action decoder is 6-layer MIP, with each hidden layer being 1024-dimensional Each linear layer followed by BatchNorm and ReLUU Skip connections are used between the first and third layers, as well as between the third and fifth layers.
E.2.4 Active Rendering Action
We propose the active rendering technique to enhance the quality of visual perception and facilitate object rearrangement. In this section, we provide more details on how to compute active rendering action, specifically applied to the neck adjust the camera pose. The primary regulation focuses on the camera's 's target view, which is the forward direction. To determine this, we compute the center of the object point cloud and use it the target viewpoint. The direction from the camera position to the point cloud center represents the expected forward direction of both the camera and the head. lead ambiguous actions.
Since the neck joint is 3-DoF spherical joint, single regulation To address this, we introduce the second regulation that controls the side direction of the head. is perpendieular the plane formed by the upward torso and the camera view. These two regulations result in unique head rotation. Finally, inverse kinematics used to solve neck actions.
E.2.5 Learning Process
We train Human VLA by cloning actions from MeeimannVLA-Teachcee At each step, the Human VLA- Teacher predicts an action using privileged information. It is mixed with an active rendering action to yield the label for supervision. Human VLA is optimized to minimize the mean square error.
We adopt a DAgger
39 framework to manage the online learning process. DAgger iteratively
schedules a mixed policy Ach (1 s at epoch to explore the environment and expand training dataset. can alleviate the covariate shift problem between different policies. We
an exponential function to schedule the mixing factor Br 30 X.
19

---

6.0
Actionn Reredteing
10
MAD Action Rendering
Beto
Arap bym Mm
045
0.8
0.00
100
di sipssing Style Reward clipping
0.40
1003
2000
4000
80000
10000
Figure 9: Learning curve comparison w/ and w/o style reward clipping.
Figure 10: Learning curve comparison w/ and w/o active rendering. The process is dominated by the teacher policy in the early stage with high 3 and demonstrates the reward upper bound.
Figure 1I:
Comparison w/ and w/o path planning. The green humanoid without path guidance fails to get close to the sofa, while the yellow humanoid with path guidance learns to go around the central table. Instruction: Move the pillow to the sofa.
E.2.6 Hyperparameter Setting
We provide hyperparameter table for Human training in Tab. 6
Additional Results
F.1
Generalization
The primary test set of HITR contains task-level unseen tasks including new compositions of objects in the scene, new placement of objects, and regenerated new text instructions from LLM describing new compositions and new spatial relations. We have proven the generalization ability of Human VLA in unseen tasks.
However, generalization in any unseen without any similar pattern in the seen data is super challenging, which also an ultimate goal of embodied AI research. We build extra tiny unseen data to make additional analysis and further disclose our method. We make additional testing data: (1) Unscen texts
20

---

Table 7: Unseen data analysis.
Success Rate (%) f
Precision (cm)
Execution Time (s)
Useen Text
65
50.4
5.4
Unseen object (visual)
50
72.3
6.2
Unseen object (geometry)
20
118.8
7.9
Unseen scene layout
35
88.5
6.8
Instruction: Move the red chair in front of the table.
Instruction: Put the box on the floor in the bottom shelf on the left.
Figure 12: Additional qualitative results.
generated for training tasks, manually reviewed to be distinct from training data. (2) Unseen objects by changing visual appearance in training tasks. (3) Unseen object category (cup) with different geometry. (4) Unseen scene layouts by repositioning static large objects.
Results are reported in Tab. We find that our work suffers less from unseen texts and unseen visual appearance. But generalizing unseen object categories and execution in the unseen scenes remains main challenge.
F.2 Learning Curves
We plot the learning curves in Fig. and Fig. 10 our method has been proven to converge faster in task completion with style reward clipping. In Fig. 10
demonstrate the efficacy of our method. In Fig. 9
active rendering improves perception quality and facilitates the learning of the student policy.
F.3
Qualitative Ablation
qualitative ablation about the path planning module represented in Fig. Specifically. the green humanoid fails navigate close to the goal receptacle, i.e., the sofa. However, the yellow humanoid guided by planned waypoints to go around the center table and place the pillow on the sofa.
F.4
Additional Qualitative Results
We provide additional qualitative visualizations in Fig, 12 to disclose results.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This work studies vision-language directed object rearrangement. It have been described the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide complete details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
Data and code are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ Cataaae//CoSSSSeeeeeeeee
wwt.Subbmiiiioooiiioooooooio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide complete details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance is reported Sec. 5.3. We report the standard deviation in 10 repeated runs to validate the robustness of our method. For consistent result format with InterPhys, they are included the main table.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute costs are in Sec. 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper conforms NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited works of related assets. License mentioned in the appendix.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiment.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing experiment.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "1ccS00lKNR", "reviewer": "Reviewer_2Cr4", "summary": "This paper proposes HumanVLA, a framework for training humanoid controllers powered by vision and language. First, a teacher control policy is trained to control a simulated humanoid to carry objects to specific positions. Then, this policy is distilled into a vision-language-action model that uses vision to guide the movement of the humanoid and objects. A few techniques such as active rendering, reward balancing, etc., are proposed to improve performance; a dataset containing sequences of object-carry locomotion is also proposed.", "questions": "How does the agent plan its path around a new room if it can is encouraged to always look at the object? The environment-awareness and target-location awareness of the agent is not properly addressed.", "limitations": "Limitations on the language instructions side is not adequately discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper paints a very promising picture for simulated humanoid control and vision-language models. The formulation (vision + language + proprioception) provides an enticing research direction for embodied AI.\n- The proposed solution (teacher-student) is a well-tested formula for humanoid control and is performing well in the proposed task. The vision element has not been explored much for humanoid control due to its high dimensionality but is incorporated in the current pipeline.\n    - The proposed curriculum learning, reward clipping, etc., while small innovations, contribute to the overall performance of the pipeline.\n- Experiments show that the model achieves a high success rate compared to previous SOTA (InterPhys) in box arrangement when given privileged information. The success rate for the vision-language-action model is also promising.", "weaknesses": "- The major weakness of this work is qualitative evaluation. Only four video demos are shown without any information about the text prompt or egocentric rendering. Thus, there are limited ways to know how well the proposed VLA method performs in terms of generalization.\n    - Along this line, since the evaluation dataset is provided by the authors, the diversity of the text prompts & tasks is unknown and not demonstrated.\n- The VLA part of the proposed method is relatively weak and understudied. Given only BERT text encodings, it is very hard to imagine that the MLP-based agent could complete complex tasks without any guidance. How does the agent know where the target location is? Does the agent start with a scanning phase where it locates where the target position is? For such a low error in location placement (~20cm) and such a coarse language instruction, the only way the MLP-based agent could succeed is by memorizing the training data.\n    - The proposed VLA agent has no memory and no planning capability, and the vision part essentially acts as an object classifier. The language instruction performs a similar role. Also, the active rendering encourages the agent to always look at the object, so there is little way it can interpret the scene layout or plan paths. In order to really prove “generalization,” truly unseen instructions and scenes need to be shown, and its success rate separately reported. At the moment, there are no real indications that the proposed method can generalize to unseen scenarios.\n- L20: “. specific object dynamics, such as carrying a box [15] and throwing a 22 ball [47].” - the proposed method also only handles one type of interaction, carrying objects with two end effectors.\n\nOverall, I feel like this work shows great potential in providing a task formulation for embodied AI; however, it is a little overclaimed at this moment in terms of its language and planning capabilities. I recommend scaling down on the formulation (e.g. focus on known scenes and objects) instead of claiming that it can tackle unseen rooms and instructions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Qualitative results**\n\nThanks for your suggestions. \nThe corresponding text prompts for demos align with Fig. 4 and Fig. 12.\nWe provide an extra qualitative result with text and egocentric rendering in the rebuttal PDF.\nDue to the unavailability of demo submission in the rebuttal window, we will add it in the final version.\n\n**W1.1: Evaluation datasets and diversity**\n\nExperiments are conducted on the new proposed HITR dataset, but we kindly remind that there is no other public dataset to support humanoid-based object rearrangement research.\nTo demonstrate our dataset, we illustrate details of scenes and moving objects in Fig.6 and Fig. 7. Examples of texts and tasks are available in Fig. 4 and Fig. 12. The diversity of texts lies in objects (pot, chair, box...), visual attributes (blue, red...), the spatial relationship between object and receptacle (center, left, bottom...).\n\n**W2: Target location**\n\nThe HITR dataset ensures that the object is visible in the first view. \nThe initial orientation in HITR is randomly sampled within 30 offset degrees from the object's initial position.\nOur active rendering technique encourages a localized object view at each step and boosts object localization in the whole execution trajectory.\nWe will add details.\n\n**W2: Placement error and coarse instruction**\n\nWe kindly remind the placement error for VLA model is **40cm**, not 20cm (See Line 279).\nWe agree using coarse instructions instead of precise coordinates is ambiguous for VLA model, and thus criteria relaxation is performed.\n\n**W2.1: Memory and planning capability**\n\nThanks. We encode history actions as the memory part. \nWe agree there is room for improvements by ad-hoc module design.\nInserting extra modules, such as Embodied LLM, for explicitly memorizing and planning in the scene will be of great value to the system.\nWe will add discussions and leave it to future works.\n\n**W2.1: Investigations in Unseen**\n\nOriginally, we conducted task-level unseen experiments in the test set and claimed its generalization performance.\nTask-level unseen includes: new compositions of objects in the scene, new placement of objects, and regenerated new text instructions from LLM describing new compositions and new spatial relations.\nOur approach can generalize in task-level unseen.\n\nIn addition, we agree it would be super challenging to generalize in any unseen without any similar pattern in the seen data. It is also an ultimate goal of embodied AI research. We make additional analysis on generalization in unseen data to further disclose our method. \nWe make additional testing data: \n(1) Unseen texts generated for training tasks, manually reviewed to be distinct from training data. \n(2) Unseen objects by changing visual appearance in training tasks.\n(3) Unseen object category (cup) with different geometry.\n(4) Unseen scene layouts by repositioning static large objects.\n\n|   | Success Rate (%) | Precision (cm)  |  Execution Time (s) |\n| --| -- | -- | -- |\n| Useen Text                            | 65  | 50.4    | 5.4 |\n| Unseen object (visual)                | 50  | 72.3    | 6.2 |\n| Unseen object (geometry & category)   | 20  | 118.8   | 7.9 |\n| Unseen scene layout                   | 35  | 88.5    | 6.8 |\n\nResults are reported in the above table. We find that our work suffers less from unseen texts and unseen visual appearance. But generalizing to unseen object categories and execution in the unseen scenes remain a main challenge.\n\n**W3: L20 \"specific object dynamics\"**\n\nWe will revise the text.\nBut we kindly remind our work can rearrange diverse objects, where the diversity of objects is not supported by previous works.\n\n**W4: Scaling down on the formulation**\n\nThanks for your suggestions. \nWe will tune down our claim and disclose more limitations and failures in the final version.\n\n**Q1: Plan its path around a new room**\n\nThe path planning capability is learned from the training room layouts.\nThough our active rendering encourages an object-oriented view, we deploy a large (90-degree) FoV camera, the background information is still available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8COjFj4r1r", "reviewer": "Reviewer_vb1s", "summary": "In this paper, the authors address the task of room arrangement with a humanoid agent and propose a teacher-student framework to achieve vision-language guided object rearrangement. The teacher policy utilizes privileged state information, including object states, goal states, and waypoints derived from A* planning. Goal-conditioned reinforcement learning together with AMP is employed to train a human-like policy that guides the humanoid agent in completing the task. This teacher policy is then distilled into a student policy, which relies on high-level language instructions and ego-centric view images instead of ground truth state information. A DAgger-like training paradigm is used, and an active rendering technique is developed to focus the camera on objects, ensuring informative ego-centric view images. The authors construct an object rearrangement dataset as a test benchmark for the proposed framework. Experimental results on this benchmark demonstrate the effectiveness of the framework, although generalization to novel tasks and environments remains challenging.", "questions": "* Regarding the teacher policy: could you justify which modules or techniques help in achieving (slightly) higher success rate, better precision, and less execution time than the InterPhys baseline? Since the proposed teacher policy training paradigm share the similar training paradigm in the InterPhys, but it is not justified clearly about the major differences between the InterPhys and the teacher policy implemented in this work.\n* Regarding the student policy: \n    * Without the waypoint information, how is the agent capable of searching and navigation? Does this mean that the agent learns to recognize objects via the vision-language distilling?\n    * The active rendering technique is suggested to encourage the camera to focus on the object state, but does this still access to the privileged object state information?\n    * It is good to see the discussions on the generalizing performance to unseen tasks, and the performance kind of actually makes sense, and I believe it would provide more insightful views if we can investigate how can the policy generalize to unseen scene layouts, unseen visual appearances, unseen language instructions, unseen object geometries individually.\n* The sphere-like hand can limit the capacity of the agent, and the authors also mentioned that one future work can be include dexterous manipulation. Could the authors give some comments on what additional challenges could be by involving the dexterous manipulation skills?", "limitations": "Yes, the authors have addressed the limitations,", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The writing is clear and easy to follow.\n2. This work is addressing a very promising problem setup - the visual-language guided policy learning, and it can have the potential to be applied to the real humanoid.\n3. The build dataset and benchmark can facilitate the community research in human-scene interaction.", "weaknesses": "1. The major methodologies employed in this work, including the AMP+goal-conditioned RL and teacher-student policy distillation were widely-used in many previous works, but still it is good to see this new application in the humanoid room arrangement task.\n2. More investigations into the generalization ability of the current pipeline is desirable, see more in the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Widely-used methodologies**\n\nWe agree some methodologies like AMP, RL, and teacher-student distillation are widely used in applications. \nBut we also investigate new innovative insights like style reward clipping, carry curriculum, active rendering, and more to apply these techniques in the challenging humanoid-based object rearrangement. \n\n\n**Q1: Improvements over InterPhys.**\n\nWe introduce four new techniques compared to InterPhys in the teacher policy, as detailed on Page 5.\nFor box loco-manipulation, the slight improvement mainly comes from style reward clipping (prioritizing task execution) and path planning (navigating complex scenes).\nFor the general rearrangement of diverse objects, our geometry encoding and carry curriculum techniques facilitate multitasking and effectively result in significant improvements.\n\n\n\n**Q2.1: Searching and navigation.**\n\nThe student policy learns from teacher-student distillation.\nIt can learn scene layout patterns and plan navigable paths from training tasks via distillation.\nWe also note the searching and navigation ability can be boosted by ad-hoc module design, such as the integration of Embodied LLMs, but we leave it to future works.\n\n**Q2.2: Privileged object state in active rendering**\n\nPrivileged object states are only used in HumanVLA training for action supervision. \nHowever, the state information is NOT accessed in the inference phase. \n\n**Q2.3: Investigation in the unseen.**\n\nOriginally, we conducted task-level unseen experiments in the test set and claimed its generalization performance.\nTask-level unseen includes: new compositions of objects in the scene, new placement of objects, and regenerated new text instructions from LLM describing new compositions and new spatial relations.\nOur approach can generalize in task-level unseen.\n\nIn addition, we agree it would be super challenging to generalize in any unseen without any similar pattern in the seen data. It is also an ultimate goal of embodied AI research. We make additional analysis on generalization in unseen data to further disclose our method. \nWe make additional testing data: \n(1) Unseen texts generated for training tasks, manually reviewed to be distinct from training data. \n(2) Unseen objects by changing visual appearance in training tasks.\n(3) Unseen object category (cup) with different geometry.\n(4) Unseen scene layouts by repositioning static large objects.\n\n|   | Success Rate (%) | Precision (cm)  |  Execution Time (s) |\n| --| -- | -- | -- |\n| Useen Text                            | 65  | 50.4    | 5.4 |\n| Unseen object (visual)                | 50  | 72.3    | 6.2 |\n| Unseen object (geometry & category)   | 20  | 118.8   | 7.9 |\n| Unseen scene layout                   | 35  | 88.5    | 6.8 |\n\nResults are reported in the above table. We find that our work suffers less from unseen texts and unseen visual appearance. But generalizing to unseen object categories and execution in the unseen scenes remain a main challenge.\n\n\n\n**Q3: Dexterous manipulation**\n\nThe current humanoid model has 28 degrees of freedom (DoFs). \nHowever, the number of DoF may exceed 70 for a humanoid robot with two dexterous hands.\nThe high dimension of the action space is a primary challenge.\nHand motion data are required to train dexterous actions. More effort should be paid to collecting hand data, even hand-object interaction.\nOn the algorithm side, adversarial training suffers from model collapse and generates less expressive actions. This phenomenon will be exacerbated as the state dimension increases. Though techniques like separated hand prior modules and tracking-based imitation can alleviate the issue, expressive whole-body dexterous controls remain very challenging.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rtU8y7wWUY", "reviewer": "Reviewer_hR7q", "summary": "This work aims at building a vision-language-action (VLA) model that can control a humanoid to interact with dynamic objects without ground-truth state information. The proposed method first trains a teacher policy with ground-truth information. It then distills the policy to a VLA model via imitation learning. Additionally, the paper presents a new dataset and evaluates the paper against a recent baseline.", "questions": "Please address the points in the weaknesses. Additionally, I have the following questions:\n\n1. Could you please comment on whether your method can be suitable for human 3D body motion synthesis (e.g., SMPL-X like the synthesis in OMOMO)?\n\n2. Would it be possible to directly train VLA to simplify the training procedure?\n\n3. Could you apply your method to longer activities that involve a sequence of goals? If not, what could be missing?", "limitations": "There is not much discussion on limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The work is well-motivated and addresses two fundamental challenges: interacting with dynamic objects and the lack of ground-truth state information in real-world scenarios.\n\n+ The method works well based on the demo videos and the quantitative evaluation. Each component of the method is well-motivated.\n\n+ There are sufficient details about the method and the dataset.\n\n+ The dataset can be a valuable resource for future research.", "weaknesses": "- The paper combines many existing methods together and builds a very complex system. It introduces some interesting components such as active rendering, but it is unclear how general the system is for a broader class of synthesis. For instance, can this system be applied to any humanoid-object control? What about one humanoid interacting with multiple objects (e.g., one agent carrying two objects in both hands) or multiple humanoids interacting with the same object (e.g., two agents carrying a sofa)? Can you extend the object category to those that are unseen during training?\n\n- Related to the first point, there is a lack of discussion on failure and limitations. \n\n- It would be good to show videos of the baseline results for a clearer comparison.\n\n- What is the sim2real gap? I assume that the end goal is to control a real-world humanoid to complete the tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Border class of synthesis.**\n\nOne-agent multi-object: Our framework does not lose generalizability in muti-object interaction, which can be achieved by task reward design. However, from a model mechanism perspective, it will be more suitable with two dexterous hands.  \n\nMulti-agent one object: It can be achieved by integrating our pipeline into a multi-agent learning framework.  \n\nSince we are the first work in vision-language humanoid, we leave these border synthesis applications to future works.\n\n\n**W1: Unseen categories during training.**\n\nOriginally, we conducted task-level unseen experiments in the test set and claimed its generalization performance.\nTask-level unseen includes: new compositions of objects in the scene, new placement of objects, and regenerated new text instructions from LLM describing new compositions and new spatial relations.\nOur approach can generalize in task-level unseen.\n\nIn addition, we agree it would be super challenging to generalize in any unseen without any similar pattern in the seen data. It is also an ultimate goal of embodied AI research. We make additional analysis on generalization in unseen data to further disclose our method. \nWe make additional testing data: \n(1) Unseen texts generated for training tasks, manually reviewed to be distinct from training data. \n(2) Unseen objects by changing visual appearance in training tasks.\n(3) Unseen object category (cup) with different geometry.\n(4) Unseen scene layouts by repositioning static large objects.\n\n|   | Success Rate (%) | Precision (cm)  |  Execution Time (s) |\n| --| -- | -- | -- |\n| Useen Text                            | 65  | 50.4    | 5.4 |\n| Unseen object (visual)                | 50  | 72.3    | 6.2 |\n| Unseen object (geometry & category)   | 20  | 118.8   | 7.9 |\n| Unseen scene layout                   | 35  | 88.5    | 6.8 |\n\nResults are reported in the above table. We find that our work suffers less from unseen texts and unseen visual appearance. But generalizing to unseen object categories and execution in the unseen scenes remain a main challenge.\n\n\n**W2: Failure and limitation.**\n\nWe have discussed some limitations in Appendix A. \nMore failures and limitations pointed out by reviewers will be supplemented.\n\n**W3: Baselines**\n\nThanks for your suggestion. \nA qualitative case comparing our method and baseline can be found in Figure 11.\nWe will add comparative demo in the final version but are unfortunately not allowed to submit demos in the rebuttal window.\n\n**W4: sim2real gap**\n\nThe sim2real gap includes the robot model and environmental data. \nWhile we utilize a simulated humanoid model, exploring real-world embodiment structures is essential. However, our method is generally applicable across similar mechanical structures of humanoids. \nBesides, efforts in scaling up real-world data, like scanning and decomposing the real world, are also crucial.\n\n\n\n**Q1: Human 3D body motion synthesis**\n\nOur work is a kind of human 3D body motion synthesis, i.e. physical motion.\nThe mentioned SMPL-X like synthesis in OMOMO is kinematic motion. \nWe have made a thorough discussion of two streams in Section 2 Related Works.\nBoth kinds of works are to generate plausible motions and human-scene interactions.\nA key distinction lies in the manipulation of object states: kinematic motion, usually for computer graphics, allows for direct editing of object states, whereas in physical motion, altering the object state necessitates indirectly controlling the humanoid to interact with.\nTo this end, we think physical motion and interaction are more challenging than kinematic motion.\nIt is not fair to directly apply our holistic pipeline to kinematic SMPL-X based synthesis.\nHowever, we think some motivations, such as adversarial training and curriculum training, can be shared for kinematic synthesis.\n\n**Q2: Directly train VLA**\n\nDirectly training VLA using current techniques and computes is very hard.\nFirst, from a computation perspective, rendering an image is usually 10 times slower than physics itself in simulation. Besides, VLA inferencing is 20 times slower than state-based network. One-stage visual RL requires large parallel environments on heavy computation loads. Our two stage system leveraging state-based RL (large-scale parallelization on light task) and behavior cloning (small parallelization on heavy task) turns out to be efficient.\nSecond, from a learning perspective, vision-language modality is a coarse, high-dimensional, and composite representation. On the contray, state based RL is precise and unambiguous. We anticipate direct Visual RL is hard to converge.\nCombining these facts, we believe our framework is most suitable and affordable within our computes.\n\n\n**Q3: Longer activity**\n\nThanks. \nApplying our method to longer activities is possible by using sequences of tasks. \nEfforts on benchmarking long-horizon tasks are required.\nWe have discussed this limitation of our current form in Appendix A and leave long-horizon benchmarking to future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response

We sincerely thank all reviewers for dedicating their time to review our work.
And we highly appreciate their positive ratings and recognition of our work:

- Our work addresses fundamental challenges for humanoid and points out a very **promising** and **potential** research direction of embodied AI.
- We propose new techniques, well motivated with small innovations.
- Our data efforts have value in facilitating future research.

We also welcome constructive comments by reviewers and add additional results to disclose failures and limitations of our system in unseen generalization. We will add discussions in the final version.
In addition, as the very first work directed at vision-language humanoid, we leave research on ad-hoc policy module design, scaling large data and large models, long-horizon tasks, dexterous manipulation, and multi-humanoid collaboration to future works.

Meta-review: The general idea of the paper is to develop a vision-language-action model to control a human avatar performing rearrangement tasks. The paper also proposes the Human-in-the-Room dataset which includes various rearrangement tasks. The reviewers unanimously rated the paper as Weak Accept. The reviewers had concerns such as the gap between simulation and reality, diversity of the proposed dataset, and generalizability of the pipeline. Generally, the rebuttal addressed the concerns well. However, there are some inaccurate statements such as “there is no other public dataset to support humanoid-based object rearrangement research”. Frameworks such as Habitat 3.0 study rearrangement tasks with humanoids. Overall, the paper has interesting contributions, and the AC follows the recommendation of the reviewers.
Decision: Accept (poster)