Endowing GPT-4 with a Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical World
Authors: Yingzhao Jian
Zhongan Wang
Yi Yang
Hehe Fan
Venue: ICLR 2026 Poster (2026)
Keywords: human-scene interaction
VLM agent
motion generation
Primary area: generative models

Abstract: In this paper, we explore how to empower general-purpose Vision-Language Models (VLMs) to control humanoid agents. General-purpose VLMs (e.g., GPT-4) exhibit strong open-world generalization, and remove the need for additional fine-tuning data. To build such an agent, two key components are required: (1) an embodied instruction compiler, which enables the VLM to observe the scene and translate high-level user instructions into low-level control parameters; and (2) a motion executor, which generates human motions from these parameters while adapting to real-time physical feedback.
We present BiBo, a VLM-driven humanoid agent composed of an embodied instruction compiler and a diffusion-based motion executor. The compiler interprets user instructions in context with the environment, and leverages a chain of visual question answering (VQA) to guide the VLM in specifying control parameters (e.g., motion captions, locations). The diffusion executor extends future joint trajectories from prior motion, conditioned on both control parameters and environmental feedback.
Experiments demonstrate that BiBo achieves an interaction task success rate of 90.2\% in open environments, and improves the precision of text-guided motion execution by 16.3\% over prior methods. BiBo handles not only basic interaction but also diverse motions, and even dancing while striking at a sandbag. The code will be released upon publication.

Published as a conference paper at ICLR 2026
ENDOWING GPT-4 WITH A HUMANOID BODY: BUILD- ING THE BRIDGE BETWEEN OFF-THE-SHELL VLMs AND THE PHYSICAL WORLD
Yingzhao Jian, Zhongan Wang, Yi Yang Hehe Fan* College of Computer Science and Technology, Zhejiang University. *Corresponding author. hehefan@ www.juuuuuc... edu. cn
ABSTRACT
Humanoid agents often struggle to handle flexible and diverse interactions in open environments. common solution is to collect massive datasets to train a highly capable model, but this approach can be prohibitively expensive. In this paper, we explore an alternative solution: empowering off-the-shelf Vision-Language Mod- els (VLMs, such as GPT-4) to control humanoid agents, thereby leveraging their strong open-world generalization to mitigate the need for extensive data collec- tion. To this end, we present BiBo (Building humanold agent By Off-the-shelf VLMs). It consists of two key components: (I) an embodied instruction com- piler, which enables the VLM to perceive the environment and precisely translate high-level user instructions (e.g., "have rest") into low-level primitive commands with control parameters (e.g.. "sil casually, location: (1. 2). facing: 90" and (2) a diffusion-based motion executor, which generates human-like motions from these commands, while dynamically adapting to physical feedback from the envi- ronment. In this way. BiBo is capable of handling not only basic interactions but also diverse and complex motions. Experiments demonstrate that BiBo achieves interaction task success rate of 90.2% in open environments, and improves the precision of text- guided motion execution by 16.3% over prior methods. The code is available at http:://gthub..co//haddw
INTRODUCTION
Humanoid agents, as a medium between digital intelligence and the physical world, have attracted
extensive research interest, particularly in the domains of scene perception (Huang et al. 2024b Qi
jet al. 2025b) and interaction (Xiao et al. 2023 Tevet et al. 2024). With recent adncccss humanoid
agents are capable of performing text- rext-guided motions (Tevet et al. [022] Yuan et al.) 2024) and
executing interactive tasks under predefined plans (Xu et al., 2024 Pan et al. 2025). However, flex- ibly handling user-intended scene interactions in open and dynamic physical environments remains a significant challenge. A straightforward strategy is to collect large-scale human-scene interaction data (Bhatnagar al., 2022 Jiang al. 2024) and train highly capable model, as commonly done for robotic arms or wheeled platforms (Firoozi et al. 2025 Team et al. 2025). Unfortunately, due to the structural complexity of humanoid bodies and the vast diversity of physical world, such data-centric scaling becomes prohibitively expensive and difficult to generalize.
In contrast, off-the-shelf general- purposs Vision Language Models (VLMs), such as GPT- 4 (Achiam et al.) 2023), Gemini (Team et al., 2023), and Qwen (Bai et al. 2023). demonstrate open-world reasoning and adaptability across wide variety of tasks, without specific finetuning. This observation naturally raises intriguing question: Can bypass costly data collection by directly leveraging these powerful offeth VLMs to control humanoid agents, thereby enabling versatile interaction in the physical world?
Motivated by this question, we introduce BiBo (Building humanold agent By Off-the-shelf VLMs), framework designed to endow off fff-the-heeff VLMs with the capability to control humanoid agents. BiBo is composed of two core components: 1) an VLM-drrive embodied instruction compiler and 2) diffusion-basee based motion executor, which jointly bridge the gap between high-level human

---

Published as a conference paper at ICLR 2026
User Inst.
Physical Environment
Move plant bookcase
Composite Action
Long Horizon
Organize bookshelf
Long-range Interaction
Lie down big sofa and place hand on table
4. and wave hand
5. Walk to the front of sofa
Multi-joint Control
Precise Control
6. Take a book from the case
Dynamic Object
Byy Agent
sit and wave hand
Physical Feedback
Bi Bo!
State
Image
location: (1, 2)
facing: 90
Inst. Compiler
Motion Executor
0o
High-level User Inst.
Off-the-shelf VLM
Structured Command
Motion Diffusion
Generated
Motion
Figure 1: BiBo a humanoid agent powered by an off-the-shelf VLM. It consists of an embodied instruction compiler (Inst. Compiler) and diffusion-based motion executor. When the user provides a high-level instruction, the compiler observes the environment and translates it into the structured command for the executor. The executor then generates future motions for the humanoid agent, conditioned on both the command and the physical feedback from the environment. In this way, BiBo is able perform diverse types of physical scene interactions.
instructions and low-level motor control required for physical humanoid interactions. This design is conceptually analogous to computer. where the compiler and the assembler work together, operat- ing the hardware to accomplish the tasks specified by programming language.
In computing system, a compiler translates source code written by high-level programming lan- guages into low-level assembled language. Inspired by this, as in Fig. [ the embodied instruction compiler in BiBo is designed to translate high-level natural language instructions into low-level ex- ecutor commands, according to environmental context. To achieve this, the compiler first represents an action as a structured set of descriptors, encompassing the motion caption, key joint configura- tions, and other contextual details. Building on this structured representation, the compiler drives the VLM to reason over each descriptor in a coarse-to-fine manner, thereby producing an accurate and structured command that specifies the user intented action.
This generated command is then passed to the motion executor, which functions much like an as- sembler. Just translating assembly commands into machine code, the motion executor interprets commands into full-body humanoid motions. Unlike a rigid rulc-based assembler, our executor leverages a diffusion generator. Each time it receives a command, the generator extends future joint trajectories from the current motion, enabling diverse motion style and on-the-fly control.
However, during execution, collisions or external forces may cause the actual executed motion to deviate from the initially generated sequence. To handle such feedback, prior approaches (Tevet al. 2024 Chen et al. 2024) extending future joint trajectories from the executed motion, rather than from those previously generated but unexecuted. This strategy enforces the diffusion to account for environmental context, but also introduces discontinuities between the current and previous gen- erated motions. We resolve this by developing novel application of the Latent Diffusion Model (LDM) (Chen et al. (023). In our method, the diffusion extends future latent from the actual exe- cuted motion, enabling environmental awareness. A Variational Autoencoder jointly decodes the latents of the previous and current generated motions, ensuring smooth transition.
According to the experiments, BiBo achieves an interaction task success rate of 90.2% under ran- dom generated physical environments using an off-the-shelf VLM (i.e., GPT-4o). Moreover, BiBo improves the precision of text-guided motion execution by 16.3% over prior methods. It handles complex motion execution, while also enabling infinite long-sequence synthesis and real-time inter- active control through user instructions. In summary, our main contributions threefold:
We empower off-the-shelf VLMs for humanoid control through embodied instruction compiler and a diffusion-basee motion executor, bridging the gap between VLM reasoning and low-level physical execution.
2

---

Published as a conference paper at ICLR 2026
The compiler introduces structured humanoid action representation, enabling coarse-to- fine embodied reasoning, while advancing humanoid behavior planning and modeling.
We develop novel application of LDM to incorporate environmental feedback in mo- tion generation, achieving state-of-the-art unlimited-length motion synthesis and offering insights for applying LDMs in broader domains.
2
RELATED WORK
2.1 HUMAN SCENE INTERACTION
When interacting with scene, humanoids perceive environments through training real-world data, reinforcement learning (RL), and large language models (LLMs). Recent advances combine them: (1) using LLM to guide RL policies (Xiao et al. 2023 Shi al. 2024), but still limits motion diversity; (2) using RL trackers (Luo et 2022 2023) to follow diffusion afgtusiongeneratee motions (Tevet al., 2024), but causes discontinuities between generated and tracked motions. BiBo employs off-the-shelf VLM to guide a latent diffusion model (LDM), promoting generalization and diversity, which achieve both smoothness and physical plausibility.
2.2 TEXT TO MOTION GENERATION
In text-to-motion, approaches can be broadly categorized into fixed- and arbitrary-length generation. For fixed-length generation, frameworks such as VAEs (Petrovich et al. 2021b Bie et al.) 2022).
masked modeling (Pinyoanuntapong et al., 2024 Guo et al.] 2024). and diffusion models (Tevet
et al. 2022 Zhang et al. 2024; Dai et al.) 2024 Chen et al.) 2025 have been extensively explored. However, humanoids perform continuous arbitrary-lengte motion following user commands. To this end, some works adopt autoregressive next-token prediction (Jiang al. 2023 [Zhang al. (023)), achieving high fidelity, while others (Chen et al. 2024] Han et al.] 2024 Xiao et al. (2025) employ
diffusion to extend future joint trajectories from past motion, improving efficiency. Bißo use latent
diffusion with few denoising steps, enabling both high-fidelity generation and real-time control.
3 METHOD
3.1
OVERVIEW
BiBo is humanoid agent powered by an off-the-shelf Vision-Language Model (VLM). As shown in comprises two main components: an embodied instruction compiler and a diffusion-based motion exccutor. Given high-level user instruction, the compiler first collects observations of the current scene, and then prompts the VLM to generate a caption of the next motion to be executed. Next, the compiler guides the VLM to refine motion details through a three-stage visual question- answering (VQA) process. Finally, formats these details into a command based on a structured motion representation, thereby instructing the executor to generate the corresponding motion.
Fig. [
The executor takes the command as a condition, extending future joint trajectories from the cur- rent motion. Due collisions external forces, the actual executed motion may differ from the predicted trajectories. To adapt to this feedback, we incorporate the actual performed motion into diffusion by developing novel application of the Latent Diffusion Model (LDM). In diffusion, the model extends future motion latents from the actual executed motion, thereby adapting to the scene feedback; in the VAE, the model jointly decodes the previously generated and currently executed motions, ensuring smooth transitions between previous and current generated motions.
3.2
EMBODIED INSTRUCTION COMPILER
As in Fig. the embodied instruction compiler enables the VLM to translate high-level user in- structions into low-level executor commands, based on environmental observation. consists of a three-stage visual question-answering process. The VLM first determines the next motion to cute, and analyzes its basic attributes, such as the motion caption and target object. Then, it reasons about the agent pose. Finally, the VLM locates the target positions of the key joints. The output of
3

---

Published as a conference paper at ICLR 2026
Embodied Instruction Compiler
State
Image
② Pose Reasoning
Command of Next Motion
Caption: Arrange
Joint Target Positions:
Ent
mobler Zole
Avol
plant
but fishef
Direction?
Facing: 90
Location: 2.5)
C. L.ond:12.2. 3.
Chonde (2, 3
EXPC
Inst.
Front
1 (c..f,o.J))
The bookshelf
messy.
Arrnnge bookshea
Wext Medion:
Distance?
0.5
Facing
Joint Generation
Attr Anolyzing
jbject
Object
Caption:
Facing?
Om label (8)
bookshelf
Joints
Fright hand
Toword
14
15
Coption
Caption
lobel
Off-the-sheif VLM
Embodied Instruction Compiler
Figure 2:
The embodied instruction compiler takes in user instructions and environmental obser- vations, and directs the VLM to generate the next motion command through structured three- three-stage visual cueestion-nswweree
process. In the first stage, analyzes the basic attributes of the motion caption, key joints, target object). In the second stage, it reasons about the agent's 's pose during the interaction. Finally, it specifies the target positions for the key joints.
the compiler is a executor command C, including the caption c, location I E IR?, facing direction [-T, n]. and joint targets (;) j is a key joint), where Pj R33 is the joint target:
C = {c. I, f, J}.
The command C serves as structured and simplified humanoid action representation, which reduces generation complexity while preserving the key information of a motion, controlling the diffusion to generate an interactive motion that fulfills the instruction. Details can be found in Sec. C
Basic Attribute Analysis. In this step, the compiler inputs the user instruction with scene images and agent's status, and prompts the VLM to analyze the attributes of the next motion be exe- cuted. These attributes include motion caption c, an anchor object for agent localization, the key joints j involved, and other complementary details that facilitate subsequent reasoning. To enhance accuracy, the final result is selected through majority voting across five parallel VLM instances.
Agent Pose Reasoning. The pose refers to the location and facing direction f of the agent. To predict the pose, directly outputting coordinates and angles is a straightforward way. However, current off-the-shelf VL.Ss struggle to handle numbers like 3D coordinates (Huang et al., 2024a Qi al. 2025ab. As result, transform it into visual identification task, which more familiar to VLMs. As in Fig. 2
we put labels around the anchor object o, each corresponds to a position or direction. By choosing label from the image, the VLM roughly locates the agent in the scene.
Key Joint Generation. When interacting with objects, we typically focus on a few key joints and their relative position to specific target points (e.g., when using hand dryer, the hands are placed about 0.2m beneath the air outlet). Inspired by this, for each key joint, we first place a 8 x 8 grid of labels over the image of the anchor object, and allow the VLM select one the target point. Next, the VLM generates the joint's 's direction and distance relative to the target point. We provide set of predefined directions: [up, down, left. right, forward, backward. toward the object center, along the surface normalj. This simplification reduces generation difficulty while covering most cases.
3.3 MOTION DIFFUSION EXECUTOR
The Motion Diffusion Executor is Latent Diffusion Model, composed of a VAE and a Diffusion module. When command C' comes, the VAE first encodes both the previously generated motion My E Fx and the actual executed motion M, E FxD (i.e. the execution result of M, under physical environment) into latent representations Sg E R I. and S, E I respectively:
S. (nncode(MM.). S, Encoder(MM (M₃)-
F is the number of frames of the motion. L is the number of latent tokens, where each token correspond to F frames. D and H are the dimension of motion frame and latent. Then, the command C, together with the latent of the executed motion S, guide the denoising process to produce the latent of future motion S E R The latents of the previous and current generated motion S, and S are jointly decoded by the VAE to obtain the future joint trajectories M, FxD.
4

---

Published as a conference paper at ICLR 2026
Compiler
Motion Diffusion Executor
Phys. Env
Next Motion Caption
BeRT +g
Diffusion
caption Token
Tracking
Control Parameters (L.J)
MLP
Transformer
Obstacle
Actual Executed Motion M,
Encoder
control Token
K
Tokens
X N
Encoder
Collision!
causal self-attn
Init zaro
future Takens S,
Act. Mo.
Generated Motion M,
(HF/L
gen.
Gen, Mo.
XM
Resample
Decoder
Previous Generated
Motion (recon) MY
Generated Future Motion M,-
UUF
Inle vactor
cross-atti
continuous
physical Gwared
causall self attn
M
The motion executor is Latent Diffusion Model. When receiving the command (motion caption and control parameters) from the compiler, the Diffusion extends the future latents Sy from the actual executed motion tokens S., conditioned on the command tokens 8m and Then, the previous and newly generated latents are jointly decoded by the VAE decoder. The decoder use casual attention, where each motion frame latent token only attend to its preceding tokens or frames. After IK optimization, a tracking policy drive humanoid joints to execute the newly generated motion My in physical environment, producing the next execution result.
Figure 3:
s DCffusion(C, Sw), [M, M2) Decoder([Sg: :s))).
Sy RLLH is the generated latent of future motion. MG M, RF is the reconstructed previous generated motion. [:] represents the concatenation across length (or frame) dimension. By conditioning on S., the future motion M is guided to account for physical feedback in M., while joint decoding enforces its continuity with Mg.
We additionally use inverse kinematics (IK) post-optimization improve control accuracy. Finally, reinforcement learned tracking policy drives humanoid joints to follow the generated joint trajec- tories physical scene. For more details, please refer to Sec. D.5
VAF Design. We use Transformer encoder-decoder architecture and propose causal self-cross atten- tion. Specifically, during the attention process, each latent token or motion frame can only attend to its preceding tokens frames. This design ensures the continuity between the previous and current generated motion M, and My. Specifically, as in Eq. since the VAE ensures continuity in its decoded motions, the the generated future motion M2 continuous with reconstructed previous motion Mg. Moreover, due to the causal mechanism:
[M, My] = Decoder([S,.) Sy]) = M₁ = Decoder(S,) 5,) + M = Mg.
(4)
My is continuous with M and M ~ My. Therefore, M, is also continuous with Mg.
4 EXPERIMENTS
In this section, we analyze the capabilities of BiBo from two perspectives: task completion and motion quality. For task completion, we establish set of tasks, and assess the success rate of BiBo and comparison methods under random generated scenes. For motion quality, we adopt standard motion quality metrics to quantitatively evaluate the synthesized and executed motions, and conduct case studies with visualizations to analyze the qualitative results. For more details and results, please refer to Sec. and
4.1 TASK COMPLETION
Task Setting. We define six types of single interaction tasks. Task success if the required criterion met for over 30 frames. The tasks include:
Reach is considered successful if the agent reaches within 0.5 meters of the target location.
5

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of task success rates for different methods under randomly generated scenes and initial poses. A single task involves navigating to the interaction position and performing the interaction, whereas a composite task consists of multiple simultaneous or sequential single interac- tions. BiBo (our) performs online planning during evaluation, while other methods use ground truth action plan. The bold and underline represent the best and second-best performance, respectively. BiBo achieves the highest success rate across all tasks.
Method (%)
Single Interaction
Composit Medhum
Keach
Warch
Sit
Sleep 1
Touch 1
Laf
Simple
Hard
UnilISI Xiao el al. 2023
93,28
81.03
85.11
69.62
Human VIA Xx
2124
56.98
44.90
TokenI (Pan et al 2025
94.55
72.95
33.33
48.19
CLD (Tevet
2024
85,83
87.76
76.99
34.67
42.55
7.71
26.47
7.05
2.38
BiBo (ours)
99.18
99.62
95.84
94,89
86.05
65.42
58.82
36.54
27.78
BiBo (ours. plan)
96.75
87.23
70.41
61.76
44.23
Watch evaluates the understanding of object orientation. success if the agent stands in front of the target and facing to it, with an angular error of less than π n/6.
Scenes
Task Type
Comp
Sit & Sleep evaluate interaction with objects of varying shapes and functions. Sit success if the hips are within 0.1m of the seat area, subject to an upward force, and the torso faces forward. Sleep success if the legs, arms, and torso are all within 0.1m of the sleep area, with the av- erage force directed upward and the torso facing upward. The angular errors should be less than ग 7/4.
Wateh
Touch
Scene Info
EEa
Reach
Tasks
Single: Sleeping on (label 1)
Task Difficulty
Simple: Relax the room.
Medium: Move plant near shelff
Mora
Hard: and turn on lamp simultoneouslly Grab book and refax on the sofo. Turn off alf tamps and return the book.
Simple
Touch is successful if the correct hand is within 0. of the target, and subject force.
Mediuu
Target Object
Task Steps
Lift evaluates dynamic object manipulation. Success if the target being lifted more than 0.25m above the ground.
400
60
200
30
We additionally introduce composite tasks, each consist- ing of multiple interactions. They succeeds if all interac- tions are accomplished in the specified order. Composite tasks are categorized into simple, medium and hard. 1) Simple tasks consist of 4 interactions. 2) Medium tasks include user intent understanding (e.g., the room too dark) and dynamic object manipulation (i.e., lift, transport). 3) Hard tasks involve long-horizon (> 10 interactions) and simultaneous interactions with multiple objects (e.g., sit on sofa and put hand on the side table).
0
plonoe
storage
onn.
Figure 4: Summary of the random gen- erated scene dataset. The tasks are con- structed by semi-automatic approach. The dataset contains various object cat- egories, task types and difficulties, eval- uating a wide range of interaction abili- ties of humanoid agents.
Dataset. To evaluate BiBo's ability to accomplish different tasks in physical environments, we ran- domly construct 100 scenes using InfiniGen (Raistrick et al. (002)). These scenes include different rooms and layouts, containing 73 object categories with randomized parameters controlling their shapes and styles. For each scene, we construct 6- 18 single interaction and 1-3 composite tasks by a semi--utoonn automatt approach. Specifically, as in Fig. 4 volunteers first annotate task examples, after which the LLM generates tasks conditioned on the examples and scene information. This process results in a total of 1,365 single tasks and 162 composite tasks. All tasks are given through natural language, and evaluated three times under different initialization. Details can be found in Sec. B.
Comparison Method. As comparison methods, UniHSI (Xiao et al.) 2023) focuses on contact interactions, HumanVLA (Xu al. 2024) follows the VLA paradigm to address transportation tasks, TokenHSI (Pan et al. 2025) trains task tokens to manage multiple skills, and CLoSD (Tevet al. 2024) employs motion diffusion, enabling diverse scene interactions. As these methods lack an applicable planner, they use a programmatically generated ground-truth plan, with details provided in Sec. For BiBo, we evaluated both the ground truth plan and online planning.
Result. As shown in Tab. BiBo achieves an average success rate of 90.2% in single interac- tion, and 41.0% in composite task. Compared with other methods, BiBo achieves an average im- provement of 12.5% and 29.1% on single interaction and composite tasks, respectively. In online planning, BiBo achieves success rates close to those of the ground-truth plan (within 4.38%).

---

Published as a conference paper at ICLR 2026
Table 2: Impact of different components in compiler and executor on the task success rates. Act. and Gen. represents the actual executed motion and previous generated motion. The bold and underline represent the best and second-best performance, respectively. The results demonstrate the effectiveness of designs in BiBo.
Single Interaction
Composite Task
Method (%)
Reach
Warch
Sit
Sleep
Touch
Lift
Simple 1
Medium
Hard 1
BiBo (ours, wio Voting)
99.18
98.87
91.13
88.67
85.82
59.75
49.51
32.05
22.22
BiBo (ours, w/o Label)
97.00
90.21
48.59
46.44
64.89
58.73
26,96
17.31
7.94
(ours, wlo Act.)
98.09
84.18
73.78
81.80
16.02
10.32
BiBo (ours, wio Gen.)
98.64
99.25
95.62
93.78
84.40
56.58
57.35
30.77
23.81
BiBo (ours, w/o IK)
98.82
99.25
95.96
92.89
48.94
6.80
31.37
13.46
3.17
(ours)
99.18
95.84
94.89
86.05
Table 3: Comparison between text-to-motion methods on the HumanML3D. We evaluates effi- ciency, motion quality, and physical plausibility. 1. and - means the higher, smaller and closer to the ground truth is preferred. Arbi, and Phys. indicate the support for arbitrary- and physical plausible generation. The bold and underline represents the best and second- bestnd-best performance.
Emficlenyy
Motion Quality
Physical Plausibility
Method
AUT
Arbi.
FID 1 K.F.TTT R.P.22
Diversity
Phys. Fen. 1 Float
Skate
Ground Trick
0.001
0.514
0.706
0,800
9.503
0.00
22.9
0.21
MDM Tevett et al. 2022
24.74
X 0.423
0.406
0.603
0.719
9.559
K
0.15
28.6
0.33
MotionEUM etall
2024
0.042
0.072
0.510
0.703
0.797
9.598
0.65
27.4
0.81
MotionStreanler Xillo all 2025
2.584
0,084
0.432
0.615
0.716
9.549
0.17
23.2
0.68
MoGenTS Yuan et ni.
2024
0.836
X 0.046
0.521
0.713
0.804
9.617
X
0.80
26.2
0.96
McCon TYao
x
3.279
0.309
0,504
0.614
8.010
0.25
32.0
0.29
DiP (Tevet et al. 2024
0,257
/
0,210
0.466
0.653
0.754
9.570
X
0.14
24.5
0.65
CLoSD Tover ot al. 2024
2.873
2,861
0.367
0.553
0.665
8.256
0.30
20.2
0.01
BiBo (ours)
0,076
0.542
0.738
0.829
9.606
0.32
25,3
0.74
BiBo (ours, Phy.)
1,883
0.411
0.604
0.716
8,298
0.19
20.6
0.01
Experimental results demonstrate that effectiveness of designs in BiBo, revealing the potential of general-purpose VL.Ms in controlling humanoids.
Table 4:
Comparison of control accuracy across methods using MAE. means smaller is better. Bold is the best performance.
Ablation Study. We conduct ablation studies to val- idate the proposed designs in BiBo. Specifically, in the compiler, we introduce voting mechanism, and leverage image labels to facilitate the visual reason- ing. In the executor, we generate future motion con- dition on the actual executed motion and previous generated motion, and apply IK for precise control. The results in Tab show that Voting and Label improve task success rate by 4.1% and 22.9%. IK plays crucial role in precise joint control, while exccuted and generated motions affect tasks that rely on physical feedback and precise interaction.
Method
Head L Hand L
Foot L
DiP Tevee et al 2024
0,0663
0.0830
0.0540
MotieaL (Dat ala
2024
0,0952
0.1470
0.0955
BiBo (ours)
0,0310
0,0571
0,0335
BiBo (ours, w/o LDM)
0,0705
0.0918
0.0608
4.2 MOTION QUALITY
Setting. We evaluate the motion quality from both quantitative and qualitative perspectives. On the quantitative side, we report Average Inference Time per Sequence (AITS) reflects the computational overhead (Chen et al.) 2023); Fréchet Inception Distance (FID), R Precision and Diversity evaluate the motion fidelity and diversity (Guo et al. 2022b); Penetration, Float, and Skate reflect the physical (023);; Mean Absolute Error (MAE) evaluates the control precision. Details
plausibility (Yuan et al.
can be found in Sec. B.2.3 On the qualitative side, we conduct human evaluations and visual inspections, and further perform case studies that analyze both successful and failure cases.
Dataset. We adopt the HumanML3D dataset. Following common practice, we use motion se- quences whose lengths fall inside [40, 200] frames. total of 24.545 motion episodes paired with 66,633 motion captions from the training split used to train the motion diffusion executor, while 4,646 motion episodes paired with 12,536 captions from the test split are reserved for evaluation.
Comparison Method. We adopt state-of-the-ar methods for both fixed-length and arbitrary length generation, including physical and non- non-physical approaches. Details are provided in the Sec.
Quantitative Result. As in Tab. BiBo handles on-the-fly control (> 20Hz), and demonstrates advantages (non-physical +3.5% and physical +7.3% relatively) in text alignment (R.P.) across comparison methods. It improves motion realism (FID) in real-time arbitrary-length generation
7

---

Published as a conference paper at ICLR 2026
Table Motion discontinuity evalu- ated by average joint acceleration a.
Table Impact of different components in BiBo on mo- tion quality, evaluated using HumanML.3. and indi-
cate higher and smaller is preferred, respectively. Phys.
Method (xri /
shows support for physical plausibility. Bold indicates the best performance.
CLoSD (Tovet 2024
0.0610
BiRo (ours, w/o LDM]
0.0879
BiBo (ours, w/o Causal)
0.0626
BiRo (ours, w/o Gen.)
0.0698
Phyx.
4
R.P.@1
1
BiBo (ours, w/o Act.)
0.0370
Ground Trath
X
0.001
0.514
0.706
0.800
BiBo (ours)
BiBo (ours, wio LDM)
X
0.258
0.467
0.662
0.762
Table Number of preferred motions interactions in User study.
BiBo (ours, wio LDM, Phy.)
/
2.138
0.376
0.561
0.674
BiBo (ours, wio Causal)
X
0.101
0.526
0,716
0.801
BiBo (ours, wlo Causal, Phy.)
2.377
0.376
0.571
0.679
BiBo (ours, wlo Gen., Phy.)
2.312
0.382
0.577
0.689
Method
Count
DiPICLoSD) Teeve
20
(ours, w/o Phy.)
0.616
MotionI.CM Dat ar. 2024
53
(ours)
BiBo (ours)
77
BiBo (ours. Phy.)
1,883
0.411
0.604
0.716
UniHSI Xiao 2024]
HumanVLA [Xu 2024]
BiBo
BiBo (ours)
(ours)
(a) Getting up from the sofa
(b) Lean the armrest
(c) Corry the box
MoConVQ (Yao 2024]
CLoSD [Tevet 2024]
BiBo
BiBo (ours)
(ours)
(d) Strike and kick
(f) Turn the lamp
(9) Grab book
wave hands
Figure 5: Visualization of executing results of comparison methods. Compared with BiBo, UniHSI generates less natural motions, while Human requires stricter initial positioning for transporta- tion. MoConVQ shows limited motion activity, and CLoSD struggles to achieve precise control.
by 63.8% relatively, demonstrates comparable physical plausibility to CLoSD. As Tab. BiBo achieves the highest control precision. These results demonstrate that the proposed executor pro- vides an effective medium for bridging general-purpose VI.M with the physical world.
Ablation Study. We employ LDM and causal attention mitigate the discontinuity between future and current motion, and use previous generated motion and the actual executed motion to pro- mote smoothness and environmental awareness. We conduct ablation studies to demonstrate the contribution of these designs. Specifically, discontinuity can be manifested as an abrupt change in joint velocities. As result, quantify discontinuity by computing average joint acceleration à = E, (1Ip) n+ + p" n-1 2p (12) / f at the initial frame n of the future motion, where p E R is the position of joint j at frame n. and t is the frame interval in seconds.
According to the results in Tab. and 7 both LDM and causal attention improve motion quality. Incorporating previous generated motion effectively reduces discontinuity during motion transitions. Incorporating actual executed motion may affect generation quality, but enhance adaptability to environmental interactions as in Tab.
Qualitative Result and Case Study. We conduct a user study to evaluate generation quality through questionnaires. The questionnaire consists pairs of motions and scene interactions generated by BiBo, MotionLCM. DiP(CLoSD) under the same prompts. Each pair of motions is displayed in the same row with the left-right order randomized, and participants are asked to select the one with higher generation quality. The selections from 30 volunteers summarized in Tab. 6 and scene interactions generated by BiBo are preferred.
The motions
We conduct visual evaluation with the comparison methods in Fig including MoConVO. Human- VLA, UniHSI, and CLoSD. In (a) and (b), UniHSI produces unnatural movements, whereas BiBo generates natural standing and leaning motion. In (c), when initial agent pose is not aligned with
8

---

Published as a conference paper at ICLR 2026
(1) Gen. Only
(2)
Act. Only
(3)
Act.&Gen. (ours)
Result
Figure 6: Visual comparison between the executed result of different motion generation method, where the red balls in the image represents generated motion. Act. and Gen. denotes extending future motion from actual executed motion and previous generated motion, respectively. Extending only from the generated motion fails to account for physical feedback, which may lead to falls. In contrast, extending only from the executed motion introduces discontinuities, resulting in jitter. Our method addresses both issues by incorporating physical feedback while avoiding discontinuities.
the target, Human fails to pick up the box, while BiBo performs human-like locomotion and completes the task. In (d) and (e), BißBo accurately follows the text prompt to perform the strike and raise hand actions. For (f) and (g), BiBo achieves higher control precision compared with CLoSD.
We further visualize how three strategies for extending future motion respond environmental feedback: (1) relying only on previously generated but not executed motion, (2) relying only on previously executed motion, and (3) integrating both by using LDM. In the experiment, place desk in front of the agent, and instruct it to raise a hand and then slap downward, which lead to hand-desk collision.
As in Fig. 6 for (1), the generated future motion (represented by the red balls) ignores the desk collision, continuing drive the hand downward and ultimately causing the agent lose balance. For (2), the generated future trajectory (frame 2 n) exhibits discontinuous jumps relative to the preceding motion (frame < n), producing jitter that bounces the hand off the desk surface. By comparison, our method in (3) preserves motion continuity between frames n 1 and while adapting to physical collision by gradually redirecting the hand upon desk surface frame + 3. This smooth transition reduces jitter and allows the hands to maintain contact with the table.
CONCLUSION
We introduce BiBo, framework that empowers off-the-shelf Vision-Language Models to control humanoid agents. Our key insight is that VLMs can control humanoid agents without costly data collection task-specific training. To achieve this, BiBo comprises two novel components, an em- bodied instruction compiler that compiles high-level instructions into executable commands, and diffusion motion executor that generates motions consistent with the physical environment. Exper- iments show that BiBo achieves high success rates across multiple task designs and maintains high text-to-motion fidelity while performing complex interactions.
Limitations and Future Work. First, our executor is trained on a cext-to-motion dataset of limited size, which may restrict generalization capability. With the availability of larger-scale motion datasets (Lin et al.) 2023 Fan et al.] 2025), there is potential to further enhance robustness and generalization. Second, while our model incorporates environmental feedback through motion exe- cution results, explicitly modeling environmental geometry
height maps (Cen et al. 2024) or basis point set (Yi et al. 2024) features remains an important direction for future exploration. Third, focus human framework to broader interaction modes, such as hand-object interactions (Chao et al. 2018) and human-human interactions (Liang al. 2024). We leave these directions future studies.
interactions in this paper, but there is potential to extend

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (92570101).
REFERENCES
Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. Medec: A benchmark for medical error detection and correction in clinical notes. arXiv preprint vrXiv:2412.19260 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprini urXiv:2303.08774 2023.
Anthropic. The claude 3 model family: Opus, sonnet, and haiku. Technical report, Anthropic, 2024. URL http://wwwwwaaaoooooooo/o//////a 3-aidd-- Accessed: October
www.
2025.
Andreas Aristidou and Joan Lasenby. Fabrik: fast, iterative solver for the inverse kinematics problem. Graphical Models, 73(5):243-260, 2011.
Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Giil Varol. Teach: Temporal action composition for 3d humans. In 2022 International Conference 3D Vision (3DV), pp. 414-423
IEEE, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint a3X2v:250..13923, 2025.
Bharat Lal Bhatnagar, Xianghui Xie, Ilya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pp. 15935- 15946, 2022.
Xiaoyu Bie, Wen Guo, Simon Leglaive, Lauren Girin, Francese Moreno-Noguer, and Xavier Alameda-Piin Pineda. Hit-dvae: Human motion generation via hierarchical transformer dynamical vae. arXiv preprint arXiv:2204.01565 2022.
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, al. Gr0Ot nl: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.1 14734, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. no. vision- lanoon-lnguagg-attee flow model for general robot control. arXiv preprint
2024.
Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y Galliker, et al. Wo.5: vision- language-action model with open-world generalization. In 9th Annual Conference on Robot Learning. 2025.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from single image. In Computer Vision ECCV 2016, Lecture Notes in Computer Science. Springer International Publishing, October 2016.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real- control scale. arXiv preprint arXiv:22/2.06811 2022.
10

---

Published as a conference paper at ICLR 2026
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-langie language- aagtinnccttionn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818 2023.
Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang, Shuai Zhu, Hujum Bao, and Xiaowei Zhou. Generating human motion in 3d scenes from text descriptions. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 1853-1866, 2024.
Joongwon Chae, Zhenyu Wang, Lian Zhang, Dongmei Yu, and Peiwu Qin. Grid- audd-augmented vision: A simple yet effective approach for enhanced spatial understanding in multi-modal agents. arXiv preprint arXiv:2411. 1. 18270, 2024.
Yu- Yei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human- object interactions. In 2018 ieee winter conference on applications of computer vision (wacv), pp. 381-389. IEEE, 2018
Kerui Chen, Jianrong Zhang, Ming Li, Zhonglong Zheng, and Hehe Fan. Clusterstyle: Modeling intra-style diversity with prototypical clustering for stylized motion generation. arXiv preprint arXiv:2512.02455. 2025.
Rui Chen, Mingyi Shi, Shaoli Huang, Ping Tan, Taku Komura, and Xuelin Chen. Taming diffusion probabilistic models for character control. In ACM SIGGRAPH 2024 Conference Papers, pp- 1-10, 2024.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18000-18010, 2023.
Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
Wenxun Dai, Lin Chen, Jingho Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real-time controllable motion generation via latent consistency model. In European Conference on Computer Vision, pp. 390-408. Springer, 2024.
Jacob Devlin, Ming- Weig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Pengxiang Ding, Jianfci Ma, Xinyang Tong, Binghong Zou, Xinxin Luo, Yiguo Fan, Ting Wang, Hongchao Lu, Panzhong Mo, Jinxin Liu, et al. Humanoid-vla: Towards universal humanoid control with visual integration. arXiv preprint (rXiv:2502.144795 2025.
Ke Fan, Shunlin Lu, Minyue Dai, Runyi Yu, Lixing Xiao, Zhiyang Dou, Junting Dong, Lizhuang Ma, and Jingbo Wang. Go to zero: Towards zero-shot motion generation with million-scale data. arXiv preprint arXiv:2507.0000 2025.
Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song. Ashish Kapoor, Karol Hausman, et al. Foundation models in robotics: Ap- plications, challenges, and the future. The International Journal of Robotics Research, 44(5): 701-739, 2025.
Canxuan Gang. Strong and controllable 3d motion generation. arXiv preprint arXiv:2501.18726, 2025.
Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou. Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Action2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM international conference on multimedia, pp. 2021-2029, 2020.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang. Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5152-5161, June 2022a.
11

---

Published as a conference paper at ICLR 2026
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5152-5161, 2022b.
Chuan Guo. Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Gener- ative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1900-1910, 2024.
Bo Han, Hao Peng, Minjing Dong, Ren, Yixuan Shen, and Chang Xu. Amd: Autoregressive motion diffusion. In Proceedings of the AAAI Conference Artificial Intelligncee volume 38, pp. 2022200222222222 2030, 2024.
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In Proceedings of the International Conference on Computer Vision 2021, October 2021.
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scems interactions. In ACM SIGGRAPH 2023 Conference Pro- ceedings, pp. 1-9, 2023.
Jonathan Ho and Tim Salimans. arXiv:2207.12596 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Seokhyeon Hong, Chaelin Kim, Serin Yoon, Junghyun Nam, Sihun Cha, and Junyong Noh. Salad: Skeleton-aware latent diffusion for text-driven motion generation and editing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7158 7158-7168, 2025.
Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, et al. Chat-scene: Bridging 3d scene and large language models with object identifiers. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024a.
Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Conference Machine Learning (ICML), 2024b.
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16750-16761, 2023.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion foreign language. Advances in Neural Information Processing Systems, 36:20067-20079, 2023.
Lei Jiang, Ye Wei, and Hao Ni. Motionpem: Real-time motion synthesis with phased consistency model. arXiv preprint arXiv:2501. 19083, 2025.
Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1737-1747, 2024.
Aliasghar Khani, Arianna Rampini, Bruno Roy. Larasika Nadela, Noa Kaplan, Evan Atherton, Derek Cheung, and Jacky Bibliowicz. Motion generation: A survey of generative approaches and benchmarks. arXiv preprint vrXii:2507.054/0 2025.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint 0022444406009246 2024.
12

---

Published as a conference paper at ICLR 2026
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion- multi- human motion generation under complex interactions. International Journal of Computer Vision, 132(9):3463-3444 2024.
Jing Lin, Ailing Zeng. Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36:25268 252280, 2023.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pp. 740-755. Springer, 2014.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1 11:::444:122:::66
October 2015.
Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. Dynamics-regulated kinematic policy for egocentric pose estimation. Advances Neural Information Processing Systems, 34:25019- 25032, 2021.
Zhengyi Luo, Ye Yuan, and Kris M Kitani. From universal humanoid control to automatic physically valid character creation. arXiv preprint 1XXi::2206009866 2022.
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real- time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10895-10904, 2023.
Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision- language-action models for embodied ai. arXiv preprint arXiv:2405. 24t.:2200..4093,, 14093, 2024.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF interna- tional conference computer vision, pp. 5442-5451, 2019.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021.
Khaled Mamou, E Lengyel, and A Peters. Volumetric hierarchical approximate convex decomposi- tion. Game engine gems, 3:141-158, 2016.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 193, 2023.
Liang Pan, Jingbo Wang. Buzhen Huang, Junyu Zhang, Haofan Wang. Xu Tang, and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes. In 2024 International Conference on 3D Vision (3DV), pp. 1498-1500 IEEE, 2024.
Liang Pan, Zeshi Yang. Zhiyang Dou, Wenjia Wang. Buzhen Huang, Bo Dai, Taku Komura, and Jingbo Wang. Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization. arXiv preprint 2an::22033.9991 2025.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics- physics-based character control. ACM Transactions on Graphics (ToG), 402 2021.
13

---

Published as a conference paper at ICLR 2026
Mathis Petrovich, Michael J Black, and Güil Varol. Action-conditooned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10985-10995, 2021a.
Mathis Petrovich, Michael J. Black, and Gül Varol. Action-conditioned 3D human motion synthesis with transformer VAE, In International Conference Computer Vision (ICCV), 2021b.
Ekkasit Pinyoanuntapong, Pu Wang, Minwoo Lee, and Chen Chen. Mmm: Generative masked motion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1546-1555, 2024.
Abhinanda Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramires and Michael J Black. Babel: Bodies, action and behavior with english labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 722-731, 2021.
Zhangyang Qi, Zhixiong Zhang, Ye Fang, Jiaqi Wang, and Hengshuang Zhao. Gpt4scene: Un- derstand 3d scenes from videos with lannual models. arXiv preprint arXiv:5501.01228, 2025a.
Zhangyang Qi, Zhixiong Zhang, Yizhou Yu, Jiaqi Wang, and Hengshuang Zhao. Vln-r1: Vision- language navigation via reinforcement fine-tuning. arXiv preprint arXiv:2506. 17221, 2025b.
Alexander Raistrick, Lahav Lipson, Zeyu Ma, Lingjie Mei, Mingzhe Wang, Yiming Zuo, Karhan Kayan, Hongyu Wen, Beining Han, Yihan Wang, Alejandro Newell, Hei Law, Ankit Goyal. Kaiyu Yang. and Jia Deng. Infinite photorealistic worlds using procedural generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12630-12641, 2023.
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint Xii:220010026100 2022.
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a gener- ative prior. arXiv preprint arXiv:2003.014/8 2023.
Wentao Shi, Xiangnan He, Yang Zhang, Chongming Gao, Xinyue Li, Jizhi Zhang, Qifan Wang. and Fuli Feng. Large language models are learnable planners for long-term recommendation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1893-1903, 2024.
Ayumi Shiobara and Makoto Murakami. Human motion generation using wasserstein gan. In Proceedings of the 2021 5th International Conference on Digital Signal Processing, pp. 278- 282, 2021.
Han Shu, Wenshuo Li, Ychui Tang, Yiman Zhang, Yihao Chen, Houqiang Li, Yunhe Wang, and Xinghao Chen. Tinysam: Pushing the envelope for efficient segment anything model. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 20470-20478, 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint orXiv:2010.02502 2020.
BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, et al. Robobrain 2.0 technical report. arXiv preprint arXiv:2507.02029, 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen- Or, and Amit Bermano. Human motion diffusion model. arXiv preprint 2rXiv:2209.14916 2022.
Guy Tevet, Sigal Raab, Setarch Cohan, Daniele Reda. Zhengyi Luo. Xue Bin Peng, Amit H Bermano, and Michiel van de Panne. Closd: Closing the loop between simulation and diffu- sion for multi-task character control. arXiv preprint arXiv:2410.03444 2024.
14

---

Published as a conference paper at ICLR 2026
Wenjia Wang, Liang Pan, Zhiyang Dou, Zhouyingcheng Liao, Yuke Lou, Lei Yang, Jingbo Wang, and Taku Komura. Sims: Simulating human-scene interactions with real world script planning. arXiv preprint arXiv:2411.19921, 2024a.
Zehan Wang, Ziang Zhang, Tianyu Pang, Chao Du, Hengshuang Zhao, and Zhou Zhao. Orient anything: Learning robust object orientation estimation from rendering 3d models. arXiv preprint arXiv:2412. 18605, 2024b.
Lixing Xiao, Shunlin Lu, Huaijin Pi, Ke Fan, Liang Pan, Yueer Zhou, Ziyong Feng, Xiaowei Zhou, Sida Peng, and Jingbo Wang. Motionstreamer. Streaming motion generation via diffusion-bart autoregressive model in causal latent space. arXiv preprint arXiv:2503. 15451, 2025.
Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiang- miao Pang. Unified human-scene interaction via prompted chain-of-contacte arXiv preprint arXiv:2309.07918 2023.
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. arXiv preprint 10.00800, 2023.
Sirui Xu, Zhengyuan Li, Yu-Xin Yu-Xiong Wang, and Liang- Yan Gui. Interdiff: Generating 3d human- object interactions with physics-informat diffusion. In Proceedings of the IEEE/CVF Interna- fional Conference on Computer Vision, pp. 14928-149444 2023.
Xinyu Xu, Yizheng Zhang. Yong-Lu Li, Lei Han, and Cewu Lu. Humanvla: Towards vision- language directed object rearrangement by physical humanoid, 2024. URL https: org/abs/ 6406. 19972
Haoru Xue, Xiaoyu Huang. Dantong Niu, Qiayuan Liao, Thomas Kragerud, Jan Tommy Gravdahl, Xue Bin Peng, Guanya Shi, Trevor Darrell, Koushil Sreenath, et al. Leverb: Humanoid whole- body control with latent vision-language instruction. arXiv preprint arXiv:2506. 1375/, 2025.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.1 1441, 2023.
Heyuan Yao, Zhenhua Song. Yuyang Zhou, Tenglong Ao, Baoquan Chen, and Libin Liu. Moconvq: Unified physics-based based motion control via scalable discrete representations. ACM Transactions on Graphics (TOG), 43(4):1-2) 2024.
Hongwei Yi, Justus Thies, Michael J Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. In European Conference Computer Vision, pp- 246-263 Springer, 2024.
Weihao Yuan, Weichao Shen, Yisheng HE, Yuan Dong, Xiaodong Gu, Zilong Dong, Liefeng Bo, and Qixing Huang. Mogents: Motion generation based on Neural Information Processing Systems (NeurIPS). 2024.
joint modeling. In
Ye Yuan, Jiaming Song. Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guiee guided human motion diffusion model. In Proceedings of the IEEE/CVF international conference computer vision, pp. 16010-16021, 2023.
Jianron Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14730-14740, 2023.
Lvmin Zhang and Maneesh Agrawala. Packing imput frame context in next-frame prediction models for video generation. arXiv preprint arXiv:2504. /2626, 2025.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE transac- fions on pattern analysis and machine intelligence, 4615):4155-4128, 2024.
15

---

Published as a conference paper at ICLR 2026
Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, et al. A survey on vision-late laaoon-lnnaag-aattion models: An action tokenization perspective. arXiv preprint arXiv:2507.01925, 2025.
Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, and Alois C Knoll. Opendrivevla: To- wards end-to-end autonomous driving with large vision language action model. arXiv preprint arXiv:2505.234633. 2025.
Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion under- standing planning generation and beyond. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 1357-1366, 2024.
Wenjie Zhuo, Fan Ma, and Hehe Fan. Infinidreamer: Arbitrarily long human motion generation via segment score distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14688-14698, 2025.
16

---

Published as a conference paper at ICLR 2026
Supplementary Materials for Endowing GPT-4 with Humanoid Body: Building the Bridge Between Off-the-Shelf VLMs and the Physical World
CONTENTS
Introduction
2
Related Work
2.
Human Scene Interaction
Text to Motion Generation
3
Method
Overview
3.2
Embodied Instruction Compiler
Motion Diftusion Executor
Experiments
Task Completion
4.2
Motion Quality
5
Conclusion
Supplemantary Related Work
20
Motion Diffusion Model
20
A.2
Large Visior-Langugee-Ation Model
................................
20
A.3
Human Scene Interaction
20
A.3.1
Physical and Non-Phy HSI Tasks
A.3.2
RL Policy vs. Diffusion Generator LLM's Executor
20
A.4
Arbitrary Length Motion Generation
Datasets
Random Generated Scene Dataset
21
B.I.I Statistics
21
1,2
Scene Construction
B.
1.3
Task Generation
23
MI.3n
25
B.2.J
Statistics
25
B.2.2
Motion Representation
25
B.2.3
Metrics Definition
25
17

---

Published as a conference paper at ICLR 2026
B.3
ScanNet ScenePlan
27
B.3.1
Scene Format
27
B.3.2
Task Format
28
B.4
AMASS
28
B.4.1
Statistics
28
B.4.2
Implementation
28
B.5
BABEL
28
B.5.1
Statistics
28
B.5.2
Implementation
29
Embodied Instruction Compiler
29
Online Control Loop
VLM-based Action Planning Module
30
C.2.1
Off-the-sheli Vision Language Model
30
C.2.2
Details of Basic Attribute Analysis
31
C.2.3
Details of Agent Pose Reasoning
31
C.2.4
Details of Key Joint Generation
33
C.2.5
Multiple Action Merging
34
C,2
Rule-based Reflection
36
C.3
Humanoid Agent State Machine
37
C.3.1
Navigation State
37
3,2
2
Interaction State
37
C,4
Navigation Module
38
C.4.1
Mapping Module
38
C.4.2
Path Planning with Modified A
38
D
Diffusion-based Motion Executor
39
D.I
Diffusion Preliminary
39
D.2 Diffusion Architecture
39
D.3 VAE Architecture
40
D.4 Training Details
40
D.5 IK Post Optimization
40
D.6 Tracking Policy
Comparison Methods
41
Task Completion
E.I.I
UniHSI
41
Human VLA
E.1.
TokenHSI
42
18

---

Published as a conference paper at ICLR 2026
E.1.4
CLoSD
42
E.1.5
Ablation Study
43
E.2
Motion Quality
43
MotionL
43
E.2.2
MotionStreamer
44
1.2.3
MoGinTTS)
44
E.2.4
MoConVQ
44
E.2.5
Dip & CLoSD
44
E.2.6
Double Take
45
E.2.7
Ablation Study
45
Supplementary Experiments
45
Rendering
45
Variants of VAE
45
Variants of Off the shelf VLMMs
46
Accuracy of the Stree-Stage VQA
47
Supplementary Ablation Study
48
F.6
Control Accuracy
48
Hyperparameter Selection
49
F8
Motion Prefix
49
F.
Case Study
51
10
Design of User Study
51
F.1
Additional Comparison Methods
................................
52
E11.1
Non-physical Human Scene Interaction
................................
52
F.1.2 RL-based Methods
................................
52
12
Length of The Previous Motion Context
................................
54
Evaluation on Robotic Platform
54
F13.1 Migrating to humanoid robot body
54
F.1.22
Real-world Deployment of Embodied Instruction Compiler
55
Additional Scene Interaction Evaluation
56
E.141
Additional Benchmark
56
1.1222
Interaction Accuracy
57
1..3
Interaction Naturalness
57
14.4
Interaction Coherence
57
F.15
Additional Generation Quality Evaluation
58
F.15.1
Generalization to AMASS
58
E,152
Generalization to BABEL
58
16
Real-time Performance
59
19

---

Published as a conference paper at ICLR 2026
A SUPPLEMANTARY RELATED WORK
A.1 MOTION DIFFUSION MODEL
Diffusion (Ho et al. 2020; Song et al. 2020) has emerged as a powerful generative framework for motion synthesis (Khani et al. 2025). It improves generation diversity compared to variational au- toencoders (VAEs) (Petrovich et al. 202 and generative adversarial networks (GANs) (Shiobara & Murakami 2021 while being more efficient than GPT-like next-token prediction (Zhang et al.) 2023) and BERT-like masked modeling (Guo et al., 2024) methods. While pioneering works di-
rectly denoised Gaussian noise into motion sequences (Tevet al., 2022 Liang al.] 2024 Yi
et al. 2024), recent approaches extend diffusion with gradient guidance (Xie et al. 2023) ) and Con-
trolNet (Gang 2025) for controllability, latent diffusion models (LDMs) (Xiao et al.) 2025 Hong et al., (2025) for enhanced motion quality, and flow matching (Consistency Models) (Dai et al.
2024
Jiang et al. 2025) for faster inference. In this work, we adopt LDM architecture with Classifter Free Guidance and few-step denoising, supporting real-time control while achieving high-fidelity.
A.2 LARGE VISION-LANGUAGE-ACOA
MODEL
Large Vision Language Models (VLMs) have demonstrated cross-domain generalization capabil-
ity (Achiam et al. 2023 Team et al. 2023 Bai al. 2023), enabling them to perform action plan-
ning for embodied agents conditioned on environmental context and user instructions (Yao et al., 2024 Xiao et al. 2023) Wang et al.) 2024a). Among these methods, Vision Aision-Language-Aee
(Ma) et al. 7224gnhoggg et al. (025) attracts extensive research attention. They typically learn action to- ken by finetuning VI.M on collected action data (Kim et al., 2024) Black et al.) 2024) 2025), thereby driving low-level action executor. Unlike low degree of freedom (DoF) platform (e.g. robot arms, vehicles) (Brohan et al., 2022 2023; Zhou et al.) 2025), humanoids process higher dimensional ac- tion space, requiring strong executor and extensive finetuning data (Bjorck et al. 2025 Ding al.) 2025). BiBo explores an alternative approach to bypass finetuning for action tokens. It employs an embodied instruction compiler that guides off-the-shel VLM output structured action com- mands. These commands drive an executor trained on open-source human motion dataset, thereby performing diverse scene interaction.
A.3 HUMAN SCENE INTERACTION
A.3.1 PHYSICAL AND
HSI TASKS
Physical HSI methods (Hssann et al. 2021 Yi et al., 2024 Cen et al.| 2024; Jiang et al. 2024] Huang et al. [0233 Xu et al.) 2023) focuses on generating motions that are consistent with the scene. During execution, they directly set joint positions or rotations without involving physical feedback. In this way, the generated and the executed motions the same, but not physical plausible. The quantitative evaluation of non- non-physical HSI involves generation quality, control accuracy, and scene consistency, Physical HSI methods (Hassan et al. 2023) Pan et al., 2024 [aang et al.) 2024a Xiao
et al. 2023 Tt al. 2024 Pan et al.] focus on both motion- scene consistency and physical plausibility. During execution, they apply torques to the joints, and the final executed motion is influenced by physical environmental feedback, which is physical plausible may differ from the generated one. In addition the non-physical evaluation, it also includes the task success rate. BiBo is a physical HSI system. According to the experiments in Sec. BiBo achieved plausible results in both generation quality and task success rate.
A.3.2 RL POLICY vs. DIFFUSION GENERATOR AS LLM'S EXECUTOR
RL Policy (Hassan et al. 2023 Xu al. 2024 Pan al. 2025) offers physically plausible sence interaction and high success rate on specific tasks. Existing methods typically prompt LLM to select from a fixed skill set (Xiao et al. 2023 Wang et al. 2024a) or rely on a predefined action template, which makes it difficult to scale to more diverse motions (e.g.. dance, boxing). Diffusion generator can synthesize a wide variety of sighfiidliity motions (Xu et al.
2023 Yi et al.
2024 Tevet et al.)
2024). Beyond skill-set based planning, recent approaches attempt to let the LLM generate motion 2024). However,
VQ tokens (Yao et al, 2024 or textual motion caption (Cen et al.) 2024 Zhou et al.
few methods consider the surrounding scene support physical interaction. BiBo uses VLM to
20

---

Published as a conference paper at ICLR 2026
guide diffusion generator, and the generated motions are tracked by an RL policy. This method generates diverse and natural motions, while maintaining scene awareness and physical plausibility.
A.4 ARBITRARY LENGTH MOTION GENERATION
Arbitrary length motion generation (Chen al. 2024 Han al. 2024 involves producing motion sequences that can extend indefinitely, commonly applied in fields like robotics and computer ani- mation. Arbitrary-length motion generation methods be roughly divided into two types: segment composition and future extension.
Segment composition methods (Athanasiou et al.) 2022 Shafir et 2023 Zhuo et 2025 first generate all key motion segments, and then connect them by generating transitions. However, com- pared with future extension, these approach may have limitations for motions exceeding the segment length limit, and cannot change control signals in real-time.
Future extension methods predict the next segment based on the previous one. They include extend- ing from a static pose (Hassan et al. 2021 Yi et al. 2024), fixed-length context (Shafir et al.. [023] Zhuo et al., 2025), and autoregressive generation (iiao al.) 2025). Extending from a static pose begins with single pose and generates future motion, but this introduces gaps due to the lack of dynamic context. Using a fixed-length context balances efficiency and continuity, but is limited by the context size. Autoregressive generation, which relies previously generated frames, captures long-range context and ensures smooth continuity, though it less efficient.
Arbitrary length physical motion TTevet et al. 2024) generation introduces additional challenge, as the generated and executed motions may not be consistent. Extending from generated motion cannot adapt physical feedhacks, while extending from executed motion introduces a gap between the previous and current generated results. BiBo adopts a lightweight fixed-length context to ensure real-time performance. extends from both generated and executed motion, enhancing continuity and physical adaptation.
B DATASETS
B. RANDOM GENERATED SCENE DATASET
B.1.1 STATISTICS
The scene dataset comprises 100 scenes spanning diverse room types and layouts, including 50 living rooms and 50 bedrooms. The floor areas range from 12.30 to 99.199... 18m2 with an average of 49.12m², featuring various floor plan geometries as illustrated in Fig. 7 The objects include diverse categories such as ornaments (e.g., plant containers, trinkets), containers (e.g., shelves, cabinets), tables (e.g., TV stands, desks), planar surfaces (e.g.. monitors, wall panels), as well as furniture (e.g., sofas, beds) and miscellaneous items. The distribution of these categories is shown in Fig. 4
Each scene contains 6-18 single-interaction tasks and 1-3 composite tasks, resulting in a total of 1,365 single tasks and 162 composite tasks. The single-interaction tasks include 297 sit, 150 sleep, 282 touch, 367 reach, 177 watch, and 294 lift tasks. Notably, reach tasks are also embedded within other interaction tasks improve testing efficiency. The composite tasks comprise 68 simple, 52 medium, and 42 hard cases, with the length distributions shown in Fig. 4 Examples of tasks are shown in Tab. 7 The evaluation criteria for task success are described in Sec. 4.1 The comparison with existing datasets is shown in Tab,
B.1.2 SCENE CONSTRUCTION
The scenes are generated using InfiniGen!] (Raistrick et al. 2023), which produces Blender-format blennd scene files. For each generated scene, all light sources and cameras removed to ensure a consistent simulation environment. Each object in the scene, including walls and furniture, is then exported
obj mesh file representing its visual geometry, and decomposed into convex collision bodies using the VHACD algorithm (Mamou et al. 2016). The visual meshes and their cor- responding collision bodies are subsequently organized into a unified urdf asset file. These URDF
hutps//infnigen.orr/
21

---

Published as a conference paper at ICLR 2026
Figure 7: Visualization of the randomly generated scenes and corresponding floor plans. The top two rows show bedroom scenes, and the bottom two rows show living room scenes. In the scene images, each object is annotated with red label. In the floor plans, light gray regions denote the lable &
while dark gray regions represent the objects. Comparison of the
dataset bewteen different HSI methods. We compare the scal- ability of Object (Obj.), Scene (Sec.), and Task, the variety and quantity of Object (O.), Interaction (I.), and Layout (L.), as well as the maximum step count and whether planning during evaluation (Plan.). The proposed evaluation method is diverse and scalable.
Method
Dataset
Scalability
Diversity 1
Difficulty 1
Ohj.
Type
Type
Dift.
Type
Step
VLA
HTTR
23
76
63
UniHSI
ScanNet
448
100
CLoSD
Multitask
BiBo (ours)
Ran. Gen.
/
73
6146
100
100
1527
6
15
files, along with metadata such object positions and orientations, imported into IsaacGym Pre- view to construct the physical simulation environment. The initial position of the humanoid agent dynamically determined runtime introduce spatial diversity and prevent initialization bias.
The scene and object generation process follows InfiniGen's default indoor generator, with minor modifications to facilitate task construction. Specifically, the footrests are removed from sofas and blankets are removed from beds. Object extraction is conducted through the Blender Python API 4. and all meshes are simplified fewer than 10,000 faces to reduce computational overhead. The simplified meshes are subsequently processed using Trimesh for vertex filtering and mesh repair, ensuring watertight topology and the removal of abnormal geometries.
During the import process, since the built-in VHACD and SDF collision modules in IsaacGym ex- hibit limitations in geometric approximation and compatibility, we employ PYVHACI to manually
"httss//doos..bendeee hwwpso//eweo.ooooooooooooeeeeone "hatss//trimeshhorry http:///httt..ccmmooooo/////y/at
22

---

Published as a conference paper at ICLR 2026
Listing 1: Example of task ISON schema. The first object name does not contain index, indicating that it can correspond to any bookstack or bookcolumn in the scene. It contains an (*), indicating that the target object of the last interaction can be the same as that of the first interaction. The two intermediate interaction objectives are placed in the same second-level array, representing that they must be achieved simultaneously.
"prompt": "Grab book, then sit en the couch whi turning on the light, and finally place the book somewhere.", "mi "mission":
II [["ebject"; "bookx", "type" "touch" HI, [("ebject"; "sofal", "oype"""sit""" I"object" "Lammp1""" "lamp1", "type": [[ """bjjec"; "book", "touch"]""
"touch")
construct the collision bodies. Given an object mesh as input, VHACD decomposes into up to 64 convex hulls that approximate the object's external geometry. Each convex hull is further ab- stracted by its axis-aligned bounding box (AABB), which is encoded into the URDF as geometry box element define the collision shape. Decorative items such as trinkets and plants are defined as dynamic assets, with a density of approximately 50kg/m², comparable to wooden boxes. Large furniture such shelves and walls are defined as static assets stabilize the simulation and reduce computational cost.
To determine the humanoid agent's initial position during evaluation, Shapely is used construct 2D polygons representing the floor and scene objects. After applying a uniform padding operation to prevent boundary collisions, random point located inside the floor polygon but outside all object polygons is sampled as the agent's 'ss starting position. This ensures valid initialization across diverse layouts while preventing overlap with scene geometry.
B.1.3 TASK GENERATION
Task Format. All tasks are stored in JSON jsson format, shown in Lst. | It contains two fields: prompt and mission. The prompt field natural language instruction of the task, and the mission field is two-level array of interaction objectives. The first level represents interac- tions that must be completed sequentially, and the second level represents interactions that can be completed simultaneously. Each interaction objective includes the target object name target and the interaction type type. There are five available interaction types, including /watch, sit, sleep, touch, lift). Note that the reach task is implicitly included in these interaction types, since the agent should first navigate to the target object before performing the corresponding interaction.
The object name follows the format category [index] [+]. The category component is required and specifies the category the target object, wildcard matching all unvisited objects of that type in the scene. The [index] component is optional and designates specific object instance. The [*] component is also optional and indicates that, after this interaction, the object will not be excluded from subsequent wildcard matching.
Single Interaction. We use rule-based program to construct single-interaction tasks. For each interaction type, the program queries the scene for eligible interactive objects and generates responding prompts accordingly. Specifically, the target objects for the watch task are directional planar surfaces such as TV screens or wall paintings, with example prompts like "watching TVI." " The sit task targets seating furniture such as sofas or beds, e.g., "sitting sofal." The sleep task targets beds. The touch task involves ornamental objects in the room, with example prompts such as "grab book from bookstack]" " or "turn lampl." " The lift task targets large ornaments containers, with prompts like "lifting large plant containerl."
Composite Task. The composite tasks are constructed using a semi-automatic pipeline, where volunteers manually annotate 5 simple, 3 medium, and hard task examples. The remaining tasks
23

---

Published as a conference paper at ICLR 2026
Box 1. Prompt for Composite Task Generation
SYSTEM*:
You are an intelligent task generator. Your goal is to create interaction tasks that humanoid agent can perform within a given scene.
Each task should be in JSON format with two fields:
prompt: natural language instruction of the task, can be both abstract and concrete
mission: two-level array of interaction objectives. The first level is sequential, while the second level is simultaneous.
Each interaction objective contains two fields: type: one of [watch, sit, sleep, touch, lift]
target: name of target object coategory[index[["" particular object (optional), omit match all uninteracted objects of that category. [*] indicates this object should not be omitted in subsequent matching.
"category" is required. [index] specifies
Task difficulty criteria:
simple: contain < 4 steps (including navigating to another object between two interactions, e.g., "sit sofal and then sit on sofa2" steps)
medium: 4-10 steps, or contains dynamic object manipulation (e.g., lift, transport) hard: 10 steps, contains simultaneous interactions with multiple objects (e.g., sit sofa and put a hand on the side table)
USER#: Example of simple task:
[examples]
Example of medium task:
[examples]
Example of hard task:
[examples]
The multi-view scene images are provided, each image contain object labels, corre- sponding to:
1: sofa (-0.4, 5.1)
2: desk (-0.2, 3.8), containing: [nature shelf trinket1]
- 3: monitor (1.1, 5.3)
You are currently generating an simple task, with abstract prompt and 2 interactions. No simultaneous interaction. No dynamic object manipulation. Please analyze before outputting the final task, and enclose your final answer in >>> and <<<.
are then generated by GPT-4o based on the provided examples and scene information. All generated tasks are manually reviewed to ensure that they are achievable. The task difficulty categorized according to the criteria described in Sec. 4.1
Composite tasks are composed of multiple single interactions that occur either sequentially or con- currently. Compared with single interactions, they feature more diverse interaction patterns, such as "laying on the coffee table", "leaning on the bookshelf". or "sitting while turning on the lamp".
During task generation, the GPT-4o is provided with several components: example tasks, Blender- rendered multi-view scene images with object labels, and list of objects with their corresponding coordinates and parent-child relationships. To ensure a balanced difficulty distribution, random number generator instead of GPT-4o to determine task difficulty. Specifically, we explicitly specify the task difficulty to be generated and the corresponding criteria to be satisfied.
To capture multi-view scene images, we first compute the positions of all wall corners based on the floor polygon. Cameras are then placed along the biscctors of these corners height of 2 m, capturing images with 30 downward tilt and a 75 field of view (FOV). For placing object labels, the point cloud of each object is first projected onto the camera plane to generate mask. Then
24

---

Published as a conference paper at ICLR 2026
OpenCV to compute the distance from each pixel within the mask to its boundary, and the pixel with the maximum distance is selected as the center of the object label.
Based on these inputs, it outputs tasks in JSON format. The generated JSON file is then parsed and automatically validated by querying the scene to check (1) whether the target objects exist, (2)
whether the interaction type belongs to the predefined interaction list, and (3) whether the coordi- nates of simultaneously contacted target objects are within a Im distance. Finally, all generated tasks are manually reviewed ensure correctness and executability. The prompt is shown in Box, [
B.2
HUMANML3D
B.2.1 STATISTICS
We train the diffusion-based motion exccutor the HumanML3D dataset (uo et al. 2022b which comprises 14,616 human motion sequences paired with 44,970 natural language descriptions. HumanML3D covers everyday activities, spatial interactions, and complex body dynamics. The motion data are extracted from HumanAct12 (Guo et al. 2020) and AMASS (Mahmood et al. 2019). augmented through mirroring to double the dataset size, and normalized into 263-dimensional relative-coordinate motion representation.
The dataset is divided into 24,843 motion sequences for training and validation and 4,382 for testing. After selecting motion sequences with lengths ranging from 40 to 200 frames, a total of 24,545 mo- tion episodes paired with 66,633 captions are used for training and validation, while 4,646 episodes paired with 12,536 captions are reserved for testing.
B.2.2 MOTION REPRESENTATION
We follow Tevet et al. 2022), each motion episode is represented as a sequence of joint positions, rotations and velocities in 3D space, sampled at 20 frame per second (FPS). Each motion frame x E R encodes a single pose and is defined as:
I = (fai Pz: 1:z, jr.
where ra is the root angular velocity around the Z-axis; fu and rz are root linear velocities in the XY- plane; ry is the root height. The joint features include local joint positions jp R33(-1), rotations j. E R)((-1), and velocities j. R all defined relative to the root. Additionally, f e IR denotes binary foot contact indicators for four foot joints (two per leg).
B.2.3 METRICS DEFINITION
We evaluate the generative capability of BiBo across four dimensions: 1) motion quality. 2) physical plausibility, 3) generation efficiency, and 4) controllability. Specifically, motion quality is measured by Fréchet Inception Distance (FID), Precision, and Diversity. Generation efficiency assessed using Average Inference Time per Sequence (AITS). Physical plausibility is evaluated with Penetra- tion, Float, and Skate. Controllability is measured by Mean Absolute Error (MAE). The generation quality evaluation is implemented based on the codebase of MotionLCM (Dai et al.) 2024), while the physical evaluation is based on CLoSD TTveet et al. 2024).
R Precision. R Precision measures the retrieval accuracy by evaluating whether the ground-truth motion is ranked within the top-k retrieved motions. We compute R Precision over the entire dataset by accumulating embeddings from all batches. The dataset is then divided into groups of size R = 32 (the batch size for Precision calculation). For each group containing R text-motion pairs, we compute the Euclidean distance matrix D(s) E R between text embeddings E($) E Rxd and motion embeddings ((()) E R Rxd,
E
m. Ila.
(5)
where E.9) and E(9) denote the i-th text embedding and j-th motion embedding in group g. respec- tively. For each text query in group g, we rank all motions within the group by their distances and
hhttss//openev.orr/
25

---

Published as a conference paper at ICLR 2026
compute the top-k precision:
R
R-Precision ((s)
1 R
Σ
Irank ( - k).
(6)
where rank(D (g) is the rank of the ground-truth motion for text query within group 9, and [[]]] 3.3 the indicator function. The final Precision is computed as the average over all groups:
is
G
R- R-Precision @k
R-Precision @ k(0))
G
9=1
where G the total number of groups.
FID. Fréchet Inception Distance (FID) measures the distribution distance between generated and ground-truth motions in the embedding space. We compute the mean Mg, Mgt Rd and covariance matrices gg Eyt Raxd for generated and ground-trute motion embeddings, respectively. The FID is then defined as:
FID |g Mgt 1 Tr(E,, Egg - Egt
(8)
where Tr(-) denotes the matrix trace. Lower FID values indicate better motion quality.
Diversity. Diversity measures the variability of generated motions by computing the average pair- wise distance between randomly sampled motion embeddings. Given N generated motion embed-
dings [e we randomly sample T pairs of indices (kk. without replacement and compute:
Diversity
Σ ci. Cjull2,
where the number of diversity samples (typically T 300). Higher diversity values indicate more diverse motion generation.
AITS. Average Inference Time per Sequence (AITS) measures the generation efficiency by com- puting the average time required to generate complete motion sequence. For each sequence i, we record the inference time & from the start of the diffusion process to the completion of VAE decoding. The AITS is computed as:
AITS Σ
4i
(10)
where is the total number of generated sequences. Lower AITS values indicate faster generation.
Penetration. Penetration measures the depth of ground penetration in generated motions. For each
frame in sequence we compute the lowest joint height (...)) = miny Jjrrj where Ja,.j. g-coordinate (height) of joint frame t in sequence The penetration depth for frame t is:
Pi. max (0. ( + +))
(i.()
1000,
(11)
where 0.005 m is tolerance threshold, and the result is converted from meters to millimeters. The overall penetration metric is the mean over all frames:
Penetration 1
E.
Piçti
(12)
where L; is the length of sequence i. Lower penetration values indicate better physical plausibility. Float. Float measures the floating height of characters above the ground. Similar to penetration, we
compute the lowest joint height (i.t) for each frame. The floating distance for frame t is:
fi, max(0, 1 (8.9 7) x 1000,
(13)
26

---

Published as a conference paper at ICLR 2026
where 0.005 is the tolerance threshold. The overall float metric is:
N
L
Float
fla-
(14)
E.
Lower float values indicate that characters are closer to the ground, which is more physically plau- sible.
Skate. Skate measures foot sliding artifacts during ground contact. For each consecutive frame pair (t, t + 1) in sequence we identify the contact joint j = arg min, Ji..j.y at frame t (the joint with the lowest height). If both frames t and t + have the contact joint below the contact threshold 0.005 m, compute the horizontal sliding distance:
81, = [[] 10 [m.,] J (o.))//m==) /2 x 1000 Jundi S and J. Jettaa ST
(15)
otherwise
where Jusst [s.2]] denotes the x and coordinates of the contact joint, and the result is converted from meters to millimeters. The overall skate metric the mean over all consecutive frame pairs:
14-1
Skate E,(,
- Σ
(16)
Lower skate values indicate less foot sliding, which more physically plausible.
MAE. Mean Absolute Error (MAE) measures the controllability by computing the average absolute emoo between predicted and target joint positions controlled joints. For each controlled joint with active condition mask m,,j E (0, 1), we compute the absolute error:
N
MAE, Σ. m.
m
pred R,
Jeet
1... 1,t.j
(17)
where J.red and [..j the predicted and reference joint positions, respectively, and the L1 norm. The overall MAE is computed as the mean over all controlled joints:
||
denotes
MAE
MAE,: JEJ.
(18)
Jn
where Je the set of controlled joints (e.g., head, wrists, feet, pelvis). Lower MAE values indicate better controllability.
B.3 SCANNET SCENEPLAN
ScanNet ScenePlan (iaoo et al. 2023) consists of 10 scenes from ScanNet dataset, which are real- world scanned data. The benchmark includes simple, 23 medium, and 57 hard task sequences, involving multiple interaction categories, such sit, sleep, and touch. An interaction is considered successful when the character makes contact with the specified object part.
B.3.1 SCENE FORMAT
Each scene ScanNet contains static scene meshes, geometry/instance segmentation, camera poses, and RGB/Depth images. Each geometry segmentation consists of series of points representing part an object. Instance segmentation contains a series of geometry segmentations and object type labels, representing object.
We extract object information from the scene based on the instance segmentation, and exclude non- interactive objects such as walls, ceilings, and floors. Each object is assigned unique instance ID to specify interaction targets in tasks. Following UniHSI, we import the scene's 's mesh as the ground mesh in IsaacGym, and assign consistent physical parameters with UnIHSI.
As the rendering quality of mesh is low, we use original RGB and Depth images during VLM inference. When navigating, mark areas with height greater than 0.1 as obstacle pixels.
27

---

Published as a conference paper at ICLR 2026
B.3.2 TASK FORMAT
Each task in the ScenePlan specifies test seene and provides list of objects and tasks. The object list annotates the interactive objects or waypoints in the scene. Each object contains part annotations (e.g.. for a couch, the seat surface, back, and left arm) and the position for interaction. Waypoints can be seen virtual objects, where the interaction position represents the coordinates of the waypoint.
The task list includes navigation or object interaction. Each task consists of contact list represent- ing simultaneous contact events. A contact event contains five keys: object name, part name, contact joint, whether the object is in contact, and the contact normal. For waypoints, the contact list length I, and all events except for the object name are set to "none", indicating that the agent should move towards the waypoint without interaction.
BiBo uses an off-the-shelf VLM to observe the scene and parse instructions. It can autonomously navigate and generate interaction positions and joint targets. Therefore, we only retain the interac- tive objects in the input, without using the ground truth waypoints or interaction pose annotations. Specifically, assign each object unique IDD (e.g., couchl) when parsing the scene. During execution, we generate interaction instructions based on the object category (e.g., for couchl, the instruction would be "sit couchl").
B.4 AMASS
B.4.1 STATISTICS
AMASS (Mahmood et al. 2019) constitutes a comprehensive and diverse database of human mo- tion, which systematically unifies 25 distinct optical marker-based motion capture datasets through a unified framework and parameterization scheme. This consolidated resource provides immedi- utility for multiple applications, including character animation, motion visualization, and the generation of training data for deep learning architectures.
The AMASS dataset represents substantial advancement
existing human motion reposito- ries, encompassing more than 62.87 hours of motion capture data. The database spans 501 unique subjects performing 18,253 distinct motion sequences, thereby providing unprecedented scale and diversity for human motion research and applications. To the best of our knowledge, we are the first validate motion generation quality using the full AMASSS dataset. The preprocessed AMASS dataset (without mirror augmentation) contains 7,582 motion sequences, totaling 40.0 hours.
B.4.2 IMPLEMENTATION
We download all A subsets (including the latest subsets), prioritizing the SMPLH (Romero ctt al. 2022 10 format, and using the SMPLX (Pavlakos et al. 2019 format when SMPLH is unavailable. Following the preprocessing pipeline from HumanML3D. we first process all motion clips into joint coordinate sequences. Then, we remove prefixes and suffixes motions (typically T-poses) based their similarity the initial/final frames. Finally, segment long clips below 200 frames for evaluation pipeline adaptation.
B.5 BABEL
B.5.1 STATISTICS
BABEL (Punnakkal et al. 2021 is large-scale motion capture dataset that bridges the gap be- tween precise 3D human motion and semantic action understanding by providing hierarchical lan- guage labels for approximately 43 hours of AMASS mocap sequences. The dataset contains 28,000 sequence-leve labels and 63,000 frame-level labels across 250+ action categories, enabling research in action recognition, temporal localization, and motion synthesis.
"https//amass.ir is que.mpg.de.
"https:/maao.is.t http://maaoo.s.uu....... .s..ue...tp................ de/
"httpss//sppl-x -x.is is. wt.tuu.mpg.de. 12 Phuts://hahel.ii due.mp..de/
28

---

Published as a conference paper at ICLR 2026
processes this data and converts it into motion generation dataset with frame-level annotations for both transitions and actions. The dataset is partitioned into 22,923 training motions (20.1 hours) and 8,343 evaluation motions (7.3 hours). The motion representation employs root translation combined with 6D rotations for each joint, encompassing 22 joints and resulting in 135-dimensional feature space.
TEACH (Athanasiou al. 2022 13
BiBo uses a 263-dimensional model, which cannot achieve lossless conversion with BABEL's 135 dimensions. Therefore, we start from the original motion data create 263-dimensional BABEL dataset.
Rather than extracting each transition-action pair into motion segment separately, we greedily select the longest segments from transition start to action end to maximally preserve shorter valid actions. In total, 13,877 text-motion pairs are extracted, amounting 27.4 hours.
B.5.2 IMPLEMENTATION
we first clean invalid motions such as T-poses before and after all motion sequence. Then, we segment motions into clips that start from a transition and end an action. Clips shorter than 40 frames are dropped, while those longer than 200 frames are truncated.
The text annotations in BABEL are brief and homogeneous. We use additional templates concate- nate multiple action captions within the same segment, and employ various predefined templates to enhance text diversity. Finally, the words in the annotated text may not match the vocabulary of the text embedding in evaluation pipeline. We establish a mapping table from annotated text words to vocabulary words based on similarity search, and use an LLM fix incorrect mappings.
C EMBODIED INSTRUCTION COMPILER
C.I ONLINE CONTROL LOOP
The embodied instruction compiler takes as input the user's instruction and the scene observation, and outputs the next structured action command for the executor to perform. To achieve this, Sec. Sec.3.2) introduces a three-stage VQA pipeline designed to progressively fill in the structured action com- mand. In implementation, the embodied instruction compiler can be further divided into three mod- ules: (1) an action Planning Module, which includes the off-the-shelf VLM and the three-stage VQA process; (2) an humanoid agent state machine controlled by the Planning Module; and (3) Navigation Module that provides path-planning services for the state machine. These three mod- ules, together with the executor and the physical environment, form an online control loop for the humanoid agent.
Specifically, the Planning Module invoked when a new user instruction arrives, when the previous interaction is completed, or when 150 frames have elapsed since the module was last invoked. It reads the current user instruction, scene information, and agent state (including all the already exe- cuted action commands and the ongoing action), and decides whether to skip, start new action, or end the current one. When a new action starts, the system applics the three-stage VQA process to generate the action command, which is then passed to the state machine.
The state machine is consist of navigation and interaction states. Upon receiving an action command, it switches between navigation and interaction according to the interaction type, the target object, scene information, and the agent's 's current state (position and ongoing action). In the navigation state, it outputs action commands based the Navigation Module's 's path-planning results; in the interaction state, it issues action commands that drive the executor to synthesize the next interaction.
During navigation, the Navigation Module first constructs pixel obstacle map of the scene and then performs path planning using a modified * algorithm. In our implementation, we introduce an additional repulsion term in the A cost function to guide the trajectory away from obstacles and ensure safe navigation.
29

---

Published as a conference paper at ICLR 2026
Box User Instruction Refinement
SYSTEM
You are state-of-the-art intelligent humanoid robot. Given a vague command, you can interpret it into sequence of clear, executable instructions according the scene.
Definition of Clear Instruction:
Imperative sentence.
Have exactly one verb.
If the verb denotes an interactive action, it must explicitly state the object. If not stated in the origin command, you should choose one from the object list.
The object stated in the sentence must exists in the provided object list.
Make sure every verb in the original command included in the instruction sequence.
Don't miss details like adjectives, directions, and numbers.
USER*: Vague Command: Jump in place. Sit, and sleep.
Objects:
sofal (5.0 ,5.00)
2: simple bookcasel (6.0, 4.5), containing [book column1, book column2] Clear Instructions:
ASSISTANT*: Jump. Sit on the sofal. Sleep on bedl.
USER: -MMlti-view Images>
Vague Command: <User Instruction>.
Objects: <Scene Description>>
Clear Instructions:
Figure 8: Visualization of the labeled images: the left side presents multi-view images with labels indicating directions relative to the object, while the right side shows the labeled image used for determining target position of key joints.
C.2 VLM-BASED ACTION PLANNING MODULE
C.2.1 OFF-THE-SHELF VISION LANGUAGE MODEL.
BiBo employs GPT-4o, accessed via the official AP In the Python environment, utilize the OpenAl API library and manually construct both the conversation-prompp framework and the chain-of-thought procedure. BiBo also be integrated with other variants of VLMs (e.g.. Qwen (Bai et al. 2023 2025), Claude (Anthropic) 2024), and Gemini (Team et al. 2023)). In our experiments, we compare the scaling capabilities VLMs of different sizes and examine how tai- lored prompt designs influence models of comparable capacity. Specifically, we employ Claude 3.5 So T6] Qwen2.5-VIl and GPT-40 mini as large-, medium-, and small-scale comparison methods, respectively.
14 "hupp:/plaafrrm.opeai..cmm
http://wwwwaaaaoooo//o/ooooop/o 17 Phtps://qwenaallapppaafoom
30

---

Published as a conference paper at ICLR 2026
Figure 9: Visualization of agent pose reasoning process and the BEV image inputs for VLM.
Visualization of the navi- gation result of the modified A*
Figure 10:
C.2.2 DETAILS OF BASIC ATTRIBUTE ANALYSIS
In Basic Attribute Analysis, the compiler takes as input the user instruction, scene information, and the agent's s status. The VLM refines the user instruction based on these inputs as Box. / and decides whether to skip() (), start () new action, or end () an ongoing action, using the prompt in Box. 3 When an action starts, the VLM first generates brief summary of the action as the motion caption. Then, it analyzes the basic motion attributes according to the scene objects and the agent's s current state, facilitating the subsequence reasoning process, using the prompt Box, and
5
Scene Information. It includes multi-view images and corresponding scene description. The multi-view images are captured in the same manner as described in Sec. [B.13] by capturing images from all wall corners to maximize the coverage of the scene contents. Each image is labeled with object IDs. The scene description contains object IDs, coordinates, and parent-child relationships among objects. The parent- child relationships are computed using union-fina algorithm (Cormen et al. 2022) based on the containment relationships among the objects' 2D polygonal shapes, and the object with the largest (or similar) area and the lowest height is selected as the parent.
Agent Status. It includes all previous planning results and executed actions, the currently executing action, the agent's pose, nearby objects, and objects held by the agent. The previous planning results are provided through multi-round conversation history, while the currently executing action is identified by its motion caption, elapsed time, and execution result. All objects are specified by their names and corresponding IDs.
Motion Attributes. They include the related objects, the key joints involved, the interaction type, and whether IK should be enabled. The related objects are categorized into three types: target, at, and by. The target is the target object of interaction, at refers to the anchor used for agent localization, and by denotes the tool being used, For example, take book (target) from the shelf (ar) or turn on the TV (target) by the remote (by). The target is chosen as the anchor if there are key joints include the head, mands,feet, and pelvis (the latter being controlled by the agent's 's position and facing direction). The interaction types are divided into four categories: contact, non-contact, distal, and manipulate. contact involves physical contact (e.g., sitting), non-contact refers to proximity without contact (e.g., hand dryer use), distal denotes remote interactions without approaching the target, and manipulate refers to interactions that change the object's 's state (e.g., carrying).
objects. The
C.2.3 DETAILS OF AGENT POSE REASONING
In Agent Pose Reasoning stage, the VIM take the rendered scene images with labels as input, and sequentially infers: (1) the agent's direction relative to the anchor object, (2) the agent's 's distance relative the anchor object, and (3) the agent's 's facing direction. The success of this reasoning process depends on the choice of scene viewpoint as well as the placement and annotation of the labels within the rendered images.
Agent's Direction Relative to Anchor. We first capture multi-view rendered images of both the anchor and target objects. The images rendered with an object-eentered setup a 30% depression angle, sampled at 45 intervals with a 75% field of view, and occluded views are excluded, as shown in Fig. 8
31

---

Published as a conference paper at ICLR 2026
Box Planning for The Next Action
SYSTEM*:
You are state-of-the-art intelligent humanoid robot.
You currently in scene with the following objects, and the layout of the scene is pro- vided in the image:
<Multi-view Images>
<Seene Description>>
You should perform sequence of actions fulfill the following instructions, step by step:
<User Instruction
In each step. the sensor will provide you with your position, the objects you are holding, your current states and action elapsed time. You should decide your next action command accordingly, including:
1. skip(): Stay in the current state.
2. start(action): Start doing the specified action. Will add the action to your current state,
which means you are doing the action.
3. stop(action): Stop doing the specified action. Will remove the action from your current
state, which means you are no longer doing the action.
Example of actions:
1. Jumping
Leaning against walll
Taking hammerl from toolbox1
4. Standing on the right of tablel three meters away
Constraints:
Each step you can only give one command.
Each action should be indivisible, which containing no more than one verb.
For interactive actions, you must explicitly specify the target object, as example 2, 3, 4. Multiple actions in current state means performing multiple actions at the same time.
You are encouraged to directly output the command without any additional explanation.
USER#: Scene: <Scene Description>>
Position: origin (0, 0)
State: [("Standing", 3s, done)]
Holding: (left hand: none, right hand: none]
ASSISTANT*:
stop(" "oo(("Standing""
USER: ... (Multi-round Conversation)
Then, we employ Orient Anything WWan et al. 2024b), a model fine-tuned on DINOv2 (Oquab) al. 2023) that can predict object orientation and corresponding confidence scores in zero-shot manner. For each view, Orient Anything outputs an estimated orientation, which we discretize into directional bins. The confidence scores of all views within each bin summed, and the bin with the highest aggregated confidence selected as the object's front direction. the overall confidence below a threshold, the object is considered to have clear orientation.
Subsequently, eight labels are uniformly placed around the anchor object at 0.4 m intervals and 45" spacing, excluding non-traversable locations. These labels are projected onto the anchor's multi- view rendered images and provided to the VLM, together with textual descriptions indicating their spatial relations (if applicable) the anchor and target, e.g., "Label 1 in front of the monitor." The VLM then selects one label as the agent's 's relative direction to the anchor object. We use the prompt Box B For non-distal interactions, the prompt explicitly states that the label represents a standing location rather than direction to skip the next reasoning step, non-distal interaction is expected to occur within 0.4m range around the object.
32

---

Published as a conference paper at ICLR 2026
Box Basic Motion Attribute Analysis (Object)
U You are a state-of-the-arr intelligent humanoid robot, <VLM Instance ID>> The following command describes interaction with objects marked with "<" and ">", Please identify the roles these objects.
The roles include "target", "by", and "at".
"target" is the intented interaction target.
- "by" means the interaction done by using these objects.
"at" means the interaction is done at these objects.
Please answer in the following format:
larget: <target>>
by: [<object>>, <object2>,...
at: [<object3>, <object4>,...
Example:
Question 1: Grab the <clothI>frrm the <wwahhing machine
Answer 1:
target: <cloth1>
<waashing machinel>
Question 2: Turn on the <light1> the <switch1>.
Answer
target: <lightl>
by: <switch1>
Sentence: <Motion Caption
Agent's Distance from Anchor. As shown in Fig. 9 construct simplified bird eye view (BEV) of the scene using the polygons of floorplan and objects, where walkable areas and obstacles filled with different colors, and the positions of the agent and objects are indicated by specific labels. Along the previously predicted direction, series of distance labels are placed starting from 0.5m and spaced Im intervals, excluding positions that are not reachable. The prompt template is shown in Box. 7 which provides each label's 's distance and direction relative to target objects (if target is distinct from the anchor). The VLM selects one label as the final standing location. For non-distal interactions, this step is skipped, and the distance is fixed at 0.4m.
Agent's Facing Direction. Using the BEV map, we place arrows around the standing location as candidate facing directions, shown in Fig. We use the prompt in Box. In the prompt, for each candidate we provide (i) its angle (relative to global reference) and (ii) the object points toward, restricting objects to those relevant to the current interaction. The VLM then selects one candidate as the final facing direction. The BEV imput is omitted at this stage for non-distal interactions.
C.2.4 DETAILS OF KEY JOINT GENERATION
This process generates joint target positions relative to the object surfaces, as in Box. 9 Given the agent's 's location relative to the object, we select the scene image rendered from the corresponding viewpoint and uniformly place an 8x8 grid of labels on it, as shown in Fig. sponding to point on the object surface. For each key joint, the VLM selects one label as an anchor point, from which it infers the direction and distance between the joint and the anchor. These values collectively define the joint's 's target position.
Each label corre-
For contact and manipulate types of interactions, we adopt a simplified strategy. For objects with size smaller than 0.25 x 0.25 x 0.25m, the key joint generation process is skipped, and the target position directly set at the object center. For larger objects, the reasoning for direction and distance omitted. Instead, the model determines whether the agent should exert force the object. If not, the target position is placed on the object surface; so, is positioned toward the object center, guiding the agent apply force the object.
33

---

Published as a conference paper at ICLR 2026
Box Basic Motion Attribute Analysis (Motion)
SYSTEM*:
You are state-of-the-art intelligent humanoid robot, VLLM Instance ID>. You can assess the details of action to perform it accurately.
The details include:
Interaction Type: one of [contact, non-contact, long-range, manipulation]. Contact refers to contact with the target object (if present), non-contact denotes proximity without contact, long-range indicates interact at a distance, and manipulation involves moving the object.
Key Joints: the most important joints involved in the interaction, available options: [' pelvis', 'left_foot', 'right_foot', 'left hand', 'right hand", "head"]
Use Inverse Kinematic: true or false, enable for precise stable interactions (e.g., touching small objects, carrying), disable for high dynamic motions (e.g., dancing)
Constraints:
No more than two contact points. there may be more, choose most important two.
One hand cannot manipulate multiple objects.
You are encouraged to directly output the details without any explanation.
US Interaction: Turn on <lamp1> on <tablel>.
State: You are holding book1 in your right hand.
Details:
ASSISTANT*: Interaction Type: contact
Key Joints: hand"]
Use Inverse Kinematic; true
USER: Interaction: <Motion Caption>>
State: You are <Agent State>.
Details:
Box 6. Inferring Agent's Direction Relative Anchor
USER#:
You're the state-of-the-art intelligent humanoid robot. When you interact with an object, you can determine your own position relative to it.
The scene layout is provided in the images, which are photos of the <Anchor Object>> taken from different perspectives. The photos contain red circular markers labeled with integer indices, each marker represents direction relative to the <Anchor Object>.. In different photos, the marker with the same index represents the same direction. <Multi-view Images>
Marker Directions:
<Direction Label Descriptions>
For example:
LLbee is in the front of monitor and desk
Label 2 is in the front-left of monitor and desk
When performing <Motion Caption>>, which direction should you stand in? Your answer should be marker index enclosed in >>>>>> >and <<<.
C.2.5 MULTIPLE ACTION MERGING
For actions occurring concurrnttyy (e.g., when action has not yet ended while another starts), the system first checks for potential conflicts after performing basic attribute analysis. Conflicts are defined under the following conditions:
34

---

Published as a conference paper at ICLR 2026
Box Inferring Agent's Distance from Anchor
You're the tfte-of-the-ar intelligent humanoid robot. When you interact with an object, you can determine your own position relative to it.
The provided image is bird-eye-view map of the scene.
<BEV>
Red markers indicate the <TTrgget Object>, while blue markers denote the IDs of scene objects. These objects follows: <Senne Description
The green markers in the image represent a set of candidate standing locations: <Distance Label Descriptions>
For example:
There distance to monitor is:
Label : 0.5m
Label 2 1.5m
When you performing <Motion Caption>, which position should you stand in? Your answer should be marker index enclosed in >> >nd <<<.
Box 8. Inferring Agent's 's Facing Direction
USER#:
You're the state-of-the-art intelligent humanoid robot. When you interact with an object, you can determine your facing direction.
The provided image is a bird-eye-view map of the scene. <BEV>
Red markers indicate the <Tagget Object>, while blue markers denote the IDs of scene objects. <Seene Description>>
The green arrows in the image represent a set of candidate facing directions: <Facing Label Descriptions>>
For example:
Arrow 1: 0%, facing directly to the monitor.
Arrow 5: 180", facing away from the monitor.
When you performing <Motion Caption>>, which direction should you face? Your answer should be an arrow index enclosed in >>>>>>>>>>>>>>>> and <<<.
The two actions involve the same key joint;
Both are non-distal interactions and their target objects are more than 1m apart;
Their anchor objects are more than 1m apart.
If no conflict is detected, the system proceeds to generate new action command through the sub- sequent pose reasoning and joint generation processes, and updates the existing action command accordingly.
35

---

Published as a conference paper at ICLR 2026
Box 9. Inferring Target Position of Key Joints
SYSTEM*:
You are wate-of-the-ar intelligent humanoid robot. When interacting with an object, you can determine the target position of specific joint. This process involves two steps: (1) locating a target point on the target object, and (2) specifying the joint's position relative to that target point (including its direction and distance).
Example I: the laptop
Joint: right hand
Target Point: keyboard
Direction: up Distance:
Example 2: use the hand dryer
Joint: right_hand hand
Target Point: outlet
Direction: down Distance: 0.2 m
USER#:
The image shows a View Direction view of the <Tarrget Object>. It contains an 8x8 grid of numbered labels (indexed from to 64), with each label corresponding point on the surface of the <TTargtt Object>.
<Image>
You are performing <Motion Caption>. Please identify the target position of your <Joint Name>> in the following format:
Target Point: a noun or noun phrase (can include adjectives and other modifiers to better describe its features) describing part of the <'Target Object>> that your <Joint Name>>
refers to.
Label: one label index in the image corresponding to the target point.
Direction: the direction of your <Joint Name> relative to the target point. Available options: [up, down, left, right, directed into the image, directed out of the image, toward the object center, along the surface normal]. Distance: the distance of your <Joint Name from the target point, using meter (m).
Specifically, when generating tion combining the ongoing and newly initiated motions (c.g., if sit has not ended and tun on the lomp starts, the merged caption becomes and turn the lamp). During Agent Pose Reasoning, the rendered image centers on all anchor objects, and the merged motion caption is used in the VQA process of VLM. During Joint Pose Generation, the system regenerates the key joint target positions for the ongoing actions and overwrites them in the corresponding action commands.
action command, the system constructs merged motion cap-
All action commands are stored concurrently in the humanoid state machine's 's action command list. The state machine's ' final output command uses the merged motion caption, adopts the location and facing of the latest generated action command (since the reasoning already considers the merged caption), and directly combines the key joint positions of all action commands.
C.2.6 RULE-
RULE-BASED BASED REFLECTION
To enhance the stability of the three-stage VQA process, we incorporate a reflection mechanism. Reflection is triggered in the following cases: (a) failure to follow the expected QA format; (b) conflicts with ongoing actions; and (c) implausible spatial relationships (e.g.. a contact positioned enror is detected, the system responds based on the error type:
too far from the anchor). When format violations result in a simple rollback, while logical inconsistencies trigger a rollback followed by the explicit insertion of the reflection result into the conversation history, as in Box. TO
36

---

Published as a conference paper at ICLR 2026
Box 10. Example of Explicit Insertion of Reflection Results Conversation History
ASSISTANT: start("Sitting on sofal.")
USER: Command execution failed.
Reason: You are performing "Sleeping on bed1.", and it is impossible to perform "Sleeping on bedl and sitting on sofal.", as bedl and sofal are located too far apart.
Possible solutions:
stop "Sleeping bed1." try another action
C.3 HUMANOID AGENT STATE MACHINE
The State Machine consists of two states, Navigation and Interaction, and maintains an action com- mand list that stores all currently executing actions. During each frame update, the State Machine first converts the positions and directions each action commands from local coordinate system, defined relative to its anchor object, into global coordinate system. Then, it employs the the method in Sec. to convert the command list into a merged action command. Next, the system evaluates the difference between the current agent pose (i.e., location and facing) and the target value.
When the distance to the target less than 0.5m and the deviation facing direction is within 45% the State Machine transitions into the Interaction state and marks the corresponding action command as executing. Once all executing non-distal actions with target objects are completed, the system switches back to the Navigation state.
C.3.1 NAVIGATION STATE
The State Machine first invokes the Navigation Module to generate a planned path. It identifics the point this path closest to the agent's current position as the starting point, and then selects the next waypoint that is both nearest to the start and farther than 0.5m away.
The direction from the current position to this next point is used as the facing direction. If the angular deviation between the current and target facing directions exceeds 45", the target location is fixed the current position, and the motion caption is to "A person slowly turming around in place." Otherwise, the target location is set to the next waypoint, and the motion caption is set to "A person walking." When the distance to the next path point exceeds 1.2m, the moving speed is When the distance falls below 1.2m, the speed decreases linearly until it reaches Om/s at 0.2m.
Im/s.
The navigation command further merged with concurrently executing distal, manipulation, non- interactive actions (i.e., actions without target objects, which are excluded from the second stage of Basic Attribute Analysis and categorized as the Type-V V action), allowing navigation and motion execution to proceed simultaneously.
C.3.2 INTERACTION STATE
First, for non-distal motions, the target location is to a position 0.3m away from the target object, guiding the agent to walk closer to the target. When the positional eтror drops below 0.2m, the interaction action command start executed. For contact actions involving force exertion, the joint target position is first assigned to the contact point on the object's surface. Once the agent is within Im of this point, the target position is set the object center.
After 60 frames of execution, the system evaluates whether each interaction is done based on pre- defined rules: a contact action done the relevant joint is applying force, is within 0.25m of the target position, and within 0.1m of the target surface; non-contact action is done when the joint within 0.1m of the target position; distal action is done if its duration exceeds 120 frames; and manipulation is done when the object's movement exceeds 0.1m while remaining within 0.1m of the hand. The execution duration and completion status of all actions are continuously fed back to the planner to support subsequent decision-making.
37

---

resolution of 512 512, Published as a conference paper at ICLR 2026
Algorithm 1: The Modified A Path Planning with Repulsion Term
Input: Start point Pstart goal point Proal, obstacle distance map M, repulsion ratio a, repulsion distance 3, scene-to-pixel scaling ratio y
Output: Path p (p,....,71 Pr)
O - (M- = 0):
M < m - -,,, 3:
M < (1 a a;
Initialize open set with Paant and set (P = 0:
// g is current cost
while open not empry do
P: < arg minp (()):
// f is estimated cost
if p: poot then break; foreach neighbor p' of Pt do if Op') true then continue;
g(p') + (Pt) + +M Mp':; f(p') g(p') a h(p'):
// h is heuristic function if (p') improves then update open set and parent pointer;
Reconstruct path P from parent pointers; return P
C.4 NAVIGATION MODULE
The Navigation Module performs path planning based on the input agent position, target position, scene floorplan, and all object information. First, it constructs a navigation map using the floorplan and the convex decomposition of objects, while optimizing both the agent and target positions to ensure they located outside obstacles. It then performs pathfinding using modified A* algorithm that incorporates a repulsion term to encourage paths farther from obstacles, followed by polyline simplification step that converts the dense pixel path into sparse waypoints.
C.4.1 MAPPING MODULE
For each scene object, the module applies the current position and rotation to transform the coordi- nates of the 64 convex AABBs obtained from decomposition (refer to Sec. B.1.2). and projects them onto the Y--plane. This calculation is GPU-accelerated with PyTorch The projected polygons converted into Shapely geometries, padded by 0.1 to form the 2D polygonal outlines of the objects. The floorplan and object polygons are then processed with PyClipper 20 for clipping, and discretized into a pixel obstacle map. Pixels covered by the floorplan but not by any object represent navigable areas, while the rest correspond to obstacles. The map discretized scaled isotropically according to the longest axis of the floorplan.
If the agent position lies within an obstacle, the module searches in four directions (up. down, left, right) to find the nearest accessible area, marking all traversed pixels as navigable. If the target position lies within an obstacle, shifted outward according to the direction of the corresponding stand location relative to the object.
Finally, OpenCV is employed to compute an obstacle distance map, which records the distance from each navigable pixel to the nearest obstacle. Specifically, a morphological dilation operation iteratively applied to the pixel obstacle map. where navigable pixels are assigned a value of 0 and obstacles 1. The number of dilation iterations before pixel is removed corresponds to its distance to the nearest obstacle. The process terminates once all pixels become 1.
C.4.2 PATH PLANNING WITH MODIFIED A
The modified * algorithm utilizes this obstacle distance map for pathtinding. Specifically, a re- pulsion term is introduced into the cost function, increasing the step the agent approaches
"https//ppyorcc..or// Phanss//gthhaccmmm//oooookootnnr
38

---

Published as a conference paper at ICLR 2026
obstacles. This design encourages the generation of paths that stay farther from walls, thereby low- ering the overall path cost. The complete algorithm is presented in Alg. [
We use Shapely to simplify the path from dense pixels into sparse waypoints. We then iterate through each pair of waypoints. If the line connecting them maintains a minimum distance greater than 0.5m from the nearest obstacle (check by padding the line into rectangle with width of 0.5m and performing a polygon intersection), the intermediate points are bypassed. This process yields the final simplified path.
D DIFFUSION-BASED MOTION EXECUTOR
The executor generates future motion extending the currently performing one, condition the structured motion command from the compiler. The core of the executor a latent diffusion model, which includes Variational Autoencoder (VAE) that transforms human motion into latent motion tokens, and Latent Diffusion Model (LDM) that performs a diffusion process in the latent space.
D.1 DIFFUSION PRELIMINARY
Our denoising network follows the DDIM (Song et al. 2020) framework with fixed forward diffu- sion and a learned reverse denoising process. Let the future-motion latent be denoted by S = Sy. and let (3.) be a variance schedule with a B, and a; II Ols.
Forward Process. Starting from clean data SJ. the forward (noising) Markov chain adds Gaussian noise over T steps:
= NNS);: v1 - 54 S 111). This can be written in closed form as:
{S$ ( (S); vor 59. (1 a.) 1)
Reverse Process. At sampling time, we sample the terminal noise from standard normal distribu- tion and iterate backward:
((S). 8m, 8r)
ST ~ 1),,1),
ST'=
S Sg.
Va-1
vI at-1
We employ Classifier-Free Guidance (CFG) (Ho & Salimans 2022 modulate the control strength, using the condition set (8m, 8c) within the noise predictor E(-).
Training Objective. The reverse model eg trained predict the injected noise each timestep via the DDPM (Ho et al. 2020) objective:
Lximple = E, s9.c [|| [ll--co($) co t, 8m. $c)]
where S is obtained from the forward process as above. After T denoising steps, the procedure yields the clean future-motion latent S = 59.
D.2 DIFFUSION ARCHITECTURE
We adopt 9-layer Transformer Encoder architecture with skip connections the diffusion back- bone. Each layer uses 4 self-attention heads, with a model hidden size of 256 and an feed-forward network (FFN, linear layer) dimension of 1024. Starting by setting the future motion tokens S Gaussian noise, where T is the total denosing steps. The diffusion timestep T is represented via a sinusoidal positional embedding passed through a small MLP. The diffusion denoiser JF iter- atively denoises S using DDIM scheduler, condition command C and the latent of preceding actual executed motion S., Specifically, encode the motion captions using pretrained text en- coder (Devlin et al. 2011)). where the [CLS] token is taken as the caption token 8m, E "F Other control parameters are encoded with MLP, and their representations control token 8,. 8m and s. are concatenated with S.. together conditioning the denoising process S/- F(S) [Sa, 8m, {c]) by cross attention with S
summed to form the
39

---

Published as a conference paper at ICLR 2026
D.3 VAE ARCHITECTURE
The proposed Variational Autoencoder (VAB) utilizes a shared 9-layer # architec- ture for both encoder and decoder. Each layer employs 4-head multi-head attention with model dimension of 256 and a 1024-dimensional feed-forward network. We use the GELU activation function and set the dropout rate to 0.1. To enforce temporal causality, we implement two key mechanisms. First, a causal mask is applied within the self-attention layers to the concatenated se- quence of latent and frame tokens. Second, for cross-attention, we introduce an growing memory window that expands its size in correspondence with the target timestep, restricting access to only the relevant prefix of latent variables. These designs effectively prevent future information leakage, ensuring faithful and tome-cons sequence generation.
D.4 TRAINING DETAILS
Our implementation uses PyTorch the primary deep learning framework for training and infer- ence. We leverage the Transformer 21 library for tokenization and text embedding. The diffu- sion process is implemented with the Diffusers toolkit, including noise schedulers and sampling pipelines. All experiments run on a CUDA-enables backend
All training is conducted on the HumanML3D dataset (Guo et al. 2022a). Motion sequences processed at 20 Hz. Each sequence is concatenated with a stance sequence at the beginning to serve as the initial state, enabling the model to learn initiate a motion from scratch. When trianing with control conditions, we apply random conditioning by sampling from 60 predefined combinations of control signals including trajectory, heading, velocity, and key joint positions, enabling flexible spatial control during generation. During generation, following CLoSD (Tevet et al.) 2024 the length of previous motion context to 20, and generation length 40.
we set
We use the Adam optimizer with parameters 31 = 0.9, 32 = 0.999, weight decay=0, and e = 10-8 The learning rate is initialized to 10-4 and follows cosine decay schedule with linear warm-up over the first 2000 steps. We utilize a DDIM sampler, apply gradient clipping with a threshold of 1.0, and set the unconditional probability to 0.1. For inference, we use 5 DDIM steps and a guidance scale of 7.5. The model is trained for up 3000 epochs with batch size of 256. All experiments are run with a fixed random seed of 1234 for reproducibility.
D.5 IK POST OPTIMIZATION
To enable precise control of end effector joints, we introduce an Inverse Kinematics (IK) post- processing step. This step optimizes the posture of arms or legs when the compiler issues precise contact command. operates on the generated pose for each frame, with the optimized motion serving as the reference for the tracking policy.
Our method employs a custom variant of the FABRIK (Aristidou & Lasenby 2011) algorithm. Specifically, for the upper limb, we solve for a 3-DOF kinematic chain consisting of the left shoulder, elbow, and wrist, with the shoulder joint acting as a fixed root. For the lower limb, we solve for a 3-DOF kinematic chain consisting of the hip, knee, and ankle, with the hip joint serving as the fixed root. The joint lengths are derived from the initial pose and remain constant. Unlike standard FABRIK, our primary IK target is the wrist joint, not the hand end-effector. This design choice provides a stable base for subsequent hand orientation and grasping.
The iterative optimization process begins by placing the end joint (wrist for the upper-limb, ankle for the lower-limb) directly at the target position. Then, standard backward and forward passes are executed to satisfy the kinematic constraints of the limb segments. The position of the end effector is subsequently reconstructed, where it is placed at fixed distance from the solved joint position and oriented along the vector of the corresponding segment (forearm for the upper-limb, thigh or shin for the lower-limb). For targets beyond the limb's reachable workspace, the chain is fully extended and pointed towards the target. The entire iterative refinement process is capped at a maximum of 50 iterations, with tolerance of 1
" lnttss//hugginggfacecom http::/hhggggnnnooc.//////////na 2 hhtaps//hugginniaccco//eeeeeeeee hwtpp:///./ooooccocccerrrrr 23 hhans:/ddwvoopprrrniiicooooomood
40

---

Published as a conference paper at ICLR 2026
D.6 TRACKING POLICY
We utilize the PHC tracking policy (uo et al., 2021), enabling humanoid agents to accurately track the generated joints in physical space. We mitiiaize the policy network with weights from CLoSD (Tevet et al.] 2024). which is trained using Isaac Gym simulator (Makoviychuk et al.) 2021
E COMPARISON METHODS
E.1 TASK COMPLETION
E.1.1 UNIHSI
UniHSI (iiao et 2023 represents each human-scene interaction task as a sequence of contact interactions. It employs AMP (Peng et al. 2021) achieve contact goals. Each contact contains standing position, a contact flag (if not in contact, the agent navigates to the standing position), and set of contact joint information. Each joint information contains a joint name, contact normal and 200 surface points on the contact area (which, in the original task definition, correspond the target object and specific part index of that object). During task execution, the agent also needs to obtain nearby scene height information.
We use the official GitHub repository import scenes in IsaacGym using the same asset options as in the scene configuration, and addi- tionally sample each object's ' point cloud using Trimesh to construct the contact surfaces and scene height map. To build the ground -tuth task plan, we first procedurally generate contact surfaces and standing points, then create navigation path based on the scene map, and finally convert it into the UniHSI task format.
25 and the released model checkpoints. During evaluation, we
As shown in Fig. II] contact surfaces are generated according to object categories and interaction types. For sofas, we decomposed the backrest, armrests, and seat cushion by point-cloud height, take the 0.5 0.5m region at the center front of the cushion the pelvis-joint contact surface for the Sit task, and set the standing point 0.5m in front of the cushion's 's front edge center. For beds, identify the mattress, pillow, and headboard based on the height of the point cloud, with contact normals facing upward. For the Sleep task, the central 0.5 x 0.5m region of the mattress is used as the pelvis contact surface, and the pillow's 's upper surface point cloud the head contact surface. For the Sit task, we use the 0.5 0.5m region at the center front of the mattress as the pelvis contact surface, with upward normals and the standing point 0.5m front of the mattress's front edge center. For Touch tasks, where the target objects are small, we use the object's 's point cloud as the contact surface for the left or right hand. We uniformly sample 36 candidate standing points along 0.5m-radius circle around the object and select the first reachable one the standing position, with the contact normal pointing from the object to the standing point. Tasks of type watch, lift, and composite cannot be executed by UnIHSI. Navigation is implemented using the same A algorithm as in BiBo.
As in Fig. the qualitative evaluation of UniHSI covers two interaction scenarios: (a) getting up from the sofa and (b) leaning on the armrest. In (a), we use the same configuration as the sitting task, placing the standing point 0.5 in front of the cushion's 's front-edge center at a height of 0.86 m. In (b), UniHSI and BiBo use the same configuration, with the pelvis and hands set to make contact with the upper part of sofa's 's armrest.
E.1.2 HUMANVLA
Human (uu et al. 2024 performs object transportation tasks based on visual observation and natural language descriptions. Its task format consists of a prompt embedding, the initial and target poses of scene objects, the target object name, and the waypoints before and after transportation. At runtime, the model takes as input the current and target poses of the target object, the next waypoint, the rendered scene image, and the BERT (Devlin et al. 2019) embedding of the prompt.
httppp///dweelooorr..ddddco/smss gym 25 hhtps://kiiangggiihhuuiiiiiiii
41

---

Published as a conference paper at ICLR 2026
Figure 11: Visualization of task execution process in UniHSI and the ground truth contact surfaces in random generated scene.
nu
Visualization of Human- VLA and the path planning result.
Figure 12:
Figure 13: Visualization of the task execution process in TokenHSI, illustrating the height map and target position.
We the official repository and the released HumannLA-Teacher model checkpoint (the Student model struggles to complete the task). The randomly generated scenes and tasks are organized according the Human task format. Walls are removed, and the asset import options and settings remain consistent with the original scene configuration. Since HumanVL.A is sensitive to interaction poses, we modify the A# pathfinding by setting the destination 1.5 m front of the target object, and adding an extra waypoint 0.5 m in front of the target to encourage direct approach toward it. The task exccution process and pathfinding result are shown in Fig. 12
During the qualitative evaluation shown in Fig. 5] the agent is initialized at position of (-3m, 2m) relative to the target, facing along the +y-axis. The waypoint set at (-Im, 0m) relative to the target. The box has size of 0.36 0.36 0.5m, with 0.5m representing its height.
E. TOKENHSI
TokenHSI (Pan et 2025) achieves multi-interaction generalization and integration by fine-tuning lask-specific tokenizers and action heads. Its task format consists of scene objects and a task plan, where each step the plan is defined by an action type and its corresponding parameters. Specif- ically, the traj action includes a list of path waypoints, the sir action specifies the interaction point, seat bounding box, position, and orientation, and the carry action defines the target box, its bound- ing box, and the destination position. During execution, the model also samples local height map around the agent as part of its observation to enhance environmental awareness.
We adopt the official repository 27 and the Longterm 4 Basic Skills checkpoint, constructing all tasks according to the TokenHSI format. The scene is first converted into an file structure compatible with TokenHSI, which includes each object's height map, mesh, bounding box, facing direction, up vector, and sit position. When importing scenes, the same asset options as those in the original scene configuration are applied. The constructed tasks follow the same format in TokenHSI, encompassing object poses and the complete action plan. Trajectory waypoints for traj actions are generated using the same A* pathfinding algorithm in BiBo, while interaction point generation follows the procedure used in UniHSI. The task execution process is illustrated in Fig. 13
1.1.4
CLOSD
CLoSD (Tevet al. 2024) use motion diffusion to drive tracking policy (Luo et al.) 2023) to perform human-scene interaction in physical environment. It is controlled by a rule-based finite state machine (FSM). The FSM determines the next action based on the current state and programmatic
"https//gittab.om//llllaaaa/a// VLA " http:://hhanggaan//gggghhhiiooof
42

---

Published as a conference paper at ICLR 2026
script, outputting the subsequent 60-frame action command that includes the motion caption, target positions of the feet, hands, and head, as well as the standing position and facing direction.
We employ the official implementation and the Muliitask model checkpoint. The state sequence is generated according to the scene objects and task. Specifically, for each interaction task, we define an update function and transition rule. The update function outputs the next action command and checks whether a transition should occur. When all interactions in the current group meet their transition conditions, the FSM proceeds to the next group of interactions (where interactions within the same group occur simultancously). The transition rules follow the same criteria of task success, as described in Sec.
28
In the reach state update function, the A algorithm is first applied to perform pathfinding, If the agent is oriented toward the next waypoint within 45 cone, the target position set to that way- point (clipped to maximum distance of 1 1.2m), and the motion caption prompt is "a person is walk- ing." Otherwise, the facing direction adjusted toward the next waypoint, and the prompt set to "a person is turning around in place." The configurations for sit, sleep, and touch actions follow those defined in UniHSI. For the lift action, the target positions both hands are initially placed 0.3m to the left and right sides of the object, then directed toward the object's center to facilitate grasp- ing. Finally, the pelvis height is set to 0.86m (the normal standing height) to guide the agent to stand upright. A reach state inserted between consecutive interactions to enable navigation across interaction points.
E.1.5 ABLATION STUDY
In the ablation study presented in Sec. 4.1 we analyze the impact of several components in both the compiler and executor. In the compiler, we ablate the voting mechanism and the use of image labels, while in the executor, we ablate the use of actual executed motion input, previously generated motion input, and IK post-optimization.
To ablate the voting mechanism, we use single VLM to analyze motion attributes instead of ag- gregating outputs from multiple VLM instances. The image label is ablated by prompting the VLM to directly predict image coordinates and facing angles. For the executor, we conduct ablations by extending future motion using only the previously generated motion or the actual executed motion. Specifically. the same motion sequence is provided as input to both the Diffusion Model and the VAE during inference. Finally, we ablate IK post-optimization by directly using the raw generated motion without applying inverse kinematies refinement.
E.2
MOTION QUALITY
E.2.1 MOTIONLCM
MotionLCM (Dai et al. 2024) adopts a Transformer- based Latent Diffusson architecture, incorpo- rates Consistency training enable few-step generation, and integrates control signals through ControlNet module.
We use MotionLCMM v2, adopting the official implementation and following the provided train- ing scripts and configurations to train the VAE, Consistency Model, and ControlNet. In addition to the original TM2T metrics, we introduce two additional evaluation pipelines assess Physical Plausibility and Control Accuracy.
The Physical Plausibility evaluation follows the implementation provided in CLoSD, whereas the Control Accuracy evaluation is performed under the same experimental setting as BiBo. Specifically, a single joint is randomly selected from the head, hand, or foot, and the ground-truth joint position at frame 40 is used as the control signal. The mean absolute error (MAE) between the generated joint position and the control signal is then calculated to quantify the control error.
Training and evaluation is conducted a workstation with 8x RTX 4090 GPUs, 256 GB DDR4 RAM, and an Intel(R) Xeon(R) Gold 6226R CPU 2.90GHz, running Ubuntu 24.04 with CUDA 11.8 and PyTorch 2.3.
hhtps//gihuuh..mm/Daii Www.:n/MMttionl..ccccm
43

---

Published as a conference paper at ICLR 2026
E.2.2 MOTIONSTREAMER
MotionStreamer (Xiao al. 2025 adopts latent diffusion architecture, where Transformer- based diffusion model performs next-token prediction, and CNN-based VAE decoder reconstructs the motion from tokens, enabling the generation of high-fidelity motions of arbitrary length.
We use the official implementation The original version employs a 272-dimensional motion rep- resentation [ which is incompatible with evaluation pipeline. Therefore, modify it to the dis-eimnsi motion representation in HumanML33, and train the model following the pro- vided scripts and configurations. The generated motions exported into the evaluation pipeline assess motion quality and physical plausibility.
Training is conducted and an Intel(R) Rc)n(R) Gold 6226R CPU @ 2.90 GHz, running Ubuntu 24.04 with CUDA 11.8 and PyTorch 2.3. Evaluation is performed on the same machine MotionLCM.
workstation equipped with 8x RTX A6000 GPUs, 256 GB DDR4 RAM,
E.2.3 MOGENTS
MoGenTS (Yuan et al. 2024) is based on a masked modeling framework, consisting of a Motion Transformer and CNN-based vector quantization VAE (VQ-VAE). It introduces a masking mech- anism that leverages both spatial skeletal relationships and temporal dependencies, enabling high- quality motion generation. We use the official implementation and the released checkpoints. The generated motions are then exported to the evaluation pipeline to assess motion quality and physical plausibility. The evaluation conducted on same machine as MotionL.MM.
E.2.4 MOCONVQ
MoConVQ (Yao al., 2024) is end-to-end physical motion generator built upon VQ-VAE architecture, which operates within its own simulation environment. We employ the official imple- mentation and released checkpoint for all experiments. For quantitative evaluation, adopt the results reported in CLoSD to ensure consistency and comparability across methods. In the qualitative experiments illustrated in Fig. 5 we evaluate MoConVQ under two representative prompts: (d) "A person is boxing with someone, and kicking at the air." and (e) "A person is sitting on sofa and waving hand above the head." Since the simulation environment of MoCon does not natively support static mesh colliders like sofa, we follow the approach in this repository and use rectangular block with the same height as the sofa cushion and very large mass as substitute. The evaluation is conducted on the same machine as MotionLCM.
directly
E.2.5 DIP CLOSD
DiP (Tevet et al., 2024) is real-time motion diffusion model capable of generating motions of arbitrary length, built upon Transformer architecture. CLoSD employs a tracking policy to follow the motions generated by DiP physical environment. DiP extends future motion from the actual executed motion, thereby adapting physical feedback.
We use the official implementatior In the text-to-motion experiments in Tab. we adopt the released r2m checkpoint, while in the control accuracy experiments in Tab. 4 we use the multi-target checkpoint and adopt the same setting as other comparison methods. The motions generated by DiP and CLoSD are exported using the provided scripts and evaluated through unified evalua- tion pipeline as other comparison methods. The evaluation is conducted on the same machine as MotionLCM, using IsaacGym Preview 4.
35
"' https//gggtbbboooo///eeya///a/up "http://gihuuu.oo//huuaaaaaaa wttp::ihwwww.ccccca/acaaaaamaaaa http:://gituuh.oo///n//eoooreoot
44

---

Published as a conference paper at ICLR 2026
Table Comparison between different VAEs on motion recon- struction error, using HumanML3D. Bold and underline is the best and second best, respectively.
Table 10: Impact of latent di-
mension of the Causal VAE on motion reconstruction error.
Method
MotionLCM
MoGenTI
VAE
w/ Causal
w/ A.
w/ VQ
Latent Dim
32
64
128
FID
0.025
0.006
0,021
0.012
0.031
0.107
FID 4
0.015
0012
0.010
2
27.44
16.35
20.36
7,58
31,72
39.82
MPJPE
8.04
7.58
6,76
E.2.6
DOUBLE TAKE
Double Take (Shafir et al. 2023 is diffusion inference strategy that enables pretrained motion diffusion model to generate motions of arbitrary length. In the first take, it synchronously generates all segments of a long-sequence motion, using gradient guidance during inference ensure coherent transitions at the connections between adjacent segments. In the second take, it uses masking to separately diffusion-optimize the transitions.
We use the official implementation of the Double Take 36 inference process, employing the released MDM pretrained on HumanML3D with 263-dim motion format. During inference, we follow the Double Take inference process to obtain motion sequences and adopt the HumanML3D evaluation pipeline.
E.2.7 ABLATION STUDY
In Tab. and sequences, and ablate Causal Attention by removing the attention mask during both training and inference. The inputs of actual executed motion and previously generated motion are further ablated by using only one of them during inference, following the same setting described in Sec. E.1.5
ablate the LDM by directly training the diffusion model motion
F SUPPLEMENTARY EXPERIMENTS
RENDERING
In the experiments, Blender Python API 4.2.0 EEVEE Next pipeline render the scene images during the planning process.
For visualization, adopt three approaches:
1. Headless server environments: We perform visualization rendering in Blender using the EEVEE Next pipeline, where humanoid body represented skeletal lines.
2. Debugging environments and Fig. 16 17
and
18
We use the built-in visualization window
provided by IsaacGym for rendering.
3. Case study in Fig. and video demonstrations: We conduct simulations in IsaacGym, export the root states and meshes all objects, and render them in Unity. The humanoid body is reconstructed in Unity based on IsaacGym XML--base humanoid body file.
F.2 VARIANTS OF VAE
Setting. We experimented with several types of VAEs to convert motion sequences into latent tokens for higher reconstruction quality. Specifically, tested:
1. The original VAE architecture with a Skip- Trip-Transformer encoder-decoder structure.
2. Adding causal attention, described in Sec. 3.3
3. Introducing causal autoregressive next-token prediction decoding specifically, like the behavior of GPT. Instead of feeding zero tokens at once and decoding all motion frames simultaneously, we iteratively append one zero token time and decode frames across N steps. All other hyperparameters remain identical with the original setting.
s hupps//unityccom/
45

---

Published as a conference paper at ICLR 2026
Table 11: Comparison between different VLM variants on task success rate in random generated scene. Bold and underline represent the best and second best performance, respectively. Gray color denotes using ground-truth action plan, which excluded in the comparison.
Method (%)
Reach
Warch
Single Internction
Sit f
Sleep
Touch
Lift
Simple
Composite Task
Medium
Hard
BiBo (ours, w/ Sonnet 3.5, 1752)
99.18
98.31
87.54
91.33
70.57
59.52
50.00
34.62
19.05
BiBo (ours, w/ 5- 72B)
98.09
98.87
72.73
78.67
68.44
55.10
25.00
11.54
14.29
4PT-4
16.00
Bißo (ours, w/ GPT44, 200B)
99.18
99.62
95.84
94,89
$6.05
65.42
58.82
36,34
27.78
Bißo
plan)
87.23
44.23
42.86
Box 11. Prompt Example for Visual Grounding (JSON Coordinates)
SYSTEM You precise visual grounding model that outputs JSON coordinates only.
USER#: You are given an image. Your task is locate the <Object Category>> in the image. Please output the approximate center coordinates of this object.
The coordinate system follows the image pixel convention:
The origin (0, 0) is at the top-left corner.
The x-axis increases the right.
The y-axis increases downward.
Return coordinates in pixels relative to the image resolution.
Return only a JSON object, for example:
("x": 180, "y": 240)
4. Incorporating vector-quantized VAE (VQ-VAE), implemented using residual- quantization VAE (RQ-VAB) with 1,024 code entries, residual quantizers, and an ex- ponential moving average (EMA) decay factor of μ 0.99.
We train these VAE models on the train split of HumanML3D. and evaluate FID and Mean Per Joint Position Error (MPJPE) on the test split. FID evaluates the similarity between the distribution the original and reconstructed dataset, and MPJPE evaluates the reconstruction error.
Results. According the experimental results in Tab 9 the (2) Causal Attention configuration achieves the best performance, with MPJPE of 7.58mm. We further test models with different latent dimensions under the Causal Attention setting, and the results in Tab. that larger latent dimensions further improve performance. To balance computational efficiency and reconstruction accuracy, we choose latent dimension of 64.
F.3 VARIANTS OF OHF-THE-SHELF VLMS
Setting. We evaluate BiBo's scaling capability across VLMs of different sizes, and assess the gen- cralizability of prompt design. Specifically, we use the large-scale Claude Sonnet 3.5 (~ 175B), the medium-scale Qwen2.5-V1. (~ 72B), and the small-scale GPT-4o mini (~ 8B). The parameter size of Qwen2.5-VI. publicly available (Bai al. 2025). while the sizes of Claude Sonnet 3.5 and directly replace the API endpoint of GPT-4o with that of each comparison VLMs. Each task is evaluated once under a randomly initialized pose.
GPT-4o mini are referenced from Abacha al. (2024). 4. In implementation,
The results in Tab. I1 show that when using the prompt originally designed for GPT-4o, BiBo achieves comparable task success rate the similarly scaled Claude Sonnet 3.5, demon- strating generalization capability, As the size the VLM increases, the task success rate improves, reflecting scaling ability.
Results.
46

---

Published as a conference paper at ICLR 2026
Box 12. Prompt Example for Visual Grounding (Numerical Labels)
SYSTEM
You are a precise visual grounding model that identifies numerical labels in an image and outputs only structured results.
USER#: You are given image that contains 64 numerical labels (each labeled with a number from to 64).
Your task:
Select the label that lies the visible surface of the <Object Category>.
Avoid choosing any number that lies on occluding or overlapping objects that block the target.
Then, output only one label number (an integer between and 64) that best fits the requirement.
Output format: Wrap your final answer between >>> and <<< for easy parsing.
F.4 ACCURACY OF THE THREE-STAGE VQA
Setting. For basic motion attribute analysis, incorporate voting mechanism to enhance robust- ness. In the agent pose reasoning process, a series of textual label descriptions (e.g., object orien- tations predicted by Orient Anything) are introduced improve the VLM's spatial understanding. During key joint position generation, we overlay a grid of numerical labels on the image to facilitate visual grounding.
We conduct ablation studies on these components to evaluate their individual contributions. The voting mechanism is ablated by using only a single VLM instance for analysis, and the textual label descriptions are ablated by using only the raw image labels without any additional contextual information the prompt. The effect of the label grid examined by comparing the joint generation accuracy with and without the grid, using the prompts shown in Box 11 and Box 12 respectively.
The experimental designs are specified follows:
Basic Attribute Analysis. We randomly sample 100 test cases during the task evaluation process and manually annotate the corresponding ground-truth plans. Each test case in- cludes a motion caption, an agent state, scene observation, and a ground-trut plan. The VLM is prompted as described in Box foxxfand and Box 5 The output of the VLM is considered correct if matches the ground-truit annotation for each motion attribute.
2. Agent Pose Reasoning. Following the same evaluation process as in (1), we select 100 interaction cases involving the agent's 's position and facing direction. For the watch task, we exclude the evaluation of agent-anchor distance since multiple plausible spatial configura- tions may exist.
3. Key Joint Generation. This is formulated as a visual grounding task. We construct test cases using the COCO 2017 dataset (Lin et al. 2014 and evaluate whether the coor- dinates predicted by the VLM fall within the ground-trui truth instance boundaries, allowing tolerance of 10 pixels. Only images that contain a category with exactly one instance are used, where the instance area should occupy 4%-25% of the entire image. Based on these criteria, we filter 500 samples from the 5,000-image validation split and perform the evaluation using GPT-4o mini for cost efficiency.
Results. According to the experimental results in Tab. 12 13 and 14 the voting mechanism achieved a relative improvement of 10.3% in basic attribute analysis accuracy. The combination of Orient Anything and the rule-based textual label description yielded 31.9% relative improvement in pose reasoning accuracy-aae an enhancement primarily observed in the model's understanding of object ori- entation. Meanwhile, the label grid led to 55.9% relative improvement in localization performance,
ss hhtps///cneoddatsst.org
47

---

Published as a conference paper at ICLR 2026
w/o Grid (original)
w/ Grid (ours)
Figure 14: Impact of the label grid on visual grounding of VLM. The result shows that label grid effectively enhance the grounding accuracy.
Table 12: Impact of vot- ing mechanism on motion at- tribute analyzing accuracy.
Table 13: Impact of label de- scription on agent pose rea- soning accuracy.
Table 14: Impact of label grid on key joint position genera- tion accuracy.
Method
Accuracyt
Method
Accuracy!
Method
Accuracy!
Analyzing (wlo Voting)
78%
Pose (wlo Description)
69%
Joint (wlaa Label Grid)
42.2%
Analyzing
86%
Pose
91%
Joint
65.8 %
which aligns with the conclusions of Yang et al. 2023 and Chae et al. (2024). The visualization of the grounding results shown in Fig. 14
F.5 SUPPLEMENTARY ABLATION STUDY
Setting. For the compiler, we employ a VLM for planning, thereby incorporating visual information. During the reasoning process. a rule-based reflection mechanism (refer to Sec. C.2.6 integrated for error detection and self-correction. For the executor, an additional moving-speed control signal introduced. We evaluate the contributions of these components through ablation and test the task success rate after each ablation. Each task runs once in the ablation groups.
The visual information ablation is performed by removing all image inputs in the first two VQA stages while retaining the label descriptions (e.g., the relative distance and orientation of each label with respect to the anchor). Since key joint generation is based on image, we keep visual information in the last VQA stage. To ablate the reflection mechanism, all rule-based detections are removed. For the moving-speed ablation, the corresponding condition mask the motion diffusion set to zero, and all speed-related descriptions (e.g., walking slowly, turning slowly, refer to Sec. C.3. are removed from the compiler.
The results in Tab. 15 show that visual information and reflection both contribute to im- proving the planning quality of the compiler, while the moving speed control effectively enhances locomotion in the scene.
Results.
F.6 CONTROL ACCURACY
Setting. First, we evaluate the control accuracy of BiBo on HumanMI.3D, The control conditions include the facing direction measured in rad, the moving speed measured in m/s, and the joint positions measured m. The control error is represented using the Mean Absolute Error (MAE). Second, sine-cosine encoding is adopted for the facing direction during training. We compare the training convergence curves of raw facing angle and the sine-cosine representation.
Results. The results in Tab. 16 demonstrate that the control parameters provide effective spatial guidance for motion generation. As shown in Fig. [5] employing sine-cosine facing direction en- coding improves both the training stability and convergence.
48

---

Published as a conference paper at ICLR 2026
Table 15: The impact of visual information, reflection and speed control on task success rate. Bold and underline represent the best and second best performance.
Single
Interaction
Composite Task
Reach 7
Watch 7
Sit
Sloap 7
Touch 1
Lift
Simple 1
Mediam
Hard T
BiBo w/o Visual Info.)
96.19
76.24
54.41
Biße (ours, wio Reflection)
96.46
95.48
56.87
81.33
81.56
61.56
50.00
32.69
23.81
BiBo Speed Control)
73.57
72.73
51.77
9.52
Dil5e (ours)
99.18
99.62
95.84
94.89
86.05
65.42
58.82
36.54
27.78
Table 16: The control error (MAE) of BiBo HLmanML.3D dataset, including facing direction (Rot), moving speed (Vel.) and joint positions.
Joint Position
Cond
Rot. trad)
Vel. (m//)
Hand
1. 0.032403
MAE 4
0.177207
0.015499
0.024649
0.033833
0.054909
0.057240
0.032687
Condition Loss Comparison
Diffusion Loss Comparison
0.200
Using Raw Angle
Using Raw Angle
Sine-cosine
0.150
0.125
0.100
0.075
0.4
0.050
0.2
0.025
0.000
0.0
0
5000
10000
15000 20000 25000 30000 35000
40000
5000
10000
15000
20000 25000 30000 35000 40000
Step
Figure 15: Visualization of the training loss variations with and without using sine-cosine encoding of the facing direction. For both the Diffusion Loss and Condition Loss, the sine cosinesine encoding yields more stable training and faster convergence.
F.7 HYPERPARAMETER SELECTION
Setting. We investigate the effects of different learning strategies and model sizes on motion gen- eration. For the learning rate, we experiment with 5e-4, 1e-3, and 2e-3. For the model size, we select hidden dimensions portionally. For the 51 512-dimensional model, we use 8 attention heads. All other parameters remain consistent with the current configuration.
128, 256, and 512, with the feed-forward layer dimension scaled pro-
We evaluate both computational efficiency and generation quality. Computational efficiency is mea- sured by the number of parameters and AITS, while generation quality assessed using FID and Top-1 ~ 3 R- R-Precision.
Results. As shown in Tab. 17 the current configuration (256 hidden dimension, 1024 forw-forward dimension, attention heads, and 5e-4 learning rate) achieves the best generation quality, while maintaining computational efficiency comparable to smaller models. Due to the limited dataset size, larger models significantly increase computational cost without yielding better performance.
F.8
MOTION PREFIX
Setting. We prepend prefix to all motion sequences in the dataset. This strategy improves data uti- lization efficiency and enhances the model's ability initiate motions from scratch or dynamically transfer to new motions. We explore three different prefix augmentation strategies:
Adding an all-zero prefix.
2. Extracting stance motion frame from the dataset and replicating it for 20 frames.
49

---

Published as a conference paper at ICLR 2026
Table 17: Impact of hyperparameter combinations on generation efficiency and motion quality, using HumanML3D, Bold and underline denotes the best and second best performance.
Efficiency
Motion Quality
Hid. Dim.
F.E. Dim.
Head
I.R
Params (M)
AITS
FID
R.P.1T
R.P.21
R.P.3T
256
1024
4
5e-4
28.541
0.047
0.076
0.542
0.738
0.829
256
1024
Te-3
28.541
0,047
0.054
0.536
0.729
0.822
256
1024
2o-3
28.541
0.047
0.091
0.543
0.735
0.826
128
5o-4
20.584
0.044
0.127
0.505
0.703
0.804
Table 18: Comparison of different prefix strategies under two preset conditions.
Ground-trutt Motion Preset
Prefix Motion Preset
Method
FIDJ
R.P.IT
R.P.21
R.P.3T
R.P.IT
R.P.1
R.P.31
0.747
All-zero Prefix
0.086
0.511
0.713
0.809
0.106
0.501
0.703
0.799
Learnable Prefix Token
0.123
0.523
0.721
08818
0.148
0.504
0.704
0.803
Stance Motion Prefix
0.076
0.542
0.738
0.829
0.087
0.525
0.721
0.816
Lylng the bed.
Leaning shelf Waving hand.
the coffee table.
(4) Sit on the sofa and turn on the lamp ATS.
(5) Put your leg on the sofa.
(6) Leaning against the side arm the sofa.
Grab book from the middle shelf.
(8) Grab book from the shelf.
Let's
(10) large plant
Carry
Figure 16: Visualization of BiBo in IsaacGym. BiBo perform type of interaction across multiple object categories (in 3, 4), and execute multiple types of interactions on the same object (in 5, 6). It can generate diverse motions conditioned on text (in 9), and supports precise control (in 7, 8), composite action (in 2, 4), long-range interactions (in 10) and manipulation (in 11).
3. Introducing learnable prefix token.
We evaluate the generation quality on the HumanML.3D dataset. During testing, we preset an initial motion segment and let the model to iteratively extend the future motion, after which the preset portion is trimmed off to compute FID and R-Precision. Two preset motion conditions are tested: (1) using the first 20 frames of ground-truth motion (as in Tevet al. (002)). and (2) using the newly added prefix.
Results. Experimental results in Tab. 18 show that introducing prefix enhances the model's ability to generate motion from scratch. rather than relying on a preset motion history. Among the tested strategies, the mance-motion prefix yields the best performance.
50

---

Published as a conference paper at ICLR 2026
Failure Case 1: VLM fail scene underslanding
Failure Case 2: VLM fail to handle multiple spatial relations
Prompt:
Sit sofa and turn the light beside it, the desk lampl.
Prompt:
Where to stand when sitting on sola, and turning desk lamp?
VLM Answer:
Answer:
Fallure fine-grained action decomposition
Fallure Lack low-level perception
Figure 17: Visualization of failure cases of BiBo. Case and 2 are related to the limitation VLM, and Case and 4 are related the lack of explicit scene geometry modeling (e.g. scene voxel, point cloud) in executor.
F.9
CASE STUDY
Basic Interactions. We evaluate BiBo's ability perform various types of interactions, including basic interactions (e.g., sit, sleep), text-driven motion generation (e.g., dance), long-range interac- tions (e.g.. pointing at), composite actions, and manipulation tasks (e.g., transport).
The prompts used for testing and the corresponding execution results in the simulator are shown Fig. I6 According the results, BiBo demonstrates diversity interaction targets (in 4), interaction modes (in 4, 5, 6), and control modalities (in 9). It also supports precise control (in 7, 8), composite actions (in 2, 4), long-range interactions (in 10), and manipulation tasks (in 11).
For more visual demonstration, please visit our Hugging Face repository
39
Failure Cases. We visualize representative failure cases during task execution. The results are shown in Fig. 17 In case (1), the VLM misinterprets the scene layout (or mismatches textual labels and scene objects), leading to the selection of an incorrect interaction target. case (2), the VLM focuses only on the agent's position relative to the desk lamp while ignoring the sofa's orientation. In cases (3) and (4), because the executor passively receives physical feedback rather than actively parsing the geometric structure of the scene, it shows limitations in fine-grained action decomposition and proactive obstacle avoidance.
Our method still has limitations, but believe that the multimodal capabilities of off-the- VLMs and 3D encoding technologies continue to advance, these problems are expected to be re- solved in the near future.
Ablation of Speed Control. During experiment, we observe that the moving speed not only en- hances controllability, but also serves as signal indicating whether continue the motion to stop the current one, which enhance motion coherence.
As shown in Fig. 18 in (1), the agent with speed control precisely reaches the target location, while the agent without speed control overshoots the target and then turns back. In (2), the agent with speed control keeps walking status between successive walking motions, while the agent without speed control tends to pause between successive walking motions, as it cannot determine whether to stop or keep moving. The agent with speed control demonstrates better controllability and motion coherence.
F.10 DESIGN OF USER STUDY
We design User Study questionnaire to assess the perceived quality of the generated results. Each questionnaire consists of 5 groups of motions and interactions, and each group contains three sam- ples, generated by BiBo, MotionLCM, and DiP (CLoSD) using the same prompts, respectively.
The prompts are randomly selected from the HumanML3D dataset, and each questionnaire is unique The three samples are randomly arranged from left to right and displayed using a con-
51

---

Published as a conference paper at ICLR 2026
(1) Controllability
w/
speed
w/o
speed
(2) Coherence
w/
speed
w/o
speed
Figure 18: Visualization of locomotion with and without moving-speed control. In (1), the agent with speed control precisely reaches the target location, enhancing controllability. In (2), the agent without speed control tends stand still between successive walking motions, as cannot determine whether to stop or keep moving, lacking coherence.
Table 19: Comparison of BiBo and TESMO on motion quality (FID, R- R-Precision, Diversity), phys- ical plausibility (Skating) and control accuracy (Goal Reaching Error and Orientation Error). BiBo demonstrates enhanced perfomance compared with TESMO.
Method
FID 1.
R-Precision
Diversity
Foot Skating
Goal Reaching Error
Orientation Errur +
Ground Truth
0.314
9.503
0.21
0.2410
BiBo (ours)
0.076
0.542
9.606
0.74
0.0246
0,1772
BiBo (ours, Phys)
1883
0.411
8.298
0.01
sistent skeletal structure with line segments. The range of motion on the XZ-axes represented by gray ground. To prevent bias, the samples are labeled only with their prompts, without indicating generation method.
Volunteers are asked to select the best-generated sample within each group. The corresponding method receives one point for each selection. total of participants evaluate 450 samples (150 method), with maximum of 150 points available. The questionnaire is hosted the Google
Docs 10 platform, with an example provided in the Fig. 19
The results are shown in Tab, 6
F.11 ADDITIONAL COMPARISON METHODS
F.11.1 NON-PHYSICAL HUMAN SCENE INTERACTION
We compare the generation quality, physical plausibility, and controllability with non-physical human-scene interaction methods. We use TESMO (Yi et al. 2024) as the comparison method, with the data sourced from the original paper.
The results in Tab. 19 demonstrates that BiBo effectively enhances motion quality (20.465-x0.766) and control accuracy 00.1445-0.0244 comparied with TESMO.
F.11.2 RL-BASED METHODS
RL-based methods like Wang et al. (2024a) based on fixed skill set, which may be limited in diversity and motion naturalness. Bißo uses a VLM guide diffusion generator, and the generated motions are tracked by an RL policy. This method generates diverse and natural motions, while maintaining scene awareness and physical plausibility.
hhtpps//doos.google.cooo//////
52

---

Watch, Pointing at Published as a conference paper at ICLR 2026
User Study
I U
x
Please review three generated motions (Left, Middle, Right). one you think overall?
:02
Question
with amms side.
with side.
with side.
Left
Middle
Right
Figure 19: An example of the User Study questionnaire on Google Does. Each questionnaire con- sists of 5 groups of generated samples. Each group contains 3 samples generated by different meth- ods, as shown in the figure. Volunteers are asked to select the sample with the best quality to vote for the corresponding method.
Table 20: Comparison of executable skills between BiBo and RL-based methods. BiBo is capable of performing diverse motions.
Method
Idle, Walk
Jump. Jug, Crawl
Sit, Lie
Punch, Kick
Lift, Carry
Human LA (Xu all 2024
UniHST (iao 2023
/
SIMS (Wang et al 20246
X
(miss)
/
Skill. We first qualitatively compare the capabilities of the methods. As shown in the Tab. 20 RL- based methods typically have a limited skill set, while BiBo can generalize to a broader range of interaction skills.
Motion Quality. We further compare the motion quality with RL-bbasdd methods. Specifically, we place single interactive object in the scene and initialize the humanoid agent at random position in front of it, facing the object. The control signals are generated by the program. We convert the collected motion into the HumanML33 format and calculate its FID indicator of motion quality.
We choose InterPhys, UniHSI, and SIMS comparison methods, with the data sourced from SIMS. The results Tab. Tab.[2]]dmmae demonstaate that BiBo's 's design improves the naturalness of motion for LLM- based humanoid agents.
53

---

Published as a conference paper at ICLR 2026
Table 21: Comparison of motion quality for Sit, Lie, and Carry actions.
Table 22: Impact of motion context lengths Top-1 R- R-Precision.
Method
Sit 4
4
Carry
InterPPlys
2025
81.0
Motion Context Length
Top-1 R-Precision
UniLISI Oiao at (025
153.84
211.22
Frame
0.423
SIMS (Wang al 20244
20 Frames
0.542
BiBo (ours)
68.78
134.46
34.72
40
0.564
Start dancing
Sit the sofa
Stand up from the sofa
Figure 20:
Visualization of the execution results BiBo on the Unitree robot platform. The robot can perform diverse motions and is capable of performing physical scene interaction.
F.12 LENGTH OF THE PREVIOUS MOTION CONTEXT
Setting. BiBo extends future motion using 20-frame prefix composed of both generated and executed motion, which ensures motion continuity. However, we are also aware that a 20-frame prefix may be insufficient for maintaining long-horizer continuity.
Increasing the prefix length is promising direction for future work. We provide some preliminary experiments using longer context. Instead of simply concatenating past motion, we follow the idea of FramePack (Zhang & Agrawala 2025) and construct motion context at multiple frame rates improve efficiency. The evaluation is conducted on the HumanM1.31) dataset, comparing the generated results in terms of Top 1 R R-Precision.
The results in Tab. show that longer context introduce better text consistency (+4.1%), which also indicates better long-term continuity.
F.13 EVALCATION ROBOTIC PLATFORM
Our proposed method focuses on physical Human Scene Interaction (HSI) modeling, aiming to leverage off-the-shelf VLMs for humanoid agents. The potential downstream applications of this approach extend to both computer animation and real-world humanoid robot control. While our method's motivation differs from traditional robot operating systems, real-world validation can pro- vide valuable insights into its downstream application potential.
Therefore, we tested BiBo on the robot body. First, we attempted to transfer the model the robot body. Then, we performed real-world deployment. Deploying to real-world robot requires addi- tional processes such as SLAM and low-level joint motor development, which require significant time, resources, and close collaboration with the robot manufacturer. Due to resource constraints, we only deployed the Embodied Instruction Compiler.
F.13.1 MIGRATING TO HUMANOID ROBOT BODY
Implementation. We keep the Embodied Instruction Compiler and motion diffusion unchanged, while changing the Tracking Policy from the SMPL body to the robot body. Specifically, we use the Unitree GI humanoid robot platform The Unitree GI has 29 degrees of freedom (DOF), including 12 leg joints and 17 upper body joints, but no head joints.
#//nps//www..miire.cmm//l
54

---

Published as a conference paper at ICLR 2026
Table 23: Task success rates of BiBo on Unitree GI platform. Bißo robotic platform shows comparable performance with methods on vir- tual characters.
Method CL.SDD (Tevet 2024
Body
Reach 1
Sit
Stand "T"
SMPI.
100%
88%
BiBo (ours,
Unitree
100%
94%
86%
Table 24: Comparison of task success rate in
Sit on the chair in the middle
real-world environment.
Task
Reach
Watch
Sit
Figure 21: Visualization of the results of
LeVERB (uue ata 2025
20
-F-
1720
Embodied Instruction Compiler in real-world scene.
HumamoidVLA
2025
9/10
10/10
BiBo (ours)
10/10
10/10
8710
We train the Tracking Policy using the Unitree G1 training repository provided by PHC Luo et al, 2023 and refer the official Unitree RL Gym train the Tracking Policy on the generated motions and basic tasks. The proposed Motion Diffusion generates a 263-dimensional joint feature in HumanML33D format. We first convert the joint features into 22 joint coordinates, and then use Smplify (Bogo al. 2016 44 to fit the SMPL (Loper et al., 2015 skeleton. The SMPL skeleton includes 24 joints, and the features include the bone rotations. we redirect the obtained SMPL
43 Following CLoSD (Tevet
2024). we
Specifically, we use Smp By using Retarget2Humanoid skeleton to the GI robot body and iteratively optimize the rotations for the G1's 29 joints. Note that the original PHC uses 39 DOF Unitree G1 body with 19 rotatable joints in the motion dataset.
Results. We test the robot's motion execution and scene interaction capabilities using IsaacGym. Specifically, evaluate reach, sit and stand. Each task is tested 50 times under different initial position conditions. The task success rate criteria remain consistent with CLoSD.
Fig. 20 shows the visualization of the robot's execution results. The results in Tab. 23 indicate that while the robot exhibits motion diversity, it is also adaptable to the physical scene interaction.
F.13.2 REAL-WORLD DEPLOYMENT OF EMBODIED INSTRUCTION COMPILER
Implementation. Deploying the Embodied Instruction Compiler onto real hardware requires odom- etry, visual sensors, and motion control.
The MID360 Lidar SLAM service provided by the manufacturer is unavailable robot. As alternative, we use a plane-matching algorithm to match wall surfaces based on Lidar point clouds, enabling rough localization.
The Unitree G1 is equipped with a RealSense D435i RGBD depth camera on its head. However, the head cannot rotate, and the downward angle (48") is too low for scene image capturing. Therefore, we attaches stationary RealSense D435i camera to observe the scene. The camera is calibrated using visual features.
The experimental environment is relatively narrow, and some RL-policy models only take the target location as input, which leads to low controllability and increases the risk of equipment damage, For safety consideration, we use the motion controller from the manufacturer. The controller includes basic locomotion and actions, which are invoked via the RPC interface control the robot's speed or movements.
" hhttp://ghhuubcom///heeggil/u//u
43 https//gggluuccooouuuieeerroorle
"hups://smplify.ik is. tue.mpg.de/
" "huts://smpl..s.ue www..u..ep..d.....
4 "htps://gitubb.cm/// wwwp///aaa/eeeeemtttoonniiiiiiii Tecver/mttionndiiffsiioonoo
47 http:///ghhhhb.eooooo//////////n
55

---

Published as a conference paper at ICLR 2026
Table 25: Comparison of task suc- cess rate of UniHSI and BiBo on ScanNet ScenePlan dataset. The results show that BiBo achieves plausible generalization real- world scanned scene.
Success BiBo (ours)
Fallure BiBo (ours)
Success UniHSI (Maao 2024] UniHSI (Xiso 2024]
Fallure
Method
Simple
Middle
Hard
UnitISI
73.2
43.1
22.3
Figure 22: Success and failure cases of BiBo and UniHSI on ScenePlan.
BiBo (ours)
81.7
65.2
41.5
Table 26: Comparison of interaction enror between different methods. BiBo demon- strates precise interaction capability even without specifically RL-learning for contact tasks. IK is essential for precise interaction.
Table 27: Comparison of R-precision for scene interactions across different methods. Higher R R-precision indicates better alignment with the instructions. demonstrating stronger coherence. According to the results, BiBo exhibits instruc- tion coherence during interaction.
Method
Pelvis L
Left Hand i Right Hand
UniHSI
0.126
0.181
0.094
Method
f
Carry
BiBo (ours)
0.109
0.079
0,080
Unil
17.12%
9.59%
BiRo cours, IK)
0.107
0.101
BiBo (ours)
25.34%
17.1%
21.23%
Unlike in virtual scene, the real-world environment does not have object instance annotations. We additionally use the TinySAM (Shu et al. 2025 48 model for real-time zero-shot image segmentation. VLM annotates SAM prompt (a point) the label grid-enhanced image, thereby extracting the mask for the target instance. The segmented images are then fed into Orient Anything for orientation prediction. Path planning performed by generating a height map from Lidar point clouds, which is then thresholded to create an obstacle map.
various tasks in real-world scenarios, in Fig. 21 The move and watch tasks validate the compiler's ' understanding of the scene and object attributes, while the sit task validates the robot's ability interact with the scene. We use chairs with different shapes as target object, and specify them by natural language description (c.g.. the chair without tray, the chair in the middle). Due to the limitations of the motion controller, sitting down requires combined action of moving backward and sitting. As result, we placed the chair against the wall.
Results. We the robot's success
The results in Tab. 24 show that the proposed compiler achieves plausible results scene under- standing and interaction. The robot succeeds in all the walk and watch tasks, and completes the sit task with a high success rate. The failure cases in the sit task are mostly due to inaccurate localiza- tion (as use simple algorithm for odometry). The robot's 's stance deviated from the front of the chair, preventing it from completing the sitting action.
F.14 ADDITIONAL SCENE INTERACTION EVALUATION
F.14.1 ADDITIONAL BENCHMARK
Setting. We conduct experiments on ScanNet ScenePlan benchmark from UnIHSI. We evaluate each task three times under different initial conditions. For more details of ScenePlan dataset, please refer to Sec. B.3
Results.
The experimental results in Tab. 25 show that BiBo achieves plausible success rates across tasks of varying difficulty, and generalizes well to real-world scene data from ScanNet.
Failure Cases. We analyze the failure cases of BiBo and UniHSI. As shown in Fig. 22 most of
BiBo's failures come from collisions with narrow scene areas, causing the agent to fall. We attribute this to BiBo's 's exccutor passively accepting environmental feedback rather than actively observing the environment, which is a direction worth exploring in the future. On the other hand, while UniHSI actively perceives the environment through height map, it is trained on PartNet with specific object
56

---

Published as a conference paper at ICLR 2026
shapes and relies manual annotations. As result, it struggles with generalizing across different scenes and objects in ScanNet, resulting in abnormal movements.
F.14.2 INTERACTION ACCURACY
Setting. While the Contact Error proposed by UniHSI provides valuable insights, remains limited in accurately reflecting interaction accuracy. Specifically, Contact Error is defined as follows:
Contact Error
E1 io 1,4770
e
f| ll: min ||d O).
G contact
(19)
C not contact
represents an action, and dı; denotes the shortest distance from the joint to the surface point cloud of the target object. G includes navigation waypoint, whose errors 6; predominantly O. When the number of navigation waypoints increases, the contact error decreases, despite no actual improvement interaction accuracy.
Therefore, we propose Interaction Error, which is more closely related to the interaction, excluding the interference of navigation waypoints:
Interaction Error
Σ
e//
Σ
1,
C3 |
(20)
1,c:=cootact
i,c:=oontact
We test the Interaction Error of three frequently used joints - pelvis, left hand, and right hand during contact interactions. Non-contact interaction accuracy evaluation can be referred in Sec. F.6, which achieves high control accuracy.
The experiments are conducted on interactions related the specified joints. Pelvis is tested with sit and sleep task, while hands are tested with touch task.
Results. According to the experimental results in Tab. 26 BiBo outperforms UniHSI, which is specifically designed for contact tasks, in interaction accuracy. There are two reasons:
1. IK effectively improves the interaction accuracy of end effectors.
2. BiBo uses Tracking Policy, which dynamically adjusts actions to achieve higher precision in interactions.
We also observe that the RL-learning in UniHSI led to lower interaction accuracy for the left hand compared to the right hand. BiBo uses motion diffusion trained on mirror augmented data, achieving high interaction accuracy for both the left and right hands.
1.13.3 INTERACTION NATURALNESS
We evaluate the motion quality of different HSI methods. For more details, please refer to Sec. 1.1
and Tab. 21
1.14.4 INTERACTION COHERENCE
Setting. Task Coherence assessed via R-Precision. R-Precision quantifies the probability of retrieving the correct motion within data batch, given textual motion caption. For interaction motions, R-Precision indicates whether the execution conforms to the task description; for example, Carry should involve holding the object between both hands rather than pushing
To evaluate R-Precision, we collect motions from physics-based interaction processes. Since R- Precision computation requires batches containing diverse motion categories, randomly substi- tute one sample in each real-world data batch with an interaction motion, and perform retrieval using the corresponding task instruction.
Results. According to the results in Tab. 27 BiBo generates interaction motions consistent with instructions. This demonstrates that the proposed method possesses high task coherence.
57

---

Published as a conference paper at ICLR 2026
Comparison of generation quality (FID) and diversity on the AMASS dataset. + indicates that closer to ground truth better. The results show that BiBo generates motions with plausible FID without fine-tuning, and ap- proaches the real world motion distribution.
Table 28:
Table 30: Comparison of generation quality (FID) across each subset of AMASS (Part 2). The results show that BiBo demonstrates advan- lages every subset, and datasets containing more real motions exhibit greater similarity to the motion distribution generated by BiBo.
Dataset Overall FTD +
G.T.
BiBo (ours)
DiP
Dataset
BiBo
DIP
Motions
0.0000
4.799
8.9664
FKUT
3,5976
9.5299
332
Overall Diversity +
7.9887
8.0127
8.1430
5.4948
7,8679
231
GRAB
4.6874
9.9674
1840
HUMANAD
5.2315
7,9081
468
Table 29:
Comparison of generation quality (FID) across each subset of AMASS (Part 1).
HumanEva
7.2032
10,7560
KIT
4.9822
10.2390
5545
MOYO
4.6147
7.8556
1171
MPLHHMMSS
5.7478
8.1308
945
Dataset
BiBo
DIP
Motions
MPL.i
5.5547
8.9683
139
4.5684
10.2789
5.0530
13.0224
BMLhandball
4.83002
9.0686
885
SFU
6.0829
9.0587
113
BMLmovi
4.7213
10.7736
1910
SOMA
4.5367
9,6309
BioMotionLaa. NTruje
4,5839
9.2667
4323
SSM
11,2186
13,5802
CMU
4.7682
8.8602
4129
TCD handMocap
7.0114
10.4370
CNRS
10.6496
TotalCapture
4.7316
7,4846
259
DFausL67
4.9707
8,5679
138
Transitions mocap
4.3569
10.8853
125
DanceDB
5,6532
7,2548
WEIZMANN
4.9802
9,0565
Table 31: Motion generation quality comparison of different methods on BABEL dataset with 263- dim motion representation. According to the results, BiBo demonstrates advantages generation quality and diversity.
Method
Continuity
FID
Score |
R.P.3T
R.P.S 1
R.P.IOT
Diversity +
Ground Truth
5.6281
0.3188
0.4467
0.6474
8.0907
Double Take (MDM)
13.6735
8.4137
0.1721
0.2691
0.4483
6.8614
MoticnLCM CLeSD
7,8812
5,1991
0.3739
0.5003
0.6970
7.9657
11.2371
7,1783
0.1931
0.2908
0.4900)
7.4086
BiBo (ours)
5.7789
5,0965
0.3971
0.5269
0,7229
8,5351
F.15
ADDITIONAL GENERATION QUALITY EVALUATION
F.15.1
GENERALIZATION TO AMASS
Setting. To validate the generalization of our method, we test the motion generation quality on the AMASS dataset. To better demonstrate the generalization capability,
the model trained on HumanML-3D, and directly test it on AMASS without fine-tuning. Since the AMASS dataset lacks text annotations, we primarily evaluate FID and Diversity, representing motion quality and diversity.
The results in the Tab. 28 demonstrate that BiBo exhibits generalization capability on the AMASS dataset. The generated motions and diversity are more aligned with the real distribution compared to DiP 1 (FID 9.0-+4.8).
Results.
We further test BiBo's FID on each subset of AMASS. The results in Tab. 29 and show that BiBo demonstrates advantages across all subsets. Notably, the larger the dataset scale (containing richer real motions), the better the distribution similarity achieved by BiBo. These results demonstrate that BiBo is capable of generating motions with high realism and possess strong generalization ability.
F.15.2 GENERALIZATION TO BABEL
Setting. We evaluate the rext-to-motion generation performance of the proposed motion diffusion on 263-dim BABEL, using the model parameters trained on HumanML33. During inference, upon reaching the starting frame of a transition, we switch the control signal to the next motion's 's prompt, and immediately generate the next motion segment.
Comparison Methods. Double Take is a segment composition based method. DiP is a future prediction- based method. MotionLCM supports only fixed-length generation. Note that the officially released 263-dim MDM variant of Double Take. All comparison methods use the official released checkpoints trained H33.
58

---

Published as a conference paper at ICLR 2026
Table 32: Computational overhead during inference on personal laptop and workstation. The results show that the proposed model has low overhead with high inference speed.
UTU
VRAM 2,5GB
CPU UTL
Personal Laptop
83%
12.8%
2.7G
0.0068
Workstation
24%
2.1GB
10.4%
L3GB
0.048s
Results. The results in Tab. 31 show that BiBo achieves enhanced FID J. (7.9+5.8) and R-precision t (0.37~+0.39) on BABEL dataset. These results demonstrate that BiBo is capable generating realistic human motions over long horizon.
F.16 REAL-TIME PERFORMANCE
Setting. We additionally deploy and test our model on two platforms:
1. A consumer-gren grade personal laptop. The laptop is equipped with 2.3GHZ CPU, 32GB of RAM, and a 6GB RTX 3060 GPU, rumning Windows 11.
2. A graphics workstation. The workstation is equipped an 3.2GHz i9 14900K CPU, 128GB RAM, and 24GB RTX 4090 GPU, running on Ubuntu 24.04.
We load the proposed motion diffusion model at float32 precision and perform 10,000 inference runs, averaging the performance metrics. We measure GPU utilization, GPU memory utilization, CPU utilization, RAM utilization, and inference time.
The results in Tab. 11 show that the proposed method occupies only 2.1 GB VRAM and 1.3 GB RAM during inference. runs at an AITS of 0.048s on RTX4090, while maintaining an AITS of 0.066s on a personal laptop. This indicates that BiBo can perform real-time inference with low resource consumption.
Results.
59


Reviews:
[{"review_id": "81ndCq1Qfk", "reviewer": "Reviewer_u3aD", "summary": "The paper targets the interesing problem of humanoid agents which can handle flexible and diverse interactions in open environments. To avoid collecting massive dataset to train the model, the paper presents a new solution by utilizing the capability of the strong VLMs and a diffusion-based motion generator. The former is used to generate the primitive commands based on the user instruction and the latter is used to generate the corresponding motions. Experiments show the promising results of the proposed algorithm.", "questions": "Please address the questions in the weakness section. More specifically, please mainly address the questions related work the experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of utlizing the strong capability of VLM to decompose the primitive commands and later handled by a motion generator is interesting. \n\n* The experiments show the proposed algorihtm obtains promising results in the challenging problem of interaction with the open environments. \n\n* The paper is well presented and the proposed algorithm should be easy to reproduce.", "weaknesses": "* The paper relies on two components to handle the target problem. On one hand, currently, even the SOTA vlms may not be able to produce the precise primitive actions. To simply the problem, the paper presents a set of predefined actions but it still cannot guarantee a robust results. On the other hand, assume vlms can produce the accurate action motions, how to obtain a good motion is not a trivial task. It should provide more justification that why the presented motion executor can produce the desired results.\n\n* The experimental results in Table 1 is evaluated based on a setting proposed by this paper. The task as well as the setups are introduced in this paper. How about the generation of the proposed algorithm to other benchmarks? \n\n* For the results in Table 2, the result of \"Lift\" is much lower than other categories. What are the potential reasons for this? \n\n* Currently, BiBo are operated in the virtual setting. Is it possible to proivde some evaluations which can show that the proposed algorithm can generate to real-cases like robots?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MB4XWncktq", "reviewer": "Reviewer_U7bF", "summary": "This paper proposes BiBo (Building humanoId agent By Off-the-shelf VLMs) — a framework that connects general-purpose Vision-Language Models (VLMs) like GPT-4 to humanoid control. The key idea is to use powerful pre-trained multimodal models to bypass costly humanoid-specific data collection and training. BiBo has two major components: 1. Embodied Instruction Compiler – translates high-level natural language instructions (e.g., “have a rest”) into structured, low-level control commands (e.g., sitting location, facing direction, joint targets) through a three-stage visual Q&A process. 2. Diffusion-based Motion Executor – a latent diffusion model (LDM) that generates continuous, physically-plausible humanoid motion conditioned on those commands and on real-time physical feedback.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of directly plugging an off-the-shelf VLM (GPT-4o) into a humanoid control pipeline is innovative. Avoids re-training large models by adding a lightweight compiler layer.\n2.\tThe compiler–assembler analogy is clear and intuitive: the VLM acts like a “compiler” converting high-level language into structured commands, while the motion diffusion module serves as an “assembler” for physical actuation.\n3.\tThe Latent Diffusion Model with joint decoding of executed and generated latents ensures temporal continuity and environmental awareness — addressing a major weakness in previous motion diffusion frameworks.\n4.\tThe experiment is also comprehensive", "weaknesses": "1.\tThe motions shown in the video do not fully comply with physical laws. During interactions with objects, there are visible cases of hovering and penetration, which make it appear that the human keypoints are rule-based attached to the objects rather than physically constrained. The interactivity seems weaker compared with methods such as UniHSI.\n2.\tThe motion generation in the video appears to heavily depend on the VLM’s outputs. However, the VLM tends to exhibit strong hallucination problems during grounding. It is unclear how the authors constrain the frame-to-frame consistency — both in the diffusion process and within the Embodied Instruction Compiler.\n3.\tDuring locomotion, the agent sometimes floats or teleports. Yet, in Table 3, BiBo is reported to outperform others on the skating and floating metrics. Providing a more detailed definition and evaluation standard for these metrics would make the results more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAzlbREWg3", "reviewer": "Reviewer_V4s2", "summary": "This paper introduces BiBo (Building humanoId agent By Off-the-shelf VLMs), a novel framework that leverages off-the-shelf Vision-Language Models (VLMs, such as GPT-4) to control humanoid agents. The core idea is to reduce data collection and training costs by combining pre-trained VLMs with a tailored embodied system for humanoid control. BiBo consists of two main components:\n\n- Embodied Instruction Compiler: Converts high-level natural language commands into low-level structured motor commands by reasoning over scene context.\n- Diffusion-based Motion Executor: Generates smooth, human-like motion trajectories while dynamically adapting to environmental feedback using a combination of Latent Diffusion Models (LDMs) and Inverse Kinematics (IK) optimization.\nThe authors highlight BiBo's ability to perform diverse and complex physical interactions in dynamic environments, achieving a task success rate of 90.2% and improving motion execution precision by 16.3% compared to prior methods.", "questions": "Please see the weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- VLM Agent Workflow for Complex Task Understanding\n\nThe Embodied Instruction Compiler is well-designed, using a structured three-step reasoning process (attribute analysis, pose reasoning, and joint generation) to translate high-level commands into low-level motor instructions. This design allows BiBo to accurately interpret user intent and adapt to complex tasks in dynamic physical environments, such as sitting, lifting objects, or interacting with multiple scene elements. The use of voting mechanisms and multi-view representations further improves the system's robustness in understanding intricate tasks.\n\n- Novel Integration of CLoSD + IK for Diffusion Motion Updates\n\nThe combination of CLoSD (a physics-based motion-tracking policy) and Inverse Kinematics for refining humanoid motion is inspiring. The framework dynamically corrects motion trajectories by incorporating physical feedback from the environment, ensuring smooth and continuous motion even in challenging scenarios (e.g., collisions, external forces). This joint optimization approach enhances both the adaptability and precision of motion generation, particularly for tasks requiring fine-grained control (e.g., grasping or touching objects). The use of Latent Diffusion Models (LDMs) further enables the generation of high-fidelity motions while maintaining computational efficiency. This part could be considered the most inspiring in this paper.", "weaknesses": "- Unclear Execution of Motion with CLoSD for Dynamic Objects\n\nThe paper lacks clarity on how the generated motion trajectories are passed to CLoSD for execution. For instance, when an object moves unpredictably, does the system rely on CLoSD alone for tracking, or does it dynamically update the motion plan using feedback? What if the dynamic object encounters collision with hands? While the authors mention incorporating physical feedback into motion updates, the explanation of how BiBo handles motion retargeting or re-planning in the presence of dynamic objects is insufficient. This aspect deserves more detailed discussion and evaluation.\n\n\n- Limited Discussion and Citation of Related Work\n\nThe paper does not sufficiently discuss its approach to prior work in key areas, such as LLM planning, long-term task completion, or the use of diffusion models in HSI motion generation. For instance:\n\n[1] SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation. ICCV2025\n\n[2] Synthesizing Physically Plausible Human Motions in 3D Scenes. 3DV2024\n\n[3] Generating Human Interaction Motions in Scenes with Text Control. ECCV2024\n\nIt is clear that TESMO[3] is exactly the type of previous approach this paper aims to compare against: it introduces discontinuity by conditioning on past generated rather than executed results.\nAt the very least, providing sufficient discussion and citations would improve my impression of this paper.\n\n- Not complicated enough environments as claimed\n\nMost environments in demo videos feature a single piece of furniture on flat ground, not challenging enough as claimed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BeOF08n2l2", "reviewer": "Reviewer_hAat", "summary": "This paper proposes the BiBo framework, which aims to leverage off-the-shelf Vision-Language Models (VLMs) to control humanoid agents for interaction in open physical environments. The paper designs an embodied instruction compiler and a diffusion model-based motion executor to realize the translation from high-level instructions to low-level actions, and verifies their effectiveness on multiple tasks.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "See Summary.", "weaknesses": "1.The core innovations of this study are \"reducing data dependency via off-the-shelf VLMs\" and \"solving motion continuity issues using LDM (Latent Diffusion Model) + VAE (Variational Autoencoder)\". However, the generalization ability in real physical environments has not been fully verified. BiBo’s interaction capability has not been validated in real-world scenarios (non-InfiniGen-generated simulated environments), making it impossible to prove its effectiveness under complex real conditions such as lighting changes and irregular objects.\n2.The dataset used in this study has limitations, and its generalization is questionable. Experiments are only conducted on the HumanML3D dataset and InfiniGen-generated scenes, without validation on other public datasets (e.g., BABEL, AMASS). This makes it difficult to demonstrate the model’s generalization ability across a wider range of action types and scenes.\n3.The evaluation metric system is incomplete. Although FID and R-Precision are used to evaluate motion quality, quantitative analysis of key dimensions in interaction tasks—such as \"interaction naturalness\" and \"task coherence\"—is omitted. It is recommended to supplement quantitative analysis of metrics like Interaction Accuracy and Task Coherence.\n4.There is a lack of real-time performance analysis. Although the paper claims that BiBo supports real-time control (>20 Hz), it does not provide analysis of specific memory usage or GPU memory consumption. Performance benchmarks on typical hardware platforms should be provided to verify its deployment feasibility.\n5.There is no systematic analysis of the method’s limitations. The paper does not discuss BiBo’s performance in extreme scenarios (e.g., occlusion, lighting changes) nor analyze the impact of VLM reasoning errors on the entire system. Such discussions should be supplemented in the main text or appendix.\n6.In the ablation study (Table 2), control groups for \"without Pose Reasoning (w/o Pose Reasoning)\" and \"without Joint Generation (w/o Joint Generation)\" need to be added to quantify the impact of each stage on task success rates (e.g., positioning accuracy for sitting tasks, joint control accuracy for touching tasks) and clarify the necessity of the three-stage process.\n7.It is recommended to supplement failure case analysis in the main text or appendix, including the category of failed tasks, root causes (e.g., pose recognition errors, insufficient collision handling), and potential improvement directions, to enhance the transparency of the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)