InfBaGel: Human-Object-Scene Interaction Generation with Dynamic Perception and Iterative Refinement
Authors: Yude Zou
Junji Gong
Xing Gao
Zixuan Li
Tianxing Chen
Guanjie Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: Interaction Generation
Consistency Model
Human Motion
Primary area: generative models

Abstract: Human–object–scene interactions (HOSI) generation has broad applications in embodied AI, simulation, and animation. Unlike human–object interaction (HOI) and human–scene interaction (HSI), HOSI generation requires reasoning over dynamic object–scene changes, yet suffers from limited annotated data. To address  these issues, we propose a coarse‑to‑fine instruction‑conditioned interaction generation framework that is explicitly aligned with the iterative denoising process of a consistency model. In particular, we adopt a dynamic perception strategy that leverages trajectories from the preceding refinement to update scene context and condition subsequent refinement at each denoising step of consistency model, yielding consistent interactions.
To further reduce physical artifacts, we introduce a bump‑aware guidance that mitigates collisions and penetrations during sampling without requiring fine‑grained scene geometry, enabling real‑time generation. To overcome  data scarcity, we design a hybrid training startegy that synthesizes pseudo‑HOSI samples by injecting voxelized scene occupancy into HOI datasets and jointly trains with high‑fidelity HSI data, allowing interaction learning while preserving realistic scene awareness. Extensive experiments demonstrate that our method achieves state‑of‑the‑art performance in both HOSI and HOI generation, and strong  generalization to unseen scenes. Project page: [yudezou.github.io/InfBaGel-page](https://yudezou.github.io/InfBaGel-page/).

Published as a conference paper at ICLR 2026
INFBAGEL: HUMAN-OBIECT-SCENT INTERAC- TION GENERATION WITH DYNAMIC PERCEPTION
AND ITERATIVE REFINEMENT
Yude Zou Junji Gong", Xing Gao?; Zixuan Li Tianxing Chen', Guanjic Zheng' SShnnghai Jiao Tong University, Shanghai Artificial Intelligence Laboratory, SSihhuan University, 'Shenzhen University, The University of Hong Kong
ABSTRACT
Human-object-scene interactions (HOST) generation has broad applications in embodied AI, simulation, and animation. Unlike hujan-object interac- tion (HOI) and human scene interaction (HSI), HOSI generation requires reasoning over dynamic object scene changes, yet sulfers from limited an- notated data. To address these issues, we propose coarse-to-fin instruc- tion conddiiiii interaction generation framework that is explicitly aligned with the iterative denoising process of consistency model. In partic- ular, we adopt a dynamic perception strategy that leverages trajectorics from the proceding refinement to update scene context and condition sub- sequent refinement at each denoising step of consistency model, yiekding consistent interactions. To further reduce physical artifacts, we introduce a bump-aware guidance that mitigates collisions and penetrations during sampling without requiring fne-grained scene geometry, enabling real-time generation. To overcome data scarcity, we design hybrid training strat- egy that synthesizes pseudo-HOST samples by injecting voxelized scene oc- cupancy into HO1 datasets and jointly trains with high-fidelity HSI data, allowing interaction learning while preserving realistic scene awareness. Ex- tensive experiments demonstrate that our method achieves state-of-the-art performance in both HOSI and HOI generation, and strong generalization unseen scenes. Project page: Tuueeouu.iihhu.... http:aittibbooolilaaaleeleoeooe InggaGel-page
INTRODUCTION
In daily life, human activities frequently involve manipulating objects while navigating clut- tered environments and interacting with surrounding static elements. For instance, an individual might carry a chair to a target location, avoid obstacles, place down, and then sit on it. Generating such integrated interactions, known as human-object-scomm interaction (HOSI) generation, essential to advance animation, simulation, and game applications.
HOSI generation requires reasoning about the complex interplay among the human, the manipulated object, and the surrounding scene, introducing several challenges: (i) Dynamic Object-Scene Changes: The scene context evolves continuously with human navigation, while the manipulation of sizable objects further reshapes the scene layout. These dynamic changes complicate the generation of consistent interactions among humans, objects, and scenes elements. (ii) Annotated Data Scarcity: Capturing high-quality, diverse human- object-scene motion data is challenging due to the combinatorial explosion of object types, scene configurations, and task instructions, which requires substantial human and computa- tional resources. The resulting lack of diverse datasets with comprehensive scene annotations hinders the development of generalizable generative models.
Recent advances have improved either human-object interaction (HOT) or human-scene in- teraction (HSI) generation, but they remain insufficient in addressing the multifaceted chal- lenges of HOSL. For example, recent scene-aware pipelines (Jiang al. 2024b a) adopt a one-off static scene encoding, generating motion without updating the scene state influ- enced by human or object movement. Additionally, some stage-wise decomposition methods
Work done when Yude Zou was an intern al Shanghai Artificial Intelligence Laboratory. "Correppnndigg Author.

---

Published as a conference paper at ICLR 2026
Zhang et al. 2024 Yao et al. 2025 separate locomotion from interaction, compromising temporal consistency. Planner-dppendee introduce computational costs and depend heavily planner quality. At the data level, existing datasets remain limited. Some (Li et al.) 2023) lack scene annotation, while others (Jiang et al. 2024b a) offer restricted object diversity and lack diverse large manipulable objects. These gaps motivate scene-aware and data-efficient HOSI generation framework.
approaches (Yi et al.) 2024 Geng et al. 2025
To address these challenges, we propose INFLAGEL, an autoregressive framework that gen- erates HOSI from high-level instructions such as goal locations and textual commands. Our approach adopts a coarse-to-fin generation scheme with a consistency model that performs few-step iterative refinement during denoising, conditioned on dynamic perceptual repre- sentations of the environment, achieving substantially faster inference than diffusion-based baselines. Furthermore, employ a hybrid data training strategy that eliminates the need for annotated HOST data, enabling the model learn diverse, complex, and coherent inter- actions. Although trained for HOSI, the model generalizes naturally to various HSI tasks, like locomotion and static-object interactions. The INFBAGEL (generating interaction from bump-aware guidance in real-time) is characterized by:
coarse-to-fine conditional interaction gencration framework aligned with the few-step denoising of consistency model, in which dynamic perception strategy adaptively updates scene context and a bump-aware guidance further improves physical plausibility.
A hybrid data training strategy that combines synthetic HOSI data with high-fidelity HSI daia, relaxing the reliance on complete HOST annolations and addressing data scarcily.
Spocifically, for the challenge of dynamic object-scene, INFBAGEL first generates a coarse motion sequence conditioned on the initial scene. This initial motion is then used to derive vime-iarying scene state, which in turn guide an iterative optimization process with bump- aware guidance. We begin by training scene-condil ioned diffussin model that accepts the temporal-aligned scene states as a condition and supports conditional and unconditional generation, empowering unified generation for coarse trajectory and fine motion. We further distill the diffusion model into a consistency model, whose few-step high-quality motion generation allows us to update the precise time-varying scene more reliably, and then refine the motion by the latest scone states. Furthermore, we introduce a bump-aware guidance in sampling that guides the model toward collision-free optimization.
To tackle the scarcity of annotated data, we propose a hybrid data training strategy that jointly leverages HST datasets with explicit scene geometry and HOT datasets with synthe- sized voxelized scene context. For HO1 data, we first identify the volume occupied by the human and the object, and then fill the surrounding free space with voxels, transforming the original HOI data into human-object-scone or high-fidelity mesh capture. Joint training on this mixture enables the model to discn- tangle human, object, and scene factors, achieving strong zero-shot scene generalization for HOSI generation. This hybrid stralegy broadens data scale and diversity, supplies consis- tent spatial context for collision reasoning, and provides a practical, data-efficienl training foundation. Extensive experiments demonstrated that INFBAGEL achieves the highest suc- cess rate and physical feasibility in HOSI generation, while maintaining highly competitive performance on the HOT benchmark.
tuples without manual scene annotation
Beyond such technical contributions, INFBAGEL is promising for various downstream appli- cations, such as robot learning and virtual character animation. For example, in humanoid motion and robot learning, INFBAGEL acts a text-conditioned high-level motion plan- ner that captures rich human-object-scenee interaction patterns, while low-level collrss handle physics-based tracking and ambodiment-spectt generalization, validated on the HOSI benchmark with 67 unseen scenes, demonstrates its adaptability and effectiveness in diverse real-world scenarios.
constraints. Its strong zero-shot
2 RELATED WORK
Interaction with Objects. Regarding interaction with small objects, recent research
has shifted from synthesizing hand motions (Zhang et. al. 2021 Christen et. al.) 2022 to
2

---

Published as a conference paper at ICLR 2026
full-body motions for object grasping (Taheri et al. 2022: Wu et al. 2022) ). Some methods simultancously generate human motion and predict object motion (Baa et al. 2024 Ghosh et all, 2023 however, these approaches are mainly limited to interactions with small objects, with an emphasis on hand motion. For interaction with large objects, the complexity of human motion increases, and object motion becomes non-negligible. Some methods employ reinforcement learning strategies to synthesize actions for specific tasks, such as box moving (Merel et al., 2020). Recent mimic learning methods (Xu et al., 2025 Yu al. 2025) achieve generalized character interaction. With the growing availability of human-object interaction
datasets [Bhatnagar et al.) 2022: Li et al.) 2023), certain methods (Li al., 2024b Cong
el a 2025 start generaling motions lor interactions with large objects, olten relying on sequential points or object trajectories, which restricts the model's ability to autonomously
generate diverse interactions. Other methods (Xu et. al. 2023 Peng et al.
2025 Li et al.
2025
Xue et al. 2025 attempt to simultaneously synthesize human and object motions
using multi-stage strategies.
Interaction with Scene. Early work primarily focuses on generating interactions within static scenes, extending from locomotion in the scene (Zhang & Tang 2022 Mao et al., 2022) to interacting with static objects (Zhang et al. 2020b] Wang et al., 2022b]. Sub- sequently, some methods consider both locomation and static interaction simul Laneously. Hassan et al. (2021h, Waang et. al. (2022a), ) and [Huang et al. (2023) modeled them sepa- rately, resulting in inconsistent motions. Zhao et al. (2023) and Yi al. (2024) rely on high-level path planners, making sensitivity to the quality of the planner. Wang et al. (2024b) and Cen et al. (2024) are multi-stage frameworks, which incur significant compu- tational overhcad. Recently, generating interaction with dynamic objects in the scene gains incrcasing attention. Some physics- based mothods (Hassan et al., 2023 Xu ct al. 2024 Pan et al.] 2025 Wang et al. 2024a) explore the interactions between physically simu Manullatedd humanoids and dynamic objects through reinforcement learning. However, these methods often require complex reward engineering and still struggle with motion diversity and re- alism. Kinematics- based methods Jiang et al. (2024b) and Jiang et al. (2024a) propose a unified generation framework, but simplity object motion by directly attaching objects to the hand. Yao et al. 22025) and Geng et al. (2025) further generate object motion. However, Yao al. 2025 is limited in motion and object types due to the limited HOSI dataset.
(2025) synthesizes HOSI data by matching OMOMO motions to collision-free scenes, but unmatched cases are simply treated as empty scenes. Both approaches rely on static scene perception and additional path planners, limiting their ability to handle dynamic contexts. In this paper, we focus on kinematics based generation and propose INFBA which accommodates diverse motion types and scenes through hybrid data training strat- egy. Tt is designed as unified coarse-to-fine framework with few-step consistency sampling and dynamic perception, enabling more adaptive and realistic HOSI interactions.
Geng et al.
Interaction Datasets. Current human-object interaction datasets (Li et al., 2023 Bhat- pagar et al]. 2022) generally lack scene context, which makes some methods only integrate the generated motion into the external scene by planning collision-free paths at inference time. However, these methods lack direct penetration awareness, limiting their applicability in complex 3D environments. Alternatively, Liu et al. (2024) aggregate motion-only data into paired human-occupancy interaction data to synthesize HOST data. Rather than trivial extension, our approach further incorporates real high-fidelity HST data through a hybrid training strategy, effectively learning HOSI while ensuring realistic scene awareness. Recently, some work introduces object interactions into HSI datasets. For example, [Jing ct al. (024b) includes object motion but lacks instruction-lovel annotations, and the variety of objects is restricted. [Jiang et al. (2024a) features diverse scenes and text annotations but omits object motion. Kim et al. (2025) has a limited variety of scenes, making it difficult to achieve generalized scene perception. The restricted variely of objects and scenes limits the learning of complex interactions in generation. More discussions provided in Appendix A.4
METHODOLOGY
We introduce INFBAGEL, an ruto-regressvve framework for generating human-object-scene interactions from textual instructions and goal locations, as illustrated in Fig. 1
3

---

Published as a conference paper at ICLR 2026
(a) Auto-regressive Motion Model
Sampling with aump-aware Guidance
(d) Hybrid Data
heratively Refine
Danaliiig
Real-world HSi Data
Bump-eware Guidance
MLP
Deealiing
Adking Molce
MLP
Fayy Somerr Rill
EICOOAA
Dynamic Perception
Update
D5
Dluee
Sone
Synthesized HO1 Data
C2 Buke
'Nove the
quiaa
Uhe quiuur
Add Condition
Instruction
Goal
Object
Figure 1: Overview of INFBAGEL. Our method operates through an iterative refinement
process. (a) Anto-regressive Motion Model generating arbitrary long-sequence motions con- ditioned on textual instructions, goals, object geometry, and scene context. (b) Dynamic Perception Encoder perceives the evolving environment with the temporal-aligned scene state updated by iterative sampling. (c) Bump-aware Guidance detects collisions and di- rects iterative, collision-free sampling. (d) Hybrid Data training enables robust zero-shot generalization to complex realistic scenes.
3.1 DATA REPRESENTATION
Problem Formulation. Formally, our objective is to generate a coupled motion sequence M = (M,, M,) including both human motion MM, and object motion M,,, given the 3D scene S, the interactive object geomelry O, the text instruction T, and the goal position G. For the interaction with static scene, the object motion M. will be set to empty.
Human and Object Motion. At each frame t, the human motion MM, consists of the root translation Ta 3 and 6D rotation representations RA g/x6 for J 22 selected joints. We use the parameterized human model SMMPL- (Pavlakos et al. 2019) to reconstruct the human mesh The object motion M. is represented by the translation of its centroid To E R and the rotation R, E 3 relative to the input object geometry O.
Input Conditions. We represent the interactive object geometry O using Basis Point Set (BPS) representation (Prokudin et al. 2019), which consists of 1024 vectors indicating the spatial relationship from a set of basis points to the object's ''s surface, denoted as RR. 1024x3 The 3D scene S is represented with voxel grids. The goal location is user-specified 3D coordinate E R2, including human pelvis goal and object centroid goal. For static scene interaction tasks, the geometry and goal of interactive object is set to empty. The text instruction T encoded into a 768-dimensional feature using CLIP (Radford et al. 2021
3.2 DYNAMIC PERCEPTION ENCODER
Building on the above objective, the traversable space evolves the human and object move, so one-off static encoding is insufficient for HOSI generation. To address this chal- lenge, we introduce coarse-to-fine generation strategy with dynamic perception encoding mechanism showed in Fig. [b). INFBA GEL can generate coarse motion trajectory to derive frame-wise, time-varying scene states that capture how the environment changes over time, and then optimize the motion based updated scene state correspondingly:
Following Jiang et al. (0024b) and Jiang et al. 2024a), we represent scene states by 3D occu- pancy voxel. Within each generation window, use five scene voxel grids to represent local scene information, three dynamic and two static. Specifically, the two static voxel grids re- spectively capture the start and goal regions, while the three dynamic voxels centered at the pelvis positions uniformly sampled from intermediate frames within the temporal window. The dynamic voxels masked out during coarse prediction, and updated iteratively the
4

---

Published as a conference paper at ICLR 2026
prediction is refined. Each voxel grid is 3D array {0, [0,1,2)NxNxx)) where 0, and denote traversable, non-traversable, and interactive object occupied cells, respectively, and is the grid size. We then employ ViT (Dosovitskiy et al. 2020) to independently encode each scene voxel grid to a 512-dimensional embedding. To incorporate precise spatial cues, we further fuse the positional encoding of the central voxels, i.e., the corresponding pelvis/ goal positions, with respective scene embeddings. More details are provided in Appendix A.6
3.3 AUTO-REGRESSIVE MOTION MODEL
Regarding the coarse prediction, existing methods usually introduce additional modules or algorithms, which cost for long horizons. To achieve unified generation, we exploit the iterative generation nature of the diffusion model to perform coarse-to-fine generation, where the dynamic scene conditions are updated based on the results from the previous iteration, and each iteration further refines the generation according to the updated scene state. Specifically, we train a diffusion model conditioned on a combination of scene, text, goal and object information, with random mask applied to the embeddings of dynamic voxel grids. This mechanism enables the model to seamlessly handle both static and dy- namic scene perceptions, generating coarse trajectories without dynamic embeddings and producing fine-grained motion refinements when dynamic cues are available.
However, the diffusion model typically requires a large number of sampling steps to generate a clean sample, which compromises generation efficiency and hinders the accurate update of scene states. To address this limitation and construct more efficient framework, we distill the diffuusio model into a consistency model, which can generate high-quality motion sequences within only a few steps. Notably, each step in the consistency model directly outputs clean sequence, unlike the noisy intermediate states of diffusion models, thereby providing a more reliable perception of the evolving scene for subsequent refinement.
Specifically, the consistency model So serves learning to directly map any noisy sample x,, along a denoising trajectory back to its clean origin Ko, with on representing the diffusion step. Under the classifier-free guidance setting, the consistency model fo (Xr,, Tu: w)) ~ & jointly models the conditional and un- conditional settings, where the interpolation weight controls the guidance strength and C,, denotes the conditions including scene, textual instruction, goal, and object informa- tion. To effectively transfer the knowledge of multi-step diffusion model into fe, we employ consistency distillation (CD), which enforces local consistency between the outputs of fe at adjacent PF-ODE steps. Mathematically, following Luo et al. (2023), the distillation objective minimizes
distilled version of the diffusion model,
CCD C.,,
d(fo(Xr.))) (df.c./ Tua Wr,, for (x): Tn-1, Cr. Crn-
(1)
where the parameters 0' of the larget network for are updated with the exponential moving average of the trainable parameters 9 of the online network fe, and d(..-) is the la distance, Here, x is estimated from Xr, by the teacher diffusion model using DDIM (Song et al. 2020
sampling (VV) under classifier-free guidance with strength E [Wmin Wmaz]]
X Xr, + (1 + W/ Tri C...) - WXV((x Tri C
(2)
where C,,, and Cº represent the full condition and without the dynamic scene condition, respectively.
To further enhance the physical realism of the generated motion, inspired by Li et al. (202bb), we incorporate auxiliary supervision on human joints and object poses. Using forward kinemattcs (FK) to the ground-true motion Maso and the predicted motion Mayos we compute the positions of hands and feet, as well as the transformed object vertices. The loss is then calculated as follows:
Cjoints [KK((M)) (M&(M))
(3)
Cob) [K.(MM) - [M.))
(4)
5

---

Published as a conference paper at ICLR 2026
The total objective is computed weighted sum of such loss functions, with hyperparam- eter Abs A as loss coefficients
LCD + Cjontss + AoCobj-
(5)
During inference, we leverage the consistency model to directly generate reliable motion. To synthesize long-horizon interactions, as illustrated in Fig. aa), we utilize the efficiency of the consistency model by adopting an auto-regressive generation strategy. Rather than generating the entire sequence at once, the model produces motion segments sequentially. This design improves the efficiency and effectiveness of updating the scene state, making real-time iteration feasible.
3.4 BUMP-AWARE GUIDANCE
iterations, strict collision avoidance remains challenging. To steer refinement toward collision-free solutions while presorving the efficiency of consistency- based optimization, we introducc a lightweight bump-aware guidance (Fig. (F)
Despite
Specifically, each sampling iteration, the consistency model predicts the clean data X which derive M₁ and M, to reconstruct the human joints and object points. Given the voxelized scene S, we detect bumps when overlaps occur between human or object points and occupied voxels, and use the distance with closest free space to calculate gradient with respect to the current motion to nudge samples away from obstacles. Following Dhariwal & Nichol 2021 and Ho et al. (002), the process is formulated as:
N0 = No + Yan Vx., boump
Lhump Σ D(V(p))
Pe (MM
where 7r, is the guidance scale related with Tas V(.) denoles the voxel occupied by the point. D(-) returns the distance from the voxel center to the closest free voxel center, which is O if the voxel itself is free.
Bump-aware guidance does not rely on high-fidelity meshes. The regular structure of voxel grids facilitates the pre-oomputation of a distance map, thereby avoiding the costly online nearest point search required by detailed meshes and reducing computational overhead. It integrates scamlessly into the consistency sampling loop, progressively reducing penetration while enhancing overall physical plausibility.
3.5 HYBRID DATA TRAINING
Another major bottleneck for HOSI generation is the scarcity of datasets that simultancously offer diverse scenes, a wide variety of manipulable objects, and lextual annotations, which limits generalization if trained on single source. To broaden coverage while preserving the unified interface OS, T),,, we introduce a hybrid data training strategy that jointly combines existing high-quality HSI and HOI datasets, as shown in Fig. d)). This approach allows us to train a model that is proficient in both complex scene navigation and interaction with static or dynamic object, without requiring single, all-cncompassing HOSI datasot.
To learn diverse and complex scene-level interaction, we leverage HSI datasets such as LINGO (Jiang et 2024a). These datasets provide high-fidelity scene meshes, diverse environments, and detailed lextual instructions for tasks like locomotion and interaction with static objects (e.g., sitting on chair). Training on this data enables our model to ground its understanding of motion in realistic, complex geometric contexts and to learn long-horizon planning and navigation behaviors.
To master diverse dynamic object interactions, we incorporate large-scale HOI dataset (Li ct al, 2023) with LINGO. Since HOI dataset typically lack scene information, we synthesize minimal yet effective scene context for each sample. Following the methodology Liu et al. (2024), we identify the spatial volume occupied by the human and the manipulated object throughout the motion. The surrounding free space are then voxelized to create plausible

---

Published as a conference paper at ICLR 2026
occupied space. This process transforms standard HOI data into human-object ooe--ooooeeeeeeee scene tu- ples without manual scene annotation, effectively teaching the model to respect the spatial constraints imposed on the human and object motion.
The HSI data provides the "macro-level" understanding of interaction within a static world, while the synthesized HOI data offers the "micro-level" knowlodge of object manipulation. By jointly training on this heterogencous data mixture, our model learns consistent collision reasoning for spatial context and yields strong zero-shot generalization to unseen scenes.
4 EXPERIMENTS
4.1 EVALUATION SETTINGS
Dataset and Metrics. We construct an HOST benchmark as testing set., comprising 469 sequences, featuring 7 categories of large, manipulable objects from OMOMO (Li et al. 2023 located within 67 diverse and unseen indoor scenes from TRUMANS (Jiang et al, 2024b.. Specifically, each sequence is constructed by first randomly sampling a unique start- end pair within a scenc, and then pairing it with a sampled initial state of human as well as interactive object and textual description from the OMOMO-test set. This approach effec- tively preserves the diverse motion types inherent in OMOMO and the diversity of scenes from TRUMANS. Further details regarding the HOSI benchmark, such as the feasibility validation of the synthetic sequences, and the composition of training set are provided in Appendix A.5 We assess the generated motion from three perspectives. For task comple- tion, we measure the final distance of the human (Tr) and the object (T,) to their respective goals, along with the task success rate (S%); considered successful if both distances are below 10 cm. For motion and interaction quality, we assess foot sliding (FS), contact percentage (C%) and human-object penetration (Pbody) from CHOIS (Li et al. 2024b). To evaluate scene awareness, we mcasure seene penetration for both the human and the object using metrics from DIMOS (Zhao al., (023). including the average, max penetration vertices depth sum (P meaa//mma in any Trame during the sequence, measured in meters (m), and the percentage of frames with penetration (Py%). For the unified penetration measure, we calculate Poody by summing the total depth of penetration from any frame through vertices, rather than the average value, mcasured in meters (m). For time cost, we follow (Dai et al. 2024 to report the average inference time per sentence (AITS) and FPS.
Baselines. We compare INFBAGEL with two scene-aware baselines, TRUMANS (Jiang el al., 2024b) and LINGO (Jiang el al.) 2024a). For a fair comparison, we augment both with object geometry, goal location, and output heads for object-motion prediction. We also adapt TRUMANS to accept text instructions and object goals (it originally relies on hard joint constraints rather than goal conditioning). All models are trained on the OMOMO dataset for evaluation across both HOSI and HO1 with the same model. For HOSI, training uses OMOMO enriched with synthesized scenes. To assess our hybrid-data training strategy, we additionally train on a mixture of synthesized OMOMO and the LINGO HSI datasct.
4.2 EXPERIMENTAL RESULTS AND ANALYSIS
Table 1 and Table 2 present the quantitative results on the HOSI benchmark. INFBAGEL significantly outperforms TRUMANS and LINGO across nearly all metrics. Notably, INFBAGEL achieves a success rate of 83.16%, a substantial improvement over LINGO's 53.09% and TRUMANS' 1.92%, demonstrating INFBAGEL's superior ability to generate goal-directed interactions in complex scenes. In terms of physical plausibility, INF BAGEL achieves the lowest foot sliding and the highest. contact percentage with the lowest penetra- tion between the human and object, indicating more realistic human motion and interaction quality. Furthermore, INF INEB exhibits remarkable scene awareness, with human-scene and object-scene penetration metrics significantly lower than those of the baselines, high- lighting its offectiveness in avoiding collisions within the environment. LINGO achieves slightly lower human goal distance (Tx), which reflects deliberate trade-off where our model prioritizes collision-fre frresi motion over strict goal adherence presented in the ablation studies. As shown in Fig. 2 our INPBAGEL generates motions and interactions that are
7

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative results of HOSI task, where the human moves the object from one place to another in cluttered realistic scene. by hybrid data.
Blue highlights the improvement brought
Method
Task Accuracy
FSJ
HO Interaction
HS Penetration!
OS Ponetration!
Ta
To 4 S%
O% 1 Poody
Pmean
Pmaz
Pyx
Pmean
Pmaz
Pyn
Trained Synthesized OMOMO Dataset
TRUMANS
71.15
1.92
0.71
46.19
4.91
10.02
70.49
93.81
56.03
218.69
34.41
LINGO
2.75
13.03
53.09
0.57
53.83
5.72
7.62
61.19
96.15
38.07
167.95
22.93
INeBAGEL
4.75
8.14
83.16
0.13
78.18
3.96
3.39
36.97
28.19
16.62
109.61
22.72
Trained on Hybrid Dataset
LINGO
3.04
17.97
39.45
0.37
53.43
7.58
7.09
37.45
91.96
22.55
123.56
20.56
INFBAGEL
0.15
76.96
5.05
37.09
26.71
93.37
Table 2: Comparison of generation speed. 'D'' denotes Diffusion Model and 'G' denotes Guidance. The average duration the sequence is about 193 frames.
Metrics
TRUMANS
LINGO
INFBAGEL DM
INFBAGEL w/o
INFBAGEL
AFTSL
5.84
6.46
57.17
1.30
6.75
FPSt
31.57
28.86
3.38
148.54
28.75
Lift the plastic box. move the plastic box, and put down the plastic box (Trained on synthesized OMOMO dataset)).
(a) TRUMANS
(b) LINGO
(c) INFBAGEL
Drag the suitcase, and set the suitcase down (Trained on synthesized OMOMO dataset).
(d) TRUMANS
(e) LINGO
(f) INFBAGEL
put down chair, (Trained hybrid dataset).
(g) LINGO
(h) INFBAGEL
Figure 2: Qualitative comparison. Top 2 rows: Comparison on human-object interaction scenes. Bottom row: Comparison on complex multi-stage task involving moving chair and then sitting on it.
physically plausible and semantically correct, outperforming bascline methods. Specifically, both TRUMANS (a/d) and LINGO (b/e) exhibit severe object-scene penetration. In con- trast, INFBAGEL (c/f) produces nearly collision-free motion. Furthermore, while LINGO (g) fails to generate plausible interactions in both HOI and HSI phases, INFBAGEL (h) successfully executes the mor-horizon task by moving the chair to the goal without collision and then seamlessly transitioning to a sitting pose. These results demonstrate the superior physical plausibility and task semantics understanding of our method. To further assess our

---

Published as a conference paper at ICLR 2026
Table 3: Ablation on key components: Dynamic Perception Encoding (DP) and Guidance (G). C denotes Contact Guidance (Li et al.) 2024b); B denotes our Bump-aware Guidance. A Diffusion Model baseline with DP included to study effects on dynamic perception, guidance is omitted due to inconsistent sampling steps and weighting.
Components
Task Accuracy
HO Interaction
HS Penetration
OS Penetration)
FSL
FPSt
DP
G
Th 4
To L Sx f
C% f Pondy 4
Pmean
Pmaz
Prs
Pmean
Pmax
Prs
2.71 7.57 71.22
0.19
65.99
3.33
6.49
56.18
35.18
28.78
146.10
23.01
180.22
x
2.72
6.32 86.35 0.14
68.09
1.46
6.38
55.81
29.82
26,00
138.12
23.01
148.54
C
2.91
6.39
85.50
0.13
79.34
3.91
6.30
57.17
28.21
25.61
139.93
23.25
29.31
C+B
4.75
8.14 83.16
0.13
78.18
3.96
3.39
36.97
28.19
16.62
109.61
22.72
28.75
Diffusion Model 2.12
6.63 84.22 0.14 65.14
3.96
7.20
58.00
31.14
28.48
139.69
23.82
3.38
Table 4: Ablation study on different parameter settings in our method, including the models with and dynamic scene representations (Voxel) at different sampling steps (Step).
HO Interaction
HS Penetration
OS Ponetration
Parameters Syg f
FSJ
AITS
FPS
Cisy f
Poady
P Pmean
Pmax
Pmeen
Pmaz
Voxel=1, Step=1
68.87
0.15
66.25
4.68
6:27
58.04
27.38
139.36
0.12
1703.79
Voxe==1, Step=8
60.98
0.16
71.29
4.27
4.07
45.52
18.64
113.30
3.11
62.35
Voxel 1, Step- 16
62.26
0.15
77.32
4.20
3.81
41.87
16.74
113.31
6.57
29.53
Voxel=3, Step=1
87.85
0.14
67.62
4.78
6.15
56.61
28. 16
142.22
0.12
1570.94
Voxelm3, Step
82.94
0.15
73.44
4.05
4.35
45.48
20.53
117.34
60.35
Voxel=3, Step=16 16
83.16
0.13
78.18
3.96
3.39
36.97
16.62
109.61
6.75
28.75
model's capabilities, we include an additional evaluation the standard HO1 benchmark in the Appendix A.2 where our method still achioves state-of-the-art performance.
Does hybrid data training strategy help? As shown in the second group of Table I both models trained on hybrid data achieve significantly reduced human and object penetra- tion with scene. This indicates that incorporating HSI data, containing high-fidelity scene geometry, enhances the model's understanding of complex environmental constraints and improves its ability to avoid collisions while navigating the object in scenes. These results demonstrate that our hybrid data strategy contributes to learning generalizable scene rep- resentation, enabling the model to perform complex human-object-ssene Interaction tasks effectively in unseen environments. Notably, the task type in the LINGO data is different from the HOSI test set, making a slight trade-off in success rate and human-object inter- action metrics (C% and Phody); this is expected as the model learns to prioritize scene-level physical plausibility over strictly adhering to interaction priors learned from HOT data. More exploration of the hybrid data ratio is provided in Appendix A.3
4.3 ABLATION STUDY
We conduct collection of ablation studies to evaluate the contributions of each component on the HOST benchmark across three dimensions. Specifically, (1) to examine the effect of dynamic perception encoding (DP), we compare a model using only static scene
encoding (i.e., without DP) with one adopting DP, as presented in Table 3 and further explore the impact of the number of dynamic voxels in DP, as shown in Table 4
(2) We
further study the effect of bump-aware guidance (B) and contact guidance (C) on
improving physical plausibility, represented as DC+C and DP+C+B in Table 3
(3) Finally,
we compare the distilled consistency model with standard diffusion model, both using
DP, as shown in Table 3 and study the influence of sampling steps Table Qualitative comparisons of different variants provided in Fig. 3
Does dynamic perception encoding help? Comparing the first two rows of Table 3 reveals the impact of our dynamic perception encoding. Disabling this component (row 1) leads to a notable degradation across multiple metrics compared to the model with DP enabled (row 2). In particular, scene penetration by both the human and the object increases, indicating that without dynamically updating its perception of the scene, the model becomes less capable of avoiding collisions and performing fine-grained interaction.
9

---

Published as a conference paper at ICLR 2026
Push the smalltable, and set down (Trained synthesized OMOMO dataset).
(a) INFBAGEL Diffusion
(b) INFBAGEL
(c) INFBAGEL
(d) INFBAGEL
Figure 3: Qualitative comparison in ablation study. Replacing/renanne removing specific modules: (a) diffusion model instead of consistency model, (b) static perception instead of dynamic perception, and (c) without bump-aware guidance, all resulted in collisions with the scene.
As shown in Fig. 3 (b) and (d), only the static scene encoding results in penctration between the human and scene, while this does not occur when dynamic scene encodings are included. In addition, the overall task success rate drops sharply from 86.35% to 71.22%, and foot sliding worsens from 0.14 0.19. This observation is further supported by the results in Table ! Reducing the temporal voxel number from three to one causes decline in success rate and worse motion quality in terms of foot sliding. These results underscore that dynamic perception is crucial for generating physically plausible, goal-directed motions in complex and evolving environments.
Docs bump-aware guidance help? As shown in Table 3 (rows 2-4), introducing con- tact guidance (C) leads to notable improvements human-object (HO) interaction metrics, including Cz and Phody- Furthermore, incorporating the proposed bump-aware guidance (B) yields consistent and substantial improvement in both HS and OS penetration metrics, while maintaining the HO interaction performance nearly unchanged. Fig 3 (c) and (d) further demonstrate the necessity of bump-aware guidance. These results validate the fectiveness of bump-aware guidance in enhancing the physical plausibility of human-scene and object-scene interactions. However, we also observe that there is a slight degradation in task accuracy metrics such as the goal distance (T), and To), which highlights trade-off introduced by the guidance. Stricter collision constraints enhance physical plausibility by mitigating penctration, but at the expense of reduced motion flexibility, occasionally hin- dering precise goal adherence.
Does consistency model help updating scene state? The comparison between the second and the last rows of 3hows that replacing the distilled consistency model (CM) with standard diffusion model causes a drastic drop generation speed, making iterative refinement computationally infeasible for real-time applications. In contrast, the CM's few- step generation capability makes efficient iteration possible. Further, the quality of the previous sample used for scene state updates is paramount. The CM yields a higher-quality motion estimate, which provides a more accurate basis for updating the dynamic scene state. This leads to better scene awareness, evidenced by the reduction in both human penetration and object penetration with scene. These findings confirm that the consistency model is not. just accelerator; its ability to provide high-quality, few-step motion predictions is essential for reliable and effective dynamic scene state updating.
CONCLUSION
This paper presents INEBAGEI, a novel framework for human-object-scene interaction gen- eration that addresses the critical challenges of dynamic object-scene changes and data scarcity. Our coarse-to-fine approach utilizes the few-step denoising consistency model, achieving dynamic perception of the scene. To enhance physical plausibility, introduce a lightweight bump-aware guidance that effectively guides the model toward collision- optimization. Furthermore, our hybrid data training strategy combines real-scene HSI data with synthesized HOI data, effectively overcoming data limitations and enabling zero-shot. generalization.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62401367 and in part by Shanghai Artificial Intelligence Laboratory.
REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility of this work. To this end, will publicly release the complete source code, pre-trained models, and the HOSI (Human-Object-Seeee Interaction) benchmark dataset that we have constructed after the paper published. Do- tails about the used dataset, including the hybrid data processing pipeline for converting HOI data into HOSI tuples, as well as the construction details of the HOSI benchmark, are described in detail in Appendix A.5 Our model architecture, including the specific implementation of the dynamic perception encoder and the motion consistency model, elaborated in Section 3 of the methodology, and more detailed implementation parameters and network structures are provided in Appendix [A.6] Finally, all training hyperparameters, inference scttings, including iterative optimization and collision-aware guidance parameters can be found in Appendix A.7
REFERENCES
Bharat Lal Bhatnagar, Xianghui Xie, Tlya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Physically plausible full-body hand-object interaction synthesis. In 2024 International Conference on 3D Vision (3DV), 2024.
Zhi Cen, Huaijin Pi, Sida Peng, Zehong Shen, Minghui Yang, Zhu Shuai, Hujun Bao, and Xiaowei Zhou. Generaling human motion in 3d scenes from text descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jianqi Chen, Panwen Hu, Xiaojun Chang, Zhenwei Shi, Michael Kampffinyer, and Xiaodan Liang. Sitcom-crafter: A plot-driven human motion generation system in 3d scenes. In The Thirleenth International Conference on Learning Represenlations.
Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, and Otmar Hilliges. D-grasp: Physically plausible dynamic grasp synthesis for hand-object interac- tions. In Proceedings of the TEEE/CVP Conference on Computer Vision and Pattern Recognétion (CVPR), 2022.
Peishan Cong, Ziyi Wang, Yuexin Ma, and Xiangyu Yue. Semgeomo: Dynamic contextual human motion generation with semantic and geometric guidance. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlem: Real-time controllable motion generation via latent consistency model. In European Conference Computer Vision (ECCV), 2024.
Zekai Deng, Ye Shi, Kaiyang Ji, Lan Xu, Shaoli Huang, and Jingya Wang. Human-object interaction with language model guided relative movement dynamics. arXiv e- prints, pp. arXiv 2503, 2025.
Prafulla Dhariwal and Alexander Nichol. Diffuusin models beat gans on image synthesis. In Advonces in neural information processing systems (NeurIPS), 2021.
Christian Diller and Angela Dai. Cg-hoi: Contact-guided 3d human-object interaction generation. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2024.
11

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Marthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint or arXiv:201. 1.100.111222 11929, 2020.
Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J. Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand- object manipulation. In Proceedings IEFE Conference on Computer Vision and Pattern Recognation (CVPR), 2023.
Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingbo Wang, Tai Wang, Jinkun Cao, Xiaolin Hu, Si Liu, Jifeng Dai, and Jiangmino Pang. Coohoi: Learning cooperative human-object interaction with manipulated object dynamics. arXiv preprint arXiv:2406. 14558, 2024.
Zichen Geng, Zeeshan Hayder, Wei Liu, and Ajmal Mian. Unihm: Universal human motion generation with object interactions in indoor scenes. arXiv preprint arXiv:2505.12774, 2025.
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-objeci interactions. In Computer Graphics Forum, 2023.
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2021.
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scene interactions. In ACM SIGGRAPH 2023 Conference Proceedings, 2023.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Flect. Video diffusion models. 2022.
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengru Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Nan Jiang, Zimo He, Zi Wang, Hongjie Li, Yixin Chen, Siyuan Huang, and Yixin Zhu. Autonomous character-scene interaction synthesis from text instruction. In SIGGRAPH Asia 2024 Conference Papers, 2024a.
Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Jeonghwan Kim, Jisoo Kim, Jeonghyeon Na, and Hanbyul Joo. Parahome: Parameterizing everyday home activities towards 3d generative modeling of human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Jiye Lee and Hanbyul Joo. interactions in complex 3d environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Synthesizing human-scene
Hongjie Li, Hong-Xte Xing Yu, Jiaman Li, and Jiajun Wu. Zerohsi: Zero-shot 4d human-scene interaction by video generation. arXiv preprinl arXiv:2412 18600, 2024a.
Jiaman Li, Jijjun Wu, and Karen Liu. Object motion guided human motion synthesis. ACM Transactions Graphies (TOG), 42:1-11, 2023.
12

---

Published as a conference paper at ICLR 2026
Jiaman Li, Alexander Clegg, Roozbeh Mottaghi, Jiajun Wu, Xavier Puig, and C Karen Liu. Controllable human-object interaction synthesis. In European Conference on Computer Vision (ECCV), 2024b.
Lei Li and Augela Dai. GenZI: Zero-shot 3D human-scene interaction generation. Proceed- ings of the IEEE/CV /VF Conference on Computer Vision and Pattern Recognátion (CVPR), 2024.
Quanzhou Li, Jingbo Wang, Chen Change Loy, and Bo Dai. Task-oriented human- object interactions generation with implicit neural representations. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision (WACV), 2024c.
Shujia Li, Haiyu Zhang, Xinyuan Chen, Yaohui Wang, and Yulong Ban. Genhoi: General- izing text-driven 4d human-object interact ion synthesis for unseen objects. Xiv preprial arXiv;2506.0 15483, 2025.
Libin Liu and Jessica Hodgins. Learning basketball dribbling skills using trajectory opti- mization and deep reinforcement learning. ACM Transactions on Graphics (TOG), 37: 1-14, 2018.
Xinpeng Liu, Haowen Hou, Yanchao Yang, Yong-Lu Li, and Cewu Lu. Revisit human-scene interaction via space occupancy. In European Conference Computer Vision (FCCV), 2024.
Zhenyu Lou, Qiongjie Cui, Tho Wang, Zhenbo Song, Luoming Zhang, Cheng Cheng, Haofan Wang, Xu Tang, Huaxia Li, and Hong Zhou. Harmonizing stochasticity and determinism: Scene-responsive diverse human motion prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in newral information processing systems (NeurIPS), 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- so Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, in press:1-22, 2025a.
Guanxing Lu, Zifeng Gao, Tianxing Chen, Wenxun Dai, Ziwei Wang, Wenbo Ding, and Yansong Tang, Manicm: Real-time 3d diffuusion policy via consistency model for robotic manipulation. arXiv preprint arXiv:2406. (06::2406601166, 2024.
Jiaxin Lu, Chun-Hao Paul Huang, Uttaran Bhattacharya, Qixing Huang, and Yi Zhou. Hu- moto: A 4d dataset of mocap human object interactions. ar preprint arXiv:2544.104 10414, 2025b.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao Latent consistency models: Syuthesizing high-rosolution images with few-step inference. arXiv preprint ar arXiv:2310.0,23 2023.
Xintao Lv, Liang Xu, Yichao Yan, Xin Jin, Congsheng Xu, Shuwen Wu, Yifan Liu, Lincheng Li, Mengxiao Bi, Wenjun Zeng, et al. Himo: A new benchmark for full-body human interacting with multiple objects. In European Conference on Computer Vision (ECCV), 2024.
Wei Mao, Miaomiao Liu, Richard Hartley, and Mathieu Salzmann. Contact-aware human motion forecasting. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Josh Merel, Saran Tunaauuvuakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG), 39:39-1, 2020.
13

---

Published as a conference paper at ICLR 2026
Aymen Mir, Xavier Puig, Angjoo Kanazawa, and Gerard Pons-Moll. Generating continual human motion in diverse 3d scenes. In 2024 International Conference on 3D Vision (9DV), 2024.
Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes. In 2024 International Conference on 3D Vision (3DV). 2024.
Liang Pan, Zeshi Yang, Zhiyang Dou, Wenjia Wang, Buzhen Huang, Bo Dai, Taku Ko- mura, and Jingbo Wang. Tokenhsi: Unified synthesis of physical human-scene interactions through task tokenization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Georgios Paschalidis, Romana Wilschut, Dimitrije Antié, Omid Taheri, and Dimitrios Tzionas. 3D Whole-body grasp synthesis with directional controllability. In Internaiional Conference on 3D Vision (3DV), 2025.
Georgios Pavlakos, Vasilcios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Xiaogang Peng, Yiming Xie, Zizhao Wu, Varun Jampani, Deqing Sun, and Huaizu Jiang. Hoi-diff: Text-driven synthesis of 3d human-object interactions using diffusion models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Paltern Recognition (CVPR), 2025.
Sergey Prokudin, Christoph Lassner, and Javicr Romero. Efficient learning on point clouds with basis point sets. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICCV), 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010. 025002 2020.
Wenfeng Song, Xinyu Zhang, Shuai Li, Yang Gao, Aimin Hao, Xia Hou, Chenglizhao Chen, Ning Li, and Hong Qin. Hoianimator: Generating text-prompt human-object animations using novel perceptive diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In International Conference on Machine Learning (ICML), 2023.
Omid Taheri, Nima Ghorbani, Michael J. Black, and Dimitrios Tzionas. GRAB: A dataset of whole-body human grasping of objects. In European Conference on Computer Vision (ECCV), 2020.
Omid Taheri, Vasileios Choutas, Michael J Black, and Dimitrios Tzionas. Goal: Generat- ing 4d whole-body motion for hand-object grasping. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2022a.
Wenjia Wang, Liang Pan, Zhiyang Dou, Jidong Mei, Zhouyingcheng Liao, Yuke Lou, Yifan Wu, Lei Yang, Jingbo Wang, and Taku Komura. Sims: Simulating stylized human-scene interactions with retrieval-augnan auugeetee script generation. arXiv preprint 1xXiv:2411.19921, 2024a.
14

---

Published as a conference paper at ICLR 2026
Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprint 0rXiv:2312.09 2023.
Yinhuai Wang, Qihan Zhao, Runyi Yu, Hok Wai Tsui, Ailing Zeng, Jing Lin, Zhengyi Luo, Jiwen Yu, Xiu Li, Qifeng Chen, et al. Skillmimic: Learning basketball interaction skills from demonstrations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recogrition (CVPR), 2025a.
Yuan Wang, Yali Li, Xiang Li, and Shengjin Wang. Hsi-gpt: A general-purpose large scene- motion-language model for human scene interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), 2025b.
Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language-conditioned human motion generation in 3d scenes. In Advances in Neural Informalion Processing Systems (NeurIPS), 2022b.
Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say, interact as you can: Languago- guided human motion generation with scene affordance. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), 2024b.
Yan Wu, Jiahao Wang, Yan Zhang, Siwci Zhang, Otmar Hilliges, Fisher Yu, and Siyu Tang. Saga: Stochastic whole-body grasping with contact. In European Conference on Computer Vision (ECCV), 2022.
Zhen Wu, Jiaman Li, Pei Xu, and C Karen Liu. Human-object interaction from human-level instructions. arXiv preprint ar arxiv:246.. 10066...0 17840, 2024.
Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. In The Twelfth International Conference on Learning Representations, 2024.
Sirui Xu, Zhengyuan Li, Yu-Xiong Wang, and Liang-Yan Gui. Interdiff: Generating
3d human-object interactions with physics-informed diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liang-Yau Yan Gui. Intermimic: Towards uni-
versal whole-body control for physics- based human-object interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Xinyu Xu, Yizheng Zhang, Yong-Lu Li, Lei Han, and Cewu Lu. Humanvla: Towards vision- language directed object rearrangement by physical humanoid. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Haibiao Xuan, Xiongzheng Li, Jinsong Zhang, Hongwen Zhang, Yebin Liu, and Kun Li. Narrator: Towards natural control of human-scene interaction generation via relationship reasoning. In Proceedings of the IEFE/ /CFF International Conference on Computer Vision (ICCV), 2023.
Mengqing Xue, Yifei Liu, Ling Guo, Shaoli Huang, and Changxing Ding. Guiding human- object interactions with rich geometry and relations. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2025.
Wei Yao, Yunlian Sun, Hongwen Zhang, Yebin Liu, and Jinhui Tang. Hosig: Full-body human-object-seenee interaction generation with hierarchical scene perception. arXiv preprint or arXi::2006.015 2025.
Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 43:1-18, 2024.
Hongwei Yi, Justus Thies, Michael J. Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. arXin:2404. 10685, 2024.
15

---

Published as a conference paper at ICLR 2026
Runyi Yu, Yinhuai Wang, Qihan Zhao, Hok Wai Tsui, Jingbo Wang, Ping Tan, and Qifeng Chen. Skillmimic-v2: Learning robust and generalizable interaction skills from sparse and noisy demonstrations. In Proceedings of the Special Interest Group Computer Graphics and Interactive Techniques Conference Conference Papers, 2025.
Ling-An An Zeng, Guohong Huang, Yi-Lin Wei, Shengbo Gu, Yu-Ming Tang, Jingke Meng, and Wei-Shi Zheng. Chainhoi: Joint-based kinematic chain modeling for human-object interaction generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), 2025.
Haozhuo Zhang, Jingkai Sun, Michele Caprio, Jian Tang, Shanghang Zhang, Qiang Zhang, and Wei Pan. Humanoidverse: A versatile humanoid for vision-language guided multi- object rearrangement. arXiv preprint arXiv:2508.1694 2025a.
He Zhang, Yuting Yc, Takaaki Shiratori, and Taku Komura. Manipnet: noural manipulation synthesis with hand-object spatial representation. ACM Transactions on Graphics (TOG), 40:121:1 10:111:11121:1,, 2021.
Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang, Generating person- scene interactions in 3d scenes. In International Conference on 3D Vision (3DV), 2020a.
Siwei Zhang, Yan Zhang, Qianli Ma, Michael J Black, and Siyu Tang. Place: Proximity learning of articulation and contact in 3d environments. In 2020 International Conference on 3D Vision (3DV), 2020b.
Xiaohan Zhang, Sebastian Starke, Vladimir Guzov, Helisa Dhamo, Eduardo Pérez Pellitero, and Gerard Pons-Moll. Scenic: Scene-aware semantic navigation with instruction-guided control. Xiviv preprint arXiv:24122 15664. 2024.
Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In Proceedings of the IEEE/CVE /CFF//VFF/C Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yonghao Zhang, Qiang He, Yanguang Wan, Yinda Zhang, Xiaoming Deng, Cuixia Ma, and
Hongan Wang. Diffgrasp: Whole-body grasping synthesis guided by object motion using diffusion model. In Proceedings of the AAAT Conference on Artificial Intelligence, 2025b.
Jiahe Zhao, Ruibing Hou, Zejie Tian, Hong Chang, and Shiguang Shan. His-gpt: Towards 3d human-in-scene multimodal understanding. arXiv preprint arXi::2503. 12955, 2025.
Kaifeng Zhao, Shaofei Wang, Yan Zhang, Thabo Beeler, and Siyu Tang, COINS: Composi- tional human-scene interaction synthesis with semantic control. In European conference on computer vision (ECCV), 2022.
Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Yufei Zhu, Yiming Zhong, Zeminn Yang, Peishan Cong, Jingyi Yu, Xinge Zhu, and Yuexin Ma. Evolvinggrasp: Evolutionary grasp generation via efficient preference alignment. arXiv preprint arXiv:2503..440 14329, 2025.
A APPENDIX
A.1 MORE QUALITATIVE RESULTS
More qualitative results are presented in Fig. 4 with more scenes, object types and motion types, including lifting over head, kicking and interaction with static scene.
As shown in Fig. 5 INFBAGEL is largely auvionmennt-ndeeeeddee voxelized scene representation, rather than relying detailed scene geometry with a fixed scene category. As long as a new environment, whether a household, store, or hospital, can be voxelized, the same network can be applied.
because it conditions on
16

---

Published as a conference paper at ICLR 2026
Lift the smalitable your head, walk, put smalitable
Kick the trashcan. and it down.
(c) Walk, and lie down on the bed.
Figure Qualitative results on different scenes, motion types and object types. The top two rows (a/b) show diverse hunan-object interactions including lifting over head and kicking. The last. row shows static scene interaction.
A.2 HOI EXPERIMENT
Experiment. Settings. To assess INFBAGEL against specialized HOT methods, we eval- uate on the standard HOI benchmark OMÔMO, Following CHOIS, we assess the results from multiple perspectives. For condition matching, we adapt the metries to our condition, measuring the object's ''s final position error (T.) and the human waypoints crror (Tay). Hu- man motion quality is evaluated using foot sliding (FS), Fréchet Inception Distance (FID), and R-precision (Rpree) to measure physical plausibility and text-motion consistency. For interaction quality, use contact precision, recall, F1-score (Cprec/ree/f1) and contact per- centage (C%), and measure the penetration between the human body and the object (Poody). Finally, the difference from ground truth is quantified by the mean per joint position error JMPJPE), the root translation error (Troot) and the object pose error (Tabj. Ootz). There are large number of noise points the SDF of some objects in OMOMO, which makes the the entire sequence meaningless. Therefore, skipped the calcula- tion of Phody on noise objects, including woodchair, whitechair, largeboz, largetable, plasticbox and trashcan. For the unified penetration measure, calculate Poody by summing the to- tal depth of penetration from any frame through vertices, rather than the average value, measured in meters (m).
calculation of Poody
17

---

Published as a conference paper at ICLR 2026
Move monitor, and put down the monitar the table, store)
(b) Put the tripod horizontally down. then pick up the fallen tripod. (In physical therapy hospital)
Figure 5: Qualitative results in socially-inerraciive scenes, including a store and physical therapy room. These unseen scenes are chosen from LINGO, displayed in default white due to the lack of texture.
Baseline Comparison. We compare INFBAGEL and LINGO (Jiagg et al. 2024a), by treating them in an empty scene, including CHOIS [Li et al. 2024b) and ROG [ et al. 2025 two standard methods for HOT generation.
CHOIS generates synchronizod human and object motion from a language description, initial states, and sparse object waypoints. To ensure realism, utilizes object geometry loss to better match the generated motion to the waypoints and incorporates guidance terms to enforce hand-object contact constraints during sampling. To aligu with our method, modify it to be conditioned on human waypoints and object goal.
ROG generates human-object interactions using rich geometric detail. It selects key points from the object's mesh to create a detailed geometric representation and constructs an in- tcractive distance field (IDF) to capture interaction dynamics. A diffusion- model with spatial and temporal attention mechanisms is used to guide the motion gener- ation process, ensuring movements are relation-aware and sernantically correct. Similarly, adapt ROG conditioning to add human waypoints and a object goal.
relation
We evaluate INEB with different iteration steps (1, 8, and 16) with the guidance of CHOIS, to demonstrate the effectiveness of our iterative optimization. Competitive results can be achieved through simpler guidance and fewer optimization steps (CHOIS and ROG used 10 optimization steps).
Results on HOI. Table 5 presents the quantitative results on the OMOMO dataset. INF- BAGEL consistently outperforms all baselines across most metrics. A key advantage of our method its exceptional efficiency. With just a single sampling step (INFBAGEL 1), our model achieves performance comparable to CHOIS across several key metrics (e.g., Cn of
18

---

Published as a conference paper at ICLR 2026
0.66), while operating at astonishing speed of over 1500 FPS. This highlights the high quality of the imitial generation from our consistency model.
Furthermore, the results clearly demonstrate the benefit of our iterative optimization. As the number of sampling steps increases from 1 to 16, we observe a significant and consistent improvement in motion and interaction quality. Specifically, INFBAGEL 16 surpasses all basclincs, achicving the best scores in FID, contact quality (Crees C₁I, C%), and human- object penetration (Phody), all while maintaining a real-time generation speed (29.31 FPS) comparable to CHOIS. This confirms that our framework not only provides high-quality initial estimate but also effectively refines it toward more physically plausible and accurate interactions, showcasing a superior balance of speed and quality.
Table 5: Performance Evaluation on the HOI Dataset.
Coudition Matchingl
Human
Interaction
GT Difference,
Method
FPSt
Tay
Pprce
FID pyree Cree 1Cp CC Poody IMP.PET,,o Toky Oaby
Roal Motion
b,00
0.66
0.00
LINGO
17.37
3.76
0.41
10.28
28
CHOIS
5.28
5.73
0.32
0.68
1.01
0.78
0.63
0.66
0.53
2.52
13:93
8.42
17.74
0.92
28.01
ROG
26.68
19.43
0.49
0.03
0.46
0.80
0.68
0.70
0.54
5.86
13.37
21.28
31.89
1.14
9.07
Ior RaGin
2.93
4.19
0.36
0.63
3.41
0.79
0.64
0.66
0.52
2.83
11.92
8.88
15.40
1.02
1588.71
INFBAGEL
2.99
3.89
0.34
0.67
1.78
0.78
0.88
0.70
0.56
2.61
11.97
8.47
15.59
1.02
60.91
INFBAGRL. 16
0.60
A.3 ABLATION STUDY ON HYBRID DATA RATIO
To investigate the impact of different ratios of synthesized OMOMO data to LINGO data in our hybrid data training strategy, we conducted an ablation study by training INFBAGEL with three different ratios: 1:0 (only synthesized OMOMO data), 1:0.5, and 1:1. The evaluation results are presented in Table 6
The results show that the gain from HSI data scene understanding has limit, too much HSI data may compromise the model's ability to learn object manipulation from HOI data, indicating trade-off between scene-level physical plausibility and task-specific interaction priors A balanced ratio of 1:1 or 1:0.5 between synthesized OMOMO data and LINGO data achieves the best overall performance across all metrics.
Table 6: Ablation study on the different ratios of synthesized OMOMO data to LINGO data.
Method
Task Accuracy
FS1
HO Interaction
Penetration
OS Penetration.
Ts. 4
T. 4
S% f
C% 1
Poudy 4
Pmean
Penx
Pmean
Pmax
P/s
INIBAGEi (1:0)
4.75
8.14
83.16
0.13
78.18
3.96
3.39
36.97
28.19
16.62
109.61
22.72
IN=BAGEL (1:0.5)
4.37
7.94
81.45
0.15
76.96
3.17
37.09
26.74
12.45
93.37
22.32
INFBAGEL (1:1)
4.8
9.44
69.72
0.18
76.48
1.01
3.13
39.39
27.87
16.00
105.76
23.94
A.4 EXTENDED RELATED WORK
A.4.1 INTERACTION WITH EVERYTHING
Interaction with Small Objects. Early research primarily focused on synthesizing hand motions (Zhang et al. 2021 2025bb. With the emergence of motion datasets encompassing full-body hand-object in- teractions (Fan et al. 2023 Taheri et al., 2020 Lv et al. 2024). the focus expanded to synthesizing full-body motions for object grasping Taheri et al. 2022 Wu et al. 2022). Some methods (Braun et al. 2024 Ghosh et al.) 2023: Li et al.
Christon et al. 2022
Paschalidis et al.
2025 Zhang et al.
2024c) simultancously
generate human motion and predict object motion; however, these approaches are mainly limited to interactions with small objects, with emphasis hand motion.
19

---

Published as a conference paper at ICLR 2026
Interaction with Large Objects. For interaction with large objects, the complexity of human motion increases, and object motion becomes non-negligible Some methods employ reinforcement learning strategies to synthesize actions for specific tasks, such as box moving (Merel et al. 2020) or basketball sport (Liu & Hodgins 2018 Wang et al., 2025a). Recent
mimic learning methods (Xu et al. 2025 Yu et al. 2025) have achieved generalized character
interaction. With the growing availabiliy of human--bbect interaction datasets (Bhatnagar
et a.l 2022 Li et al., 2023; Lu et al. 2025b), certain methods (Li et al.) 2024b Cong et al,
2025 have started generating motions for interactions with large objects, often relying on sequential points or object trajectories, which restricts the model's ability autonomously
generate diverse interactions. Other methods (Xu et al.) 2023 Diller & Dail 2024 Song
et al1 2024 Peng et al. 2025 Li et al. 2025 Xue et al., 2025 Zeng et al. 2025) allempl
to simullaneously synthesize human and object motions but require additional models for optimization, unable to achieve real-time generation.
Limitation of HOI dataset. Due to the general lack of scene annotations in datasets, human-object interactions are typically placed into scenes by plan collision-free paths (Li ct all. 2024b; Wu ct al. 2024 However, these methods lacks direct perception awareness, limiting their applicability in complex and dynamic 3D environments.
A.4.2 INTERACTION IN E'VERWWIERE
Static Interaction in Scenc. Early work primarily focused on generating interactions within static scenes, extending from locomotion in the scene (Zhang & Tang
2022 Mao
el al., 2022) to interacting with static objects (Zhang et al.) 2020b Xuan el al.] 2023
Zhao el al. 2022 Wang et al. 2022b), such as sitting and lying down. Subsequently, some
methods considered both locomotion and static interaction simultaneously. Hassan et al.
(2021), Wang et al. (2022a), Huang et al. (2023). Mir et al. (2024) and Zhang et al. (0224)
modeled them separately, resulting in inconsistent motions. Huang et al. [2023] Zhao ct al. (002)) Yi et al. (2024) and Lou et al. (2024) relied on
high-level path planners for locomotion, making sensitivity to the quality of the planner. Zhang et al. (2020a), Wang
et al. (222bb) and Cen et al. (2024) are multi-stage frameworks, which incurred signilicant
computational overhead. Recently, Some methods (Zhao et al.] 2025 Wang et al. 2025b
Chen et al.) based on LLMs have shown strong generalization ability in static HSI.
Dynamic Interaction in Scene. Recently, interactions with dynamic objects in the scene gains increasing attention. Some physics-basee based methods {Lee & Joo, 2023 Hassan et al.
2023 Pan al. [2024] Xu et al.) 2024 Xiao et al. 2024 Deng et al., 2025 Pan et al. 2025
Wang et al.) 20244; Gao et al. 2024 Zhang et al. 2025a explore the interactions between physical simulated humanoids and dynamic objects through reinforccment. However, these methods often require complex reward functions engineering and still struggle with motion diversity and realism. Kinematics- bbseaeedd methods Jiang et al. (2024b) and Jiang el al. (2024a) propose unified generation framework, but simplity object motion, by simply attaching to the hand. [i & Dai (2024) and Li et al.] (2024a) achieve generalized generation by the powerful prior of image video generation models, but is limited by time-consuming generation and complex post-processing. Our most similar work are Yao et al. (2025) and Geng al. (2025), which generate object motion. However, these methods only employ one-time static scene awareness and lack object perception of the scene, also restricted by the limited object types, text, and scene annotations in the dataset.
Limitation of HSI dataset. Due to the high cost of data collection, current scene- annotated datasets are limited. [iu al. (2024) aggregate motion-only data into paired interaction data, substantially improving scene diversity. Our method is not simply an extension, but further combines enhanced HOT and HST data on the basis of this method, decoupling the data required for the HOSI task through a hybrid training strategy. Recently, some work introduces object interactions in scene. Jiang et al. (2024b) includes object motion but lacks instruction-level annotations, and the variety of objects is restricted. Jiang et al. (2024a) features diverse scenes and text annotations but omits object motion. Kim et al. (025) has limited variety of scenes, making it difficult to achieve generalized scene perception. The restricted variety objects and scenes limits perception and generation for dynamic interactions in complex scenes.
20

---

Published as a conference paper at ICLR 2026
A.4.3 GENERATION ALL AT ONCE
Diffusion- based acceleration methods (Song et al. 2020 Lu et al.
2022:
2025a have demon-
strated strong capabilities in various domains, such video generation Wang et al.
2023
2024 Zhu et al. 3D reconstruction (Ye et al.) 2024), and Embodied AI (Lu et al.
2025 p.
Consistency models achieve high-quality samples with few-step sampling and also
be
optimized through iterative generation (Song et al. 2023 Luo et al. 2023), introduced into the field of motion generation by |aai al. (2024) recently, but the aspect of interaction generation still awaits exploration. We use the inherent nature of consistency models to achieve a unified framework with dynamic perception through few-step generation, and a bunp-aware sampling guidance for iterative optimization, while improving efficiency.
A.5 DATASETS AND BENCHMARK DETAILS
Training Datasets. Our hybrid data training strategy utilizes two existing high-quality datasets: (1) LINGO (Jiang et al.) 2024a]: This is a large-scale human-scent interaction (HST) dataset that provides diverse indoor scenes (high-fidelity meshes), rich text instruc- tions, and corresponding 3D human actions. We use it to train the model's 's navigation and interaction capabilities in complex static environments. (2) OMÓMO (Li et al.) 2023 :
This is large-scale human-object interaction (HOI) dataset that includes full-body motion capture data for interacting with various manipulable objects (especially large objects). We use it to learn fine, physically plausible human-object interaction dynamics.
Due to the lack of scene information in the OMOMO dataset, we adopted a synthetic strat- egy to convert it into a human-object-scene (HOSI) tuple suitable for our unified framework. The process follows the method of Liu et al. 2024 with the specific steps as follows:
For each motion soquence in OMOMO, occupied by the human (via the SMPI-X mesh) and the manipulated object throughout the entire time series. We create a bounding box around this dynamic occupancy volume and voxelize the space outside the bounding box, marking it as non-traversable "walls" or " "obstacles." This synthesizes minimal but logically reasonable scene context for the HOI data without a scene, enabling the model to learn basic spatial obstacle avoidance constraints during training. The synthesized data is processed into the same (S, O, T.T.99 TG) format with the LINGO dataset, where S) is the hesized voxelized scene, O) is the object geometry, T) set according the original data, and g) is set to the pose of the motion endpoint.
first calculate the union of all space points
HOSI Benchmark Construction. In order to comprehensively evaluate the model's gen- eralization ability on complex, unseen HOSI lasks, we have constructed new benchmark. We selected 67 diverse indoor scenes from the TRUMANS dataset and excluded the objects with dirty SDFs from the OMOMO dataset, choosing 7 categories of large manipulable objects (such as boxes, chairs, tables, etc.). In each scene, randomly select start and goal position for each object type, and choose a pair of initial and target poses from the OMOMO-test including human and object, ensuring that both initial and goal poses have collision in the scene. We ensure that there is a passage between the start and target points, but the path may need to detour. The initial motion and text instruction are sampled from OMOMO-test and therefore inherits diverse motion types, including lifting, kicking, dragging, rotating and so on, ensuring comprehensive and representative evaluation. By combining different scenes, objects, starting points, and target points, we have collectively generated 469 independent test sequences, each of which constitutes a unique HOST task.
A.6 MODEL ARCHITECTURE
Motion Generator. Our motion generator is based on Transformer encoder architecture, which includes layers and 16 attention heads. Each frame's 's raw motion state of the human and interactive object is first embedded into a 512-dimensional latent space via MLP, where the diffusson process is applicd. The resulting noised motion tokens, along with conditional tokens (scene, text, goal and object geometry), are fed into the Transformer for denoising. The denoised latent tokens are finally decoded by another MLP to reconstruct the motion state in the original space. Other conditions are embedded using the corresponding
21

---

Published as a conference paper at ICLR 2026
encoder. An embedding of diffusion timestep is added to the condition tokens and all tokens undergo a positional encoding before being fed into the Transformer. It generates motion sequences in autoregressive manner, by replacing the first two tokens of the current window to the last two frames from the previous window without noise.
Scene Perception Encoder. We use Vision Transformer (ViT) architecture (Dosovit- skiy ct a 2020) with 6 Transformer layers and 16 attention heads to encode scene Ieatures from 3D voxel inputs. The input consists of five NxNxN voxel grids (3 dynamic and static), which are flattened and processed into a patch sequence, with N-32 denoting the number of voxels per spatial dimension. Each 32x32x32 grid represents a cubic region of 1.2 meters per side centered at the pelvis/goal, so each element corresponds to 3.75 centimeter cube indicating occupancy or emptiness. The VIT outputs a 512-dimensional embedding for each voxel grid, which is further fused with the positional encoding of its correspond- ing pelvis or goal position to incorporate precise spatial cues. These five 512-dimensional embeddings jointly constitute the scene condition.
Text Encoder. First, we use the CLIP encoder (Radford et al., 2021 to convert the MLP to transform
lextual instruction into encoding with 768-dimension. Then, this encoding into 112-iimnsioon as the final text condition.
Goal Encoder. We use different MLPs to encode pelvis and object goals into two 512- dimensional vectors. For the pelvis goal, we retain only the 2-dimensional horizontal coor- dinates input for the model. For the object goal, the 3-dimensional target coordinates serve as input for the model. For motion that do not involve object, we mask the object goal tokens as zeros.
Object Geometry Encoder. We use an MLP to project the object BPS representatin from the 1024x3 vector to 512-dimensional embedding, which serves the object geometry condition.
A.7 IMPLEMENTATION DETAILS
Training Details. We used the Adam optimizer for training, with a learning rate set to 0,0001, and a batch size of 2048. The model was trained for about 500 epoches on 4 NVIDIA A100 GPUs. During training, We randomly mask the dynamic scene encoding with a ratio of ten percent. For distillation, we use DDIM the ODE solver with total 25 timesteps. The scale w of classificr-free guidance (CFG) is uniformly sampled from the interval [0, 3.0) during training. The weight settings for the total loss function are A 1 and Ao = 0.5 to balance the consistency loss and the kinematic auxiliary loss.
Inference Details. In reasoning, we first use 1 step to generate a rough motion trajectory (without using dynamic scenes as a condition). Then, we perform 15 steps for optimization with guidance. In each iteration, we the actions generated in the previous iteration to update the dynamic scene voxels. Bump-aware guidance is activated in each sampling iteration. The guidance scale "r, is related to the sampling step Tn, providing stronger guidance in the early stages of sampling. Pre-computation of the distance map makes the additional overhead of this module extremely minimal.
A.8 LARGE LANGUAGE MODEL USAGE STATEMENT
We used the large language model as a general- purposs language-polishnng tool. The model was prompted to improve grammar, word choice, and sentence fluency selected portions of the manuscript. All generated suggestions were manually reviewed and edited by the authors, who take full responsibility for the final text.
22


Reviews:
[{"review_id": "CPqeWTZVVh", "reviewer": "Reviewer_2xtj", "summary": "The paper proposes a method to generate paired human-object interaction motion in the context of a static scene. The input is the static scene geometry, interaction object geometry, text instruction, and goal position of the object. The dynamic human-object-scene interaction is represented as a sequence of five voxel grids, representing the scene and object occupancy at the start position, goal position, and (optionally) 3 intermediate timesteps along the human-object interaction trajectory. Voxel grids at intermediate timesteps can be masked out for unconditional trajectry generation. Then, an auto-regressive consistency model is trained via consistency distillation to generate human and object motion conditioned on these inputs. To ensure strict collision avoidance, the denoised trajectories outputted by the consistency model are iteratively refined to avoid colliding with the static scene geometry, by introducing guidance from each colliding voxel to the nearest free-space. The method is trained on a combination of real-world HSI data (LINGO) and synthetic HOI data (OMOMO).", "questions": "- The exact output format of the dynamic perception encoder has many missing details and is not fully clear in Sec. 3.2. It would be helpful to refer to the appendix in the main text and provide mathematical notations with the shape of each variable. My understanding is, each \"voxel grid\" is a 3D array {0,1,2}^(NxNxN) where N is the size of the voxel grid? And there are five voxel grids, corresponding to the start position, goal position, and three intermediate timestamps? How are the intermediate timestamps sampled?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Human-object-scene interaction generation is important and challenging problem.\n- The paper achieves substantially improved results compared to LINGO and TRUMANS.\n- The method is technically sound and the proposed components are effective: the dynamic perception encoder helps improve task success rate, the consistency model improves generation speed, and the bump-aware guidance reduces scene penetration.", "weaknesses": "- Despite the dynamic scene encoder and bump-aware guidance, the method does not seem to generate physically plausible manipulation motions. In Fig2(c) and Fig2(e), the crate is at a strange angle, hand poses do not seem to be predicted, and the hands are either penetrating with the crate or not in contact.\n- The results are shown for a single task of moving an object from the start location to the goal location through a static scene. Therefore, it is not clear to me that the text instruction is necessary or useful in the problem formulation. More qualitative or quantitative examples for diverse interaction types (such as those seen in OMOMO - kicking and dragging objects, lifting over your head or in different manners) would strengthen the applicability of the method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ebkN5j3m9n", "reviewer": "Reviewer_9B2s", "summary": "The paper introduces a coarse-to-fine, instruction-conditioned generation framework using a consistency model that iteratively refines the human-object-scene interaction (HOSI). \n\nA dynamic perception strategy is key, where scene context is continuously updated using trajectories from the preceding refinement step to condition the subsequent refinement, ensuring consistent interactions.\n\nExperiment results show that the hybrid data training strategy overcomes data limitations by combining\nreal-scene HSI data with synthesized HOI data, achieving zero-shot scene generalization.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The core idea is to integrate a dynamic perception strategy and a coarse-to-fine iterative refinement scheme, which is aligned with a consistency model's denoising process. \n\nThis novel approach ensures that the scene context (which changes due to object and human movement) is updated at each step, leading to more physically consistent and realistic HOSI.\n\nSuch a method is robust, allowing the model to learn diverse and complex interactions without the dependence on a massive, fully annotated HOSI dataset, enabling strong generalization to unseen scenes.", "weaknesses": "Need to show some benefits on the downstream tasks such as robot learning, gaming, humanoid motion learning, e.t.c.\n\nHow is the model integrated with real physics? \n\nAnd how is the model transfer to different simulation socially-inteactive environments (indoor vs outdoor, household, store, hospital, e.t.c.)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BeuAGimGbU", "reviewer": "Reviewer_f1br", "summary": "The problem this paper aims to tackle is the (1) limited annotation in dataset and the (2) dynamic scene variation (the movement of objects changes the scene as well), for a task termed as human-object-scene interaction generation. The proposed approach consists of several contributions/effective modules, termed as the dynamic perception, bump-aware guidance, the consistency guided iterative refinement, with the help of hybrid-data-training strategy. It seems that using the proposed method achieves good results in the new benchmark HOSI and the HOI as well.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength\n-\tThe technical method seems correct\n-\tThe motivation is well-presented", "weaknesses": "Weakness \n\n-\tIn table 3, it seems that the C+B provides inferior results. It does not convince me that all components achieve better performance, esp in the task accuracy metric. Maybe more explanations should be provided. \n\n-\tIt seems that the baseline methods in comparison seem limited. More evaluations should be performed comparing with more methods. \n\n\n-\tThe proposed contributions seem a lot. These makes the contributions diluted for some perspectives. The relationship among these contributions, and evaluation of their effectiveness one by one is necessary. Otherwise we cannot understand how each component tackles which portion of data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WtStJTdslP", "reviewer": "Reviewer_73R8", "summary": "This paper presents InfBaGel, a unified framework for human–object–scene interaction generation that combines a coarse-to-fine consistency model with bump-aware guidance for physically plausible motion (mostly for collision free). It also introduces a hybrid data training strategy that mixes real and synthetic data to reduce annotation needs and improve generalization across diverse scenes.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation and contribution is clear: to generate higher quality HOSI data and to tackle the data scarcity of this field.\n2. Experiment shows that the method is significantly better than previous works in terms of collision avoidance.", "weaknesses": "1. Limited qualitative results. Only two action sequences are presented in the paper, which makes it difficult to fully assess the model’s performance.\n2. The method part confuse me a bit. Specifically, for Section 3.2 (“Motion Consistency Model”), does this section correspond to the Auto-regressive Consistency Model shown in the figure? Additionally, in line 277, how are human joints and object points constructed from human motion only? Is this perhaps a typo?\n3. In Figure 2(c), there is a noticeable penetration between the hand and the box. Is this due to training solely on the OMOMO dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)