Unified Human-Scene Interaction via Prompted Chain-of-Contacts
Authors: Zeqi Xiao
Tai Wang
Jingbo Wang
Jinkun Cao
Wenwei Zhang
Bo Dai
Dahua Lin
Jiangmiao Pang
Venue: ICLR 2024 spotlight (2024)
Keywords: Human-Scene Interaction
Chain-of-Contacts
Unified
LLM
Primary area: applications to robotics, autonomy, planning
TL;DR: A unified Human-Scene Interaction framework that supports versatile interactions through language commands

Abstract: Human-Scene Interaction (HSI) is a vital component of fields like embodied AI and virtual reality. Despite advancements in motion quality and physical plausibility, two pivotal factors, versatile interaction control and the development of a user-friendly interface, require further exploration before the practical application of HSI. This paper presents a unified HSI framework, UniHSI, which supports unified control of diverse interactions through language commands. The framework defines interaction as ``Chain of Contacts (CoC)", representing steps involving human joint-object part pairs. This concept is inspired by the strong correlation between interaction types and corresponding contact regions. Based on the definition, UniHSI constitutes a Large Language Model (LLM) Planner to translate language prompts into task plans in the form of CoC, and a Unified Controller that turns CoC into uniform task execution. To facilitate training and evaluation, we collect a new dataset named ScenePlan that encompasses thousands of task plans generated by LLMs based on diverse scenarios. Comprehensive experiments demonstrate the effectiveness of our framework in versatile task execution and generalizability to real scanned scenes.

Published as a conference paper at ICLR 2024
UNIFIED HUMAN-SCENS INTERACTION VIA PROMPTED CHALN-OF-CONTACTE
Zeqi Xiao! Tai Wang', JJingoo Wang', Jinkun Cao Jinkun Wenwei Zhang', Bo Dai
Dahua Lin', Jiangmiao Pang 1Ed
Shanghai AI Laboratory, Rs-Lab, NTU, SCMU
Unified and Long-horizon Control With Language Commands Inputs
vine-granularity Control
Lie on the bed, knees up
book the shelf
Lean forward and watch TV
UniHSI
Diverse Interactions with the same object
Play video games while
Multi-obj Interaction
Lean back and watch TV
Figure 1: UniHSI facilitates unified and long- ong-horizon control in response to natural language com- mands, offering notable features such as diverse interactions with singular object, multi-object interactions. and fine-granularity control.
ABSTRACT
Human-Scene Interaction (HSI) vital component of fields like embodied AI and virtual reality. Despite advancements in motion quality and physical plausi- bility, two pivotal factors, versatile interaction control and user-friendly interfaces, require further exploration for the practical application of HSI. This paper presents a unified HSI framework, named UniHSI, that supports unified control of diverse interactions through language commands. The framework defines interaction as "Chain of Contacts (CoC)", representing steps involving human joint-object part pairs. This concept inspired by the strong correlation between interaction types and corresponding contact regions. Based on the definition, UniHSI constitutes a Large Language Model (LLM) Planner to translate language prompts into task plans in the form of CoC, and Unified Controller that turns CoC into uniform task execution. To support training and evaluation, we collect a new dataset named ScenePlan that encompasses thousands of task plans generated by LLMs based on diverse scenarios. Comprehensive experiments demonstrate the effectiveness of our framework in versatile task execution and generalizability real scanned scenes.
INTRODUCTION
Clman--Scen Interaction (HSI) constitutes crucial element in various applications, including em- bodied AI and virtual reality. Despite the great efforts in this domain to promote motion quality
(Holden et al. 2017 Starke et al. 2019 2020 Hassan et al. 2021b Zhao al., 2022 Hassan al.,
X Corresponding Author. Project page this URL

---

Published as a conference paper at ICLR 2024
Wang al. 2022a) and physical plausibility (Holden et al. 2017 Starke et al. 2019 2020 Hassan et al., 2021b; Zhao et al., 2022 Hassan et al., 202 a Wang et al. 2022a). two key factors, versatile interaction control and the development of user-fmendl. interface, yet to be explored
2021a;
before HSI can be put into practical usage.
This paper aims to provide HSI system that supports versatile interaction control through lan- guage commands, one of the most uniform and accessible interfaces for users. Such a system re- quires: 1) Aligning language commands with precise interaction execution, 2) Unifying diverse interactions within a single model to ensure scalability. To achieve this, the initial effort involves the uniform definition of different interactions. We propose that interaction itself contains a strong prior in the form of human-object contact regions. For example, in the case of "lie down on the bed", it can be interpreted as "first the pelvis contacting the mattress of the bed, then the head contacting the pillow". " To this end, we formulate interaction ordered sequences of human joint-object part con- tact pairs, which we refer to as Chain of Contacts (CoC). Unlike previous contact-driven methods, which are limited to supporting specific interactions through manual design, our interaction defini- tion is generalizable to versatile interactions and capable of modeling multi-round transitions. The recent advancements in Large Language Models have made it possible to translate language com- mands into CoC. The structured formulation then can be uniformly processed for the downstream controller to execute.
Following the above formulation, we propose UniHSI, the first Unified physical HST framework with language commands inputs. UniHSI consists of high-level LLM Planner to translate language inputs into the task plans in the form of CoC and low-level Unified Controller for executing these plans. Combining language commands and background information such body joint names and object part layout, we harness prompt engineering techniques to instruct LLMs to plan interaction step by step. We design the TaskParser to support the unified execution. serves as the core of the Unified Controller. Following CoC, the TaskParser collects information including joint poses and object point clouds from the physical environment, then formulates them into uniform task observations and task objectives.
As illustrated in Fig. [ the Unified Controller models whole-body joints and arbitrary parts of objects in the scenarios to enable fine-granularity control and multi-object interaction. With differ- ent language commands, we can generate diverse interactions with the same object. Unlike previ- ous methods that only model limited horizon of interactions, like "sitting down", we design the TaskParser to evaluate the completion of the current steps and sequentially fetch the next step, result- ing in multi-round and long-horizon transition control. The Unified control leverages the adversarial motion prior framework (Peng et al., 2021) that uses a motion discriminator for realistic motion synthesis and physical simulation (Makoviychuk al., 202T) to ensure physical plausibility.
Another impressive feature of our framework is the training is interaction annotation-free Previous methods typically require datasets that capture both target objects and the corresponding motion sequences, which demand numerous laboring. In contrast, we leverage the interaction knowledge of LLMs generate interaction plans. It significantly reduces the annotation requirements and makes versatile interaction training feasible. To this end, we create novel dataset named ScenePlan. encompasses thousands of interaction plans based on scenarios constructed from PartNet (Mo) et al. 2019) and ScanNet (Dai et al.] (017)) datasets. We conduct comprehensive experiments on ScenePlan. The results illustrate the effectiveness of the model in versatile interaction control and good generalizability on real scanned scenarios.
2 RELATED WORKS
Kinematics- Human-Scene Interaction. How to synthesize realistic human behavior is long-standing topie. Most existing methods focus on promoting the quality and diversity of hu-
manoid movements (Barsoum et al. 2018 Harvey et al. 2020; Pavllo et al.) 20188) Yan et al. 2019 Zhang et al.) 2022at Tevet et al., 2022b Zhang et al. 2023b) but do not consider scene influence. Recently, there has been a growing interest in synthesizing motion with human-scene interactions, driven by its applications in various applications like embodied AI and virtual reality. Many previ- methods (Holden et al. 2017 Starke et al., 2019: 2020 Hassan et al. 2021b: Zhao et al. 2022 Hassan et al., 2021a| Wang et al. 2022a] Zhang et al., 2022b Wang et al. (022b)) use data-driven kinematic models to generate static or dynamic interactions. These methods are typically inferior in
2

---

Published as a conference paper at ICLR 2024
Table I: Comparative Analysis of Key Features between UniHSI and Preceding Methods
Methods
Unified Inseraction
Language
Long horizon Transition
Interaction Amnotation free
Control
Multi- Interactions
Input
NSM Starke et all
2019
(pelvis, hands) (pelvis) (pelvis, hands)
SAMP Faasiun di
20213
COLCHI
HUMANISE Wang A si
ScenDiffuse Fronng et
PADL Junivsy
InterPlyy Hassai at
(pelvis, head, hands)
Qurs
15 (whole-body)
physical plausibility and prone to synthesizing motions with artifacts, such penetration, floating, and sliding. The need for additional post-processing to mitigate these artifacts hinders the real-time applicability of these frameworks.
Physics- based Human-Scene Interaction. Recent advances in physics- based methods (c.g., (Peng et al., 2021] 2022 Hassan et al. 2023 Juravsky et al. 2022 Pan et al. 2023) hold promise for ensuring physical realism through physics-aware limulators. However, they have limitations: 1) They typically require separate policy networks for each task, limiting their ability to learn versatile interactions within a unified controller. 2) These methods often focus on basic action-based control, neglecting finer-grained interaction details. 3) They heavily rely on annotated motion sequences for human-scems interactions. which can be challenging to obtain. In contrast, our UniHSI re- designs scene interactions into a uniform representation, driven by world knowledge from our high-level LLM Planner. This allows us to train unified controller with versatile interaction skills without the need for annotated motion sequences. Key feature comparisons are in Tab. T
Languages in Human Motion Control. Incorporating language understanding into human motion control has become a recent research focus. Existing methods primarily focus on ccene-agnostic motion synthesis (Zhang et al. [2022a] Chen et al. 2023 Tevet et al. 2022a b Zhang al.) 2023a b Jiang et al. 2023) (Athanasiou et al. 2023). Generating human-scene interactions using language commands poses additional challenges because the output movements must align with the com- mands and be coherent with the environment. /Zhao et al. (2022) generates static interaction gestures through rule-based mapping of language commands specific tasks. Juravsky al. (2022) utilized BERT (Devlin et al. (018) to infer language commands, but their method requires pre-defined tasks and different lowelevel policies for task execution. [Wang et al. (2022b) unified various tasks in CVAE Yao et al., 2022) network with a language interface, but their performance was limited due challenges grounding target objects and contact areas for the characters. Recently, there have been some explorations on LLM-based agent control. BBoohan et al. (2023) uses fine-tuned VLM (Vision Language Model) to directly output actions for low-level robots. |Rocamonde al. (2023) employs -e.e-geneeatedd cos-similarity as RI. training rewards. In contrast, UniHSI utilizes large language models to transfer language commands into the formation of Chain of Contacts and design robust unified controller to execute versatile interaction based on the structured formation.
3 METHODOLOGY
As shown in Fig. 2 UniHSI supports versatile human-scene interaction control following language commands. In the following subsections, we first illustrate how we design the unified interaction formulation as CoC(Sec. 3.1.. Then we show how we translate language commands into the unified formulation by the LLM Planner (Sec. 3.2p. Finally, we elaborate on the construction of the Unified Controller (Sec. 3.3
3.1 CHAIN OF CONTACTS
The initial effort of UniHSI lies in the unified formulation of interaction. Inspired by Hassan et al. (2021b), which infers contact regions of humans and objects based on the interaction gestures of humans, we propose a high correlation between contact regions and interaction types. Further, interactions are not limited to single gesture but involve sequential transitions. To this end, we can universally define interaction as CoC C, with the formulation as
(1) where S, is the tth contact step. Each step S includes several contact pairs. For each contact pair, control whether joint contacts the corresponding object part and the direction of the contact.
C = S2...,,
3

---

Published as a conference paper at ICLR 2024
Instruction: would like to play video games for a while. After that, I will go to sleep.
LLM Planner
Step n:Play video games
surface Pair 2, left hand keyboard
1, pelvis
LLMS
Pair 3, right hand on keyboard
Unlabeled Motion Datasets
Scene Information
Chain of Contacts
Unified Controller
Fetch
Next
Randomly Fetch
TaskParser
St
5t
<Pair Obs
<air Reward>
Discriminator
<Pair Obs
<Pair Reward>
St
Train & Infer
ar
Control Policy
Environment
Only Train
Step
Step 2
Step
Step 4
Step
Step 6
Step 7
Figure 2: Comprehensive Overview of UnIHSI. The entire pipeline comprises two principal com- ponents: the L.I Planner and the Unified Controller. The L.LM Planner processes language inputs and background scenario information to generate multi-step plans in the form of CoC. Subsequently, the Unified Controller executes CoC step by step, producing interaction movements.
We construct each contact pair with five elements: an object object part p. a humanoid joint j. the contact type e of j and p. and the relative direction from j to p. The contact type includes "contact", "not contact", and "not care". The relative direction includes "up". "down". "front", "back", "left", and "right". For example, one contact unit Jo,,,ss c, d} could be (chair, seat surface, pelvis, contact, up}. In this way,
formulate each S as
S = Pfo. d1), (0,, (0,,Pa, C22 d2),...
CoC is the output of the LLM Planner and the input of the Unified Controller.
3.2 LARGE LANGUAGE MODEL PLANNER
We leverage LLMs as our planners to infer language commands L into manageable plans C. As shown in Fig. the inputs of the LLM Planner include language commands C. background scenario information 13, humanoid joint information J together with pre-set instructions, rules and examples. Specifically, B includes several objects O and their optional spatial layouts. Each object consists of several parts P, i.e., chair could consist of arms, the back, and the seat. The humanoid joint information is pre-defined for all scenarios. We use prompt engineering to combine these elements together and instruct LLMS output task plans. By modifying instructions in the prompts, can generate specified numbers of plans for diverse ways of interactions. We can also let LLMs automatically generate plausible plans given the scenes. this way, we build our interaction datasets to train and evaluate the Unified Controller.
3.3
UNIFIED CONTROLLER
The Unified Controller takes multi-step plans C and background scenarios the form meshes and point clouds input and outputs realistic movements coherent the environments.
4

---

Published as a conference paper at ICLR 2024
Instruction: I want play video games for whil, then go sleep.
Step 1: close to the chair.
of background Information) room OBJECTS: [bed, table, laptap]. The [OB) ECT: laptop] is upon the [OB] ECT: table) The [OB) ECT: bed] has PARTS: [pllow, mattress)... The human OINTS: (pelvis, hip, knee. left foot.. lend background Information]
Step 2: Sit on chair.
Pair 1: khair, seat surface, pelvis, contact, up)
Given the instruction and background information, generate 1 task plan according the following rules and examples. Istart rules]l. Each plan should be composite 2. Each step should contain meaningful joint-part pairs...end rules] examplesTend exemmpees)]
steps.
Step 3: Play video games.
Figure 3: The Procedure for Translating Language Commands into Chains of Contacts
Preliminary. We build the controller upon AMP (Peng et al. 2021) AMP is conditondt re- inforcement learning framework incorporated with an adversarial discriminator to model the motion prior. Its objective is defined by a reward function IR(-)
(3) The task reward R defines the high-level goal 9 agent should achieve. The style reward π encourages the agent to imitate low-level behaviors from motion datasets. w and wD are empirical weights of RC and R respectively. 8t, a₁2 s are the state at time t, the action at time f, the state at time t ++1 1, respectively. The style reward R is modeled using an adversarial discriminator D, which is trained according the objective:
R( St. Gr St+1, 9) = RR (st, a, 9)+1,9) + (ARRSss 8(+1).
arg min - (E.M()))))41) [log (D(s), ((1)) (a [log (1 D( ,811)) D
(4)
V ((6) (=(aM
where (;. and d (s, 1c+.) denote the likelihood of a state transition from 8, to 8 in the dataset .M and the policy respectively. wSP is empirical coefficient to regularize gradient penalty, 8 = ((x) is the observation for discriminator. The style reward S = R.)(.) for the policy then formulated as:
(S 8141) = log(1 D(s s s441).
(5)
We adopt the key design of motion discriminator for realistic motion modeling. In our implemen- tation, we feed 10 adjacent frames together into the discriminator to assess the style. Our main contribution to the controller parts lies in unifying different tasks. As shown in the left part of Fig.
(a), AMP (Peng et al. 202T). as well as most the previous methods (Juravsky et al., [022] Zhao et al. 2023). design specified task observations, task objectives, and hyperparameters to train task- specified control policy. In contrast, we unify different tasks into Chains of Contacts and devise a TaskParser process the uniform representation.
TaskParser. As the core of the Unified Controller, the TaskParser is responsible for formulating CoC into uniform task observations and task objectives. It also sequentially fetches steps for multi- round interaction execution.
Given specific contacting pair {o, p. c, d]. for task observation, the TaskParser collects the corresponding position v E R of the joint j. and point clouds oP E Rmxx of the object part from the simulation environment, where m is the point number of point clouds. It selects the nearest point "h E vP from v² to v2 as the target point for contact. We formulate task observation of the single pair as -p- v., c, d]. For the task observation in the network, we map e and into digital numbers, but we still use the same notation for simplicity. Combining these contact pairs together, we get the uniform task observations 8 [[0 oi C1, did, (w2 v2g C2, d2); ..., (v.. v Cn, dn }.
The task reward G = [- is the summarization of all contact pair rewards:
we Re, k R Σ k
n.
(6)
We model each contact reward R2 according to the contact type cg. When c, = contact, the contact reward encourages the joint j to be close the part p, satisfying the specified direction d. When
5

---

Published as a conference paper at ICLR 2024
Model 1
Design
Task 1
Unified Interface
Model 2
Design 2
Task 2
TaskParser
Model 3
Design
Task 3
Unified Policy
Previous:
Ours: Unified designs for all tasks
Isolated designs for different tasks
Framework Comparison
(b) Ego-centric Heightmap
Figure 4: Design Visualization. (a) Our framework ensures a unified design across tasks using the unified interface and the TaskParser. (b) The ego-centric height map in ScanNet scene is depicted by green dots, with darker shades indicating greater height.
Table 2: Performance Evaluation on the ScenePlan Dataset.
Success Rate (%)
Contact Error Mid
Success Steps
Source
Simple
Mid
Hard
Simple
Hard
Simple
Mid
Hard
PurtNet (Mo 2019]
91.1
63.2
39.7
0.038
0.073
0.101
2.3
6,1
Adaplive Weighils wo Heightmap
21.2
5.3
0.181
0.312
0.487
0.7
0.0
61.6
45.7
0.068
0.076
1.8
0.0
ScanNet Dai et
2017
76.1
43.5
32.2
0.067
0.101
0.311
1.8
2.9
4.9
C notcontact, we hope the joint j is not close to the part If c = not care, we directly set the reward to max. Following the idea, the h contact reward R3 is defined as
exis (xpe(xxxx)) [ []) + Wai mam.ma.xx.dd.... 0),
C* contact
R = 1 exp(-wsss dsl)),
not contact
Ck not care
where dę DP - v indicates the Eth distance vector, dy is the normalized unit vector of d2. de the unit direction vector specified by direction dy, and Cg is the h contact type. Wdia. Wddr Wak corresponding weights. We set the scale interval of R₃ [0, 1] and use exp ensure it.
Similar to the formulation of contact reward, the TaskParser considers step to be completed All k = 1, 2..,... satisfy: if Ck = contact : ||dæ|| < 0.1 and dıds > 0.8, if Ca = not contact || 0.1, if cg not care, True.
Adaptive Contact Weights. The formulation of includes lots of weights to balance different con- tact parts of the rewards. Empirically setting them requires much laboring and is generalizable to versatile tasks. To this end, we adaptively set these weights based on the current optimization process. The basic idea to give parts of rewards that are hard optimize high rewards while lowering the weights of easier parts. Given R1. R2, .... Ra, we heuristically set their weights to
1 RR
Wk
71 2k-122.... Rr.
Ego-centric Heightmap. The humanoid must be scene-aware to avoid collision when navigating or interacting in scene. We adopt similar approaches in Wang et al. (2022a): Won al. (2022); Starre et al. 2019) that sample surrounding information as the humanoid's 's observation. We build square ego-centric heightmap that samples the height of surrounding objects (Fig. (b)). important to extend our methods into real scanned scenarios such as ScanNet (Dai et al. 2017 in which various objects are densely distributed and easily collide.
EXPERIMENTS
Existing methods and datasets related to human-scene interactions mainly focus short and limited tasks (Hassan et al., 2021a Peng et al., 2021 Hassan et al. 2023; Wang et al. 2022b). To the best of our knowledge, commands as input. To this end, we construct a novel dataset for training and evaluation. We also conduct various ablations with vanilla baselines and key components of framework.
are the first method that supports arbitrary horizon interactions with language

---

Published as a conference paper at ICLR 2024
Simple
Medium
close to the bed
Sit on the bed
Lie Down on the bed
Get close the chair
Lean back to the chair
Cross the leg
Lean forward
Stand
In meditating
Hard
Get close
Stand up
on the couch
the toilet
the garbage can
Rubbish
the couch
the couch
Figure 5: Visual Examples Illustrating Tasks Varying Difficulty Levels.
4.1 DATASETS AND METRICS
To facilitate the training and evaluation of UniHSI, we construct novel ScenePlan dataset com- prising various indoor scenarios and interaction plans. 'hee indoor scenarios are collected and con- structed from object datasets and scanned scene datasets. We leverage our L.MM Planner to generate interaction plans based on these scenarios. The training of our model also requires motion datasets to train the motion discriminator, which constrains our agents to interact in natural ways. We follow the practice Hassan et al. (2023) evaluate the performance of our method.
ScenePlan. We gather scenarios for ScenePlan from PartNet (Mo et al. 2019) and ScanNet (Dai eet al., 2017) datasets. PartNet offers indoor objects with fine fine-grained part annotations, ideal for LLM Planners. We select diverse objects from PartNet and compose them into scenarios. For Scan- Net, which contains real indoor on fragmented area annotations. We then employ the LIM Planner to generate various interaction plans from these scenarios. Our training set includes 40 objects from PartNet, with 5-20 plausible interaction steps generated for each object. During training, we randomly choose 1-4 objects from this for each scenario and select their steps interaction plans. The evaluation consists of 40 PartNet objects and 10 ScanNet scenarios. We construct objects from PartNet into scenarios either manually randomly. We generated 1. 1,040 interaction plans for PartNet scenarios and 100 interac- tion plans for ScanNet scenarios. These plans encompass diverse interactions, including different types, horizons, and multiple objects.
scenes, collect scenes and annotate key object parts based
Motion Datasets. We use the SAMP dataset (Hassan et al. 2021a) and CIRCLE (Araújo et al.,
2023) as our motion dataset. SAMP includes 100 minutes of MoCap clips, covering common walk- ing, sitting, and lying down behaviors. CIRCLE contains diverse right and left-hand reaching data. We use all clips SAMP and pick 20 representative clips in CIRCLE for training.
Metrics. We follow Hassan et al. (203)) that Success Rate and Contact Error (Precision
in Hassan et al. (2023) as the main metrics to measure the quality of interactions quantitatively. Success Rate records the percentage of trials that humanoids successfully complete every step of
the whole plan. In our experiments, we consider a trial of n steps to be successfully completed
humanoids finish it in 10 seconds. We also record the average error of all contact pairs:
ContactError
Σ er// Σ1
=
m0n - || O).
|d
not contact
contact
(9)
(,c:500
We further record. Success Steps, which denotes the average success step in task execution.
4.2 PERFORMANCE ON SCENEPLAN
We initially conducted experiments on ScenePlan dataset. To measure performance in detail, categorize task plans into three levels: simple, medium, and hard. We classify plans within 3 steps as simple tasks, those with more than steps but with single object medium-level tasks, and those with multiple objects as hard tasks. Simple task plans typically involve straightforward interactions. Medium-level plans encompass diverse interactions with multiple rounds of transitions. Hard

---

Published as a conference paper at ICLR 2024
Table 3: Ablation Study on Baseline Models and Vanilla Implementations.
Success Rate (%)
f
Contact Error
Methods
Sit
Lie Down
Reach
Sit
Lie Down
Reach
NSM Sit Starke al..
2019
75.0
0.19
SAMP- Sit [Hassan
2021a
75.0
0.06
SAMP Lie Down
(aasaan (Hasana
202
50.0
0.05
InterPhys Sit (Hasaan et 2025
93.7
0.09
InterPhys Lie Down Tassan al.
2023
80.0
0.30
AMP (Peng 2021
2021)-4
77.3
0.090
AMP-LiE Down
21.3
0.112
AMP-Reach
98.1
0.016
Aanilla Combination (VC)
62.5
20.1
90.3
0.093
0.108
0,032
UniHST
94.3
81.5
97.5
0.032
0.061
0.016
Success Rate (96
AMP-Sit
AMP AMP-L
60
AMP-Sit AMP-VCC-It
2000
4000
8000
9900
10000
Training Steps
Success Rate (%)
Ours-Sit
Ours-Lie
AMP-VC-Lie
Training Steps
(a) Visual comparisons on task performance
(b) Comparisons on Success Rate v.s. Training Steps
Figure 6: Visual Ablations. (a) Our model exhibits superior natural and accurate performance compared to baselines in tasks such as "Sit" and "Lie Down". (b) Our model demonstrates more efficient and effective training procedures.
task plans introduce multiple objects, requiring agents navigate between these objects and interact with one or more objects simultaneously. Examples of tasks are illustrated in Fig. 5
As shown in Table 2 UniHSI performs well in simple task plans, exhibiting high Success Rate and low Error. However, as task plans become more diverse and complex, the performance of our model experiences noticeable decline. Nevertheless, the Success Steps metric continues increase, indicating that our model still performs well parts the plans. It's important to note that the scenarios in the ScenePlan test set unseen during training, and scenes from ScanNet exhibit modality gap with the training set. The overall performance on the test set demonstrates the versatile capability, robustness, and generalization ability of UniHSI.
4.3 ABLATION STUDIES
4.3.1 KEY COMPONENTS ABLATION
Choice of LLMs for UniHSL. We evaluated different Language
Model (LM) choices
for the LLM Planner using 100 sets of language com- mands. We compared task plan Execution Success Rate (ESR) and Planning Correctness (PC) among hu- mans, S0PT33..OpenA12222 (020). and GPT-4OpenAAl (2023) across 10 tests per plan. iC evaluated by humans, with choices of "correct" and "not correct". GPT-4 outper- formed GPT-3.5, but both I.MMs still lag behind human performance. Failures typically involved incomplete planning and out-of-distribution interactions, like GPT-3.5 occasionally skipping transitions or generating out-of-distribution actions like opening laptop. While using rules in prompts and GPT-4 mitigate these issues, errors still occur.
Table 4: UniHSI with different LLMs.
LLM Type
ESR (%) f
PC (%) 1
Human
73.2
w. GPT-3.5
35.6
49.1
GPT-4
57.3
71.9
Adaptive Weights. debonsssttt that removing Adaptive Weights from controller leads to substantial performance decline across all task levels. Adaptive Weights are crucial for optimiz- ing various contact pairs effectively. They automatically adjust weights, reducing them for unused
8

---

Published as a conference paper at ICLR 2024
easily learned pairs and increasing them for more challenging pairs. This becomes especially vital as tasks become more complex.
Ego-centric Heightmap. Removing the Ego-centric Heightmap results in performance degradation, especially for difficult tasks. This heightmap is essential for agent navigation within scenes, enabling perception of surroundings and preventing collisions with objects. This is particularly critical for challenging tasks involving complex scenarios and numerous objects. Additionally, the Ego-centric Heightmap key to our model's ability generalize to real scanned scenes.
4.3.2 DESIGN COMPARISON WITH PREVIOUS METHODS
Baseline Settings. We compared our approach to previous methods using simple interaction tasks like "Sit," "Lie Down," and "Reach." Direct comparisons are challenging due to differences in train- ing data and code unavailability for closely related method (Hassan et al. (023). We integrated
key design elements from Hassan al. (2023) into our baseline model (Peng al. 2021 to ensure
fairness. Task observations and objectives were manually formulated for various tasks, following
Hassan al. (023)) with task objectives expressed as:
R 10.7Rneat + 0.3R if distance > 0.5m
(10)
0..7Ruea + 0.3, otherwise
In this equation, faa encourages character movement toward the object, and Ruear encourages specific task performance when the character is close, necessitating task-specific designs.
We also created a vanilla baseline by consolidating multiple tasks within a single model. We com- bined task observations from various tasks and included task choices within these observations. We randomly selected tasks and trained them with their respective rewards during training. This experi- ment involved a total of 70 objects (30 for sitting, 30 for lying down, and 10 for reaching) with 4096 trials per task and random variations in orientation and object placement during evaluation.
Quantitative Comparison. In Table UniHST consistently outperforms matches baseline im- plementations across various metrics. The performance advantage is most pronounced in complex tasks, especially the challenging "Lie Down" task. This improvement stems from our approach of breaking tasks into multi-step plans, reducing task complexity. Additionally, our model benefits from shared motion transitions among tasks, enhancing its adaptability. Figure 6 (b) shows that our methods achieve higher success rates and converge faster than baseline implementations. Im- portantly, the vanilla combination of AMP (Peng et al. 2021 results in a noticeable performance drop in all tasks while our methods remain effective. This difference is because the vanilla combi- nation introduces interference and inefficiencies in training, whereas our approach unifies tasks into consistent representations and objectives, enhancing multi-task learning.
Qualitative Comparison. In Figure 6 (a), we qualitatively visualize the performance baseline methods and our model. Our model performs more naturally and accurately than the baselines in tasks like "Sit" and "Lie Down". This is primarily attributed to the differences in task objectives. Baseline objectives (Eq. [0P model the combination of sub-tasks, such as walking close and sitting down, as simultaneous processes. Consequently, agents tend to perform these different goals simul- taneously. For example, they may attempt sit down even if they or throw themselves like projectile onto the bed, disregarding the natural task progression. On the other hand, our methods decompose tasks into natural movements through language planners, resulting in more realistic interactions.
in the correct position
5 CONCLUSION
UniHSI is unified Human-Scene Interaction (HSI) system adept at diverse interactions and lan- guage commands. Defined as Chains of Contacts (CoC), interactions involve sequences human joint-object part contact pairs. UniHSI integrates a Large Language Planner for command transla- tion into CoC and a Unified Controller for uniform execution. Comprchensive experiments show- case UniHSI's effectiveness and generalizability, representing significant advancement in versatile and frienfry HSI systems. Acknowledgement. We acknowledge Shanghai AI Lab and NTU S-Lab for their funding support.

---

Published as a conference paper at ICLR 2024
REFERENCES
Joao Pedro Araújo, Jiaman Li, Karthik Vetrivel, Rishi Agarwal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg. and Karen Liu. Circle: Capture in rich contextual environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
21211-21221, 2023.
Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Gül Varol. Sine: Spatial composition of 3d human motions for simultaneous action generation. arXiv preprint arXiv:2304. 10417, 2023.
Emad Barsoum, John Kender, and Zicheng Liu. Hp-gan: Probabilistic 3d human motion prediction via gan. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 1418-1427, 2018.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- laaioon-lnnuug--ccttioo models transfer web knowledge to robotic control. arXiv preprint 3rXiv:2307.158//1 2023.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18000-18010, 2023.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnner Scannet: Richly-annotate 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Jacob Devlin, Ming-Way Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Félix G Harvey, Mike Yurick, Derek Nowrouzezahraii, and Christopher Pal. Robust motion in- betweening. ACM Transactions on Graphics (TOG), 39(4):60-1, 2020.
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 11374-11323 11384, 2021a.
Mohamed Hassan, Partha Ghosh, Joachim Tesch Dimitrios Tzionas, and Michael J Black. Populat- ing 3d scenes by learning human-scent interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14708-14718, 2021b.
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scene interactions. arXiv preprint arXiv:2302.00883, 2023.
Daniel Holden, Taku Komura, and Jun Saito. funetioned neural networks for character con- trol. ACM Transactions on Graphics (TOG). 36(4):1-13, 2017.
Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song- Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16750-16761, 2023.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as foreign language. arXiv preprint arXiv:2306. 14795, 2023.
Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Padl: Language-directed physics- based character control. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 108. 10470, 2021.
Kaichun Mo. Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object under- standing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 909-918, 2019.
10

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-3: Generative pre-trained transformer 3. Attp:///penaa..com/eeesrr gpt- 3 2020.
OpenAl. Gpt-4 technical report, 2023.
Liang Pan, Jingbo Wang. Buzhen Huang, Junyu Zhang, Haofan Wang. Xu Tang. and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes. arXiv preprint arXiv:2308.0000 2023.
Dario Pavilo, David Grangier, and Michael Auli. Quaternet: A quaternion-based recurrent model for human motion. arXiv preprint 1.XXiv:8805.064880 2018.
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4):1-20, 2021.
Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 11(4):1-17, 2022.
Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint arXiv:2310. 12921, 2023.
Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. ACM Trans. Graph., 38(6):209-1, 2019.
Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. Local motion phases for learning multi-contact character movements. ACM Transactions on Graphics (TOG), 39(4):44-1, 2020.
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Ex- posing human motion generation clip space. In Computer Vision-ECCV 2022: 17th Euro- pean Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pp. 358-374. Springer, 2022a.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022b.
Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10460-20469, 2022a.
Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language-conditioned human motion generation in 3d scenes. Advances in Neural Information Processing Systems, 35:14959-14977 2022b.
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics-based character controllers using conditional vaes. ACM Transactions on Graphics (TOG), 41(4):1-12, 2022.
Sijie Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin. Convolutional sequence generation for skeleton-based action synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4394-4402, 2019.
Heyuan Yao, Zhenhua Song, Baoquan Chen, and Libin Liu, Controlvae: Model-based learning of generative controllers for physics-based characters. ACM Transactions Graphics (TOG), 41 (6):1-16, 2022.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang. Yong Zhang, Hongwei Zhao, Hongtao Lu,, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint 2208 15001, 2022a.
11

---

Published as a conference paper at ICLR 2024
Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. Couch: Towards controllable human-chair interactions. European Conference on Computer Vision, pp. 518-535. Springer, 2022b.
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are general-purpose purpose motion generators. arXiv preprint 2rX. 10900, 2023b.
Kaifeng Zhao, Shaofei Wang, Yan Zhang, Thabo Beeler, and Siyu Tang. Compositional human- scene interaction synthesis with semantic control. In European Conference Computer Vision, pp. 311-27. Springer, 2022.
Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes, arXiv preprint 23..:::203..22111,, 2023.
12


Reviews:
[{"review_id": "UshyGbF7B6", "reviewer": "Reviewer_gSqi", "summary": "The paper contributes the first unified framework that can generate physically plausible HSI motions in static scenes given compounded language commands. To examine the method's effectiveness, the paper constructs a new dataset containing thousands of interaction plans on existing 3D scenes. Experiments show that the proposed framework outperforms baselines on different single-step tasks, and also exhibits its ability to handle hard task plans and good generalizability.", "questions": "* In Figure 5, there are some interactions among different human joints such as \"cross the leg\" and \"lean forward in meditating\". How to model such interactions in the method?\n\n* In the method design, the style reward examines the reality of every two adjacent frames. I am curious whether it is sufficient without evaluating longer frame ranges that include more integral human motion semantics.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "(1) The proposed framework is unified without task-specific designs. It can directly support language instruction input without manual task decompositions.\n\n(2) The method achieves impressive performance on unseen real-world scenarios, indicating its good generalizability.\n\n(3) The paper is well-organized and easy to follow.", "weaknesses": "(1) The evaluation metrics aim to examine whether a method can complete the task. However, both the Success Rate and the Contact Error are defined on $c_k,d_k$ that are generated by the LLM Planner. If the LLM Planner fails to generate reasonable chains of contacts or contact pairs, the evaluation may be misleading.\n\n(2) The current evaluation focuses on contact information that only relates to local human body parts. I think it is also important to examine the global reality of the generated motion. The evaluation for motion reality may include whether the motion is human-like and whether it is semantically faithful to the language command.\n\n(3) In the definition of task observations, the human joint $v_k^j$ corresponds to the nearest object surface point $v_k^{np}$. However, $v_k^{np}$ may not be a reasonable contact point for $v_k^j$. Understanding object geometries and affordances is important to finding ideal contact areas that human prefers.\n\n(4) The method is limited to static scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RlSbNZQTwQ", "reviewer": "Reviewer_tgmd", "summary": "The paper proposes a method to generate physical human-scene interactions from language inputs. The method is capable of generating multiple tasks with a single model. \n\nA large language model (LLM) is used to translate the language prompt (e.g go sit on the chair) into a plan of multiple sequential steps (e.g. go closer to the chair, make contact between hip and seating area in the chair). Each step is defined as the contact between a human joint and an object part. The paper calls this: chain of contact.\n\nThe authors derive observation and reward based on the contact pair. A humanoid agent is trained using AMP (Peng et al. 2021) to maximize the sum of rewards for all the steps specified by the LLM. \n\nTo train the humanoid agent, the authors collected a dataset of plans by prompting the LLM in different scenarios. These scenarios require access 3D scene data (i.e. object meshes, scene layout, object parts).", "questions": "- Can this approach be applied to kinematic models? What would need to change?\n- What is the importance of having a reward for the \"not care\" contact?\n- What is the difference between $\\hat{d_k}$ and $\\bar{d_k}$? I don't see how the directional part of the reward in Eq.7 enforces the desired behavior.\n- In Table 2. How are the weights chosen without adaptive weights?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is novel, tackles an important problem, provides interesting results, opens the door for exciting future directions, is easy to read, and is technically sound.\n- The main novelties of the paper are:\n\n1. Using LLM as a task planner. i.e. using LLM to convert textual prompts into smaller executable steps.\n2. Representing all the steps as contact pairs. i.e chain of contact\n3. Unifying the design of the observation and reward for all tasks.\n\n- The method is capable of generating multiple interactions with the same model.", "weaknesses": "- The motion quality looks worse than previous works like InterPhys, AMP, or NSM. \n- The paper mentions the use of prompt engineering to generate the plan data. It is not entirely clear how was this done. How difficult was this? How sensitive is the model to changes in the prompt? I think this process involves several steps which were not discussed in detail. Thus I think it will be hard to reproduce this part.\n- The LLM expects examples of a task plan with each prompt. It is unclear how many examples were needed in total. How were these examples generated? manually?\n- The comparison in Table 3 seems weak. I understand the difficulty since the code for InterPhys is unavailable. However, I still have many questions about the comparison. How many test objects were used and how many trails were run? Did you use similar numbers as the ones used by InterPhys Hassan et al 2023? In addition, the performance of AMP seems quite low. InterPhys is an extension of AMP so I expect AMP to perform better than the provided numbers. I suspect the model has not been trained properly.\n- The method does not handle interaction with moving objects.\n\nMinor weaknesses\\comments for improvements\n- There are three recent(possibly concurrent) works that use LLM as an agent planner. It would be helpful to the reader to discuss the differences between the proposed methods and their approaches. \n1. Athanasiou et al.\"SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation\"\n2. Brohan et al.\"RT-2: New model translates vision and language into action\"\n3. Rocamonde† et al. \"VISION-LANGUAGE MODELS ARE ZERO-SHOT REWARD MODELS FOR REINFORCEMENT LEARNING\"\n- The authors argue for the value of language input. Language is useful for some applications but it is not necessarily a better or more user-friendly interface. For many users, a joystick, brush, and fingers might be more comfortable. Especially in games or digital design.\n- The contact pair contains 5 elements. Then it is not a pair. Maybe it is better to be called a contact tuple or a set.\n- It would be nice to see an example of when the human plan succeeded but the LLM one. It would be useful to put both plans side by side", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sKHerBeAjr", "reviewer": "Reviewer_YGEs", "summary": "The paper proposes to interact with virtual scene interfaces via language commands. Its proposed UniHSI prompts an LLM to translate user commands into (human part, touched object) sequences. An example is \"play video games\" -> (chair, seat surface, pelvis, contact, down)  \n-> (table, keyboard, left hand, contact, down) -> (table, keyboard,\nright hand, contact, down). They are then executed by a second model.  \nThis modularity allows modeling more interactions than before, and doing to automatically. The authors also propose a new dataset, Sceneplan, which combines PartNet and ScanNet into the above sequences of multiple interactions.", "questions": "In Table 3, is there no other SOTA method that allows combinations of Sit, Lie Down, and Reach (other than the AMP vanilla baseline)? Maybe other reviewers from the field could also comment.\n\nExamples of hard-to-parse sentences:\n* \"The framework is built upon the definition of interaction as Chain of Contacts (CoC): steps of human joint-object part pairs, which is inspired by the strong correlation between interaction types and human-object contact regions.\"  -> These are three sentences in one.  \nHow about \"Interactions intuitively describe contacts between human joints and object parts. Thus, we define sequences of such contacts as Chain of Contacts (CoC).\"\n* \"the initial effort involves the uniform definition of different interactions. We propose that interaction itself contains a strong prior in the form of human-object contact regions.\" -> It is confusing to use \"uniform\" in a non-statistical way when the surrounding text contains priors etc.. What do you mean by uniform throughout the text?\n* \"To facilitate the unified execution, we devise the TaskParser as the core of the Unified Controller.\" -> Please use simpler language.\n* Section 3.2 is easier to read. It would be great to write more like in this section.\n\nIn how far is the discriminator $D$ adversarial and what effect does this have?\n\nIn Equation 7, instead of using exp to ensure positivity, you could consider using Softplus, which is sometimes more numerically stable.\n\nPlease analyse the dataset quality in more detail.\n\nPlease format the reference section coherently.\n\nPlease add the Limitations from the Appendix to the main paper\n\nIt would be an option to split the dataset from the method. This way, you could focus more on either of them. E.g., a big multi-round dataset with high quality would be a great contribution.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The new Sceneplan dataset could be useful for the community, and the different difficulty levels can help drive the community\n* The UniHCI method is the first that generalizes to much more interaction combinations than previous methods (this is claimed by the authors, I am not inside the field enough to judge it) ## Weaknesses\n* The evaluation of the UniHSI method would benefit from more ablations on the design choices (loss, which module leads to the error, used motion style datasets) and comparison to more SOTA baselines (here, the authors blame code unavailability and too different training setups)\n* The proposed new method (UniHSI) is only evaluated on the proposed new dataset (Sceneplan) which both use the LLM Planner module. This may give an unfair advantage.\n* The Sceneplan dataset was automatically generated, although the involved LLM Planner in UniHSI gives much lower performance than humans (57.3 vs 73.2, \"Failures typically involved incomplete planning and out-of-distribution interactions\"). This may lower the dataset quality severly.\n* Most sentences in the paper are hard to parse, mostly because of over-complicated words (see Questions). This can be fixed in a (major) revision.\n* The reference section is inconsistent: Six different spellings for CVPR; Harvey 2020 misses where it was published; Zhang 2022a misses the arxiv identifier; in Tevet 2023 ICLR should not be referenced to as only \"ICLR, 2023\". This can be fixed in a revision.\n* The paper may be more applicable to computer vision conferences, judging by the references and by the fact that it is heavily practical/conceptual. This potential weakness can be better judged by the AC and Senior AC.", "weaknesses": "See Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7VFODJBsY0", "reviewer": "Reviewer_JJQ3", "summary": "This paper presents UniHSI, a unified framework for Human-Scene Interaction (HSI). The framework aims to support versatile interaction control through language commands, providing a user-friendly interface. The authors propose a definition of interaction as a Chain of Contacts (CoC), which represents the steps of human joint-object part pairs. UniHSI consists of a Large Language Model (LLM) Planner that translates language prompts into CoC task plans and a Unified Controller that executes the plans. The framework enables diverse interactions, fine-granularity control, and long-horizon transitions. The authors collect a dataset ScenePlan for training and evaluation, and experiments demonstrate the effectiveness and generalizability of the framework.", "questions": "- In supp tbl 1, example 1 step 2 pair 2: The OBJECT is chair and PART is left knee. Is this a typo or a design choice?\n- what does {none, none, none, none, front} and {bed, none, none, none, front} mean, and how does the UniHSI controller use this information?\n- In supp tbl 1 example 2, why is the direction between mattress and body parts “down” and the direction between pillow and head “up”?\n- Can the LLM Planner, or a quick extension of the LLM Planner support fine-grained spatial relationships such as distances and directions between objects? For example, can this method handle cases where 1) a chair is placed too closed to a table so that it is impossible to sit on it,  and 2) two chairs are placed facing opposite directions, and the agent must choose the correct one to watch tv?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The proposed method is the first to generate long-horizon and open-domain human-scene interactions.\n- The proposed method outperforms existing baselines in both success rate and contact error.\n- The method proposed to use an LLM Planner to generate a sequence of contact pairs (Chain of Contacts) as an intermediate representation, and parse that into a reward function for learning a unified controller. I find this solution both intuitive and elegant.", "weaknesses": "The paper does not provide a thorough explanation of the method. Specifically, it is unclear how different cases are handled by the LLM Planner, and how each type of interaction pair is translated to the reward. See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)