Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zeqi Xiao
authors: Tai Wang
authors: Jingbo Wang
authors: Jinkun Cao
authors: Wenwei Zhang
authors: Dahua Lin
authors: Jiangmiao Pang
reviews: 
Comments: This paper presents UniHSI, a unified framework for Human-Scene Interaction (HSI). The framework aims to support versatile interaction control through language commands, providing a user-friendly interface. The authors propose a definition of interaction as a Chain of Contacts (CoC), which represents the steps of human joint-object part pairs. UniHSI consists of a Large Language Model (LLM) Planner that translates language prompts into CoC task plans and a Unified Controller that executes the plans. The framework enables diverse interactions, fine-granularity control, and long-horizon transitions. The authors collect a dataset ScenePlan for training and evaluation, and experiments demonstrate the effectiveness and generalizability of the framework.
Strengths: - The proposed method is the first to generate long-horizon and open-domain human-scene interactions.
- The proposed method outperforms existing baselines in both success rate and contact error.
- The method proposed to use an LLM Planner to generate a sequence of contact pairs (Chain of Contacts) as an intermediate representation, and parse that into a reward function for learning a unified controller. I find this solution both intuitive and elegant.
Weaknesses: The paper does not provide a thorough explanation of the method. Specifically, it is unclear how different cases are handled by the LLM Planner, and how each type of interaction pair is translated to the reward. See questions.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper proposes to interact with virtual scene interfaces via language commands. Its proposed UniHSI prompts an LLM to translate user commands into (human part, touched object) sequences. An example is "play video games" -> (chair, seat surface, pelvis, contact, down)  
-> (table, keyboard, left hand, contact, down) -> (table, keyboard,
right hand, contact, down). They are then executed by a second model.  
This modularity allows modeling more interactions than before, and doing to automatically. The authors also propose a new dataset, Sceneplan, which combines PartNet and ScanNet into the above sequences of multiple interactions.
Strengths: * The new Sceneplan dataset could be useful for the community, and the different difficulty levels can help drive the community
* The UniHCI method is the first that generalizes to much more interaction combinations than previous methods (this is claimed by the authors, I am not inside the field enough to judge it) ## Weaknesses
* The evaluation of the UniHSI method would benefit from more ablations on the design choices (loss, which module leads to the error, used motion style datasets) and comparison to more SOTA baselines (here, the authors blame code unavailability and too different training setups)
* The proposed new method (UniHSI) is only evaluated on the proposed new dataset (Sceneplan) which both use the LLM Planner module. This may give an unfair advantage.
* The Sceneplan dataset was automatically generated, although the involved LLM Planner in UniHSI gives much lower performance than humans (57.3 vs 73.2, "Failures typically involved incomplete planning and out-of-distribution interactions"). This may lower the dataset quality severly.
* Most sentences in the paper are hard to parse, mostly because of over-complicated words (see Questions). This can be fixed in a (major) revision.
* The reference section is inconsistent: Six different spellings for CVPR; Harvey 2020 misses where it was published; Zhang 2022a misses the arxiv identifier; in Tevet 2023 ICLR should not be referenced to as only "ICLR, 2023". This can be fixed in a revision.
* The paper may be more applicable to computer vision conferences, judging by the references and by the fact that it is heavily practical/conceptual. This potential weakness can be better judged by the AC and Senior AC.
Weaknesses: See Questions below.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The paper contributes the first unified framework that can generate physically plausible HSI motions in static scenes given compounded language commands. To examine the method's effectiveness, the paper constructs a new dataset containing thousands of interaction plans on existing 3D scenes. Experiments show that the proposed framework outperforms baselines on different single-step tasks, and also exhibits its ability to handle hard task plans and good generalizability.
Strengths: (1) The proposed framework is unified without task-specific designs. It can directly support language instruction input without manual task decompositions.

(2) The method achieves impressive performance on unseen real-world scenarios, indicating its good generalizability.

(3) The paper is well-organized and easy to follow.
Weaknesses: (1) The evaluation metrics aim to examine whether a method can complete the task. However, both the Success Rate and the Contact Error are defined on $c_k,d_k$ that are generated by the LLM Planner. If the LLM Planner fails to generate reasonable chains of contacts or contact pairs, the evaluation may be misleading.

(2) The current evaluation focuses on contact information that only relates to local human body parts. I think it is also important to examine the global reality of the generated motion. The evaluation for motion reality may include whether the motion is human-like and whether it is semantically faithful to the language command.

(3) In the definition of task observations, the human joint $v_k^j$ corresponds to the nearest object surface point $v_k^{np}$. However, $v_k^{np}$ may not be a reasonable contact point for $v_k^j$. Understanding object geometries and affordances is important to finding ideal contact areas that human prefers.

(4) The method is limited to static scenes.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 4 excellent