Preference learning for guiding the tree search in continuous POMDPs
Authors: Jiyong Ahn
Sanghyeon Son
Dongryung Lee
Jisu Han
Dongwon Son
Beomjoon Kim
Venue: CoRL 2023 Poster (2023)
Keywords: POMDP
Online planning
Guided Search
Preference-based learning

Abstract: A robot operating in a partially observable environment must perform sensing actions to achieve a goal, such as clearing the objects in front of a shelf to better localize a target object at the back, and estimate its shape for grasping. A POMDP is a principled framework for enabling robots to perform such information-gathering actions. Unfortunately, while robot manipulation domains involve high-dimensional and continuous observation and action spaces, most  POMDP solvers are limited to discrete spaces. Recently, POMCPOW has been proposed for continuous POMDPs, which handles continuity using sampling and progressive widening. However, for robot manipulation problems involving camera observations and multiple objects, POMCPOW is too slow to be practical. We take inspiration from the recent work in learning to guide task and motion planning to propose a framework that learns to guide POMCPOW from past planning experience. Our method uses preference learning that utilizes both success and failure trajectories, where the preference label is given by the results of the tree search. We demonstrate the efficacy of our framework in several continuous partially observable robotics domains, including real-world manipulation, where our framework explicitly reasons about the uncertainty in off-the-shelf segmentation and pose estimation algorithms.


Reviews:
[{"review_id": "gOr-879FRUL", "reviewer": "Reviewer_TVga", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes Preference-Guided POMCPOW (PGP), and extension of POMCPOW that learns a policy and a value function to guide the tree search. In contrast to existing works, PGP aims to reduce the cost and data requirements needed to learn a policy and a value function that is used to guide POMCPOW's Monte Carlo Tree Search. At its core, PGP learns a value function and a policy (represented by transformer and VAE based networks) using preference learning, by labelling histories obtained from previous simulations according to whether they were successful in reaching the goal. The premise is that preference learning significantly reduces the data requirements to learn useful policies and value function compared to previous methods. PGP is tested on a set of POMDP problems where it shows superior performance compared to unguided POMCPOW and imitation-guided POMCPOW.", "summary_of_recommendation": "The paper proposes an interesting and novel extension of POMCPOW and its efficiency is demonstrated on a set of POMDP problems, including a real-world object fetch scenario. While in some parts the paper requires further clarifications (see comments and questions above), I believe that its contributions are substantial enough to warrant publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Section 3: \"This preference labelling scheme operates under the assumption that all actions carry a uniform cost.\" => This needs to be explained further. Isn't that a very strong assumption?\n\n- During deployment, the learned policy is only used to sample new actions, in case PW decides to add an action to a history node. Previous works (e.g. Alpha-Go or Lets-Drive) have additionally used the policy to guide the action selection strategy during the tree search. Is there a reason why PGP doesn't do that?  \n\n- The current architectures of the value and policy networks require a goal state as an input, but it is not clear why this is needed. Wouldn't a simple binary value \"goal reached\" and \"goal not reached\" suffice? As mentioned above, for problems with goal states, we don't necessarily know the exact goal state reached by a history. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a very nice paper, both clearly written and well motivated. The observation that the ranking between histories (instead of computing their actual values) is sufficient to efficiently guide the tree search makes sense and is effectively exploited in PGP. The method itself is sound and described with sufficient detail. The experiments are generally carried out well and demonstrate the effectiveness of PGP (including on a real-world object fetch scenario).\n\nOne limitation of PGP that wasn't mentioned in the paper is that the current history ranking scheme assumes that the problem includes a set of goal states and that histories are labelled according to whether they reach a goal state or not. However, in many POMDPs we don't have any goal states, or the information whether a history has reached a goal state can not be inferred from the history itself (due to the partial observability of states). The paper should make it clearer what the assumptions are (existence and identifiability of goal state) or whether the method can be extended to the general POMDP case.\n\nSome minor comments:\n- g_{\\theta} in Algorithm 1 is not defined.\n\n- In the videos it is not clear what's happening in the upper-left corner. Is this some representation of the beliefs?", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "_CI0eeCwTim", "reviewer": "Reviewer_kYWh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces PGP (Preference-Guided POMCPOW), a new framework for planning tasks in partially observable, continuous domains. As the main contribution, PGP improves upon POMCPOW by incorporating preference learning to guide the tree search. The authors generate a preference dataset by ranking trajectories based on reaching the goal state. They then use this dataset to train value and action-value functions using a transformer. They propose an energy-based method for policy learning and employ a variational autoencoder (VAE) to approximate it.\nThe authors evaluate PGP in various environments, including both simulated and real-world scenarios. They compare its performance against baselines such as unguided and imitation-guided POMCPOW. The results demonstrate that PGP, in general, achieves higher success rates and improved efficiency in the tested environments. Although the scenarios do not cover a wide range, the findings highlight the practicality of PGP for continuous POMDPs.\n", "summary_of_recommendation": "The paper introduces a novel method for preference-based dataset generation and improves upon existing approaches for continuous POMDP with preference learning. While the proposed method demonstrates increased data and time efficiency, its real-time applicability remains limited. The experiments conducted in simulation and real-world settings validate the effectiveness of the approach. Overall, the paper presents a strong combination of learning approaches in perception and planning to address a challenging task in a partially observable environment.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- The experiments show examples of settings with a single occluding object that the robot is capable of moving; then, the robot moves the occluding object out of the way and picks up the target afterwards. How might planning change in other settings where the robot cannot move the occluding object, or if there are multiple occluding objects? It would be great if you could extend this analysis (surely acknowledging the limited time).\n\n- Is it possible to extend the preference labeling scheme for actions carrying non-uniform cost?\n\n- Can you provide details about the runtimes of those algorithms? Neither the paper nor the videos are informative. \n\n- The task covered is simple, with two objects, and requiring a couple of steps. What’s the scalability of this approach? \n\n- The accompanying video could be enhanced to better explain the steps of the process.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\nOriginality: The paper proposes an original framework that combines preference learning with tree search in continuous POMDPs, addressing the challenges in complex real-world scenarios.\n\nQuality: The effectiveness of the proposed framework is supported by representative experiments in both simulated and real-world environments. The comparison with suitable baselines demonstrates the advantages in terms of effectiveness and efficiency.\n\nClarity: The paper provides a fair introduction to the problem and the proposed framework. The figures and discussions enhance the clarity of the presented concepts and results to some degree.\n\nSignificance: The paper contributes to the study of planning approaches in partially observable continuous domains, offering a somewhat practical framework for real-world applications with improved data and time efficiency.\n\n\nWeaknesses:\n\n- Dependence on observation model: The performance of the framework heavily relies on the quality of the observation model without any extra layer of representations, which can be a limitation in terms of perception and scene understanding.\n\n- Applicability to real-world tasks: While the proposed framework is more time-efficient than existing baselines, it seems to still falls short of real-time operation, which limits its practicality for many applications.\n\n- Those two points above are acknowledged by the authors, however, the paper does not discuss these issues nor possible solutions in detail.\n\n- Limited variety of real-world experiment setup: The set of object classes and environmental setups in the real-world experiments do not cover a wide enough range of scenarios, raising concerns about the generalizability of the approach.\n\n- Assumption of uniform action cost: The assumption of uniform action cost may not hold in many real-world problems, which contradicts the claim of increased practicality compared to existing methods.\n\nMinor issues:\n- Style issues in figures: Some figures, such as Figure 5, suffer from small font size and readability issues, affecting the overall presentation of the experiment results.\n\n- Unclear discussion: At line 286, authors mention that PGP finds the most optimal solution among all the baselines. However, since there is an explicit distinction made between PGP and SF-PGP, this statement becomes unclear, since in some experiments SF-PGP outperforms PGP. It would be better to replace “PGP” with “PGP and SF-PGP” or “PGP-based”. So, a clarification or rephrasing is needed to avoid confusion.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "IGH0AnmpMEI", "reviewer": "Reviewer_8hjC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Review for “Preference learning for guiding the tree search in continuous POMDPs“\n\nThe paper studies planning problems in partially observable environments. A common approach is using POMDPs, yet they are not applicable for continuous space problems. POMCPOWs present an alternative, yet are challenged in high-dimensions. The authors, present a framework to guide POMCPOW using pair-wise preferences based learning. Preferences are provide by automatically analyzing past search trees: a successful trajectory is to be preferred over a non-successful one, and between two successful trajectories preference is given to the shorter one.\nThis pairing of search tree histories allows for efficiently generating large data sets. Using reward learning allows for obtaining successful reward functions. The framework is validated in two simulation and one real-world experiments, showing its advantages over the state-of-the-art with respect to success rate and efficiency of the found solutions.\n", "summary_of_recommendation": "Overall, this is a well written and paper, relevant to the CoRL community. However, there are a few minor issues that should be addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1.\tIntroduction: There is no clear statement of contribution, and no clear problem statement. While the introduction describes the problem setup, it would be good to add the latter in clear mathematical terms.\n\n2.\tIntroduction line 75: Why is a multi-modal policy needed? There should be at least some high-level explanation that does not require reading [17].\n\n3.\tApproach: The definition of T_i  is not clear, please clarify.\n\n4.\tResults: Figure 5 is quite dense, it might be good to find a little bit of extra space throughout the paper to subcaptions below each row to improve readability. Further, the third row misses y-labels, it is not immediately clear what is shown. Also, optimum appears only in the second third row, but not in the first – why is that?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written, addressing an interesting and challenging problem and convincing empirical results. The review of related work is quite extensive and convincing, limitations are discussed in great detail. While the paper is overall convincing, I have a few comments, see below.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-iCrXVVycu2", "reviewer": "Reviewer_dipf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method to solve POMDPs with continuous action spaces. The idea is to guide the tree search to solve the POMDP using preference-based feedback where successful trajectories are preferred over unsuccessful ones, and shorter successful trajectories are preferred over longer successful trajectories. The approach is validated with a few simulated experiments and in a real-robot experiment.", "summary_of_recommendation": "This paper uses preference feedback for a task which I don't believe requires preference-feedback. The knowledge of the reward function should make preference-based learning unnecessary, and I expect a simpler regression approach to work not worse than the preference-based method.\n\nPOST-REBUTTAL UPDATE: The authors conducted an additional study that empirically shows the benefit of preference feedback over value feedback, even though this is information theoretically counter-intuitive. I updated my recommendation from weak reject to weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "The first point I raised in my review is my biggest concern about the paper. If authors can theoretically prove me wrong or at least do an ablation study that shows preference dataset is better, I will be happy to recommend the acceptance of the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "POST-REBUTTAL UPDATE: The authors addressed my major comment (#1 below) empirically and I updated my recommendation as weak accept.\n\n\nThere are two major flaws of the paper:\n\n1) The main motivation for using preference feedback is that it is easier for humans to provide (than actual scalar values). However in this work, the reward function is already known and it is possible to do standard regression to learn the value function instead of preference-based learning. The paper claims preference feedback \"can create much larger data from a single search tree than the one that regresses the value from history\". While it is true that preference dataset will be larger, it is not true that this dataset will be more informative. If that was the case, one could go further and create rankings of three trajectories to get better datasets for learning the value functions. Data processing inequality tells us that if X implies Y which implies Z, then Z cannot give more information about X than Y. In this case, true value function (X) implies the true values of the realized trajectories (Y), and these values (Y) imply the preferences between them (Z). Therefore, information theoretically, the preference dataset cannot be more informative, albeit larger, than a dataset that consists of (trajectory, value) pairs.\n\n2) The work cites some recent LLM works for preference-based learning, which is fine. But it does not give enough credit to the large body of preference-based learning in robotics. Of course, I acknowledge the fact that citations [5-7] are either in robotics or are closely related, but  there have been a lot of work in robot learning that developed different forms of preference feedback and analyzed them information theoretically. I would suggest the authors to dive deeper into that literature.\n\n3) Minor: Bibliography information is missing on reference 17.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use preference guiding in POMCPOW to improve performance where POMCPOW is a tree-search based algorithm for solving continuous valued POMDP problems. The proposed approach learns a value function and policy using preference learning to guide the tree search. In experimental evaluation, the proposed approach outperforms comparison methods. The paper is well written and motivated. Using ranks instead of actual values is a valuable idea.
Decision: Accept (Poster)